TS-Net: Assembling Task-specific Features from Multiple Feature Levels for Multi-task Learning

Multi-task learning (MTL) has become an attractive topic that leverages shared knowledge to improve performance and enhance generalization. However, most existing works neglect the varying contribution of multi-level features to sub-task representations. In this paper, we explore the impact of multilevel features on different tasks and propose a novel level-assembling MTL architecture named TS-Net. TS-Net integrates multi-level features into multi-task representations by combining task-specific and task-generic features. We first introduce a Task-Specific Feature Capturing Block (TSFCB) to aggregate task-specific features by dynamically assembling features for input samples and prioritizing more relevant feature levels. In addition, we present a Multi-Task Mixture-of-Experts (MTMoE) module to facilitate cross-task interaction. In MTMoE, task-generic features are captured and integrated with task-specific features through a gating mechanism, allowing TS-Net to effectively share knowledge across tasks. Extensive experiments demonstrate that TS-Net exhibits superior performance across a range of tasks, including detection, segmentation, and image reconstruction.