{'fraction_non_alphanumeric': 0.04613445577988633, 'fraction_numerical': 0.012936672136215547, 'mean_word_length': 4.544831223628692, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-task Attention Mechanism for Dense Multi-task Learning

Multi-task learning has recently become a promising solution for a comprehensive understanding of complex scenes. Not only being memory-efficient, multi-task models with an appropriate design can favor exchange of complementary signals across tasks. In this work, we jointly address 2D semantic segmentation, and two geometry-related tasks, namely dense depth, surface normal estimation as well as edge estimation showing their benefit on indoor and outdoor datasets. We propose a novel multi-task learning architecture that exploits pair-wise cross-task exchange through correlation-guided attention and self-attention to enhance the average representation learning for all tasks. We conduct extensive experiments considering three multi-task setups, showing the benefit of our proposal in comparison to competitive baselines in both synthetic and real benchmarks. We also extend our method to the novel multi-task unsupervised domain adaptation setting. Our code is available at https://github.com/cv-rits/DenseMTL.

: Overview of our MTL framework. The three tasks semantic Segmentation, Depth regression and Normal estimation share an encoder E. The task-specific decoders S, D and N exchange information in the "multi-Task Exchange Block" (mTEB) via an attention-based mechanism, resulting in refined features to produce final predictions.

Introduction
Recent advances in deep neural network architectures [11,13] and efficient optimization techniques [1,15] constantly set higher accuracy in scene understanding tasks, demonstrating great potentials for autonomous applications. Nevertheless, the majority of the literature focuses on pushing performance of single-tasks, being either semantic tasks like segmentation [4,12] and detection [32], or geometrical tasks like depth/normal estimation [18,9]. Very few have paid attention to a more comprehensive objective of joint semantics and geometry understanding, while in practice, this is desirable in critical applications such as robotics and autonomous driving. In those systems, we would expect to have cooperative synergies among all tasks, i.e. tasks should be processed together in one unified system rather than separately. Arguably, promoting such synergies could also bring mutual benefit to all tasks involved. For example, having disruptive changes in depth maps may signal semantic boundaries in segmentation maps; while pixels of some semantic classes, like "road", may share similar surface normals.
To this end, multi-task learning (MTL) [3,16,25] has become a promising solution as it seeks to learn a unified model that excels in average on all tasks. A common MTL design is to have a large amount of parameters shared among tasks while keeping certain private parameters for individual ones; information is exchanged via shared parameters, making possible the synergy flow. Some recent works [2,26,33] focus on new multimodal modules that further boost tasks interactions. One important advantage of such MTL models is memory-efficiency thanks to the shared parts. In this work, we adopt the same principle and design our model with shared encoder and dedicated task decoders.
Recent MTL approaches come with different solutions to improve the information exchanged or distilled across tasks, often known as multi-modal distillation. PAD-Net [34] and MTI-Net [26] have proven the effectiveness of self-attention [28] for MTL, i.e. a mechanism to self-discover most attentive signals of each task-specific feature for the other tasks. Differently, ATRC [2] advocates to better look at pair-wise task similarity to guide multi-modal distillation. We partially follow the same direction as in ATRC with our correlation-guided attention, in which we use correlations between task-specific features to guide the construction of exchanged messages. Further, we propose an unified mechanism that combines two different attentions, correlation-guided attention and self-attention, via a learnable channel-wise weighting scheme.
In summary, we propose a novel multi-modal distillation design for MTL, relying on pair-wise cross-task attentions mechanisms (coined xTAM, Sec. 3.1) combined into a multi-task exchange block (coined mTEB, Sec. 3.2). We address three critical tasks for outdoor scene understanding: semantic segmentation, dense depth estimation and surface normal estimation. In extensive experiments on multiple benchmarks, our proposed framework outperforms competitive MTL baselines (Sec. 4.2). With our new multi-task module, we also report improvement in semantic segmentation on Cityscapes where self-supervised depth plays as the auxiliary task [12]. Empirically, we showcase the merit of our proposed multi-task exchange in a new setting of MTL unsupervised domain adaptation (MTL-UDA, Sec. 4.5), outperforming all baselines.

Related Work
Multi-task learning. In the early days of neural networks, Caruana [3] introduces the idea of multi-task learning with hard parameter sharing, i.e. some parameters are shared between all tasks, some are dedicated to separate tasks. In the same spirit, UberNet [16] features a deep architecture to jointly address a large number of low-, mid-and high-level tasks. Zamir et al. [37] conduct a large-scale study with 26 tasks on four million images of indoor scenes, studying the dependencies and transferabilities across tasks. While those seminal works show great potential of multi-task learning, they do note a few challenges, most notably the negative transfer phenomenon that degrades performance of certain tasks when learned jointly [16,14]. Some works [14,5] reason that negative transfer is due to the imbalance of multi-task losses and introduce mechanisms to subsequently weight these individual loss terms. Kendall et al. [14] propose to weight multiple losses by estimating the homoscedastic uncertainty of each task. Chen et al. [5] introduce GradNorm, an algorithm that helps dynamically tuning gradient magnitudes such that the learning pace of different tasks are balanced. Differently, Sener and Koltun [23] cast multi-task learning as multi-objective optimization where the goal is to find Pareto-optimal solutions. Cross-task mechanisms. Closer to our work are methods focusing on improving exchange or distillation across tasks [33,38,26,2]; the main idea is that each task could benefit from different yet complementary signals from the others. Inspired from the success of visual attention in perception tasks [34,28,7], PAD-Net [33] uses an attention mechanism to distill information across multi-modal features. In MTI-Net [26], Vandenhende et al. extend PAD-Net with a multi-scale solution to better distill multimodal information. Zhang et al. [38] propose to aggregate affinity maps of all tasks, which is then "diffused" to refine task-specific features. PSD [39] mines and propagates patch-wise affinities via graphlets, instead we model the interactions with attention mechanisms. Bruggemann et al. [2] introduce ATRC to enhance multi-modal distillation with four relational context types based on different types and levels of attention. Similar to ATRC, our method also exploits pairwise task similarity to refine task-specific features. Differently though, we advocate to combine the pairwise similarity with cross-task self-attention via a learnable weighting scheme to learn the refined residuals that complement the original task-specific features.
Recent efforts seek for label-efficient learning paradigms to train models for urban scene understanding. Tosi et al. [25] introduces real-time multi-task network based on knowledge distillation and self-supervision training. Hoyer et al. [12] propose a novel architecture and different strategies to improve semantic segmentation with selfsupervised monocular depth estimation. One interesting finding in [12] is that multi-task learning module of PAD-Net [33] complements other self-supervised strategies and further improve segmentation performance. On this regard, we study the effect brought by our proposed module in this particular setup.

Method
In multi-task learning, the aim is to optimize a set of n tasks: i P tT 1 , ..., T n u while seeking a general good performance -as opposed to favoring a single task. Our model takes an input image and makes n predictions. Generally, this is achieved with a shared encoder and separate task-specific decoders. Here, we introduce a novel mechanism that enhances cross-task talks via features exchange, building on the intuition that each decoder discovers unique but complementary features due to its separate supervision [37]. Our bidirectional xTAM module enables information between task pairs (i, j) via the discovery of directional features, f j i and f i j . For clarity, only xTAM j i is detailed here. It relies on two attention mechanisms. Fist, a correlation-guided attention (green, upper part) to discover xtask j i , the features from task j contributing to task i. Second, a self-attention (purple, lower part) to discover complementary signal self j i from j. Gray blocks are 1x1 projection layers for compatibility, and Ó s is a s-times downscale for memory efficiency (reversely, Ò s for upscale). Best viewed in color.
We formulate in Sec. 3.1 a bidirectional cross-Task Attention Mechanism module, coined xTAM, taking as input a pair of task-specific features and returning two directional features. We then present in Sec. 3.2 our complete MTL framework for scene understanding that encompasses our multi-Task Exchange Block (mTEB, see Fig. 1).

xTAM: Bidirectional Cross-Task Attention Mechanism
Building upon recent works demonstrating task interactions [24], our module seeks to capture the shared pair-wise task knowledge while preserving their exclusive (nonshared) knowledge. Our intuition is that we can exploit features from pairs of tasks denoted pi, jq and self-discover the interactions from their correlation matrices for either directional interaction: i j and j i. Fig. 2 illustrates our xTAM component which helps knowledge distillation between tasks, taking as input the task features pf i , f j q and returning the corresponding directional features for the other task, i.e. pf j i , f i j q.
For ease of speech we describe the directional xTAM j i where task j helps task i, but emphasize that xTAM, being bidirectional, is made of both xTAM j i and xTAM i j .
Directional cross-task attention (xTAM j i ). Considering here i as our primary task, and j as the secondary, we seek to estimate the features from task j that can contribute to i. We leverage two cross-task attentions: (i) a correlation-guided attention which goal is to exploit cross-task spatial correlations to guide the extraction of contributing features of the secondary task to the primary task, and (ii) a self-attention on the secondary task to self-discover complementary signals which are beneficial for the primary task. Visualizations of the two attentions are colored in green and purple in Fig. 2, respectively. Importantly, note that each attention contributes differently to the primary task i relying either on identification of shared j and i knowledge or on exclusive j knowledge. We use 1x1 convolutional layers (gray blocks in Fig. 2) for dimension compatibility.
Correlation-guided attention. To guide feature inclusion, we rely on the spatial correlation of task features (green blocks in Fig. 2). In practice, because computing such correlation is memory-greedy, we reduce complexity by projecting the f i and f j downscaled features onto d-dimensional sub-spaces as in [28] such that K " P Q pÓ s f i q, Q " P K pÓ s 2 f j q, having Ó s the downscale operator with s the scale factor, and P Q , P K separate 1x1 convolutions. The spatial-correlation matrix C j i is then obtained by applying a softmax on the matrix multiplication K TˆQ normalized by ? d: d is the feature size [28]. Intuitively, C j i has high values where features from i and j are highly correlated and low values otherwise, which we use to weight features from f j . Subsequently, we obtain our correlation-guided attention features xtask j i by multiplying the correlation matrix C j i with the projected j features: with V " P V pÓ s 2 f j q, P V the 1x1 projection, and Ò s the upsample operator.
Self-attention. We additionally employ a spatial attention [33] which we denote as 'self-attention' to contrast with the above which takes pairs of differing tasks. Instead the following mechanism (purple blocks in Fig. 2) takes features from j alone and aims to extract private information from f j that are relevant for predicting task i: being our self-attention features. Where d is the element-wise multiplication, and σ the sigmoid function. Both F f and F m are convolutional layers which are supervised by the target task i to learn to extract relevant information from features f j . The self-attention features self j i is defined as the point-wise multiplication between the features coming from F f and the dynamic mask provided by F m .
Directional feature. To construct the final directional features f j i for the j i interaction, the two attention based feature maps are combined as: where r., .s is the channel-wise concatenation operation, and α 1..c are learnable scalars used to weight the c channels of xtask j i . All α 1..c are initialized with 0; learning will adaptively adjust the per-channel weighting. Intuitively, cross-task exchange first starts with self-attention only, then gradually adjusts the α 1..c values to include some contribution from the correlation-guided attention. This initialization strategy is important to stabilize training, especially here where we combine different types of attention.
Overall, the bidirectional xTAM block is made up of two directional blocks, xTAM j i and xTAM i j ; it outputs both f j i and f i j for each pi, jq task pair.
Discussion. Our xTAM design is different from existing multi-modal distillation modules [33,26,2]. While PAD-Net [33] and MTI-Net [26] only consider cross-task selfattention to distill multi-modal information, ATRC [2] models pair-wise task similarity, which shares a similar spirit to our correlation-guided attention. Different to those, our xTAM learns to adaptively combine cross-task self-attention and correlation-guided attention using a learnable weighting scheme. Of note, adopting the multi-scale strategy of MTI-Net [26] or having other pair-wise modules of ATRC [2] is orthogonal to our xTAM. There might be potential improvements by systematically combining those strategies with xTAM. However, we focus here on giving an extensive study of xTAM design and applications; we leave such combinations for future investigation.

Fig. 3:
Our cross-task MTL framework. We jointly predict semantic segmentation (S, blue), depth (D, orange) and surface normal (N , green). All tasks share encoder (E) and have dedicated decoders (shown as trapezoids). Our cross-task attention module is inserted within a "multi-Task Exchange Block" (mTEB) (Sec. 3.2) that takes taskspecific features as inputs and returns refined features of the same kinds. Here, three xTAM modules are used to extract pairs of directional features, where each represents the complementary signal from one task to another, e.g. f D S encapsulates depth information (D) to extracted to improve the segmentation task (S). Multi-modal directional features are combined in the square blocks, resulting in multi-modal residuals for refinement. Task specific losses (shown in red) are applied before and after our mTEB.
We focus on semantic and geometrical estimation tasks as they are critical for scene understanding, especially outdoor environments. Fig. 3 overviews our MTL framework having a unique encoder E and illustrates the interactions taking place between the three different task decoders (S, D and N ) in a three-task setup: T " tSegmentation, Depth, Normalu. We first describe our "multi-Task Exchange Block" (mTEB) that encapsulates our cross-task exchanges, and then go over our MTL training setup.
multi-Task Exchange Block (mTEB). Our block is a plugin that can be inserted at any stage in the decoder architecture, taking task-specific features as inputs, here pf S , f D , f N q, and outputting refined features, here pf S ,f D ,f N q. The block consists of one bidirectional xTAM module (Sec. 3.1) per task pair, each returning two directional features which are fused to the input task features.
Considering i to be the primary task, we concatenate the n´1 directional features f j i | jPT ztiu , contributing towards task i, along the channel dimension. The concatenated feature map is then processed with F i (depicted as square blocks in Fig. 3): a 1x1 convolution followed by a batch-normalization [13] and ReLU activation. Projected features are then fused with the main task base features, f i , via an element-wise addition denoted À , leading to the final refined featuresf i : where r...s is the channel-wise concatenation operation. Intuitively, the xTAM directional features are combined in F i to constitute multi-modal residuals that complement original features f i . By design, the refined features have same dimensions than the input task-specific features.
Although, it is possible to insert a mTEB at several levels in the decoder. We show experimentally that the overall best choice is to have a single block before the last layer.
Training. To train our framework, we use a multi-scale multi-task objective defined as a linear combination of the task losses applied at two or more stages of the decoder: before any mTEB, and on the full resolution output. Although we do not explicitly introduce losses to encourage cross-task distillation, this is made possible implicitly by the design of xTAM and mTEB. More details are provided in Appendix A.1.

Experiments
This section presents all results and ablation studies of our proposed MTL framework. We describe in Sec. 4.1 our experimental setups including the four leveraged datasets, five baselines, three set of tasks; we show as well the training details and evaluation metrics. As tasks, we consider the four dense prediction tasks of semantic segmentation, depth estimation, surface normal estimation, and semantic edge estimation, that we refer to as S, D, N , and E respectively. Our source code is publicly released.
Sec. 4.2 puts forward main experiments for fully supervised multi-task learning, including: two sets 'S-D' (segmentation + depth) and 'S-D-N ' (+ normals) on different urban datasets, in Tab. 1. As well as performance on an indoor dataset with the additional task of edge estimation in a third task set 'S-D-N -E' (+ edges), in Tab. 2.
Sec. 4.3 presents three ablation studies on (i) the position of mTEB in the decoders, (ii) different types of fusions used in mTEB, and (iii) alternative designs for the correlation-guided attention in xTAM.
We further demonstrate the benefit of our multi-task exchange framework, for semantic segmentation in Sec. 4.4 and for Unsupervised Domain Adaptation in Sec. 4.5.

Experimental setup
Common setup.
Datasets. We leverage two synthetic datasets and two real datasets: VKITTI2 [8], Synthia [21] and Cityscapes [6], NYUDv2 [19] respectively. In Cityscapes, we use stereo depth from semi-global matching as labels. Surface normals labels are estimated from depth maps following [35]. Lastly, NYUDv2 provides ground-truth semantic edge maps which we use for supervision. We train semantic segmentation on 14 classes on VKITTI2, 40 classes on NYUDv2, and 16 classes on Synthia and Cityscapes. In VKITTI2 and Synthia, we perform a random 80/20 split. For Cityscapes and NYUDv2, we use official splits and load the images at 1024ˆ512 and 576ˆ448 respectively. Whereas they are scaled to 1024ˆ320 for VKITTI2 and 1024ˆ512 for Synthia.
Baselines. Following [27] we compare our performance against single task learning networks (STL), where each task is predicted separately by a dedicated networks, and a naive multi-task learning baseline (MTL) using a shared encoder and task-specific decoders. PAD-Net [33] is a more competitive baseline than MTL. We address two variants of PAD-Net: (i) the original model used in [33] and (ii) a stronger model introduced in [12] ('3-ways PAD-Net '). Details on all models are in Appendix B.
Architecture. All networks have a shared ResNet-101 [11] encoder and a suitable amount of task decoders depending on the task set. Decoders of the STL, MTL and PAD-Net baselines are similar as described in [27]. For 3-ways PAD-Net and our model, we stick to [12] and use an Atrous Spatial Pyramid Pooling (ASPP) [4] followed by four upsampling blocks with skip connections [20].
Training details. We train all models for 40k iterations, set gradient norm clipping to 10 and reduce the learning rates by 0.1 at 30k steps as in [12]. We train our MTL models with the Adam optimizer [15] with β 1 " 0.9 and β 2 " 0.98; learning rate is set to 2.0e´4 for the backbone, and 3.0e´4 for all decoders. In other models, we use the SGD optimizer, setting the learning rates of the backbone and decoders to 1.0e´3 and 1.0e´2, respectively as in [12]. We use momentum of 0.9 and weight decay of 5.0e´4. We always report the average and standard deviation over 3 runs.
Supervision. For training, we use the cross-entropy for semantic segmentation L S and edge estimation L E [2]. The berHu loss [17] is applied on inverse normalized values for depth regression as L D pdq " berHupd f ar {d, d f ar {dq. To supervise surface normals training, we use the approach of [10]: L N pnq " 1´cospn, nq, which computes the cosine between the estimatedn and ground-truth n normal vectors as cospn, nq " n¨n{∥n∥∥n∥. Losses are weighted using our task balancing strategy.
Single-task metrics. We report a standard metric for each task: the mean intersection over union (mIoU) for semantic segmentation, the root mean square error in meters (RMSE) for depth regression, the mean error in degrees (mErr) for surface normals estimation, and the F1-score (F1) for semantic edge estimation.
Task balancing. We observe that tasks loss weighting heavily impacts the performance of multi-task models similarly to [27], though we also notice that weights vary primarily with the loss functions and their interaction, and much less on the data or model. This is fairly reasonable knowing gradients scale with the loss term magnitude. Hence, as in [27] we applied grid-search for each unique task set, and use the same optimal set of weight for all methods. For T "tS, Du we obtain tw S "50, w D "1u, for T "tS, D, Nu we get tw S "100, w D "1, w N "100u, and choose for T "tS, D, N, Eu, tw S "100, w D "1, w N "100, w E "50u. Importantly, this search is done on the MTL baseline ensuring we do not favor our method in any way.
MTL metrics. Again, our goal is to improve the overall performance of the model over a set of tasks T , which we measure using the delta metric of [27] written ∆ T . The latter measures the relative performance of a given multi-task model compared to the performance of dedicated Single Task Learning (STL) models: ∆ T pfq " 1{n ř iPT p´1q gi pm i´bi q{b i , where m i and b i are the metrics on task i of the model f and STL model respectively. While g i is the metric direction of improvement, i.e. g i "0 when higher is better, and g i "1 when lower is better.

Main results
Tab. 1 reports our main experimental results on three datasets, and two multi-task setups: 'S-D' and 'S-D-N '. On synthetic data with perfect segmentation and depth labels, our model outperforms all baselines by large margins of up to`5.69∆ SD in 'S-D' and up to`5.75∆ SDN in 'S-D-N '. In Cityscapes, having stereo depth pseudo labels, our model is outperformed by 3-ways PAD-Net , only in the 'S-D' setup, and outperforms 3-ways PAD-Net in 'S-D-N '. Quantitative results prove the effectiveness of our proposed cross-task attention mechanism.
The improvements of PAD-Net over MTL baselines confirm the benefit of selfattention mechanism [33]. Revisiting the "3-ways" architecture introduced in [12] in a different context of multi-task learning, we observe a significant leap in performance of 3-ways PAD-Net over PAD-Net; the two models are only different in decoder design. Such result emphasizes the importance of decoder in MTL.
An important remark here is that the best overall performance (i.e. delta metrics) coincides with the best individual task performances, which shows the benefit of handling multiple tasks together. It is striking to look at the 'S-D-N ' setup, where we also account for the rarely studied normal estimation task. In this three-tasks setup, in addition to the ∆ SDN pf SDN q metric, we report a partial delta measurement ∆ SD pf SDN q. This allows to compare f SD to f SDN , respectively the 'S-D' and 'S-D-N ' models considering normal estimation as an auxiliary task. We record a gap of up to +3.23 points between ∆ SD pf SDN q and ∆ SD pf SD q, showing the benefit of injecting additional geometrical cues in the form of surface normals to help the other two tasks.
We provide additional results in Tab. 2 by evaluating our method on the indoor dataset NYUDv2 and introducing a third task 'E'. Not only do we show the effectiveness of our method, we also demonstrate the benefit of multi-task learning. The partial  Table 1: MTL performance on two sets. We report individual task metrics but seek best overall tasks performance measured by ∆ SD and ∆ SDN (cf. text), computed w.r.t Single Task Learning (STL). Except for 'S-D' on Cityscapes where we are second, we outperform significantly the baselines on all delta metrics. We also report ∆ SD in 'S-D-N ' for direct comparison with 'S-D'. Notice ∆ SD pf SD q ă ∆ SD pf SDN q on all methods highlighting the importance of surface normals estimation. We highlight best and 2nd best.  Table 2: Results on NYUDv2 evaluated on three sets of tasks. We report a delta metric for each set as in Tab. 1 and provide partial metrics ∆ SD pf SDN q and ∆ SDN pf SDNE q to compare between sets and highlight the benefit of inserting additional tasks in the framework. We underline the fact that ∆ SDN pf SDN q ă ∆ SDN pf SDNE q validating the benefit of edge estimation. Ours outperforms all other methods across the three sets. delta metrics ∆ SDE pf SD q and ∆ SDN pf SDNE q underline the advantage of normals and edge estimation respectively. They are shown to provide additional valuable cues which help further understand the structure of the scene. Figure 4 shows qualitative results on the Cityscapes dataset. This is noticeable when looking at thin elements (e.g. pedestrians, bicycles, etc.) and object contours. The visuals are in fair adequacy with the quantitative analysis. More results in Appendix C.

Ablation studies
We report ablations of our cross-task distillation module on the VKITTI2 dataset on both the 'S-D' and 'S-D-N ' setups. We evaluate our method using different combinations of our module, varying the position and number of exchange blocks (Tab. 3), changing the type of fusion used (Tab. 4a), or the type of attention used (Tab. 4b).

Segmentation
Depth Normal

PAD-Net [33]
3-ways PAD-Net [12] Ours Fig. 4: Qualitative MTL results on Cityscapes in the 'S-D-N ' setup. The first row shows the input image and corresponding segmentation, depth and normals groundtruths. Segmentation results of our models are better overall, especially in boundary areas -see "bicycle", "rider" and "pedestrian". Surface normals predictions of PAD-Net is blurrier as compared to 3-ways PAD-Net and Ours.
Multi-task exchange. In Tab. 3 we report variations of our MTL framework with mTEB (cf. Sec. 3.2) positioned at different scales in the decoders, where scale s is the number of downsampling from input resolution. We observe that with a single block (upper part, Tab. 3), the ∆ SD and ∆ SDN metrics are lower if the block is placed early in the decoder (i.e. scale 4) as opposed to being later in the decoder (i.e. scale 1). This resonates with the intuition that late features are more task-specific, and thus better appropriate for cross-task distillation. Using several blocks (lower part, Tab. 3) was shown marginally beneficial, and at a cost of significantly more parameters ('Param.' column).
Multi-task features fusion. Considering a single mTEB at scale 1, we replace the original add fusion operator in Eq. (5) with either a concatenation or a product and report results in Tab. 4a. We show this choice impacts the overall delta metric. Experimentally, we notice that the gap differs on other datasets though 'add' and 'prod' remain best.
Correlation-guided attention. While our original xTAM design (cf. Sec. 3.1) uses spatial attention with Eq. (1) for cross-task correlation-guided exchange, other practices exist in the literature [26]. Hence, in Tab. 4b we replace our choice of attention in Eq. (1) using either spatial attention, channel attention, or both. Results show that channel attentions or both are less efficient here, and come at higher implementation complexity.
In Appendix A.2 we show it is better preferred to combine correlation-guided attention and self-attention in our xTAM in order to reach the best performance.  Table 3: Performance on VKITTI2 when varying the mTEB positions. The upper part shows our block is more efficient when located later in the decoders (i.e. scale 1) which we attribute to more task-specific features. Using more blocks leads to a small boost at the cost of more parameters ('Param.' column represents the size of distillation block).  Table 4: Ablation of our fusion and cross-task attention mechanism on VKITTI2.
(a) Changing our addition operation in Eq. (5) with a product or a concatenation shows slight advantage for our addition strategy. (b) We replace spatial attention in our xTAM (Sec. 3.1) with either channel only, or both spatial and channel attentions. Overall, spatial performs better with lower complexity.

MTL for segmentation
We now evaluate the effectiveness of our multi-task strategy focusing only on semantic segmentation performance -often seen as a core task for scene analysis. In Tab. 5, we report performance training with two tasks (semantics+depth) on 'Cityscapes SDE' where unlike prior results we use Self-supervised monocular Depth Estimation (SDE) with Monodepth2 [9], as in "3-ways" [12]. In the latter, a proposed strategy is to use the multi-task module of PAD-Net ("3-ways PAD-Net " in Tab. 5). We proceed similarly but replacing the above module with our mTEB (hereafter, "3-ways mTEB "). We refer to [12] and Appendix B.2 for in-depth technical details. Note that neither "3-ways PAD-Net " nor "3-ways mTEB "use the strategies of "3-ways" like data augmentation and selection for annotation. Still, Tab. 5 shows that using our module, i.e. 3-ways mTEB , achieves best results (`3.20 mIoU). This shows the increased benefit of our multi-task module. * We only report mIoU for '3-ways' because [12] does not release their weights on the full train set. Table 5: Multi-task learning with 'S-D' for semantic segmentation on Cityscapes SDE. Using the architecture of "3-ways" [12] we either insert PAD-Net ("3-ways PAD-Net ") or our mTEB ("3-ways mTEB ") as distillation block and compare the performance on semantics only, showing our multi-task exchange is better.

MTL for Unsupervised Domain Adaptation (MTL-UDA)
Unsupervised domain adaptation is the line of research dealing with distribution shifts between the source domain, in which we have labeled data, and the target domain, in which only unlabeled data is available for training. Here, we extend our experiments to the novel MTL Unsupervised Domain Adaptation setup (MTL-UDA) where the goal is to perform well in average on all tasks in the target domain. We argue task exchange is beneficial for MTL-UDA since semantic-and geometry-related tasks exhibit different behaviors and sensitivities to the shift of domain, but were shown complementary in above experiments. We leverage typical synthetic to real scenarios, namely SynthiaÞ ÑCityscapes and VKITTI2Þ ÑCityscapes, reporting results on the 'S-D' multi-task setup. Typically, while semantic segmentation degrades due to the shifts in color and texture, depth estimation is more affected by change in the scene composition and the optics sensor.
Architecture adjustments. We adopt a simple multi-task Domain Adaptation (DA) solution based on output-level DA adversarial training [29,30,22]. As output-level techniques do not alter the base MTL architecture, it allows direct understanding whether our MTL design is favorable or not for adaptation. In detail, for each task, a small discriminator network is jointly trained with the main MTL model, being two sides in an adversarial game. While discriminators try to tell from which domain the input data comes from, the MTL model tries to fool all discriminators by making outputs of source and target domain indistinguishable, which eventually helps align source/target. The complete DA architecture and training are detailed in Appendix B.3.
Baselines adjustments. We use the same baselines as in Sec. 4.2 and introduce patch discriminators to align the processed output prediction maps (see Appendix B.3). We use the STL-UDA models -consisting of a single task decoder and one output typeas baseline for measuring the delta metric. We train all methods on source and target in mix batches containing two instances from either domains.
Results. In Tab. 6 results show that the use of our multi-task exchange significantly improves performance in all scenarios and metrics. Along with MTL results, we re-port STL source, trained only on source, and STL oracle, trained on labeled target. In SynthiaÞ ÑCityscapes (Tab. 6a) our method very significantly outperforms the naive MTL-UDA baseline by`38.1∆ SD , and by`8.34∆ SD in VKITTI2Þ ÑCityscapes (Tab. 6b), also coinciding with the best individual task metrics. An interesting note is that STL-UDA is preferable over straightforward multi-task learning in UDA (MTL-UDA), which we attribute to the competing objectives in MTL-UDA.  Table 6: Multi-Task Learning Unsupervised Domain Adaptation considering a two tasks 'S-D' setup. We report ∆ SD metric w.r.t single-task learning unsupervised domain adaptation (STL-UDA). Overall, it is interesting to note that naive multi-task adaptation strategy (MTL-UDA) is lowering performance w.r.t STL-UDA, while our method outperforms all other strategies in both scenarios. To best match the source classes definition, we either use 16 or 8 semantic classes; hence the difference in mIoU for STL.

Conclusions
We address semantic and geometry scene understanding with a multi-task learning framework on the four tasks: semantic segmentation, dense depth regression, surface normal estimation, and edge estimation. We propose a novel multi-modal distillation module, namely multi-Task Exchange Block, built upon xTAM, a bidirectional crosstask attention mechanism that combines two types of cross-task attentions to refine and enhance all task-specific features. Extensive experiments in various datasets demonstrate the effectiveness of our proposed framework over competitive MTL baselines. We extend the application of our module in two practical scenarios: (i) to improve semantic segmentation of a recent self-supervised framework [12] in Cityscapes and (ii) to improve performance of multi-task synthetic-2-real unsupervised domain adaptation.

A Method details
Our method enables task exchanges to boost the general tasks performance with our multi-task exchange block (mTEB) introduced in Sec. 3.2.

A.1 Training
Considering T " tT 1 , . . . , T n u the set of n tasks to be jointly optimized, our general MTL training loss is a weighted combination of individual tasks losses and writes: where ω t is the task-balancing weight, L t is the task-specific supervision loss at intermediate scale s (i.e. L s t ) or at the final prediction stage (i.e. L final t ). Additionally, S defines the intermediate scales of supervision considering scale s to be the task intermediate output at resolution 1 2 s w.r.t input resolution. In practice, to enforce cross-task exchange without direct mTEB supervision, we set S equal to the scales at which the mTEB are inserted. In our method, we keep a single mTEB and set S " t1u.

A.2 Ablations
We study the overall benefit of our multi-task exchange block (mTEB) on VKITTI2 with our complete architecture. To further demonstrate the effect of attention mechanisms in our method, we remove self-attention from the directional features of xTAM, so that Eq. (4) now writes: f j i " rdiagpα 1 , ..., α c qˆxtask j i s . We report the performance of 'S-D-N ' with spatial cross-task attention. Metrics. We detail the four task-specific metrics below.
‚ Semantic segmentation uses mIoU as the average of the per-class Intersection over Union (%) between label s and predicted mapŝ : m S " mIoUpŝ, sq. ‚ Depth regression uses the Root Mean Square Error computed between label d and predicted mapd: m D " RMSEpd, dq, reporting the RMSE in meters over the evaluated set of images. In the SDE (Sec. 4.4) and DA (Sec. 4.5) setups, a perimage median scaling [40] is applied since the models used are not scale-aware. ‚ Normals estimation, we measure the absolute angle error in degrees between the label n and predicted mapn: m N " degpn, nq. For all datasets we retrieve labels from the depth map [35]: we unproject the pixels using the camera intrinsics and depth values, then compute the cross-product using neighboring points (from 2D perspective) [10] and average over four pairs of neighbors [36]. Cityscapes [6] provides disparity maps which we use to compute noisy surface normals labels. ‚ Edge estimation, we apply the F1-score between the predicted and ground-truth maps: m E " F1pê, eq. NYUDv2 [19] provides ground-truth semantic edge maps.  Table 7: Performance of the 'MTL' baseline model (cf. Fig. 5) for different sets of multi-task weights on VKITTI2. There are important remarks. First, uniform weighting is far from optimal. Second, best ∆ T does not always equate to optimality on individual metrics as shown by the results in bold. Ultimately, to avoid favoring a single task, we use the set of weights with highest ∆ T metric for all models, as highlighted in red .
Task balancing. Tab. 7 reports a subset of our grid-search to select an optimal set of weights for both the 'S-D' and 'S-D-N ' sets. To avoid favoring a specific task or model, the evaluation is conducted on the 'MTL' baseline model and we select the set of weights from best ∆ T metrics.

B.2 Main results
General architectures (Tabs. 1 to 4). In Fig. 5 we show the general architectures (i.e. , considering depth supervision), including the STL, MTL and PAD-Net models [12], 3-ways PAD-Net [12] and Ours. Based on the same encoder taken from a pretrained ResNet-101 backbone [11], those multi-task networks differ only in decoder design. We observe improvements in all tasks using both Atrous Spatial Pyramid Pooling (ASPP) and UNetlike connections as done in [12] (cf. 3-ways PAD-Net vs. PAD-Net in Tab. 1).
SDE architectures (Tab. 5). To allow monocular depth estimation in 'MTL for segmentation' (Sec. 4.4), we adopt the setup of [12] where intermediate depth estimation from pair of consecutive frames is supervised by a photometric reconstruction loss [9]. Fig. 6 shows the architecture used for all 3-ways variants of Tab. 5. Variants consist of swapping the yellow 'Exchange block' with either 'PAD-Net block' (3-ways PAD-Net ) or our 'mTEB' (3-ways mTEB ).
To train, we use 1.0e´5, 5.0e´5, and 1.0e´6 as learning rates for the encoder, decoder and pose estimation network respectively. The training strategy is similar to our other MTL setups, only this time we initialize all models with weights from a singlebranch model trained on self-supervised depth estimation (cf. [12]). Since the depth loss STL [27] MTL [27] PAD-Net [33] 3-ways PAD-Net [12] Ours Fig. 5: General architectures. For clarity, we only visualize two tasks in the multi-task networks. While the encoder is identical, models differ in their decoder architecture, with PAD-Net, 3-ways PAD-Net and Ours using dedicated tasks exchange blocks.
differs from the supervised one used in Sec. 4.1 we do not apply the weighting found for 'S-D' but instead resort to uniform weighting for direct comparison to [12]. Fig. 6: Architecture for Self-supervised Depth Estimation (SDE). To accommodate monocular depth on 'Cityscapes SDE', we follow the setup of [12] with added intermediate depth supervision (D 1:x ). For the two variants in Tab. 5, we use the above architecture, replacing the 'exchange block' with the desired one.

B.3 MTL for Unsupervised Domain Adaptation
Architecture and training. Fig. 7 illustrates our adversarial learning scheme with source/target data flows for multi-task UDA. We consider the two-task 'S-D' setup. As explained in main table Sec. 4.5, domain alignment is made possible with output-level DA adversarial training. In our work, alignment is done at both intermediate and final output-levels.
We follow the strategies introduced in [29,31]. Discriminators D are train on the source dataset X src and target dataset X trg by minimizing the binary classification loss: where L BCE is the Binary Cross-Entropy loss, and Q x stands for either segmentation output Q S x or depth output Q D x of the network. To compete with the discriminators, the main MTL network is additionally trained with the adversarial losses L adv , written as: The final MTL-UDA loss becomes: where λ adv is used to weight the adversarial losses and is set to 5.0e´3.
For segmentation alignment, we use "weighted self-information" map [29] computed from the softmax segmentation output P x with the formula: For depth alignment, we normalize the depth-map outputs using the source's min and max depth values, and directly align the continuous normalized maps Q D x [31].