GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based
  on Transformer Networks
  A key problem in multi-task learning (MTL) research is how to select
high-quality auxiliary tasks automatically. This paper presents GradTS, an
automatic auxiliary task selection method based on gradient calculation in
Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS
improves the performance of MT-DNN with a bert-base-cased backend model, from
0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE
benchmarks. GradTS is also time-saving since (1) its gradient calculations are
based on single-task experiments and (2) the gradients are re-used without
additional experiments when the candidate task set changes. On the 8 GLUE
classification tasks, for example, GradTS costs on average 21.32% less time
than AUTOSEM with comparable GPU consumption. Further, we show the robustness
of GradTS across various task settings and model selections, e.g. mixed
objectives among candidate tasks. The efficiency and efficacy of GradTS in
these case studies illustrate its general applicability in MTL research without
requiring manual task filtering or costly parameter tuning.
