Efficiently Identifying Task Groupings for Multi-Task Learning
  Multi-task learning can leverage information learned by one task to benefit
the training of other tasks. Despite this capacity, naively training all tasks
together in one model often degrades performance, and exhaustively searching
through combinations of task groupings can be prohibitively expensive. As a
result, efficiently identifying the tasks that would benefit from training
together remains a challenging design question without a clear solution. In
this paper, we suggest an approach to select which tasks should train together
in multi-task learning models. Our method determines task groupings in a single
run by training all tasks together and quantifying the effect to which one
task's gradient would affect another task's loss. On the large-scale Taskonomy
computer vision dataset, we find this method can decrease test loss by 10.0%
compared to simply training all tasks together while operating 11.6 times
faster than a state-of-the-art task grouping method.
