HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with
  Auxiliary Tasks
  Multi-task learning (MTL) can improve performance on a task by sharing
representations with one or more related auxiliary-tasks. Usually, MTL-networks
are trained on a composite loss function formed by a constant weighted
combination of the separate task losses. In practice, constant loss weights
lead to poor results for two reasons: (i) the relevance of the auxiliary tasks
can gradually drift throughout the learning process; (ii) for mini-batch based
optimisation, the optimal task weights vary significantly from one update to
the next depending on mini-batch sample composition. We introduce HydaLearn, an
intelligent weighting algorithm that connects main-task gain to the individual
task gradients, in order to inform dynamic loss weighting at the mini-batch
level, addressing i and ii. Using HydaLearn, we report performance increases on
synthetic data, as well as on two supervised learning domains.
