Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign
  Dropout
  The vast majority of deep models use multiple gradient signals, typically
corresponding to a sum of multiple loss terms, to update a shared set of
trainable weights. However, these multiple updates can impede optimal training
by pulling the model in conflicting directions. We present Gradient Sign
Dropout (GradDrop), a probabilistic masking procedure which samples gradients
at an activation layer based on their level of consistency. GradDrop is
implemented as a simple deep layer that can be used in any deep net and
synergizes with other gradient balancing approaches. We show that GradDrop
outperforms the state-of-the-art multiloss methods within traditional multitask
and transfer learning settings, and we discuss how GradDrop reveals links
between optimal multiloss training and gradient stochasticity.
