Representation Stability as a Regularizer for Improved Text Analytics
  Transfer Learning
  Although neural networks are well suited for sequential transfer learning
tasks, the catastrophic forgetting problem hinders proper integration of prior
knowledge. In this work, we propose a solution to this problem by using a
multi-task objective based on the idea of distillation and a mechanism that
directly penalizes forgetting at the shared representation layer during the
knowledge integration phase of training. We demonstrate our approach on a
Twitter domain sentiment analysis task with sequential knowledge transfer from
four related tasks. We show that our technique outperforms networks fine-tuned
to the target task. Additionally, we show both through empirical evidence and
examples that it does not forget useful knowledge from the source task that is
forgotten during standard fine-tuning. Surprisingly, we find that first
distilling a human made rule based sentiment engine into a recurrent neural
network and then integrating the knowledge with the target task data leads to a
substantial gain in generalization performance. Our experiments demonstrate the
power of multi-source transfer techniques in practical text analytics problems
when paired with distillation. In particular, for the SemEval 2016 Task 4
Subtask A (Nakov et al., 2016) dataset we surpass the state of the art
established during the competition with a comparatively simple model
architecture that is not even competitive when trained on only the labeled task
specific data.
