Continual Learning in the Teacher-Student Setup: Impact of Task
  Similarity
  Continual learning-the ability to learn many tasks in sequence-is critical
for artificial learning systems. Yet standard training methods for deep
networks often suffer from catastrophic forgetting, where learning new tasks
erases knowledge of earlier tasks. While catastrophic forgetting labels the
problem, the theoretical reasons for interference between tasks remain unclear.
Here, we attempt to narrow this gap between theory and practice by studying
continual learning in the teacher-student setup. We extend previous analytical
work on two-layer networks in the teacher-student setup to multiple teachers.
Using each teacher to represent a different task, we investigate how the
relationship between teachers affects the amount of forgetting and transfer
exhibited by the student when the task switches. In line with recent work, we
find that when tasks depend on similar features, intermediate task similarity
leads to greatest forgetting. However, feature similarity is only one way in
which tasks may be related. The teacher-student approach allows us to
disentangle task similarity at the level of readouts (hidden-to-output weights)
and features (input-to-hidden weights). We find a complex interplay between
both types of similarity, initial transfer/forgetting rates, maximum
transfer/forgetting, and long-term transfer/forgetting. Together, these results
help illuminate the diverse factors contributing to catastrophic forgetting.
