A Study on Efficiency in Continual Learning Inspired by Human Learning
  Humans are efficient continual learning systems; we continually learn new
skills from birth with finite cells and resources. Our learning is highly
optimized both in terms of capacity and time while not suffering from
catastrophic forgetting. In this work we study the efficiency of continual
learning systems, taking inspiration from human learning. In particular,
inspired by the mechanisms of sleep, we evaluate popular pruning-based
continual learning algorithms, using PackNet as a case study. First, we
identify that weight freezing, which is used in continual learning without
biological justification, can result in over $2\times$ as many weights being
used for a given level of performance. Secondly, we note the similarity in
human day and night time behaviors to the training and pruning phases
respectively of PackNet. We study a setting where the pruning phase is given a
time budget, and identify connections between iterative pruning and multiple
sleep cycles in humans. We show there exists an optimal choice of iteration
v.s. epochs given different tasks.
