Theoretical Understanding of the Information Flow on Continual Learning
  Performance
  Continual learning (CL) is a setting in which an agent has to learn from an
incoming stream of data sequentially. CL performance evaluates the model's
ability to continually learn and solve new problems with incremental available
information over time while retaining previous knowledge. Despite the numerous
previous solutions to bypass the catastrophic forgetting (CF) of previously
seen tasks during the learning process, most of them still suffer significant
forgetting, expensive memory cost, or lack of theoretical understanding of
neural networks' conduct while learning new tasks. While the issue that CL
performance degrades under different training regimes has been extensively
studied empirically, insufficient attention has been paid from a theoretical
angle. In this paper, we establish a probabilistic framework to analyze
information flow through layers in networks for task sequences and its impact
on learning performance. Our objective is to optimize the information
preservation between layers while learning new tasks to manage task-specific
knowledge passing throughout the layers while maintaining model performance on
previous tasks. In particular, we study CL performance's relationship with
information flow in the network to answer the question "How can knowledge of
information flow between layers be used to alleviate CF?". Our analysis
provides novel insights of information adaptation within the layers during the
incremental task learning process. Through our experiments, we provide
empirical evidence and practically highlight the performance improvement across
multiple tasks.
