Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet
  Process
  Recent research efforts in lifelong learning propose to grow a mixture of
models to adapt to an increasing number of tasks. The proposed methodology
shows promising results in overcoming catastrophic forgetting. However, the
theory behind these successful models is still not well understood. In this
paper, we perform the theoretical analysis for lifelong learning models by
deriving the risk bounds based on the discrepancy distance between the
probabilistic representation of data generated by the model and that
corresponding to the target dataset. Inspired by the theoretical analysis, we
introduce a new lifelong learning approach, namely the Lifelong Infinite
Mixture (LIMix) model, which can automatically expand its network architectures
or choose an appropriate component to adapt its parameters for learning a new
task, while preserving its previously learnt information. We propose to
incorporate the knowledge by means of Dirichlet processes by using a gating
mechanism which computes the dependence between the knowledge learnt previously
and stored in each component, and a new set of data. Besides, we train a
compact Student model which can accumulate cross-domain representations over
time and make quick inferences. The code is available at
https://github.com/dtuzi123/Lifelong-infinite-mixture-model.
