Lifelong Learning Process: Self-Memory Supervising and Dynamically
  Growing Networks
  From childhood to youth, human gradually come to know the world. But for
neural networks, this growing process seems difficult. Trapped in catastrophic
forgetting, current researchers feed data of all categories to a neural network
which remains the same structure in the whole training process. We compare this
training process with human learing patterns, and find two major conflicts. In
this paper, we study how to solve these conflicts on generative models based on
the conditional variational autoencoder(CVAE) model. To solve the uncontinuous
conflict, we apply memory playback strategy to maintain the model's recognizing
and generating ability on invisible used categories. And we extend the
traditional one-way CVAE to a circulatory mode to better accomplish memory
playback strategy. To solve the `dead' structure conflict, we rewrite the CVAE
formula then are able to make a novel interpretation about the funtions of
different parts in CVAE models. Based on the new understanding, we find ways to
dynamically extend the network structure when training on new categories. We
verify the effectiveness of our methods on MNIST and Fashion MNIST and display
some very insteresting results.
