Learning by Active Forgetting for Neural Networks
  Remembering and forgetting mechanisms are two sides of the same coin in a
human learning-memory system. Inspired by human brain memory mechanisms, modern
machine learning systems have been working to endow machine with lifelong
learning capability through better remembering while pushing the forgetting as
the antagonist to overcome. Nevertheless, this idea might only see the half
picture. Up until very recently, increasing researchers argue that a brain is
born to forget, i.e., forgetting is a natural and active process for abstract,
rich, and flexible representations. This paper presents a learning model by
active forgetting mechanism with artificial neural networks. The active
forgetting mechanism (AFM) is introduced to a neural network via a
"plug-and-play" forgetting layer (P\&PF), consisting of groups of inhibitory
neurons with Internal Regulation Strategy (IRS) to adjust the extinction rate
of themselves via lateral inhibition mechanism and External Regulation Strategy
(ERS) to adjust the extinction rate of excitatory neurons via inhibition
mechanism. Experimental studies have shown that the P\&PF offers surprising
benefits: self-adaptive structure, strong generalization, long-term learning
and memory, and robustness to data and parameter perturbation. This work sheds
light on the importance of forgetting in the learning process and offers new
perspectives to understand the underlying mechanisms of neural networks.
