Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights
  Modification
  The ability of neural networks (NNs) to learn and remember multiple tasks
sequentially is facing tough challenges in achieving general artificial
intelligence due to their catastrophic forgetting (CF) issues. Fortunately, the
latest OWM Orthogonal Weights Modification) and other several continual
learning (CL) methods suggest some promising ways to overcome the CF issue.
However, none of existing CL methods explores the following three crucial
questions for effectively overcoming the CF issue: that is, what knowledge does
it contribute to the effective weights modification of the NN during its
sequential tasks learning? When the data distribution of a new learning task
changes corresponding to the previous learned tasks, should a uniform/specific
weight modification strategy be adopted or not? what is the upper bound of the
learningable tasks sequentially for a given CL method? ect. To achieve this, in
this paper, we first reveals the fact that of the weight gradient of a new
learning task is determined by both the input space of the new task and the
weight space of the previous learned tasks sequentially. On this observation
and the recursive least square optimal method, we propose a new efficient and
effective continual learning method EOWM via enhanced OWM. And we have
theoretically and definitively given the upper bound of the learningable tasks
sequentially of our EOWM. Extensive experiments conducted on the benchmarks
demonstrate that our EOWM is effectiveness and outperform all of the
state-of-the-art CL baselines.
