Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic
  Forgetting
  One of the major limitations of deep learning models is that they face
catastrophic forgetting in an incremental learning scenario. There have been
several approaches proposed to tackle the problem of incremental learning. Most
of these methods are based on knowledge distillation and do not adequately
utilize the information provided by older task models, such as uncertainty
estimation in predictions. The predictive uncertainty provides the
distributional information can be applied to mitigate catastrophic forgetting
in a deep learning framework. In the proposed work, we consider a Bayesian
formulation to obtain the data and model uncertainties. We also incorporate
self-attention framework to address the incremental learning problem. We define
distillation losses in terms of aleatoric uncertainty and self-attention. In
the proposed work, we investigate different ablation analyses on these losses.
Furthermore, we are able to obtain better results in terms of accuracy on
standard benchmarks.
