KASAM: Spline Additive Models for Function Approximation
Neural networks have been criticised for their inability to perform continual
learning due to catastrophic forgetting and rapid unlearning of a past concept
when a new concept is introduced. Catastrophic forgetting can be alleviated by
specifically designed models and training techniques. This paper outlines a
novel Spline Additive Model (SAM). SAM exhibits intrinsic memory retention with
sufficient expressive power for many practical tasks, but is not a universal
function approximator. SAM is extended with the Kolmogorov-Arnold
representation theorem to a novel universal function approximator, called the
Kolmogorov-Arnold Spline Additive Model - KASAM. The memory retention,
expressive power and limitations of SAM and KASAM are illustrated analytically
and empirically. SAM exhibited robust but imperfect memory retention, with
small regions of overlapping interference in sequential learning tasks. KASAM
exhibited greater susceptibility to catastrophic forgetting. KASAM in
combination with pseudo-rehearsal training techniques exhibited superior
performance in regression tasks and memory retention.