{'fraction_non_alphanumeric': 0.027242942417306223, 'fraction_numerical': 0.013195098963242224, 'mean_word_length': 4.739824832560536, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Memory Consolidation with Orthogonal Gradients for avoiding Catastrophic Forgetting

The memory consolidation process enables the accumulation of recent and remote memories in the long-term memory store. In general, the deep network models of memory suffer from forgetting old information while learning new information, called catastrophic forgetting/interference. The human brain overcomes this problem quite effectively, a problem that continues to challenge current deep neural network models. We propose a regularization-based model to solve the problem of catastrophic forgetting. According to the proposed training mechanism, the network parameters are constrained to vary in a direction orthogonal to the average of the error gradients corresponding to the previous tasks. We also ensure that the constraint used in parameter updating satisfies the locality principle. The proposed model’s performance is compared with Elastic Weight Consolidation on standard datasets such as permuted MNIST and split MNIST on classification tasks using fully connected networks, and Convolution-based networks. The model performance is also compared to an autoencoder on split MNIST dataset, and to complex core50 dataset on two types of classification tasks with EWC. The proposed model gives a new view on plasticity at the neuronal level. In the proposed model, the parameter updating is controlled by the neuronal level plasticity rather than synapse level plasticity as in other standard models. The biological plausibility of the proposed model is discussed by linking the extra parameters to synaptic tagging, which represents the state of the synapse involved in Long Term Potentiation (LTP).


Introduction
The human brain continuously acquires, processes, and uses information from the world throughout its lifetime. This lifelong learning is crucial for human survival and effective manipulation in the world since the organism needs to select and remember details of the world that enhance or compromise its existence. The process of formation of memories, particularly over the longer term, known as memory consolidation, predominantly occurs during sleep [1], [2]. Memory consolidation is a time-dependent process in which the recently acquired important information is transferred from the shortterm memory and integrated and stabilized in the long-term memory [3]. The importance can be attributed to various factors such as novelty, familiarity, and saliency [4]- [6]. In the consolidation process, the newly acquired memory is enhanced and integrated with the previously existing memories in the long-term store [7].
parameters except the task-specific output layer parameters. This network only uses the new task data for preserving the network performance on earlier tasks. Before training the network on a new task, the outputs for the new task data on each old task networks are estimated. During training on the new task, both the new task network loss and the old task network loss on the output are minimized together. Kirkpatrick et al. proposed Elastic Weight Consolidation (EWC), where the network parameter change is controlled with the help of the diagonals of the Fisher information matrix corresponding to the previous tasks [31]. This approach keeps a copy of the diagonals of the Fisher information matrix and the latest learned parameters. Zenke et al.
proposed the Synaptic Intelligence model, where the importance of each parameter for the previous tasks is estimated based on the change in parameter values [32]. Based on this value, the parameters with high importance refrain from learning, and the rest of the parameters are allowed to update. In Synaptic Intelligence [32], the importance is estimated online fashion, whereas in EWC, the diagonals of the Fisher information matrix are estimated after training each task. Farajtabar et al. (2019) proposed the orthogonal gradient descent method, where the gradients of the new task are projected to a subspace in which the gradients do not affect the output of the old tasks [33]. This method maintains a fixed number of gradients for each task and uses Gram-Schmidt process for finding the orthogonal direction.
Schwarz et al. proposed the progress and compress model, where two neural networks (active column and knowledge base) are used and trained in two phases [34].
During the progress phase, only the active column is trained with the new task. The connections between layers of the active column and knowledge base are established to enhance the reuse of encoded features as it allows the transfer of old tasks. During the compression phase, the active column is transferred to the knowledge base using the distillation process as proposed by Hinton et al. [35]. Maltoni

Methods and Results
To solve the problem of catastrophic forgetting in a continual learning setting, we proposed a method in which the parameters are updated by constraining the direction of update. Instead of giving importance to individual parameters just as in earlier methods like EWC and SI, the proposed model gives more preference to a particular direction for the update of the input parameters corresponding to a specific neuron. Here the parameters are constrained to move orthogonally to the previous direction. The need for this controlled update of parameters and the implementation method is described below.
To justify this special constraint and to define the way to update the parameters for a task, it is convenient to consider neural network training from a linear perspective.
Consider the below linear regression equation (a single neuron with linear activation function).

Figure 1 Linear Regression
Let X ∈ R m be the input, W∈ R m be the parameters, and Y∈ R 1 be the output.

Theorem 1
Let = and = + Δ . Here Δ ∈ R m is the change in W.
There exists W′ which satisfies = ′ if and only if Δ ⊥ .
. In the above linear regression problem, consider the loss function ℒ = ( − ) , where X is the input vector, W is the weight vector (parameters), Y is the expected output and Y'= W T X is the estimated output.

Proposition 1
If Δ = ℒ is the gradient vector, then both the X and Δ vectors represent the same unsigned direction.

Proof.
Consider the first-order derivative of the loss with respect to W: It can be observed that X and Δ are different in terms of magnitude. But they denote the same unsigned direction. This is valid for any activation function.
For example, if sigmoid activation function is used then Y = (X T W) and the firstorder derivative will be Δ = ( − ). . (1 − ). . By combining the above two observations, we can conclude that, if the change in the parameters Δ (while training the second data), is orthogonal to the gradient corresponding to the first data, Δ , then the representations generated from the first data will remain intact upon updating. This is reminiscent of null space control when combining controllers in classical control systems. If the second controller is in the null space of the first one, then the first control's stability/performance is not compromised.
This idea can be extended to any deep neural network by restricting the change in parameters to the orthogonal direction to the input vector. In deep neural networks, the inputs of each neuron can be related to the gradients of its input parameters.
To summarize, when the network is trained on the second dataset, if the parameters change in the orthogonal direction to the gradients of the first data at the neuronal level, then the representations of the first data used to train the network can be retained. This constraint can be accomplished by adding the following term to the loss function.
where ℒ is the extra loss term used along with standard loss; Δ , , is the gradient for the first data set A; , represents the current network parameter; , , * represents the trained parameters for the data A; n represents individual neuron and 'i' represents each input parameter to that particular neuron.
When the network is trained using this approach, the copy of the trained parameters ( * ) and the gradients ( Δ ) needs to be preserved. Thus, the network maintains three times the number of actual parameters. As the network is trained in batches, after training the first batch, the average of the normalized gradients is estimated and used for further training.
Here normalization is employed to give equal importance for all the data points in the batch.
To evaluate the proposed model, the performance on alleviating catastrophic forgetting is tested using deep neural networks. In particular, a network of four fully connected layers is trained on a sequence of classification tasks. Once the network is trained on a particular task, the network is not allowed for further training on any data from the same task. The model is evaluated on the permuted MNIST dataset and split MNIST dataset. The performance is also evaluated on complex networks such as CNN and Autoencoder using split MNIST datasets. The performance is also evaluated with the complex dataset CORe50. Although some methods perform better than EWC in continual learning settings, the realization of biological plausibility is not observed in these models.
Some approaches keep copies of task-related meta-data, such as gradients for each task.
These meta-data storage increase as the number of tasks increases [33]. The replay-based approaches keep the subset of earlier trained data [38]- [40] or use a generator network for sampling from earlier tasks [24]. Among the different methods, the proposed model uses a similar regularization-based mechanism as in EWC and SI. They also explain biological plausibility in terms of synaptic plasticity. So the performance of the proposed model is compared with that of EWC [31].

Dataset
Permuted MNIST For the permuted MNIST task, 1000 images are chosen from the MNIST dataset as task 1 (T1). For each task, one fixed random permutation is chosen and applied to the input pixels (784) of all the images. In this manner, three tasks are created, each having 1000 images.

Split MNIST
For the split MNIST, the MNIST images are divided into three tasks, each having only three classes. In each task, the network has to predict one among the three classes. Each task has the images of classes [0,1,2], [3,4,5] and [6,7,8] respectively. Here also, for each task, 1000 images are chosen. The data distribution for the split MNIST dataset is shown in Table 1.

Network Architecture
The network comprises 4 fully connected layers. It receives the input of size 784 and the data is processed by fully connected layers with 256, 128, 64, 10 neurons respectively.
For the split MNIST tasks, the last layer of the network uses 3 neurons. For both the tasks, the first two layers use the rectified Linear Unit (ReLU) activation function. The third layer uses sigmoid function and the final layer uses the Softmax activation function.

Training
For the first task (A), the network is trained to minimize the conventional crossentropy loss along with L2 regularization loss (Eqn. 4).
where ℒ is the loss function for the first task (A); ℒ is the cross-entropy loss; yn is the true probability of n th class, and pn is the estimated probability for the n th class; N denotes the number of classes; represents network parameters. For permuted MNIST dataset N = 10 and for split MNIST dataset N=3.
After training the network with the first task (A), a copy of the trained parameters * and the corresponding normalized mean gradients Δ , are retained and used for further training. These gradients are estimated for task A. These mean gradients are estimated as the average of normalized gradients. This normalization of gradient vector is implemented at the level of neurons.
For the second task, the network is trained with the loss function defined in equation The performance of the proposed model on retaining the old information is compared with EWC. In the EWC, the network is trained to minimize the loss function given in Eqn. 8.
where ℒ is the loss function used by the EWC. , = Δ , represent the diagonal elements of Fisher information matrix estimated from the first-order derivatives of the loss function with respect to the network parameters for task A, is the trade-off parameter.

Results
The performance of the proposed method is compared with EWC on both the permuted MNIST and Split MNIST tasks using fully connected deep networks. Here both the methods use the same network architecture. In order to evaluate the proposed method on complex networks, a split MNIST dataset alone is used on CNN-based classifier and Autoencoder. For evaluating a complex dataset, the CORe50 dataset is used.

Permuted MNIST
From the MNIST dataset, three permuted tasks, each with 1000 data samples, are created as described earlier. The first task, T1, contains MNIST images. The remaining tasks, T2 and T3, contain two different permuted versions of T1. Here the tasks are trained in the order T1, T2, and T3. Each task is trained for a fixed number of epochs. Once the network is trained for a particular task, it will not receive any data from that task again.
The network is trained on the first task T1 to 100% accuracy. When the network is trained on tasks T2 and T3 in order, the accuracy on each task is compared between the proposed model and EWC.

Split MNIST
Here the MNIST dataset is split into three tasks (T1, T2, and T3), each having 1000 images and, each image corresponds to one among three classes as described before. Here also, the first task, T1, is trained to 100% accuracy, and the level of retaining the accuracy for all the tasks is plotted for the EWC and the proposed model while training on the tasks T2 and T3.

Convolutional Network
The performance of the above models is evaluated on Convolution-based classification networks using a split MNIST dataset with three tasks (T1, T2, and T3). The network comprises 2 convolution (+max-pooling) layers followed by two fully connected layers. The first convolution layer uses eight 3x3 filters. The second convolution layer uses 16 3x3 filters. The max-pooling layer uses stride two following each convolution layer. The output of the 2 nd convolution layer is processed by the fully connected layers with 128 and 3 neurons, respectively. Here both the convolution layers use leaky-ReLU activation function, the first fully connected layer uses sigmoid function, and the output layer uses SoftMax function.
The first task is trained to attain 100% accuracy with L2 regularization, and the performance of the proposed model is compared with EWC while training the second and third tasks sequentially.      Figure 9 shows the accuracy of the three tasks when training with EWC. After training, the T1 gets the accuracy of 79%, T2 gets 60%, and T3 gets 90%. The three tasks together get an average accuracy of 76.3% for EWC.

Auto-Encoder
To evaluate if the proposed model can retain the old information on even more complex networks, we experimented with the continual learning task on an autoencoder.
The same split MNIST settings are used here. The number of data points for each task is set to 100.
The autoencoder comprises four fully connected layers. The network receives the MNIST image of size 784. The input is encoded by two fully connected layers with 256 and 128 neurons, respectively. The encoded feature is decoded by two layers with 256 and 784 neurons, respectively. All the layers use the sigmoid activation function. The network is trained to minimize cross-entropy loss. The network is trained using Adam optimizer.
The loss for the three tasks is compared between the proposed model and EWC. Similarly, Figure 11 shows the loss for the proposed method. At the end of the training, T1 gets the loss of .111, T2 gets .123, and T3 gets .078. Here the average loss for the proposed model is 0.104.

Core50 Dataset
To evaluate the proposed model on complex datasets, CORe50 is used. CORe50 is a complex dataset generated for continual object recognition problems [37]. It contains images of 50 household items under different conditions. The input image is of dimension 128x128x3.
Two types of problems are used for evaluating the performance of the proposed model. In the New instances (NI) case, each task will have the images corresponding to the same classes, but the environment (background, illumination, orientation, etc.) is different between tasks. Here we have considered three tasks, each having 1000 images. Each task consists of images corresponding to 5 classes.
In New classes (NC) scenario, each task will have images corresponding to different types of objects (classes). Each task includes 1000 images corresponding to 5 image classes.

Network Architecture
The network comprises two convolutional (+max-pooling) layers followed by two fully connected layers. The first convolution layer uses eight 3x3 filters. The second convolution layer uses eight 3x3 filters. The max-pooling layer uses stride 2 following each convolution layer. The output of the 2nd convolution layer is processed by the fully connected layers with 1024 and 5 neurons, respectively. Here both the convolution layers use the leaky-ReLU activation function, the first fully connected layer uses the sigmoid function, and the output layer uses the SoftMax function.
The loss and accuracy of the three tasks are shown while training them sequentially.
The first task is trained with standard L2 regularization to get the maximum accuracy of 100%. Figure 14 shows the loss for the three tasks while training them sequentially using EWC. Here at the end of the training, the tasks T1, T2, and T3 get the loss of .988, 1.474, and .218, respectively, with an average loss of .8934.   Here the average accuracy for all the tasks together comes to 95.06%. Table 2 shows the comparison of average loss and accuracy between EWC and the proposed model on CORe50 new instances task.      Table 3 shows the comparison of average loss and accuracy between EWC and the proposed model on CORe50 new classes task. It may be observed that the proposed model attains a higher accuracy and a smaller loss over EWC. The biological plausibility of the proposed model can be briefly explained by comparing it to two biological processes: memory consolidation and synaptic tagging.
During initial encoding, the hippocampus stores the recently acquired memory and acts as short-term memory [42]. Memory consolidation process is initiated by the hippocampus replaying the recently acquired memory [13], [14]. This consolidation enables the accumulation of recent memory along with the remote memory in the long-term memory store, and the memory becomes independent from the hippocampus that becomes remote memory. The proposed model does not include the short-term memory and the replay mechanism, which are the characteristics of the hippocampus. These are planned to be included in future studies.
Synaptic tagging is another mechanism that can possibly be related to the proposed model. Frey et al. (1997) proposed that the memory traces set a tag for each synapse that acts as a marker for change in the synaptic plasticity for the consolidation [43]. They also suggest that only the tagged synapses are allowed for consolidation [43]. The synaptic tag denotes the eligibility of a particular synapse for consolidation [44]. Though synaptic tagging was proposed as a mechanism of the neuronal system, recent studies show that astrocytes can play a crucial role in this process. Bosworth et al. (2017) reviewed that the astrocytes are involved in synaptogenesis and synaptic pruning, thereby enhancing the process of complex network formation [45]. Navarette et al. (2012) showed that astrocytes are crucial for cholinergic induced synaptic plasticity and contribute to information storage [46]. Kol et al. (2020) showed the importance of astrocytes in the formation of remote memories [47]. Thus, astrocytes could possibly be the key to this synaptic tagging process. The proposed model uses the copy of the average gradients from the previous tasks and the copy of the latest trained parameters, which can be considered as tags for individual synapses. There is a need for more studies to evaluate this kind of neural mechanism involved in memory consolidation.

Conclusion
The proposed method gives importance to certain directions while updating the synaptic weights/network parameters (at a neuronal level) rather than giving importance to each synapse (at the synapse level). It helps selectively choose the neurons (whose change in parameters is not orthogonal to the earlier task gradients) and update the parameters corresponding to those neurons faster compared to the other neurons. The proposed model showed better performance in all the tasks while comparing with EWC. This method keeps the copy of the parameters corresponding to the latest trained task and the average gradients for all the previously learned tasks. This provides a possibility of linking the mechanism of synaptic tagging to the parameters used in the neural network models. The model follows the locality principle, where the extra parameters related to a particular neuron alone influence/control the parameter updating. Similarly, the proposed method of memory consolidation provides a theoretical framework to evaluate memory formation and retention in the form of long-term memory. For future work, one possible direction is that instead of using the gradients' average, the gradients' important direction can be identified using PCA at the neuron level, and it could improve the performance. Another possible solution is imposing sparsity constraints on the parameters and evaluating the performance.

Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author Contributions
Simulations were done by TK and RK. The main text is written by TK. VSC contributed in providing the key ideas, editing the manuscript drafts, providing insight into the model.
BR contributed in providing key ideas, correcting the manuscript drafts. RM contributed in providing key ideas, correcting the manuscript drafts.