{'fraction_non_alphanumeric': 0.03499605965967135, 'fraction_numerical': 0.011120516038644524, 'mean_word_length': 4.3888015099087765, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving the Backpropagation Algorithm with Consequentialism Weight Updates over Mini-Batches

Least mean squares (LMS) is a particular case of the backpropagation (BP) algorithm applied to single-layer neural networks with the mean squared error (MSE) loss. One drawback of the LMS is that the instantaneous weight update is proportional to the square of the norm of the input vector. Normalized least mean squares (NLMS) algorithm amends this drawback by dividing the weight changes by the square of the norm of the input vector. The affine projection algorithm (APA) improved the NLMS algorithm to weight update over a batch of recently seen samples. However, the application of NLMS and APA had been limited to single-layer networks and adaptive filters. In this paper, we consider a virtual target for each neuron of a multi-layer neural network and show that the BP algorithm is equivalent to training the weights of each layer using these virtual targets and the LMS algorithm. We also introduce a consequentialism interpretation of the NLMS and the APA algorithms that justifies their use in multi-layer neural networks. Given any optimization algorithm based on the BP over mini-batches, we propose a novel consequentialism method for updating the weights.Consequently, our proposed weight update can be applied both to plain stochastic gradient descent (SGD) and to momentum methods like RMSProp, Adam, and NAG. These ideas helped us to update the weights more carefully in such a way that minimization of the loss for one sample of the mini-batch does not interfere with other samples in that mini-batch. Our experiments show the usefulness of the proposed method in optimizing deep neural network architectures.


Introduction
LMS algorithm is the common origin of neural networks and adaptive filters. While research on neural networks has been targeting multi-layer architectures, research in the field of adaptive filters has been mainly focused on single-layer architectures. In the field of neural networks, BP [1] [2, pp. 241-245] over mini-batches is the core method for training multi-layer networks, and any improvement to this algorithm is of great practical importance. In this paper, we try to transfer the improvements to the LMS algorithm in the field of adaptive filters to the training of deep neural network (DNN)s. In the following, we review some techniques that make the behavior of the network more predictable. This can be done, for example, by reducing the oscillations of the parameters around the optimal values, by preventing their deviation from the optimal path to the minimum, or by eliminating the redundant variables from the optimization calculations.
The learning curves of the SGD algorithm usually contain many fluctuations. This phenomenon may happen due to optimization over a small subset of input vectors or mini-batches at each iteration, which gives us a rough estimation [3]. Besides, SGD shows weakness in parts of the loss surface that bends sharply in one dimension than another [4, p. 296] [5]. We want to provide more clues to the source of such instabilities.
The momentum method [1] and its extensions such as Adam, NAG [5], Adagrad [6], and Adadelta [7] help to stabilize and speed-up the training by bringing some eigen components of the system closer to critical damping [8]. These methods reduce the oscillations around the optimum value of parameters. It is possible to reduce such oscillations by guiding them to move straight towards their optimum values.
Batch normalization (BN) [9] is an algorithm that helps to optimize the error independent of the change in the distribution of the input features that they called it internal covariate shift (ICS). They assumed it as a source of unpredictability during the training, which slows down the training by forcing one to pick lower learning rates and to initialize the parameters carefully. BN normalizes each feature of the outputs by using the first two moments by the statistics of each mini-batch. It also introduces two new variables to restore the representation power of the network by finding the preferred scale and shift over the training. These initiatives enabled them to use higher learning rates to achieve the same accuracy of the state-of-theart image classification with 14 times fewer training steps. The success of BN can be because of making the outputs of each layer, hence the inputs of the next layers more predictable, by eliminating one source of the unpredictability. The possibility of using a higher learning rate for the modified architecture can be an effect rather than the cause.
LMS algorithm trains adaptive filters [10] [11] in an online fashion by reducing the errors through an iterative process. An adaptive linear element (Adaline) [10] network [11] is a single-layer neural network with a linear activation function. However, subsequent investigations revealed the sensitiveness of the learning rate selection to the norm of the input vector [10], which makes it hard or impossible to pick a learning rate to make the convergence faster while maintaining stability at the same time. Besides, after the selection of a fixed learning rate, we should expect a kind of bias towards stronger inputs. While the filter learns the stronger inputs faster, the weaker inputs need more iterations. Additionally, it may easily be confused when suddenly encounters an outlier by jumping over a completely different area of the loss surface and forgetting the earlier learned parameters.
NLMS [12, pp. 333-337] in regression tasks, which is also known as α-LMS [10,13] in classification variants, can solve this problem. It is known as a method that adjusts the learning rate to cancel the problematic effect of the norm of the input vector.
Two questions ought to be answered to extend the success of NLMS to DNNs. First, how can someone generalize the underlying idea to a network with multiple layers? The second one arose after we extended it to a multi-layer perceptron (MLP), and observed a kind of instability during the mini-batch training. The second question is, what was the cause? To address the first issue, we show that each layer follows some virtual targets. We prove that optimization with BP is equivalent to using the LMS algorithm for independent layers. The second one can be solved by considering how each sample of the input deflects another sample from following its virtual target. This makes the outputs of each layer, which become the inputs of the next layer after an activation function, unpredictable. Moreover, the groups of neurons, not just a single neuron, determines the outcome of the neural networks. For example, networks with better generalization do not rely on individual neurons [14]. By considering vectors as a group of scalar inputs in on-line training, mini-batch training should be considered as its matrix extension [3]. These findings led us to use this matrix view instead of a vector view to handle the second case.
Similar efforts have been made before, for single-layer networks. Since our solution contains pseudoinverse, we later discovered that Ozeki and Umeda generalized NLMS in adaptive filters, by considering the batch of last seen inputs at each time step and solving it with a geometric viewpoint by an affine projection algorithm (APA) [15] [12, pp. 345-350] that reached to the same pseudoinverse formula.
Finally, one way to solve the optimization problem in neural networks is to consider some restrictions. For example, in [16] they optimized the weights as a solution of a constrained optimization problem. They suggested a margin-based cost function, which made their online algorithm more aggressive. They also put restrictions on weight changes and came up with an NLMS like formula with a slack term. As another example, Atiya and Parlos put constraints on gradients w.r.t. the outputs to train the recurrent neural networks [17]. In their method, they considered the states as the control variables. The weight modifications elicited from the changes in the states. As we show in Section 3, NLMS can also be explained by the gradients w.r.t. the outputs of the neurons instead of the weights. The paper proceeds as follows. First, the notations are introduced in Section 2. Second, some past works are analyzed in Section 3 to grasp the consequentialism idea and the basis for the next section. After that, we propose our method in Section 4. Next, we investigate the effectiveness of the algorithm with some experiments in Section 5. Finally, the paper concludes in Section 6.

Notations
In this section, we provide a summary of the notations which are used in this paper. Bold letters indicate vectors, and bold capital letters represent matrices. Lower case letters denote the matrix and vector elements. For matrix M, m i j refers to the ith row and jth column.
The new method applies to any type of layer. However, we study just the feed-forward networks throughout this paper to simplify notations. Other layers are just some special cases of the fully-connected (FC) layer. For example, the convolution operator almost always is implemented as a matrix multiplication by using the im2col method for forward and col2im method for backward [18].
Consider a feed-forward network with L layers (or L − 1 hidden layers) to be trained with batch size of N. D ( ) denotes the dimension of the output for layer ∈ {0, 1, . . . , L}. As two special cases, D (0) is the dimension of the inputs of the network, and D (L) is the number of classes in a typical classification task. For layer ∈ {0, 1, . . . , L}, we refer to the input matrix by X ( ) , which is D ( ) × N, and for layer ∈ {0, 1, . . . , L − 1}, we refer to the weight matrix by W ( ) , which is D ( +1) × D ( ) . Fig. 1 illustrates a 3-layer MLP network.
In the forward pass, by starting from the first layer ( = 0), the outputs of a layer are calculated by These outputs then pass through a non-linearity or activation function φ to create the inputs of the next layer as X ( +1) = φ(Z ( ) ). When we discuss a single layer, we suppress the layer number. In the final step, loss function L finds the discrepancy between the outputs of the network (X (L) ) and desired targets, which are referred by the matrix of T. As a special case, L MSE refers to the MSE loss function.

Finally,
← − ∆ denotes the gradients in the backward pass, and − → ∆ denotes our expected changes in the forward pass. For example, ← − ∆W is the matrix of gradients of the loss w.r.t. the weights and − → ∆W is the matrix of actual changes to the weights.

Lessons from the past
In this section, we briefly review BP, LMS, and NLMS algorithms. This background knowledge is needed to introduce our consequentialism idea, which we want to extend to mini-batch training of the BP. BP assigns the needed correction to each weight in a neural network by following the long path of the chain rule to that weight [19, p. 126] [20]. Besides, it utilizes dynamic programming to store the intermediate calculations to prevent the recalculation of them when applying the chain rule.

Algorithm 1 Forward phase for a feed-forward network
Require: input X (0) , an array of weight matrices W, target matrix T, number of layers L, and activation function φ 1: for = 0, 1, . . . , L − 1 do 2: 4: end for 5: Compute Loss = L(X (L) , T) 6: return X, Z, Loss The process begins by finding the derivatives of the loss w.r.t. the outputs of the network as follow: i is the gradient of the loss w.r.t. the x (L) i , and := means definition. It can be shown in the compact form of It is followed by applying the chain rule again to the activation function of the last layer. Therefore, for the previous layer we have: Next, the weight parameters should be changed in the opposite direction of the gradients. Consequently, for i ∈ {1, 2, . . . , D (L) }, j ∈ {1, 2, . . . , D (L−1) } and learning rate µ we have: which can be show in the compact form of For a mini-batch, the final weight changes are being calculated by summation of this individual weight corrections [19, pp. 127-128] of (11) as where N is the batch-size, ← − ∆z (n) is the nth sample in the matrix of the gradients of the outputs, and x (n) is the nth sample in the input matrix. After that, to calculate the gradients for the weight parameters of the former layers, one needs the gradients w.r.t. the inputs of this layer as follow where i ∈ {1, 2, . . . , D (L) } and j ∈ {1, 2, . . . , D (L−1) }. These are the gradients w.r.t. the elements of ← − ∆x that is needed for the bottom layer. The same pattern can be repeated to obtain the BP formula. The backward phase can be simplified to by considering mini-batches. Algorithm 2 shows the pseudocode of the backward phase for a mini-batch.

Algorithm 2 Backward phase for an MLP
Require: arrays of input matrices X, weight matrices X, output matrices Z, number of layers L, Loss and learning rate µ 1:

LMS
LMS is an algorithm for on-line training of a single layer network with linear activation function, hence φ (•) = 1. Consequently, the newtork computes z = Wx. The error of each output element of this vector is calculated by which t j , z j and e j are the jth elements of the ideal target, current output, and error vectors respectively. LMS computes the MSE loss as which yields Therefore, the derivatives of the loss w.r.t. the outputs are just the elements of the error vector. For i ∈ {1, 2, . . . , D (1) }, j ∈ {1, 2, . . . , D (0) } and learning rate µ the chain rule can be followed to find the weight update of in the opposite direction of gradients. This is the µ-LMS algorithm.

NLMS
Since the inputs and targets of a single sample is all that we have at the current moment, let us foresee the consequences of applying this weight modification on the error of the current sample: This is not ideal, because the actual correction of error also depends on the square of the second norm of the input. Changing the learning rate to can eliminate it from the equation. After that, by applying this new learning rate (24) becomes Finally, this new weight update yields As an outcome, NLMS minimizes the error regardless of the power of the input. This is the key idea that we call it consequentialism that we want to generalize it to BP. Another interpretation is that it minimizes the error in the opposite direction of the gradients of the loss w.r.t. the outputs.
To sum up, we want to foresee what happens after applying the weight update formula, then to improve it by considering the consequences of our actions.

Proposed method
In this section, we show that the negative direction of the gradients of the loss w.r.t. the outputs points to some virtual targets that each layer tries to achieve. We aim to follow these moving targets for each sample of the mini-batch more accurately, without making drastic changes to BP.
These are our objectives: At first, to introduce the idea of the virtual targets. After that, we want to extend the consequentialism view to the BP in on-line mode with the help of the virtual targets. Third, we explain the difficulty of extending it to mini-batch training. At last, we introduce our comprehensive formula.

Virtual targets
Here, we want to show that the on-line training of BP with any loss function is exactly like the training of each layer of the network in isolation by µ-LMS with virtual targets. From (17), for layer we have is just a vector of numbers at the end of the calculations, and (30) resembles (24). To prove the equivalence, we first define as virtual gradients, and as virtual targets vector for layer , where := means definition. By repeating the same steps of (19) to (24) again, we obtainẽ By substituting the real values of the virtual variables from (33) and (32) we get and finally, (36) becomes which is the same weight changes in (30), and the proof is completed. These targets are our best predictions of where the output vectors should stand in the next iteration for this very mini-batch. If we successfully control the outputs of each layer, which will be the inputs of the next layer after the nonlinearities, the inputs of the next layers should be more predictable, and the error of each sample should be decreased according to our expectations.

Simple extension
The main strength of NLMS, as we saw in Section 3, was the predictable minimization of error. In our first attempt, we take a straightforward approach. We want to extend NLMS to feed-forward neural networks by normalizing each single input vector. This can be done by creating a new normalized matrix X ( ) by the following definition for column n ∈ {1, 2, . . . , N} of a mini-batch:x ( ) where x ( ) (n) is the n-th input vector, and ≥ 0 can be a small positive number for stability, otherwise we can ignore it. In BP, (36) is generalized to matrices of mini-batches as This simple extension can be refuted by a simple counterexample. Let us test it on a single layer network. By defining the matrix X with two features and batch-size of 2 as X = x 11 x 12 x 12 x 2 12 +x 2 22 x 21 x 2 11 +x 2 21 x 22 Let us look ahead by forwarding the weight changes: This yields where we assumed = 0. This equation explains how one sample can interfere with the optimization of another sample. With this matrix removed from the calculations, we expect to optimize without the mentioned bias towards stronger inputs that we mentioned earlier in Section 1. Therefore, the only way to move precisely towards the virtual targets happens when the right side matrix is an identity matrix. This requires the nondiagonal elements to be zeros that is not true most of the time.

The main proposal
Let us have a consequentialism view at BP for a specific layer to find out where the calculations deviate from our expectations. In mini-batch mode, from (36) and (38), we have Now, we want to find out what exactly happens at the next iteration if we see the same mini-batch again. From (1) we have By substituting − → ∆W ( ) in the above equation by (46), we get the real changes in the error matrix: This is where a correlation matrix emerges. Thus, X 2 2 in (26) of µ-LMS becomes (X ( ) T X ( ) ) for a mini-batch in BP. We expect from (48) to even deflect some of the outputs from the virtual targets. This redundant calculation can confuse DNNs. The next step is to devise an analytical solution to remedy this problem. The idea is to modify the weight correction rule in such a way that produces an identity matrix instead of the mentioned correlation matrix. Therefore, we extend (29) to minibatch training and solve it. We wish to have − → ∆Ẽ matrix as our desired error reduction to be just The answer is the solution to where − → ∆ W is our ideal matrix of weight changes. By solving it, we find our desired formula of where the † is pseudoinverse operator. By repeating the substitution of − → ∆W in (47) by this new weight update rule of where I ( ) is the identity matrix. Therefore, by this new update rule, the outputs of each layer should move to the exact direction of the virtual targets, and we achieved a part of our goal.
Let us rewrite the pseudoinverse of the input vector in NLMS as wherex is the normalized input vector (unit vector) of x. This explains that the x T an adaptive learning rate. Besides, it indicates how the norm of the input affects the optimization twice. First, on the backward pass, then on the next forward pass. For a single scalar input, it means that instead of multiplication, we are dividing the gradient by the input to neutralize the multiplication in the next iteration. It is worth noticing that the new algorithm does not Algorithm 3 Backward phase for a feed-forward network with consequentialism weight updates Require: arrays of input matrices X, weight matrices X, output matrices Z, number of layers L, Loss and learning rate µ 1: Compute the X ( ) † with ridge regression 5: if ≥ 1 then Weight changes normalize the inputs, but it tries to cancel their effect just on the optimization of the activations. As another interpretation, it considers outputs as variables. In other words, it modifies the weights to change the outputs in such a way that minimizes the loss. The perfect learning rate for a single layer with a linear activation function in full-batch mode is µ = 1. However, for DNNs, the changes should be infinitesimal as the main assumption of the differentiation. We use the Moore-Penrose method with ridge regression [4, pp. 45-46] [21] for two purposes. In the first place, it can easily prevent the ill-conditioning issues. Besides, it also prevents the weights from changing dramatically, which may cause the violation of the chain rule as a consequence of our modification.
Therefore, (51) can be rewritten as We refer to the complete formula of (57) as the consequentialism weight update rule. The pseudocode of the proposed algorithm is shown in Algorithm 3.

Time Complexity
The most time-intensive parts of the neural networks are the FC and convolutional layers, and we explained in Section 2 that the convolution layer can be treated as a FC layer. In addition, the computations of the other layers, activation and loss functions are negligible in the big O notation compared to these layers. Thus, we just concentrate on the forward and backward propagation of the FC layers.
In this subsection, T denotes the total operations. From (17), total operations of the backward in vanilla mini-batch SGD for each FC layer is: We dropped the layer numbers to simplify the formulas. These operaions take steps. In addition, according to (1), for the forward pass we have: with the time complexity of Overall, BP has the total time complexity of: for L layers. In contrast, for the proposed method with Moore-Penrose method we have: For the first case with Naïve Gauss elimination [22, pp. 252-258], we have: and for the second case the total operations are Therefore, for weight updates formula, we have Since the dimenions of features are usually greater than the batch-size number, we conclude: Overall, both algorithms have the same time complexity in big O terms, but the inside coefficient constants differ.

Experiments
In this section, we evaluate the effectiveness of the proposed algorithm by several measures: loss surface of parameters, the convergence path of the outputs to the targets, learning curves of an MLP on multiple datasets, and training of ResNet [23] with different optimizers. We used a 3 GHz Pentium G2030 CPU and a GTX 750 2GB GPU for our experiments. MLP experiments were carried out in tensorflow [24] and ResNet experiments were performed in Caffe [18].

Loss surface
To begin, we wanted to know how the weights converge to their optimum values in the loss surface graph. In this experiment, we created a very simple single-layer network with just two separate inputs and outputs with MSE loss function, which is depicted in Fig. 2. The loss surface of two weight parameters and the path that they travel to the optimum values with different learning rates are also shown in Fig. 3.
As the graphs show, the proposed algorithm moves the parameters directly towards the optimum values, regardless of the learning rate. By contrast, SGD moves the parameters in the direction of the gradients which is perpendicular to the ellipses in the loss surface. While a weight parameter in SGD converges faster in one direction, it travels slowly in another direction. This is why increasing the learning rate to achieve faster convergence in one direction may cause some perturbations or amplifies them in another direction. However, this quivering may inject some noise to the calculations which can be good for generalization.
For the higher learning rate in the left graph, the momentum term reduced the oscillations of SGD, but it also caused the parameters to pass the optimum values because of the accumulated momentum. Since the momentum term is usually been used alongside the higher learning rates, it also can be an unexpected source of randomness.
In the right graph, applying a small learning rate eliminated the oscilations of the vanilla SGD and reduced it for SGD with momentum at the cost of more iterations for convergence.
To conclude, because the consequentialism makes the outputs of each layer more predictable, we expect fewer perturbations in more complex models.

The path from the initial outputs to the ideal targets
In our second experiment, we tested how the outputs converge to the targets during the training, in comparison to SGD in mini-batch training. The network of this experiment has a single layer with 20-dimension inputs and 2 output neurons. The mini-batch contains 10 samples of random inputs and outputs. The training was done in full-batch mode. Fig. 4 depicts how the outputs of 2 out of 10 samples converged to their targets for both SGD and C-SGD.
As the graphs illustrate, regardless of the learning rate, the proposed method moves the outputs directly towards the targets. Conversely, SGD, not only took an indirect path, but it also demonstrated more oscillations. While this usually happens in practice with higher learning rates, the stochastic nature of mini-batch also makes it less predictable.
The right graph of lower learning rate shows that even decreasing the learning rate does not lessen the fluctuations, because one or more of the input samples can affect the other samples from behaving predictably. Thus, the learning rate is not the only cause of the observed noise of the mini-batch SGD. These plots illustrate how the combination of increasing the learning rate and deploying the mini-batch training can cause the outputs of a layer, which will be the inputs of the next layer after a non-linearity in DNNs, to oscillate around the optimum values (virtual targets). Because of this, SGD has a higher chance of escaping from a local minimum. Moreover, the injected randomness may improve the generalization.

Fully-connected neural networks
Here, we investigate the effectiveness of the consequentialism version of the SGD against the plain SGD on one MLP architecture for 3 datasets. All FC layers in the following experiments had 8 hidden layers, and the batch size was 32. Weights were initialized by He's method [23]. All layers except the last one had rectified linear unit (ReLU) activation functions. The last layer had the softmax activation. Finally, we applied the cross-entropy loss. We used the full training set to report the loss. While the networks for CIFAR-10 [25] and Fashion-MNIST [26] had 800 hidden units for each layer, the network for CIFAR-100 had 1600 hidden units [25]. The inputs of the first layer was normalized to be between 0 and 1 for half of the experiments. λ was set to 1e − 3 for the consequentialism formula. For the other half, we normalized each feature to zero mean and unit variance (marked by NF), and λ to 1e − 2. Fig. 5 shows the loss graph of SGD against C-SGD over 100 epochs for all datasets based on iterations and elapsed time.
According to these experiments, C-SGD optimizes faster than SGD in the beginning when the distribution of the feature has not changed. Moreover, the graphs display more overall stability throughout the training for C-SGD. While the normalization of the features improved both algorithms, it brings more stability to SGD and makes its convergence faster and even competitive to the C-SGD on Fashion-MNIST. In contrast, the feature normalization decreased the stability of the C-SGD. Overall, C-SGD suffers less from the changes in the distribution of the inputs. The graphs also show that both algorithms will reach to a point that the loss drops sharply, then the progress would be slow, which occurs earlier for C-SGD. More oscillations around the virtual targets may be the source of a randomness that gives SGD a higher chance to find a better local minimum at the end.
Since C-SGD needs more time because of the pseudo-inverse operations, we also compared them by elapsed time. The righthand side graphs of Fig. 5 depict the comparison between both algorithms in seconds.
According to these experiments, the C-SGD maintains its competitiveness for CIFAR-10 and CIFAR-100 even if we take the time into account, but does not perform better for Fashion-MNIST. This can happen due to the more preprocessing for this dataset that the pseudoinverse does not help a lot. For example, the images have not much skewing, rotations, and significant portions of the images are just black. But the improvement is substantial for a more complicated dataset. Therefore, it enjoys sharper drops from the first epochs for CIFAR-100 rather than CIFAR-10. Finally, another interesting observation is that for more complicated inputs of the CIFAR-100, the feature normalization does not help much both algorithms.

ResNet-20
In this section, we investigate the effectiveness of the proposed method on a deeper architecture of the ResNet-20 [23]. We implemented the consequentialism weight updates for FC and convolutional layers in Caffe [18]. All tests were done on the CIFAR-10 dataset. We chose SGD [1], Nesterov [5], and Adam [27] from popular optimizers. Fig. 6 compares the loss and accuracy of the consequentialism method alongside their plain counterpart optimizers for two different architectures, in the presence and absence of the BN layers. λ was set to 0.03 and batch-size was 128. Momentum was 0.95 for SGD and Nesterov, and 0.9 for Adam. The second momentum for Adam was set to 0.999. The values were reported for the last minibatch of 100 iterations. The inputs were normalized between -1 and 1 for plain optimizers, and between -0.5 and 0.5 for the consequentialism method. The learning rate multiplier of the FC layer was set to 0.1 for the consequentialism experiments.
For the 3 top right-hand side graphs, we deployed unmodified models with BN layers, and for the left-hand side graphs, we removed the BN layers. As the non-BN graphs demonstrate, C-SGD and C-Nesterov performs better than SGD and Nesterov, C-Adam does not improve the results of Adam optimizer. However, the C-SGD outperformed the Adam/C-Adam optimizer. BN graphs show a conflict between the new method and BN algorithm. The combination of consequentialism and BN layers was unstable. The C-Adam optimizer displays strong oscillations, but with the help of this randomness, it eventually found a better minimum.
There were also two good reasons to compare C-SGD without BN layers with SGD and BN layers. Firstly, the effectiveness of the optimizers is not our main concern, because our method is not an optimizer, but an extension that can be combined to improve the optimizers. Secondly, SGD represents the best results in both cases. Thus, we compared them at the bottom of Fig. 6. According to the graph, while the oscillations of the BN increase after 10000 iterations, the consequentialism counterpart shows more stability. As these graphs were drawn for individual mini-batches, this is a sign that our method forgets less the previously trained mini-batches during the training, with a side-effect of less randomness, hence the lower chance of escaping the worse local minima.
While C-SGD achieved 100% train accuracy without forgetting any sample after about 8000 iterations, the SGD with BN, sometimes forgot some samples. Overall, C-SGD converged faster in the beginning and maintained its stability to the end of the training, but SGD with BN finally crossed the C-SGD line after 25000 iterations. Another interesting observation is the striking resemblance between their accuracy graphs. The above tests show that our method, like BN, does not suffer from the vanishing and exploding gradients. Now, let us compare them by considering the time instead of steps. Since the previous experiments show a conflict between consequentialism and BN, in the next figure, we did not use the combination of consequentialism optimizers with BN layers at all. Fig. 7 compares the loss and accuracy of the consequentialism method alongside their plain counterpart optimizers for two different architectures, in the presence and absence of the BN layers in seconds.
According to the left-hand side graphs, while C-SGD and C-Nesterov kept their advantages even by taking the time into account, C-Adam did not perform better than Adam. However, both C-SGD and C-Nesterov achieved better results.
From the right-hand side graphs, while SGD BN did well before about 6000 seconds, C-SGD crossed it for some time, but again SGD BN took the lead by crossing it at roughly 9000 seconds. As for Adam, Adam BN outperformed C-Adam. However, Adam optimizer did not do well overall compared to SGD and Nesterov. At last, Nesterov BN performed well in the beginning until about 3000 seconds that C-Nesterov easily crossed it, and kept its advantage to about 11000 seconds.

Conclusion and Future works
In this paper, we introduced two ideas. Firstly, a new way of improving weight correction in BP by anticipating the consequences of our actions in the hypothetical next iteration. It is equivalent to minimizing the loss with gradients of the outputs, via the weight updates. Secondly, we explained that one could treat the gradients of the outputs of each layer as the changes to be made towards some virtual targets. Next, optimizing by BP is equivalent to optimizing each layer by the LMS algorithm with virtual targets. As a result, it is possible to apply the successful algorithms from single layers to the deep neural network (DNN)s.
In spite of the faster convergence of the proposed method in the beginning and more stability during the training, it could get stuck at a worse local minimum in the long run. This weakness should be addressed in future works.
We also got mixed results from the time-based experiments. While for the MLP tests in Tensorflow, it was competitive, for the ResNet tests in Caffe, the processing demand almost eliminated its advantage. At least one of these two reasons can be the cause. Either the convolution with big mini-batch sizes does not benefit from the proposed method rather than using the BN layers, or our Caffe implementation was not optimized as well as Tensorflow. However, the second case is not easy to prove or dispute.