{'fraction_non_alphanumeric': 0.04120704957525041, 'fraction_numerical': 0.013693419551160137, 'mean_word_length': 4.541877459246768, 'pattern_counts': {'":': 1, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analytic Learning of Convolutional Neural Network For Pattern Recognition

Training convolutional neural networks (CNNs) with back-propagation (BP) is time-consuming and resource-intensive particularly in view of the need to visit the dataset multiple times. In contrast, analytic learning attempts to obtain the weights in one epoch. However, existing attempts to analytic learning considered only the multilayer perceptron (MLP). In this article, we propose an analytic convolutional neural network learning (ACnnL). Theoretically we show that ACnnL builds a closed-form solution similar to its MLP counterpart, but differs in their regularization constraints. Consequently, we are able to answer to a certain extent why CNNs usually generalize better than MLPs from the implicit regularization point of view. The ACnnL is validated by conducting classification tasks on several benchmark datasets. It is encouraging that the ACnnL trains CNNs in a significantly fast manner with reasonably close prediction accuracies to those using BP. Moreover, our experiments disclose a unique advantage of ACnnL under the small-sample scenario when training data are scarce or expensive.


INTRODUCTION
N EURAL networks have become increasingly popular owing to the outstanding performance in various complex tasks, e.g., image-based tasks including image classification [1], object detection [2], [3] and image segmentation [4], [5]. However, these remarkable achievements have been built upon an exceedingly amount of training time even with the aid of modern parallel computing tools such as GPUs. A main cause is due to training based on the backpropagation (BP) [6] through multiple visits of the dataset. Moreover, such training mechanism often runs into issues related to search convergence. Firstly, updating the network through calculating gradients using BP could encounter vanishing or exploding gradients. Secondly, the nature of iteration-based learning could invite slow convergence given inappropriate initializations. Thirdly, BP learning requires multiple visits (i.e., epochs) of the dataset to acquire the necessary information. One usually needs hundreds of epochs before the training is complete, rendering the learning time-consuming and resource-demanding. Finally, the tunning of hyperparameters such as the learning rate, relies heavily on human experience, and without it the tunning could become very laborious.
These limitations of BP motivate the development of alternative training schemes. Several recent attempts adopted the analytic learning [7], [8], [9] to train the neural network.
Unlike the BP training paradigm, these analytic learning methods convert a nonlinear network learning problem into linear segments, which are then tackled by solving matrix equations. Apart from circumventing the aforementioned BP's limitations, the analytic learning also has advantages in network interpretability due to its simple matrix formulation in training [7], [10].
Conventional analytic learning started with shallow networks [11], [12], [13], [14]. A good example is the wellknown radial basis function (RBF) network [15]. Its parameters in the second layer are trained using a closed-form least squares (LS) solution after conducting a feature transformation in the first layer. The analytical solutions have also evolved towards training multilayer networks [8], [9], [16], [17], [18], [19], [20], receiving an increasing attention in recent years. For instance, the correlation projection network (CPNet) [9] partitions an multilayer perceptron (MLP) into multiple 2-layer subnetworks which are trained in a locally supervised manner. The most evident benefit of analytic learning for training neural networks is the fast single shot learning for training data of small to medium sizes.
The analytic learning has indeed shown promising fast training speed owing to its one-epoch training style. Yet, current studies can only handle the training of MLP weights, which can be referred to as MLP-based analytic learning. It is a natural motivation to extend such a technique towards more complex networks, such as CNNs. There have been efforts (e.g., [16], [21]) of involving the CNN training with analytic learning, but only the MLP weights at the final layer are analytically determined. There has not been any successful attempt for analytic learning of CNN weights. Such an analytic learning is challenging due to CNN's sparse structure (e.g., see [22] and section 2.4 [23]). The solution to CNN weight computation remains conjectural and primitive [17]. This has motivated us to explore solutions that push forward the analytic learning for fast training of more complex networks.
In this paper, we propose an Analytic Convolutional neural network Learning (ACnnL) for effective single-shot learning. Unlike existing analytic learning methods, the AC-nnL is able to obtain CNN weights using analytical solutions in a one-epoch training manner. This paper contributes as follows.
• To the best of our knowledge, the proposed ACnnL is the first analytic algorithm for training CNNs. It obtains CNN weights in closed-form while completing the training in a single epoch. • We demonstrate that the CNN structure can be treated as that of a generalized MLP, but differs in that the CNN is regularized by significantly more constraints. In this regard, we can justify to a certain extent why CNNs generalize better than MLPs (i.e., CNNs are heavily regularized).

•
In comparison with its BP counterpart, the ACnnL completes the training with a significantly faster speed (e.g., > 100×) and with a reasonably close generalization performance.
• On several benchmark datasets, the ACnnL is shown to significantly outperform the MLP-based analytic learning methods in terms of generalization. In particular, for small-sample datasets, the ACnnL is shown to excel in prediction comparing with that of BP. That is, the ACnnL is less prone to over-fitting for data-limited scenarios naturally due to the wellconditioned LS-based solutions.

MLP-based Analytic Learning
Existing methods of analytic learning are mostly restricted to the learning of MLP weights. These MLP-based methods begin at shallow networks (i.e., 2-layer networks) where the weight of the second layer is computed analytically after setting the first layer. For instance, the RBF network [15] is mainly characterized by the kernel projection in the first layer. Random-weighted networks [7], [24], [25], another category of shallow networks, are identified by the random weight selection in the first layer.
Extension of analytic learning to multilayer/deep networks has also been fruitful. The main difficulty behind analytic learning in multilayer networks is to deal with the learnings in hidden layers. One particular solution is to project the target label information into the hidden layers such that these hidden layers can guide their own supervised learnings. In [8], a kernel-and-range network (KARnet) projects the label information into hidden layers utilizing a sequence of Moore-Penrose (MP) inverses [26]. However, the KARnet requires the activation function to be invertible. Its generalization performance is less ideal especially on datasets with large sample size. The randomweighted networks are also extended to multiple layers [27] with a constraint of utilizing the invertible activation function like that in the KARnet. The correlation projection network (CPNet) in [9] adopts an alternative projection strategy named label encoding. The combination of correlation projection and the label encoding technique allows the CPNet to become one of the MLP-based state-of-the-art analytic learning methods.

Analytic Learning Beyond the MLP Structure
There have also been attempts and prototypes that incorporate analytic learning beyond the MLPs. Most of these methods are more related to transfer learning where the non-MLP networks are pre-trained elsewhere and used as feature extractors. The RBM-GI [16] adopts a feature extractor that has been pre-trained using the restricted Boltzmann machines (RBMs). Features are extracted by the RBMs and mapped to the label with an LS estimator. The deep analytic network (DAN) [21] with transfered features obtains remarkable results. However, aside from needing the feature extractor, the DAN requires fine-tunning of the network with BP to achieved the best performance. There are also methods without adopting pre-trained networks. In [10], a method named feedforward (FF) design adopts principal component analysis (PCA) to construct the CNN layers which are then followed by MLPs trained using the LS regression. The FF design trains the CNNs with significantly lower cost but expects a considerable loss of generalization performance. Other Non-BP Learnings: There are also other variants related to analytic learning (e.g., non-iterative or non-BP) in the form of stacking structure. The PCAnet [28] stacks PCA modules to learn multistage filter banks for extracting features. Like the analytic learning, the deep forest (DF) [29] also abandons the BP algorithm to improve the learning efficiency.
In summary, variants of the analytic learning attempts could be deployed across various networks structures. However, only the weights of MLP are obtained analytically due to the difficulty to formulate closed-form solutions beyond the MLP structure. In [17], the CNN is treated as a special case of a sparse MLP network, and a prototype of analytic learning for CNN weights is constructed. However, the formulation is primitive due to nontrivial assumptions and constraints. The proposed ACnnL proposed in this paper is able to train CNN weights with closed-form solutions, which closes an important research gap in the literature of analytic learning.

PRELIMINARIES
Here some background knowledge related to analytic learning is revisited. This mainly includes the LS solution, the CNN structure and its representation.

Notations
For consistency, we unify the notations used in this paper as follows. Without specification, a non-boldface character (e.g., a or A) denotes a scalar number. A boldface lowercase character (e.g., a) represents a column vector. A boldface uppercase character (e.g., A) indicates a matrix or a tensor 1 . a ∈ R K denotes a column vector a with K entries. A 3dimensional (3-D) tensor is represented, for instance, by A ∈ R N ×K×T of size N × K × T . Also in this example, A[n 1 : n 2 , k 1 : k 2 , :] indicates a sliced sub-tensor from A with N, K indexes ranging from n 1 to n 2 and k 1 to k 2 respectively. In particular, in the last dimension T , the subtensor takes all the entries by marking a colon ":".
Here we define square bracket [ ] as the concatenation operator, which concatenates multiple vectors/matrices into one. For instance, let A 1 ∈ R a1×b1 , A 2 ∈ R a2×b2 and A 3 ∈ R a3×b3 . If a 1 = a 2 = a 3 , we can have

Least Square Problem
Assume that there is a dataset with N data samples {x n1 , x n2 , . . . , x nK , z n } N n=1 . Specifically, each of the n th sample has K features with x nk being the k th feature, and z n being the label/output. Suppose that these features follow a linear combination to represent the label. That is, where n is a noise term with Gaussian distribution with zero mean. On the N -sample dataset, the formulation in (1) can be written in matrix form: where w ∈ R K , z ∈ R N and X ∈ R N ×K are the weight vector, the label vector, and the data matrix respectively, constructed via The formulation in (2) is the model for linear regression. To search for an optimal w, one minimizes the following cost function: where · 2 indicates the l 2 -norm. The best estimateŵ for (3) is given by:ŵ where γ is a factor that controls the regularization of w. The solution in (4) is optimal since the cost function z − Xŵ 2 2 + γ ŵ 2 2 is the smallest given any w ∈ R K . For γ → 0, (4) can be rewritten aŝ where † is the MP inverse operator.

Representation of Convolutional Neural Networks
The operation in CNNs is well interpreted through a graphical aid (see Fig. 1(a)), where the convolution is conducted in an iterative manner. Here we put the operation in detail as explicit formulas to prepare for the subsequent development of ACnnL.
We use X l ∈ R C l ×W l ×H l , a 3-D tensor to represent the input feature map at layer l. In detail, X l is a feature map having C l channels with W l × H l pixels in each channel. Let W [j] l ∈ R C l ×K l ×K l be the j th filter weight with kernel size K l × K l at layer l. Also let Z l be the output tensor at layer l. A convolution operation using filters {W [1] l , W [2] l , . . . , W where f conv denotes the convolution operator. Let P and S indicate the padding and striding operations. After the convolution, we have . For simplicity, in this paper we adopt zero padding and a stride of one (i.e., Z l ∈ R J×(W l −K l +1)×(H l −K l +1) ) for illustration purpose. The convolution operation can then be formulated via where is an operator conducting element-wise multiplication between two matrices, with

ANALYTIC CONVOLUTIONAL NEURAL NETWORK LEARNING
Here we present the proposed ACnnL in details. Essentially, we reformulate the iteration-based interpretation of CNN in Section 3.3 into a matrix multiplication form. Subsequently, we adopt the label encoding technique [9] to introduce the label information in each hidden CNN layer. This allows us to convert the nonlinear CNN learning into a linear one where the CNN weight can be analytically trained.
To the best of our knowledge, the ACnnL is the first analytic learning algorithm for training CNNs. For simplicity, in this paper we only discuss the vanilla CNN without considering advanced structures (e.g., ResNet [31]). Specifically, we discuss an L-layer vanilla CNN structure containing L − 1 CNN layers followed by one MLP. The general structure of ACnnL is shown in Fig. 2.

Convolution Represented by Matrix Multiplication
We shall derive an equivalent matrix multiplication form for the conventional formulation in (7) (see Fig. 1(b)). The first step is to combine the set of filter weights {W [1] l , W We flatten and concatenate the weight tensors into a 2-D matrix as follows: where W l ∈ R (C l K 2 l )×J . The notation F 2 is adopted, indicating that the flattening operation is employed twice such The convolution can also be equivalently implemented using a matrix multiplication with the filter weights flattened and the feature maps arranged in matrix form. This is similar to the operation called general matrix multiplication (or image-to-column transformation) [30]. The general structure of the proposed ACnnL. The ACnnL includes several steps. Firstly, the label information is projected into hidden CNN layers using a linear transformation to cope with dimension mismatch. Secondly, for each CNN layer, the input feature map and CNN weights are converted into 2-D matrices according to the feature and weight transformations (e.g., see Fig. 1(b)). Finally, the CNN weights are computed analytically by conducting a linearized locally supervised learning (e.g., see (14), (18) and (21)).
that the 3-D tensor W j l is brought into a column vector. For convenience, the above weight-flattening operation can be written as f wflat : For feature map X l , we do similar operations to convert it into a 2-D matrix. This include two steps. Firstly, we flatten the matrices along the channel axis on patches of the feature map , i.e., X l [c, w : w+K l , h : h+K l ] in (7), and concatenate them as follows: where c l,w,h ∈ R 1×C l K 2 l is a row vector. Next, we further concatenate these vectors into a matrix: The above steps can be compactly denoted as the following transform I which maps the feature map into a 2-D matrix. Conversely, I −1 performs the opposite operation, i.e., I −1 (I(X l )) = X l . Hence, the convolution process in (7) can be equivalently converted into a matrix multiplication form as follows: . The output tensor Z l identical to that obtained in (7) can be adopted by rearranging the elements through F −1 (F(Z l )). The transformation in (8)-(13) is similar to the operation called general matrix multiplication (or image-to-column transformation) [30] implemented by various deep learning platforms (e.g., PyTorch [32]) to accelerate the CNN operations. Owing to the above transformations, the convolution operation can be implemented in matrix product form. Such a representation is in fact a rather practical trick in computers to accelerate the implementation of CNNs. This is because the matrix multiplication can be highly parallelized using acceleration devices such as GPUs. More importantly, we are able to elevate this representation to facilitate analytic learning of the CNN weights, which is formulated in the following subsection.

Analytic Learning of CNN Weights
To formulate an analytic learning of CNN weights, the key is to convert the learning into a linear problem and provide each layer with corresponding label information. Hence, the learning in each hidden layer can be treated as a linear network learning problem where the resultant LS formulation can be solved.

Label Encoding
The label information can be projected into hidden layers utilizing the so-called label encoding technique [9]. Given an N -sample label matrix Y ∈ R N ×K (e.g., label Y has K classes), pseudo labelZ l at layer l can be generated viā where Q l is a label encoding matrix generated with each element sampled from a normal distribution. The projection is linear in order to match the dimensionality of Z l . For instance, if we need to project the label information into a hidden layer with J( In particular, the projected label can be rearranged by F −1 (Z l ) to have the same shape as that of Z l .
The encoding does not lose information since we are projecting lower-dimension data into higher ones. This also holds true for classification problems, since the output entries (i.e., number of classes) are normally in smaller number than the number of nodes in hidden layers.

Linear Formulation and Solution
By introducing supervised information into hidden layers, we can leverage the label encoding technique to achieve analytic learning. To reach this goal, the matrix representation of CNN in Section 4.1 is the key. The matrix multiplication in (13) allows us to construct a learning problem in a linear manner using the convolution weights {W [1] l , W l }. Single-sample case: For simplicity, the single-sample learning case can be formulated as It is simple to obtain the optimal solution to the linear matrix equation in (15), i.e., The above single-sample example is to straightforwardly demonstrates that the CNN filtering weights can be calculated by solving the linear matrix regression problem. Such a layer-wise learning has been proved effectively in various existing analytic learning methods. The more general version (i.e., multiple-sample case) can be extended to in a similar manner, but with additional efforts as shown below. Multiple-sample case: To fit multiple samples, the optimization in (15) must be expanded to where the superscript [·] (n) indicates the n th sample of the total N samples. The multiple-sample optimization problem in (17) appears to be of significant modification based on the singlesample case. However, it only needs a fair amount of additional work to obtain the solution. We summarize it in the following Theorem.
Proof. We take derivatives with respect to (w.r.t.) W l on the objective function in (17), which leads to Putting the derivative in (19) to 0 gives which completes the proof.
For γ → 0, we havê The form of (20) will be more useful than that of (18) in the following development (e.g., see (27)). After estimating the convolution weights, we feed the input feature map X (n) l to the activation function g l as follows: Next, the activations {X l+1 , . . . , X l+1 } serve as inputs to compute the convolution weights at layer l + 1 according to (14), (18) and (21). Also, if needed, one can reshape the activation back into the traditional 3-D tensor through The training of CNN is conducted in a stacking manner following [9], [21]. The convolution layers are mainly used for extracting useful features. After completing the training of CNN layers, an MLP layer (e.g., classifier) follows to map the extracted features to the target label.
Firstly, the 3-D feature tensor X (n) l ∈ R C l ×W l ×H l is flattened into a 2-D matrix X: Subsequently, the flatten feature X l is mapped onto label Y by optimizing where the solution iŝ For γ → 0, we haveŴ For an L-layer CNN, the weights of CNN layers or MLP layers can be obtained analytically layer-by-layer in a stacking fashion. Owing to the closed-form solutions, the training starts from the first layer and ends at the last without visiting the dataset more than once. That is, the ACnnL trains CNNs with merely one epoch, consistent with the requirements met by existing analytic learning methods [9], [10], [25]. We summarize the ACnnL in Algorithm 1. (1) 1 , X

ANALYSIS
In this section, a theoretical analysis of the proposed AC-nnL is given. The representation capacity of the proposed ACnnL is discussed. We shall provide evidence regarding ACnnL's potential to perfectly fit the training samples given sufficient parameters. More importantly, we show, from the structural point of view, that the CNN structure can be treated as a generalized MLP, but differs in that the CNN is more heavily regularized and therefore has better generalization ability.

Discussion of Representation Capability
Given sufficient parameters, the representation capacity can be shown by proving thatŴ l obtained using (20) can perfectly fit the set of training sam-ples {{X l . Subsequently, we can summarize the representation capacity in the following theorem.
Theorem 2. At layer l, If X has full row rank, the convolution layer can perfectly fit the learning samples, i.e.,Ŵ l obtained from (20) allows the objective function (with γ = 0) in (17) to become 0.
Proof. Having known that the solution in (20) can be equivalently rewritten by (27), after the training, the objective function in (17) can be replaced by If X has full row rank, naturally we have XX † = I. This allows a minimum of 0 for the objective function in (29).
Clearly, to generate a matrix with full row rank, a necessary condition is that X must be a "fat" matrix by choosing appropriate channel size C l and kernel size K l such that That is, sufficiently large C l and K l could allow X to be of full rank, thereby leading to a perfect fitting of training samples. From a linear regression point of view, a larger C l or K l could potentially increase the fitting performance as the rank of X could increase even when (30) is not yet satisfied.

Interpretation of CNN Structure through Regularization
As shown in (20), the training of CNN weights boils down to solving a linear matrix equation by inverting the data matrix X ∈ R N (W l −K l +1)(H l −K l +1)×C l K 2 l given a total of N samples. This closely resembles the MLP training measure (e.g., see (26) or [9], [25]). Such resemblance is natural given the fact that the analytic learning can only be achieved by somehow converting the network learning problem into a linear one.
There are certainly distinctions between the proposed ACnnL and its MLP counterpart. The most representative one can be described by the extent to which the two training methods are regularized. This can be elaborated from a linear regression point of view as follows.
Recall that one data sample becomes a matrix after transformation, e.g., . This differentiates it from the existing MLP-based analytic learning methods (e.g., see (26) and [9], [25]) where one sample is presented by one row vector only. We take the MLP weight learning in (24) as an example where such a solution can be interpreted by solving a regression problem constrained by N linear conditions. The ACnnL trains CNN layers by solving (17) yielding a solution given by (27). In other words, the learning is through solving a linear problem that is constrained by a total of N (W l −K l +1)(H l −K l +1) linear equations (i.e., X has N (W l − K l + 1)(H l − K l + 1) rows as shown in (28)). That is, the number of linear constraints has increased by (W l − K l + 1)(H l − K l + 1) times. For small kernel size K l in a large image (i.e., large W l and H l ), the increase can be tremendous. Hence, we may interpret the ACnnL as a more heavily regularized version than its MLP counterpart. This is why CNNs usually generalize better than MLPs. Conclusion made in [33] claims that the network structure (e.g., the CNN) is a form of implicit regularization, with no further theoretical findings provided. However, such a claim can be supported by our analysis here. Rather encouragingly, we provide evidence from a linear regression angle to validate that the structure is indeed a form of regularization. Specifically, the CNN is a form of regularized MLP by introducing more localized linear constraints.
Although the above explanation of generalization is facilitated by comparing different analytic learning methods, it should, to a reasonable extent, apply universally. As shown in (13), the CNN layer can be converted into a linear matrix multiplication form. Hence, it can be treated as a generalized version of MLP with the sole difference being the number of feature vectors/constraints (e.g., the number of rows in X l ). Such a structural resemblance does not change whether or not the proposed learning scheme ACnnL is in play.

EXPERIMENTS
To evaluate the performance of the proposed training strategy, we train CNNs to conduct classification tasks on several datasets, including MNIST, FashionMNIST, CIFAR-10 and CIFAR-100. The performance of ACnnL is assessed in comparison with that obtained by BP and by other non-BP methods. The assessment mainly includes the generalization performance and the time consumed to complete the training. The detailed comparisons are conducted mainly between BP and ACnnL as other methods cannot train CNNs. Datasets. The MNIST and FashionMNIST have 10 classes with 50,000 images of 28 × 28 gray pixels for training and 10,000 for testing. The CIFAR-10 and CIFAR-100 datasets include 32×32 color images, with 50,000 images for training and 10,000 for testing. The CIFAR-10 and CIFAR-100 have 10 classes and 100 classes respectively.
The conducted experiments aim to validate the proposed ACnnL as the first analytic learning tool for training CNNs. To avoid complicating the validation, here we only train vanilla CNN structures (LeNet or VGG-like structures shown in next subsection) without involving advanced modules such as batch normalization [34] or dropout [35]. For the BP algorithm, we adopt 3 optimizers as follows: • Vanilla SGD, i.e., BP (VSGD), without momentum or weight decay.
The network is trained for 100 epochs. The learning rate begins at 0.001 and is divided by 10 at 30, 60 and 80 epochs. During the experiments, no data augmentation techniques are applied for BP and ACnnL. The generalization performance is measured by the testing accuracy, i.e., the accuracy on the testing set, after the training process. For iterationbased methods (e.g., BP), the testing accuracy is reported at the last epoch. No validation set is adopted. All experiments of BP and ACnnL are reported by the average of 3 runs on a workstation with Intel Xeon W-3265 Processor with 256G RAM and 2080 Ti GPU with 11G RAM.

Vanilla Convolutional Neural Network Structure
We experiment on a 5-layer 2 vanilla CNN containing 4 CNN layers followed by one MLP. For convenience, the structure is represented by indicates a K l × K l CNN layer with C l channels followed by an activation function g l .
[AvgPool(P × P )] represents the average pooling of size P × P with stride P for down-sampling the feature map. MLP(K) means the MLP layer mapping the current feature to a vector of K dimensions. For the classification problems in this experiment, K is the number of classes.
We use a consistent set of structural parameters for various datasets. Specifically, we set K 1 = 5, K 2 , K 3 , K 4 = 3, C 2 , C 3 , C 4 = 2C 1 , 4C 1 , 4C 1 and P = 2 respectively. The activation functions are chosen to be LeakyReLU with slope 0.1 for negative activations. We name this unified structure "CNN-5(C)", representing a 5-layer vanilla CNN with a tunable channel parameter C = C 1 . For instance, a "CNN-

Generalization Performance Evaluation and Comparison
6.2.1 ACnnL's generalization w.r.t. the selection of γ The solution in ACnnL is constructed based on the LS technique. The regularization factor γ could play an important role in determining the generalization ability. Hence, we explore the selection of γ with a range from 10 −5 to 10 5 to evaluate its impact. As shown in TABLE 1, the ACnnL with a regularization factor γ of a value around 10 2 gives satisfying generalization performance. A too large or too small γ leads to deteriorated generalization. For convenience, we adopt γ = 10 2 in the following experiments.

Overall Generalization Comparison
Here we give an overall evaluation of ACnnL's generalization in comparison with that of BP as well as the existing non-BP methods. The non-BP methods included for comparison are MLP-based analytic learning techniques (i.e., KARnet [8], PILAE [37], ANnet [38] and CPNet [9]) and DF [29]. As reported in TABLE 2, the ACnnL gives better accuracies than DF's on MNIST (e.g., 0.9931 v.s. 0.9926) and CIFAR-10 (e.g., 0.7049 v.s. 0.6337). Among the compared analytic learning methods, the proposed ACnnL significantly outperforms its MLP counterparts in terms of generalization. The performances deviate more evidently on relatively difficult datasets (e.g., CIFAR-10) than easy ones (e.g., MNIST). For instance, on MNIST the ACnnL gains a 1% lead over the ANnet while the lead becomes 20% on CIFAR-10 (e.g., ACnnL of 0.7049 v.s. ANnet of 0.4990). This shows that the ACnnL is an effective CNN trainer with analytical solutions. It also partly demonstrates that the CNN structure is in general more powerful than the MLP for dealing with image-based classification problems.
It is observed that MLP-based analytic learning methods suffer much from over-fitting more severely than the proposed ACnnL. For instance, on CIFAR-10 the ANnet obtains a training accuracy of 0.7599 but receives a testing accuracy of only 0.4990. The proposed ACnnL achieves a training accuracy of 0.8019 with a rather comparable prediction accuracy of 0.6706. This also validates our claim in Section 5.2. That is, the ACnnL trains neural networks by imposing significantly more constraints than those in MLP-based methods, thereby heavily regularizing the optimization for enhancing network generalization.
On the other hand, as an iteration-free training strategy, it is of obligation to investigate the prediction performance deviation from the commonly used BP. As indicated in TABLE 2, CNNs trained by BP (Adam) give better generalization than those trained by the proposed ACnnL. This is reasonable as the ACnnL forgoes the global feedback information and focuses on the local one in order to achieve analytic learning. The lower layers in the CNN trained by the ACnnL do not receive feedbacks from the upper layers, leading to certain under-fitting. However, it is encouraging to see that the ACnnL outperforms the BP (VSGD) and BP (SGD), e.g., ACnnL with 0.6706 v.s. BP (VSGD) with 0.2358 and BP (SGD) with 0.5818 on CIFAR-10. This demonstrates that the ACnnL can simply bring out a network's generalization power with a very limited tuning of hyperparameters (e.g., γ). The BP needs to carefully select an appropriate optimizer (e.g., Adam in this case) before the network's generalization fully appears.

ACnnL performance w.r.t. network width
We also evaluate the generalization performance of our ACnnL w.r.t. network width. This is done by training CNN-   TABLE 3, the network's generalization performance increases with a wider network structure. Such a performance gain is not surprising as the analytic learning is less likely to experience over-fitting when increasing the number of parameters [9]. Instead, the network desires more parameters to enhance its generalization. This is also consistent with the trend of traditional deep learning [39].

ACnnL performance w.r.t. network depth
In particular, we evaluate the impact of a network's depth on the generalization performance. In TABLE 4, the evolution of generalization performance w.r.t. network depth is reported by training CNNs of various depths on MNIST and CIFAR-10. A clear pattern of an increasing performance for a deeper structure is observed in the reported numbers. This is optimistic as our analytic method preserves the most important feature of current deep learning. That is, features extracted by deeper CNNs are usually more discriminative than those obtained by shallow ones.

Speed evaluation
The ACnnL trains CNNs with merely a single epoch leading to an extremely fast training speed. To demonstrate this, we measure the running times of BP and ACnnL for training CNN5(C=128) with CPU and GPU settings respectively . As shown in Fig. 3, training the network with GPU, our ACnnL spends around 40 seconds to complete the training, which is more than 17× faster than that of BP (more than 700 seconds for 100 epochs seconds)! Such a speed growth can be justified by the fact that GPUs cannot accelerate matrix inverse in the same way they accelerate matrix multiplication. In fact, comparing the speed using GPUs is less fair as GPUs are mainly built for BP-based iterative algorithms while matrix inverse is yet to be well supported. From the cost-efficient point of view, the proposed ACnnL obtains a reasonable trade-off between the accuracy (e.g., a 6% drop from 0.73 to 0.67 for CNN-5(C=256)) and the time consumption (e.g., 17× to 113× speed increase). The ACnnL experiences a marginal generalization degradation compared with the BP. However, it is motivating to see that training a network for merely one epoch could achieve most of its generalization potential. Yet, our goal is not to replace the BP-based iteration methods. Instead, the ACnnL is more of an alternative to trade accuracy for speed in a reasonable manner for certain scenarios where computing resources are limited (e.g., no available GPUs) or fast training is required (e.g., in near real-time training applications). The advantage of high training speed and interpretability is very encouraging.
More importantly, our goal does not stop at accuracyspeed trade-off. Instead, it is of great interest to incorporate the ACnnL with the existing BP such that a comparable generalization with that of BP can be obtained with the network trained for much fewer epochs. For instance, one might adopt the ACnnL as an initialization strategy and the BP as a subsequent adjustment (e.g., BP as fine-tuning). However, currently this incorporation is not trivial due to numerical differences. For instance, CNN weights trained with BP are usually designed to be small while the weights trained with ACnnL in the current phase exhibit a relatively random pattern (e.g., some weights could have large values) resulted from the matrix inversion. Connecting the ACnnL and the BP has potential and is to be investigated in future work.

Small-sample Performance
In this subsection, we conduct experiments with smallsample settings. Instead of utilizing all the samples from the datasets, we randomly select N c (with the total samples KN c ≤ N smaller than the original size) samples from each class and train the classifier from scratch using these selected samples. The small-sample experiment resembles but differs from the recently popular few-shot learning as no "support set" [40] is provided to pre-train the network. Therefore, we exclude the existing few-shot learning methods for comparison, and focus the evaluation against the BP to highlight the impact of analytic learning. We map the evolution of testing accuracy w.r.t. N c on CIFAR-10 and MNIST in Fig. 4. As shown in the figure, the proposed ACnnL outperforms the BP for small N c values. For instance, results on MNIST in Fig. 4(a) indicate that networks trained with ACnnL give better generalization for N c < 1000 samples. This is consistent with the pattern on CIFAR-10 in Fig. 4(a). Empirically, it is evident that our ACnnL gives better generalization performance than that of BP (here we adopt Adam optimizer as it gives the best performance among the 3 optimizers) in small-sample setting. Theoretically, this can also be explained by the "over-fittingavoiding" nature of the LS technique that plays a major role in the ACnnL.
The ACnnL having outstanding prediction performance is strategically advantageous to recent deep learning developments focusing on small-sample related tasks (e.g., fewshot learning). This trend is natural as the heart of deep learning has been hinged upon availability of sufficient data. As indicated in the upper panel of Fig. 4, we can see that networks trained by BP experience a much sharper climb of generalization with N c > 100 than that with small N c values. For the BP-trained networks, there appear to be a threshold of sample size, below which the generalization would be heavily suppressed. Contrarily, the proposed AC-nnL does not share the same weakness because the LS-based solution is less likely to invite over-fitting. This allows our method to become of value with reasonable accuracy in applications for scarce data (e.g., medical images that are expensive).
To summarize the empirical evidence from the above experiments, we highlight the insights as follows: • Among the studied analytic learning methods, the ACnnL leads to outstanding generalization performance (e.g., from 50% to 70% on CIFAR-10).

•
The ACnnL shows consistent patterns with the existing deep learning trend. For instance, the generalization ability increases with a deeper or wider structure.
• The ACnnL experiences mild degradation of generalization in comparison with its BP counterpart (e.g., BP (Adam)), but receives a significant boost of training speed due to the one-epoch training style.

•
The ACnnL is advantageous for training smallsample datasets from scratch. This is empirically and theoretically true as the LS technique establishes a major component.

CONCLUSION
In this paper, we have proposed an analytic convolutional neural network learning method called ACnnL, the first analytic method for training CNNs. The one-epoch training style of ACnnL completes the training in a significantly fast manner while only experiencing a mild generalization degradation comparing with that trained using the well-known back-propagation. Such a speed-generalization trade-off is consistent with that in the existing analytic learning community. Essentially, with the derivation of AC-nnL, we have theoretically uncovered why CNNs generalize better than MLPs in terms of imposition of localized constraints by solving linear matrix functions. The experiments have validated our proposed training scheme, showing a significantly improved generalization over its analytic MLP counterpart. In addition, the ACnnL has been empirically demonstrated to excel in small-sample scenarios, which bears great potential in data-scarce applications.