{'fraction_non_alphanumeric': 0.04364411533714799, 'fraction_numerical': 0.010695987134896593, 'mean_word_length': 4.410764872521247, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

A series of recent works suggest that deep neural networks (DNNs), of fixed depth, are equivalent to certain Gaussian Processes (NNGP/NTK) in the highly over-parameterized regime (width or number-of-channels going to infinity). Other works suggest that this limit is relevant for real-world DNNs. These results invite further study into the generalization properties of Gaussian Processes of the NNGP and NTK type. Here we make several contributions along this line. First, we develop a formalism, based on field theory tools, for calculating learning curves perturbatively in one over the dataset size. For the case of NNGPs, this formalism naturally extends to finite width corrections. Second, in cases where one can diagonalize the covariance-function of the NNGP/NTK, we provide analytic expressions for the asymptotic learning curves of any given target function. These go beyond the standard equivalence kernel results. Last, we provide closed analytic expressions for the eigenvalues of NNGP/NTK kernels of depth 2 fully-connected ReLU networks. For datasets on the hypersphere, the eigenfunctions of such kernels, at any depth, are hyperspherical harmonics. A simple coherent picture emerges wherein fully-connected DNNs have a strong entropic bias towards functions which are low order polynomials of the input.

Due to the complexity of DNNs our current understanding of these features is still largely empirical. Notwithstanding, it was recently proven [Daniely et al. (2016); Jacot et al. (2018); Mei et al. (2018)] that keeping the DNN depth fixed and taking the number of parameters to infinity, learning in DNNs becomes analytically tractable. This is due to the fact that the networks' parameters, in all nonlinear layers, change in an important yet minor manner during training. Several subsequent works corroborated these results experimentally (Lee et al. (2018); Lee et al. (2019); Ben-David & Ringel (2019)) and verified that they hold at least for some real-world networks [Lee et al. (2019)].
The emerging picture is that DNNs trained with GD (full batch SGD) and an MSE loss are mappable to Bayesian Inference on a Gaussian Processes (GP) known as the Neural Tangent Kernel (NTK) [Jacot et al. (2018)]. Similarly DNNs trained using SGD at small learning rates are mappable to Bayesian Inference on a GP related to randomized DNN (an NNGP) [Lee et al. (2018); Ben-David & Ringel (2019)]. According to this, DNNs are merely a smart way of designing GP covariance-functions and a computationally efficient way of performing large scale inference with GP, which circumvents the costly matrix inversions involved (see below). An additional advantage is that internal representations are accessible in the DNN context, facilitating transfer learning.
While theoretically solid, the applicability of these results to real-world networks have been questioned by Chizat & Bach (2018) and dubbed "Lazy learning". For instance in AlexNet [Krizhevsky et al. (2012)], the parameters of the first convolutional layer undergo major changes during training despite having quite a large number of channels (96). It may thus be that AlexNet operates in a qualitatively different regime or possibly that one can gradually go from lazy-learning to feature-learning without drastically changing the overall predictions of the DNN.
With these developments in mind, it would be useful to have a clear, flexible, and systematic way of calculating learning curves for NNGPs and NTKs as well as means of departing from the Gaussian limit and accounting for finite width corrections. This would help explain to what extent does the GP viewpoint explain the aforementioned "pleasant features", and to what extent is lazy-learning different than feature-learning.
In this work we make the following contributions. (1) We derive closed analytic expressions for the eigenvalues and eigenfunctions of NNGP and NTK covariance-functions on the hypersphere. This is relevant to fully-connected networks and datasets wherein each datapoint has L2 norm of 1. Employing standard equivalence kernel (EK) results, we show how this explains generalization in fully-connected networks in the large dataset limit. (2) We develop a formalism based on statistical field theory and replicas, which allows for systematic departure from the large dataset limit and the infinite-width/channels limit. (3) Our formalism is then used to derive corrections to the learning curve which go beyond EK results and allow for further systematic improvement.

Background
Gaussian Processes are non-parametric models on which Bayesian inference can be performed analytically, up to a matrix inversion [Rasmussen & Williams (2005)]. The dimension of this matrix is the size of the dataset. The GP itself can be viewed as a Gaussian probability distribution over a space of functions (f (x)). In the DNN context the function f (x) describes the output of DNN given an input x [Cho & Saul (2009)] and can be viewed as an indirect representation of DNN's parameters. Such GPs, known as NNGP, have a clear meaning as representing the distribution over functions generated by the DNN at initialization. The NTK type GPs [Jacot et al. (2018)], have a similar yet slightly less direct interpretation as a certain sum of the NNGP along with a different NNGP wherein the DNNs activation function is replaced by its derivative. Notably the mapping between DNNs and GPs is more natural for regression problems but can also be applied to classification problems [Lee et al. (2018)].
Next we review the state-of-the-art on GP Bayesian Inference learning-curves. Learning-curves are graphs describing the generalization error as a function of data-set size (N ), averaged on all possible datasets of size N . Average learning-curves are learning-curves which are furthered averaged over all target functions/labels drawn from some prior distribution. Here we focus on explicit learning-curves wherein the resulting expression can be evaluated directly and one avoids the aforementioned matrix inversions which is computationally-costly and analytically obscure. For similar reasons we focus on cases where one has knowledge on the eigenvalues and eigenfunctions (features) of the covariancefunction. Considering average learning curves, analytic expressions are scarce except for very simple GPs [Rasmussen & Williams (2005)]. Assuming that the prior over targets matches that of the GPs (and an MSE error) lower bounds were obtained in [Opper & Vivarelli (1999)]. Considering (nonaveraged) learning-curves, using the Bayesian-PAC approach along with the Laplace approximation, Seeger (2003) managed to estimate learning-curves with a 7.5% error on binary MNIST with RBF convariance-functions. Considering MSE loss, large datasets (N ), and large noise, one has the EK result [Sollich & Williams (2005)]. The EK approximation replaces the discrete sum over data-points by an integral. This uncontrolled approximation can easily lead to false estimates (for instance at zero noise it yields zero error regardless of N , see below). In contrast, here we derive EK type results and improve upon them in a systematic manner.

Field Theory Formulation of GP learning-curves
Here we develop a field theory formalism for exploring learning curves. We begin with standard definitions of GPs and Bayesian Inference on GPs. Being Gaussian, the probability distribution on function f (x) drawn from GPs are determined by its first and second moments. The first is typically taken to be zero and second is known as the covariance function (K xx = E[f (x)f (x )], where E denotes expectation under the GP distribution). Notably, K xx of both the NNGP and NTK type can be calculated analytically for many activation functions [Cho & Saul (2009);Jacot et al. (2018)]. Furthermore, Bayesian Inference on GPs drawn from DNNs is tractable [Lee et al. (2018); Cho & Saul (2009)] and explicitly given by where x * is a new datapoint, g * is the prediction, g m are the training targets, x n are the training data-points, [K(D)] nm = K xn,xm is the covariance-matrix (the covariance-function projected on the training dataset (D)), σ 2 is a regulator corresponding to a noisy measurement of the GP and I is the identity matrix. Some intuition for this formula can be gained by verifying that x * = x q yields g * = g q when σ 2 = 0. While the above equation determines the predictions and therefore the learning-curves, it does not do so in any clear or computationally accessible manner. This fact is due to the (potentially very) large matrix inversion involved, and the additional averaging over D required.
To facilitate the analysis of Eq. 1 we turn to Feynman-path-integral, or equivalently to statistical field theory [Schulman (1996)]. These are well-studied, powerful approaches for performing integrations over a space of functions (the jargon is "paths" when x in one dimensional and "fields" when x is higher dimensional). To get some familiarity with this formalism, consider first averages over the (centered) GP itself with no dataset. Using the path-integral formalism we write it as where Df denotes integration over the space of functions, dµ x = P (x)dx is a probability measure over the inputs space and K −1 (x, x ) is the inverse covariance function To define the path-integrals one first chooses an orthonormal basis of functions φ i (x) (with respect to the measure µ x ) arranged in order of likeliness P 0 [φ i ] ≥ P 0 [φ j ] for i > j (note that this comparison doesn't require calculating the path integral in (2)). Second, one expands f = i f i φ i (x), and defines the path-integral as a series of simple integrals where F is some functional of f . Finally, one makes this last expression well-defined by taking a limit procedure where the number of integrals is gradually taken to infinity [Schulman (1996)].
Performing the above procedure we show in App. 2. that Notably, all other higher correlation functions split into products of the above correlation function due to standard properties of Gaussian integrals (Wick's/Isserlis' theorem).
Following a similar procedure, and denoting f 2 [Rasmussen & Williams (2005)] where Z is known as the partition function.
The next obstacle is dealing with the fact that expressions like Z appear in the numerator and denominator of Eq. (5). To by-pass this difficulty we introduce a source term into Z where α(x) is an arbitrary function of x. Next we note that Notably, the functional derivative ∂ ∂α(x * ) can be understood through a discretization limit: For discrete x, α(x) becomes a vector whose dimension is the number of discrete points, and the functional derivative means a standard partial derivative with respect to the discrete point x * .
The advantage of the above formulation follows from the identity and from the use the standard "replica trick" (see for instance [Gardner & Derrida (1988)]), where we assume M to be a non-negative integer, and extrapolate the limit at M → 0.
The generalization error is defined as (g(x * )−g * (x * )) 2 x1,...,xn∼µ . Therefore, in order to calculate learning-curves, one needs to be able to average quantities (like g * and g 2 * ) over all datasets of size N drawn from a probability distribution dµ x = P (x)dx. We denote this averaging by . . . µ,N . To facilitate this we begin by relaxing the requirement of fixed N and instead consider a related quantity given by the Poisson averaging of the former one where ... can be any quantity, in particular g * and g 2 * . Taking η = N , means we are essentially averaging over values of N which in a √ N vicinity of N . Following this we find that for a nonnegative integer M , Z[α] M µ,η can be re-written as where notably M "replicas" of f (x) emerged. Delegating the formal derivation to App. 4, here we just give the main observation which is that the n'th order taylor expansion of the exponent in M µ,N =n as expected. Therefore, substituting Eq. (8) into Eq. (7) and averaging using Eq. (9), g * µ,η can be written as The main benefit of Eq. (11) over Eq.
(1) is that it allows for a controlled expansion in 1/η. At large η (or similarly large N ) we expect that the fluctuations in f m (x) would be small and centered around g(x) (from a Bayesian point of view, the posterior would become very close to the target). Indeed such a behavior is encouraged by the term multiplied by η in the exponent. That be the case we can systematically taylor expand as shown in App. 4, dealing with the first order term in this expansion in an exact (Gaussian) manner yields the standard EK type results. The second order term and further terms render the theory non-Gaussian and cannot be dealt with exactly but rather through standard perturbationtheory/Feynman-diagrams. In App. 4 we show that g * µ,η is given by Assuming we know how to diagonalize K(x, x ), using a set of features φ i (x) and eigenvalues λ i Eq. (13) becomes Notably, similar expressions for g 2 * µ,η can be achieved by using two replica indices

Finite Width
While the focus of this work is on obtaining learning-curves perturbatively in 1/N , the field-theory formalism can be used for finite-width corrections as we briefly explain.
The NNGPs reflect the distribution over functions generated by DNNs at initialization [Cho & Saul (2009); Ben-David & Ringel (2019)]. It is Gaussian essentially because each of the output neurons sums over many uncorrelated activations from the preceding layer.  Rasmussen & Williams (2005) ch. 6.1] and may represent an NNGP.
As the width (W ) becomes finite, corrections to CLT appear. The most dominant ones, in 1/W , would is some function of the DNNs architecture. In principal, U (x 1 ..x 4 ) can be calculated using similar techniques to Cho & Saul (2009), and is closely related to the forth cumulant. Given U (x 1 ..x 4 ), perturbation theory can be carried in a straightforward manner as done here. For a study of finite-width corrections to NTKs see [Ethan & Gur-Ari (2019)]

Features and Eigenvalues of Fully-Connected Networks
To make the result in Eqs. (13,15) explicit and interpretable, one needs to find φ i (x) and λ i . This can be done most readily for the case of datasets normalized to the hypersphere ( x n = 1), a uniform probability measure on the hypersphere and rotation-symmetric covariance functions, meaning K(x, x ) = K(Ox, Ox ) for any O, where O is an orthogonal matrix over the space of inputs. Importantly, the NNGP and NTK GPs associated with any network whose first layer is fullyconnected, have this symmetry under rotations. Indeed, let h w (x) be the output vector of the first layer [h w (x)] i = φ( j w ij x j + b) where x j is the j'th component of the input vector x. Let z w (h) be the output of the rest of the network given h. The covariance function of NNGPs are defined by [Cho & Saul (2009) where P 0 (w, w ) is a prior over the weights, typically taken to be i.i.d Gaussian for each layer (P 0 (w, w ) = P 0 (w)P 0 (w ) and P 0 (w) ∝ e − ij w 2 ij /(2σ 2 ) ). Following this one can show where the second equality uses the definition of h w (x), the third results from an orthogonal change of integration variable w → wO, and the forth is a property of our prior over w.
Considering the particular case of a depth 2 network with one hidden ReLU layer, variance at the ith layer and no biases, we have that K(x, x ) in the (infinite) wide limit can be written as [Cho & Saul (2009)] sin θ+(π−θ) cos θ for θ=arccos(x·x ) Notably, the appearance of only x , x and x · x and their powers in the above expansion is not a coincidence, as these are the only rotationally-invariant quantities one can make out of two vectors.
Turning to the NTK type kernels, these are always made of sums of the NNGP type GPs so they inherent this symmetry as well.
Assuming uniformly distributed (on the sphere) normalized data, the covariance function is only a function of power of x · x . Dot-product kernels can be diagonalized using hyperspherical harmonics (Y lm (x)) [Azevedo & Menegatto (2015)], which are a complete (and orthonormal w.r.t a uniform measure) basis for functions on the hypersphere. For each l these can be written as a sum of polynomials in the x i 's of degree l. The extra index m enumerates an orthogonal set of such polynomials. Note that usually the hyperspherical harmonics are normalized w.r.t Lebesgue measure on the hypersphere, but in this context the normalization is w.r.t a probability measure on the hypersphere. For a kernel of the form K(x, x ) = n b n (x · x ) n the eigenvalues are independent of m and given by [Azevedo & Menegatto (2015)] where d is the dimension of the input (meaning that the inputs space is S d−1 ).
Importantly, as any covariance function of an NNGP or NTK type GP for fully-connected network (regardless of depth, width ratios or non-linearity) can be expanded in this manner, we find that the eigenvalues are given by (20) and the eigenfunctions are always simply φ i (x) = φ lm (x) = Y lm (x).
Notwithstanding the eigenvalues will vary between DNNs. For the case of Eq. (19) we get Interestingly, without biases we find that such ReLU networks do not contain any l = 3, 5, 7 . . . terms and so they are inadequate for any target function which does contain those features. The NTK covariance function of the same network has the following eigenvalues

Generalization in fully-connected networks
Here we combine the results of the two preceding sections to obtain a clear quantitative picture of learning curves and generalization in fully-connected networks. Let us consider first the standard EK result for fully-connected networks. We first expand our target (g(x)) into hyperspherical harmonics g(x) = l,m g lm Y lm (x) so the EK approximation yields Thus the GP prediction essentially cuts out all λ l σ 2 /η. Notably for input dimension d 1, Eqs. (21,22) both predict a very sharp decrease of λ l ≈ O(d −l ) for l d. Consequently there is typically a well defined threshold value (l c ) for which λ l<lc σ 2 /η and λ l>lc σ 2 /η. Taking this into account along with the fact that Y lm 's with l ≤ l c span all polynomials up to degree l c , one finds that the infinitely wide ReLU network with one hidden layer essentially finds the best polynomial of degree equal or less than l c that matches the data.
One immediate use of the correction evaluated in Eq. (15), is to estimate the range of validity of the EK results. Here we find η min ≈ (C K,η,σ 2 − σ 2 )/λ lmax , where C K,η,σ 2 = l,m 1 λ l + η σ 2 −1 (summing also over the degeneracy of λ l ) and l max is the l which maximizes m g 2 lm . This sharp cut-off on the degree of the estimated target generalizes to all fully-connected networks. Indeed note that K(x, x) is finite and therefore its integral over the hypersphere is also finite and given by dµK(x, x) = K(x, x) = lm λ l , and K(x, x) does not depend on x. The degeneracy (number of m's per l, denoted deg(l)) is fixed from properties of hyperspherical harmonics, and equals deg(l) = 2l+d−2 l+d−2 l+d−2 l [Frye & Efthimiou (2012)] which goes as O(d l ) for l d. Noting that λ l are all positive, to ensure a finite K(x, x), it must be that The following pictures emerges of what fully-connected DNNs do on datasets which are uniform on the sphere: They are simply a means of performing polynomial regression with polynomials of degree less than O(log(N/σ 2 )/ log(d) while sparing us the computationally costly N by N matrix inversion. Below we argue that following some data augmentations, the same may very well hold for real-world datasets.

Numerical Verification
We turn to numerically test the accuracy of our prediction for the generalization error. We consider a target function g(x) made out of hyperspherical harmonics with l = 1, 2 and the spectral weight distributed equally between the eigenspaces. We set g(x) = 1.
We generated datasests D N of N uniformally distributed points on the unit sphere S d−1 . For each N = 1, 2, . . . , N max we calculated the MSE of the Bayesian inference prediction of the target given D N and g(D N ), as in Eq. (1), and averaged over many D N draws. This quantity is the average MSE over datasets of size n MSE n . To compare with our analytic predictions, we performed Poisson averaging of MSE N in accordance with Eq. (9), truncating the summation at N max and taking N max − η max √ η max to avoid any noticeable truncation issues. The resulting quantity, MSE η , Figure 1: Solid lines indicate the experimental generalization error obtained by averaging the MSE of the Bayesian prediction over many datasets, them Poisson averaging with parameter η. The hyper-parameters for both kernels were chosen to be σ 2 w1 = σ 2 w2 = 1 and the input is in S 50−1 . Dotted lines indicate the traditional EK prediction of the generalization error, which is clearly inferior to the PC prediction (dashed line). The inset on the left demonstrates an inverted learning curve and how the PC prediction follows it. is shown in Figure 1. Our EK and pertubatively corrected EK (PC) predictions for the generalization error are plotted for comparison.
One immediately notices that both predictions converge to the experimental learning curve when σ 2 /η 1, and that the PC prediction is a notable improvement over the standard EK result. Another interesting feature is evident in the small η regime of our results. It has been shown [Sollich (2001)] that under certain conditions generalization error can actually increase with size of the training set in the small N regime, as is demonstrated by our experimental data. This effect is completely missed by the EK prediction of the learning curve, which is strictly monotonically decreasing. The corrected PC predictions shows potential of capturing it.
Notwithstanding, a real-world dataset would not be randomly distributed. Still one can perform PCA to extract its leading principal components and standardize these to have unit variance. As this all amounts to a linear transformation, it is plausible that such data augmentation would not hamper DNN performance. At the level of co-variances, this augmented dataset would now appear random and therefore one may speculate that the eigenvalues of the covariance-function sampled on the augmented dataset would resemble those obtained by random sampling. In App. 3, such a procedure is performed on the CIFAR-10 dataset. Both the random dataset and the augmented dataset are finally normalized to the sphere. Rather strikingly the difference in eigenvalues are minute. This raises hope that our results are more broadly applicable.

Discussion and Outlook
In this work we developed a formalism based on field theory tools for predicting learning-curves. Such a formalism may facilitate the transfer of knowledge between the physics community and ML community. Focusing on fully-connected networks and datasets which are uniform on the sphere, we derived closed expression for the features and eigenvalues of NNGP and NTK type GPs. Combining these results, along with the correspondence with DNNs, enabled us to explain why such over-parameterized networks generalize well and estimate their learning-curves. To the best of our knowledge, our learning-curves improve the current state-of-the-art on GP and DNN learning-curves. Our formalism can be used to derive further corrections and/or finite-width corrections, both of which are left for future work. Another interesting direction to pursue concerns the features and eigenvalues of CNN.

Bibliography 1 Re-stating the Notation
For completeness, here we re-state the notations used in the main-text.
Ω -Space pace of inputs.
K (x, x ) -Kernel function (covariance) of a Gaussian process. Assumed to be symmetric and positive-semi-definite.
can be assumed to be orthonormal: g (x) -The target function.

Explicit path integral computations
Here we wish to prove Eq. 3. of the main text using an explicit computation of the path integral.
3 Relaxing the uniformity assumption As discussed in the Numerical Verification section in the main-text in Fig. 1, we plot the comparising between an NNGP kernel sampled on random data and on properly augmented CIFAR10 data.

Gaussian Process Prediction as a Field Theory
Let us assume a Gaussian process (GP) with mean 0 and co-variance function K (x, x ). For a data-set D N of size N and noisy targets {g ( , it is known that the posterior mean obtained by Bayesian inference is For a data-set D N of size N and noisy targets {g (x i )} N i=1 , we present the GP canonical partition function: Where the Df notation stands for path integral. Notice that the functional derivative of log Z D N ,σ 2 [α (x)] w.r.t α (x * ) at α (x) = 0 yields: where the last equality is due to the fact that for Gaussian distributions, the expected value coincides with the most probable value. Therefore, the exact baysian inference mean:

Canonical Ensemble Formalism
for evaluating the quality of a certain GP, we're interested in the average prediction for all the data-sets of size N , meaning: Using the replica trick we obtain: for an integer M :

Grand Canonical Ensemble Formalism
We now wish to allow fluctuations in the value of N , meaning averaging over f C N,σ 2 (x * ) for different values of N . The motivation is to simplify the calculations, while averaging around a relatively confined region of N s. Let us average the canonical prediction while weighting N according to Poisson distribution with expected value η: and defining: we get: That is, the functional derivative w.r.t α (x * ) at α (x) = 0 yields the average prediction, averaged over different data-set sizes (the canonical averaging) and different data-sets for each size (the grand canonical averaging).
For a given η, the standard deviation of N is η, so the relative error is 1 √ η , decreases with η.

Deriving the Equivalence Kernel using the Grand Canonical Formalism
We wish to get rid of the exponent inside the exponent. Expending it using (first order) Taylor series:

Equivalence Kernel as Free Field Theory
Regarding the Equivalence Kernel as the free (quadratic) theory, we can denote f EK η, Let us calculate the correlations in the free theory: Therefore: and we see that the correlations are O 1 η . For rotationally invariant kernel, we get that Substituting the expressions for the variance and covariance:

Averaging f 2
This time we must use two different replica indices: Averaging w.r.t poisson distribution: 5 Various insights

Range of validity of EK
We define the validity of EK by comparing the EK result with the correction we derived earlier. When the two are comparable taking only the EK result is unjustified.