{'fraction_non_alphanumeric': 0.04083850387352333, 'fraction_numerical': 0.014407403512461878, 'mean_word_length': 4.273197781885397, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
How to Start Training: The Effect of Initialization and Architecture

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.


Introduction
Despite the growing number of practical uses for deep learning, training deep neural networks remains a challenge. Among the many possible obstacles to training, it is natural to distinguish two kinds: problems that prevent a given neural network from ever achieving better-than-chance performance and problems that have to do with later stages of training, such as escaping flat regions and saddle points [12,27], reaching spurious local minima [5,15], and overfitting [2,34]. This paper focuses specifically on two failure modes related to the first kind of difficulty: Our main contributions and conclusions are: • The mean and variance of activations in a neural network are both important in determining whether training begins. If both failure modes FM1 and FM2 are avoided, then a deeper network need not take longer to start training than a shallower network.
• FM1 is dependent on weight initialization. Initializing weights with the correct variance (in fully connected and convolutional networks) and correctly weighting residual modules (in residual networks) prevents the mean size of activations from becoming exponentially large or small as a function of the depth, allowing training to start for deeper architectures.
• For fully connected and convolutional networks, FM2 is dependent on architecture.
Wider layers prevent FM2, again allowing training to start for deeper architectures. In the • For residual networks, FM2 is largely independent of the architecture. Provided that residual modules are weighted to avoid FM1, FM2 can never occur. This qualitative difference between fully connected and residual networks can help to explain the empirical success of the latter, allowing deep and relatively narrow networks to be trained more readily.
FM1 for fully connected networks has been previously studied [8,21,25]. Training may fail to start, in this failure mode, since the difference between network outputs may exceed machine precision even for moderate d. For ReLU activations, FM1 has been observed to be overcome by initializations of He et al. [8]. We prove this fact rigorously (see Theorems 5 and 6). We find empirically that for poor initializations, training fails more frequently as networks become deeper (see Figures 1 and 4).
Aside from [29], there appears to be less literature studying FM1 for residual networks (ResNets) [9]. We prove that the key to avoiding FM1 in ResNets is to correctly rescale the contributions of individual residual modules (see Theorems 5 and 6). Without this, we find empirically that training fails for deeper ResNets (see Figure 2).
FM2 is more subtle and does not seem to have been widely studied (see [16] for a notable exception). We find that FM2 indeed impedes early training (see Figure 3). Let us mention two possible explanations. First, if the variance between activations at different layers is very large, then some layers may have very large or small activations that exceed machine precision. Second, the backpropagated SGD update for a weight W in a given layer includes a factor that corresponds to the size of activations at the previous layer. A very small update of W essentially keeps it at its randomly initialized value. A very large update on the hand essentially re-randomizes W . Thus, we conjecture that FM2 causes the stochasticity of parameter updates to outweigh the effect of the training loss.
Our analysis of FM2 reveals an interesting difference between fully connected and residual networks. Namely, for fully connected and convolutional networks, FM2 is a function of architecture, rather than just of initialization, and can occur even if FM1 does not. For residual networks, we prove by contrast that FM2 never occurs once FM1 is avoided (see Corollary 2 and Theorem 6).

Related Work
Closely related to this article is the work of Taki [29] on initializing ResNets as well as the work of Yang-Schoenholz [32,33]. The article [29] gives heuristic computations related to the mean squared activation in a depth-L ResNet and suggests taking the scales η of the residual modules to all be equal to 1/L (see §3.2). The articles [32,33], in contrast, use mean field theory to derive the dependence of both forward and backward dynamics in a randomly initialized ResNet on the residual module weights.
Also related to this work is that of He et al. [8] already mentioned above, as well as [14,22,23,25]. The authors in the latter group show that information can be propagated in infinitely wide ReLU nets so long as weights are initialized independently according to an appropriately normalized distribution (see condition (ii) in Definition 1). One notable difference between this collection of papers and the present work is that we are concerned with a rigorous computation of finite width effects.
These finite size corrections were also studied by Schoenholz et al. [26], which gives exact formulas for the distribution of pre-activations in the case when the weights and biases are Gaussian. For more on the Gaussian case, we also point the reader to Giryes et al. [6]. The idea that controlling means and variances of activations at various hidden layers in a deep network can help with the start of training was previously considered in Klaumbauer et al. [16]. This work introduced the scaled exponential linear unit (SELU) activation, which is shown to cause the mean values of neuron activations to converge to 0 and the average squared length to converge to 1. A different approach to this kind of self-normalizing behavior was suggested in Wu et al. [31]. There, the authors suggest to add a linear hidden layer (that has no learnable parameters) but directly normalizes activations to have mean 0 and variance 1. Activation lengths can also be controlled by constraining weight matrices to be orthogonal or unitary (see e.g. [1,10,13,18,24]).
We would also like to point out a previous appearance in Hanin [7] of the sum of reciprocals of layer widths, which we here show determines the variance of the sizes of the activations (see Theorem 5) in randomly initialized fully connected ReLU nets. The article [7] studied the more delicate question of the variance for gradients computed by random ReLU nets. Finally, we point the reader to the discussion around Figure 6 in [2], which also finds that time to convergence is better for wider networks.

Results
In this section, we will (1) provide an intuitive motivation and explanation of our mathematical results, (2) verify empirically that our predictions hold, and (3) show by experiment the implications for training neural networks.

Avoiding FM1 for Fully Connected Networks: Variance of Weights
Consider a depth-d, fully connected ReLU net N with hidden layer widths n j , j = 0, . . . , d, and random weights and biases (see Definition 1 for the details of the initialization). As N propagates an input vector act (0) ∈ R n0 from one layer to the next, the lengths of the resulting vectors of activations act (j) ∈ R nj change in some manner, eventually producing an output vector whose length is potentially very different from that of the input. These changes in length are summarized by where here and throughout the squared norm of a vector is the sum of the squares of its entries. We prove in Theorem 5 that the mean of the normalized output length M d , which controls whether failure mode FM1 occurs, is determined by the variance of the distribution used to initialize weights. We emphasize that all our results hold for any fixed input, which need not be random; we average only over the weights and the biases. Thus, FM1 cannot be directly solved by batch normalization [11], which renormalizes by averaging over inputs to N , rather than averaging over initializations for N . Theorem 1 (FM1 for fully connected networks (informal)). The mean E [M d ] of the normalized output length is equal to the input length if network weights are drawn independently from a symmetric distribution with variance 2/fan-in. For higher variance, the mean E [M d ] grows exponentially in the depth d, while for lower variance, it decays exponentially.
In Figure 1, we compare the effects of different initializations in networks with varying depth, where the width is equal to the depth (this is done to prevent FM2, see §3.3). Figure 1(a) shows that, as predicted, initializations for which the variance of weights is smaller than the critical value of 2/fan-in lead to a dramatic decrease in output length, while variance larger than this value causes the output length to explode. Figure 1(b) compares the ability of differently initialized networks to start training; it shows the average number of epochs required to achieve 20% test accuracy on MNIST [19]. It is clear that those initializations which preserve output length are also those which allow for fast initial training -in fact, we see that it is faster to train a suitably initialized depth-100 network than it is to train a depth-10 network. Datapoints in (a) represent the statistics over random unit inputs for 1,000 independently initialized networks, while (b) shows the number of epochs required to achieve 20% accuracy on vectorized MNIST, averaged over 5 training runs with independent initializations, where networks were trained using stochastic gradient descent with a fixed learning rate of 0.01 and batch size of 1024, for up to 100 epochs. Note that changing the learning rate depending on depth could be used to compensate for FM1; choosing the right initialization is equivalent and much simpler.
and its connection to the variance of weights at initialization have been previously noted (we refer the reader especially to [8] and to §2 for other references), the implications for choosing a good initialization appear not to have been fully recognized. Many established initializations for ReLU networks draw weights i.i.d. from a distribution for which the variance is not normalized to preserve output lengths. As we shall see, such initializations hamper training of very deep networks. For instance, as implemented in the Keras deep learning Python library [4], the only default initialization to have the critical variance 2/fan-in is He uniform. By contrast, LeCun uniform and LeCun normal have variance 1/fan-in, Glorot uniform (also known as Xavier uniform) and Glorot normal (Xavier normal) have variance 2/(fan-in + fan-out). Finally, the initialization He normal comes close to having the correct variance, but, at the time of writing, the Keras implementation truncates the normal distribution at two standard deviations from the mean (the implementation in PyTorch [20] does likewise). This leads to a decrease in the variance of the resulting weight distribution, causing a catastrophic decay in the lengths of output activations (see Figure 1). We note this to emphasize both the sensitivity of initialization and the popularity of initializers that can lead to FM1.
It is worth noting that the 2 in our optimal variance 2/fan-in arises from the ReLU, which zeros out symmetrically distributed input with probability 1/2, thereby effectively halving the variance at each layer. (For linear activations, the 2 would disappear.) The initializations described above may preserve output lengths for activation functions other than ReLU. However, ReLU is one of the most common activation functions for feed-forward networks and various initializations are commonly used blindly with ReLUs without recognizing the effect upon ease of training. An interesting systematic approach to predicting the correct multiplicative constant in the variance of weights as a function of the non-linearity is proposed in [22,25] (e.g., the definition of χ 1 around (7) in Poole et al. [22]). For non-linearities other than ReLU, however, this constant seems difficult to compute directly.

Avoiding FM1 for Residual Networks: Weights of Residual Modules
To state our results about FM1 for ResNets, we must set some notation (based on the framework presented e.g. in Veit et al. [30]). For a sequence η , = 1, 2 . . . of positive real numbers and a sequence of fully connected ReLU nets N 1 , N 2 , . . ., we define a residual network N res L with residual modules N 1 , . . . , N L and scales η 1 , . . . , η L by the recursion Explicitly, Intuitively, the scale η controls the size of the correction to N res −1 computed by the residual module N . Since we implicitly assume that the depths and widths of the residual modules N are uniformly bounded (e.g., the modules may have a common architecture), failure mode FM1 comes down to determining for which sequences {η } of scales there exist c, C > 0 so that where we write M res L = N res L (x) 2 and x is a unit norm input to N res L . The expectation in (2) is over the weights and biases in the fully connected residual modules N , which we initialize as in Definition 1, except that we set biases to zero for simplicity (this does not affect the results below). A part of our main theoretical result, Theorem 6, on residual networks can be summarized as follows. Theorem 2 (FM1 for ResNets (informal)). Consider a randomly initialized ResNet with L residual modules, scales η 1 , . . . , η L , and weights drawn independently from a symmetric distribution with variance 2/fan-in. The mean E [M res L ] of the normalized output length grows exponentially with the sum of the scales L =1 η .
We empirically verify the predictive power of the quantity η in the performance of ResNets. In Figure 2(a), we initialize ResNets with constant η = 1 as well as geometrically decaying η = b for b = 0.9, 0.75, 0.5. All modules are single hidden layers with width 5. We observe that, as predicted by Theorem 1, η = 1 leads to exponentially growing length scale M res L , while η = b leads the mean of M res L to grow until it reaches a plateau (the value of which depends on b), since η is finite. In Figure 2(b), we show that the mean of M res L well predicts the ease with which ResNets of different depths are trained. Note the large gap between b = 0.9 and 0.75, which is explained by noting that the approximation of η 2 η which we use in the proof holds for η 1, leading to a larger constant multiple of η in the exponent for b closer to 1. Each datapoint is averaged over 100 training runs with independent initializations, with training parameters as in Figure 1.

FM2 for Fully Connected Networks: The Effect of Architecture
In the notation of §3.1, failure mode FM2 is characterized by a large expected value for the empirical variance of the normalized squared lengths of activations among all the hidden layers in N . Our main theoretical result about FM2 for fully connected networks is the following. For a formal statement see Theorem 5. It is well known that deep but narrow networks are hard to train, and this result provides theoretical justification; since for such nets 1/n j is large. More than that, this sum of reciprocals gives a definite way to quantify the effect of "deep but narrow" architectures on the volatility of the scale of activations at various layers within the network. We note that this result also implies that for a given depth and fixed budget of neurons or parameters, constant width is optimal, since by the Power Mean Inequality, j 1/n j is minimized for all n j equal if n j (number of neurons) or n 2 j (approximate number of parameters) is held fixed. We experimentally verify that the sum of reciprocals of layer widths (FM2) is an astonishingly good predictor of the speed of early training. Figure 3 Note that types (i)-(iii) have the same layer widths, but differently permuted. As predicted by our theory, the order of layer widths does not affect FM2. We emphasize that 1 30 type (iv) networks have, for each depth, the same sum of reciprocals of layer widths as types (i)-(iii). As predicted, early training dynamics for networks of type (i)-(iv) were similar for each fixed depth. By contrast, networks of type (v), which had a lower sum of reciprocals of layer widths, trained faster for every depth than the corresponding networks of types (i)-(iv).
In all cases, training becomes harder with greater depth, since 1/n j increases with depth for constant-width networks. In Figure 3(b), we plot the same data with 1/n j on the x-axis, showing this quantity's power in predicting the effectiveness of early training, irrespective of the particular details of the network architecture in question.
Each datapoint is averaged over 100 independently initialized training runs, with training parameters as in Figure 1. All networks are initialized with He normal weights to prevent FM1.

FM2 for Residual Networks
In the notation of §3.2, failure mode FM2 is equivalent to a large expected value for the empirical variance

Theorem 4 (FM2 for ResNets (informal)). The mean E[ Var[M res ]] of the empirical variance for the lengths of activations in a residual
ReLU net with L residual modules and scales η is exponential in L =1 η . By Theorem 2, this means that in ResNets, if failure mode FM1 does not occur, then neither does FM2 (assuming FM2 does not occur in individual residual modules).

Convolutional Architectures
Our above results were stated for fully connected networks, but the logic of our proofs carries over to other architectures. In particular, similar statements hold for convolutional neural networks (ConvNets). Note that the fan-in for a convolutional layer is not given by the width of the preceding layer, but instead is equal to the number of features multiplied by the kernel size. Figure 4: Comparison of the behavior of differently initialized ConvNets as depth increases, with the number of features at each layer proportional to the overall network depth. The mean output length over different random initializations is observed to follow the same patterns as in Figure 1 for fully connected networks. Weight distributions with variance 2/fan-in preserve output length, while other distributions lead to exponential growth or decay. The input image from CIFAR-10 is shown.
In Figure 4, we show that the output length behavior we observed in fully connected networks also holds in ConvNets. Namely, mean output length equals input length for weights drawn i.i.d. from a symmetric distribution of variance 2/fan-in, while other variances lead to exploding or vanishing output lengths as the depth increases. In our experiments, networks were purely convolutional, with no pooling or fully connected layers. By analogy to Figure 1, the fan-in was set to approximately the depth of the network by fixing kernel size 3 × 3 and setting the number of features at each layer to one tenth of the network's total depth. For each datapoint, the network was allowed to vary over 1,000 independent initializations, with input a fixed image from the dataset CIFAR-10 [17].

Notation
To state our results formally, we first give the precise definition of the networks we study; and we introduce some notation. For every d ≥ 1 and n = (n i ) d i=0 ∈ Z d+1 + , we define N(n, d) = fully connected feed-forward nets with ReLU activations and depth d, whose j th hidden layer has width nj .
Note that n 0 is the dimension of the input. Given N ∈ N(n, d), the function f N it computes is determined by its weights and biases For every input act (0) = act The vectors preact (j) , act (j) are thus the inputs and outputs of nonlinearities in the j th layer of N . Definition 1 (Random Nets). Fix d ≥ 1, positive integers n = (n 0 , . . . , n d ) ∈ Z d+1 + , and two collections of probability measures µ = µ (1) , . . . , µ (d) and ν = ν (1) , . . . , ν (d) on R such that µ (j) , ν (j) are symmetric around 0 for every 1 ≤ j ≤ d, and such that the variance of µ (j) is 2/(n j−1 ).
A random network N ∈ N µ,ν (n, d) is obtained by requiring that the weights and biases for neurons at layer j are drawn independently from µ (j) , ν (j) , respectively.

Formal statements
We begin by stating our results about fully connected networks. Given a random network N ∈ N µ,ν (d, n) and an input act (0) to N , we write as in §3.1, M j for the normalized square length of activations 1 n d || act (j) || 2 at layer j. Our first theoretical result, Theorem 5, concerns both the mean and variance of M d . To state it, we denote for any probability measure λ on R its moments by Theorem 5. For each j ≥ 0, fix n j ∈ Z + . For each d ≥ 1, let N ∈ N µ,ν (d, n) be a fully connected ReLU net with depth d, hidden layer widths n = (n j ) d j=0 as well as random weights and biases as in Definition 1. Fix also an input act (0) ∈ R n0 to N with || act (0) || = 1. We have almost surely Moreover, if (4) holds, then exists a random variable M ∞ (that is almost surely finite) such that where C 1 , C 2 the following finite constants: 4 ≤ 1 and µ (j) is Gaussian for all j ≥ 1, then C 1 = 5, C 2 = 4.
In particular, Var[M d ] is exponential in d j=1 1/n j and if j 1/n j < ∞, then the convergence of M d to M ∞ is in L 2 and Var[M ∞ ] < ∞.
The proof of Theorem 5 is deferred to the Supplementary Material. Although we state our results only for fully connected feed-forward ReLU nets, the proof techniques carry over essentially verbatim to any feed-forward network in which only weights in the same hidden layer are tied. In particular, our results apply to convolutional networks in which the kernel sizes are uniformly bounded. In this case, the constants in Theorem 5 depend on the bound for the kernel dimensions, and n j denotes the fan-in for neurons in the (j + 1) st hidden layer (i.e. the number of channels in layer j multiplied by the size of the appropriate kernel). We also point out the following corollary, which follows immediately from the proof of Theorem 5. Corollary 1 (FM1 for Fully Connected Networks). With notation as in Theorem 5, suppose that for all j = 1, . . . , d, the weights in layer j of N d have variance κ · 2/n j for some κ > 0. Then the average squared size M d of activations at layer d will grow or decay exponentially unless κ = 1: Our final result about fully connected networks is a corollary of Theorem 5, which explains precisely when failure mode FM2 occurs (see §3.3). It is proved in the Supplementary Material.
Corollary 2 (FM2 for Fully Connected Networks). Take the same notation as in Theorem 5. There exist c, C > 0 so that In particular, suppose the hidden layer widths are all equal, n j = n. Then, E[ Var [M ]] is exponential in β = j 1/n j = (d − 1)/n in the sense that there exist c, C > 0 so that Finally, our main result about residual networks is the following: Theorem 6. Take the notation from §3.2 and §3.4. The mean of squared activations in N res L are uniformly bounded in the number of modules L if and only if the scales η form a convergent series: Moreover, for any sequence of scales η for which sup η < 1 and for every K, L ≥ 1, we have where the implied constant depends on K but not on L. Hence, once the condition in part (8)

Conclusion
In this article, we give a rigorous analysis of the layerwise length scales in fully connected, convolutional, and residual ReLU networks at initialization. We find that a careful choice of initial weights is needed for well-behaved mean length scales. For fully connected and convolutional networks, this entails a critical variance for i.i.d. weights, while for residual nets this entails appropriately rescaling the residual modules. For fully connected nets, we prove that to control not merely the mean but also the variance of layerwise length scales requires choosing a sufficiently wide architecture, while for residual nets nothing further is required. We also demonstrate empirically that both the mean and variance of length scales are strong predictors of early training dynamics. In the future, we plan to extend our analysis to other (e.g. sigmoidal) activations, recurrent networks, weight initializations beyond i.i.d. (e.g. orthogonal weights), and the joint distributions of activations over several inputs.

A Proof of Theorem 5
Let us first verify that M d is a submartingale for the filtration {F d } d≥1 with F d being the sigma algebra generated by all weights and biases up to and including layer d (for background on sigma algebras and martingales we refer the reader to Chapters 2 and 37 in [3]). Since act (0) is a fixed non-random vector, it is clear that M d is measurable with respect to F d . We have where we can replace the sigma algebra F d−1 by the sigma algebra generated by act (d−1) since the computation done by a feed-forward neural net is a Markov chain with respect to activations at consecutive layers (for background see Chapter 8 in [3]). Next, recall that by assumption the weights and biases are symmetric in law around 0. Note that for each β, changing the signs of all the weights w β to change sign. Hence, we find Note that preact Symmetrizing the expression in (9), we obtain where in the second equality we used that the weights w α,β ] = 2/n j−1 . The above computation also yields that for each d ≥ 1, It also shows that 2 is a martingale. Taking the limit d → ∞ in (11) proves (4). Next, assuming condition (4), we find that which is finite. Hence, we may apply Doob's pointwise martingale convergence theorem (see Chapter 35 in [3] Proof. Note that and, conditioned on act (d−1) , the random variables {act We apply the same symmetrization trick as in the derivation of (10) to obtain E act which after using that the odd moments of w To evaluate the first term, note that αi,1 . Since where we recall that µ . Putting together the preceding computations and using that E act we find that Var act Iterating the lower bound in this inequality yields the lower bound in (5). Similarly, using that 1 + C/n d > 1, we iterate the upper bound to obtain Using the above estimate for j a j , gives the upper bound in (5) and completes the proof of Theorem 5.

B Proof of Corollary 2
Fix a fully connected ReLU net N with depth d and hidden layer widths n 0 , . . . , n d . We fix an input act (0) to N and study the empirical variance Var[M ] of the squared sizes of activations M j , j = 1, . . . , d. Since the biases in N are 0, the squared activations M j are a martingale (see (10)) and we find E [M j M j ] = E M 2 min{j,j } . Thus, using that by (5)  This proves the lower bounds in (6) and (7). The upper bounds are similar.

C Proof of Theorem 6
To understand the sizes of activations produced by N res L , we need the following Lemma. Lemma 2. Let N be a feed-forward, fully connected ReLU net with depth d and hidden layer widths n 0 , . . . , n d having random weights as in Definition 1 and biases set to 0. Then for each η ∈ (0, 1), we have x + ηN (x) 2 = x 2 (1 + O(η)) .
Proof of Lemma. We have: where x = x x , and we have used the fact that N (x) 2 = x 2 (see (10)) as well as the positive homogeneity of ReLU nets with zero biases: Let us also write x = x (0) for the input to N , similarly set x (j) for the activations at layer j. We denote by W (j) β the β th row of the weights W (j) at layer j in N . We have: Therefore, using that x (j) is a supermartingale (since its square is a martingale by (10)): x .
Hence, we obtain: since by Jensen's inequality, Combining this with (13)  Derivation of the estimates (ii) follows exactly the same procedure and hence is omitted. Finally, using these estimates, we find that the mean empirical variance of {M res } is exponential in η :