\section{Experiments}
\label{sec:exp}
In this section, we provide empirical results on how the structure of a neural network (width and depth) impacts its amenability to large-batch training using various datasets and network architectures.
Our main findings are three-fold:
\begin{enumerate}
	\item For all neural networks we used, there exists a threshold $B^*$, such that using batch size larger than this threshold induces slower convergence;
	\item  The threshold of wider neural networks is often larger than that of  deeper ones; 
	\item When using the same large batch size,  almost all wider neural networks need much fewer epochs to converge compared to their deeper counterparts.
\end{enumerate}

Those findings validate our theoretical analysis and suggest that 
wider neural networks are indeed more amenable to large-batch training and thus more suitable to scale out.


\textbf{Implementation and Setup} We implemented our experimental pipeline in Keras \cite{Keras2015}, and conducted  all experiments on p2.xlarge instances on Amazon EC2. All results reported are averaged from 5 independent runs. 


\begin{table}[t]
	{\small
	\centering
	\begin{tabular}{|c||c|c|c|c|c|}
		\hline Dataset & Synthetic & MNIST & Cifar10 & EMNIST & Gisette \bigstrut\\
		\hline
		\# datapoints & 10,000 & 70,000 & 60,000 &  131,600 & 6,000 \bigstrut\\
		\hline
		Model & linear FC & FC/LeNet & ResNet-18/34 & FC & FC \bigstrut\\
		\hline
		\# Classes & $+\infty$ & 10 & 10 & 47 & 2 \bigstrut\\
		\hline
		\# Parameters & 16k & 16k / 431k & 11m / 21m & 16k & 262k \bigstrut\\
		\hline
		Converged Accuracy & $10^{-12}$ (loss) & 96\% / 98\%  & 95\% & 65\% & 95\% \bigstrut\\
		\hline
	\end{tabular}
\caption{The datasets used and their associated learning models and hyper-parameters.}
	\label{Tab:DataStat}
}
\end{table}


\paragraph{Datasets and Networks} Table~\ref{Tab:DataStat} summarizes the datasets and networks used in the experiments.
In the synthetic dataset, all data points were independently drawn from $\mathcal{N}(0,1)$ as described by our theory results. 
A deep linear fully connected neural network (FC) whose weights were generated from $\mathcal{N}(0, 1)$ independently was used to produce the true labels.
The task on the synthetic data is a regression task.
We train linear FCs on the synthetic dataset.
The real-world datasets we used include MNIST \cite{lecun2010mnist}, EMNIST\cite{cohen2017emnist}, Gisette \cite{chang2011libsvm}, and CIFAR-10 \cite{CIRFA10}, with appropriate networks ranging from linear, to non-linear fully connected ones, and to LeNet \cite{lecun1998gradient} and ResNet \cite{ResNetCVPR16}.

For each network, we fix the total number of parameters and vary its depth/number of layers $L$ and width $K$.
For fully connected networks and LeNet, we vary depth $L$ from $1$ to $10$ and change $K$ accordingly to ensure the total number of parameters are approximately fixed.
More precisely, we fix the total number of parameters $p$, and solve the following equations 
\begin{align*}\label{eq/fixed-p}
d_{\text{in}} \times K + (L - 1) \times K^2  +  K\times d_{\text{out}} = p.
\end{align*}
where $d_{in}$ is the dimension of the data and $d_{out}$ is the size of output.
For ResNet, we vary two parameters  separately.
We first vary the width and depth of the fully connected layers without changing the residual blocks.
Next we fix the fully connected layers and change the number of blocks and convolution filters in each chunk. We refer to the building block in a residual function described in \cite{ResNetCVPR16} as chunk. For ResNet-18/34 architecture, we use $[s_1,s_2,s_3,s_4]$ to denote a particular structure, where $s_1$ represents the number of blocks stacked in the first chunk, $s_2$ is the number of blocks stacked in the second chunk,  etc. For varying depths, we incrementally increase or decrease one block in each chunk and adjust the number of convolutional filters in each block to meet the fixed number of parameters requirement.


For each combination of depth and width of a NN architecture, we train the model by setting a constant threshold on training accuracy for classification tasks, or loss for regression tasks.
We then train the NN for a variety of batch sizes, in range of $2^i, \text{for } i \in \{5, \cdots, 12\}$.
We tune the step size in the following way: (i) for all learning rates $\eta$ from a grid of candidate values, we run the training process with $\eta$ for 2 passes over the data; and then (ii) we choose $\hat \eta$ which leads to the lowest training loss after two epochs. An epoch represents a full pass over the data.
  
\vspace{-2mm}
\paragraph{Experimental Results} 
\begin{figure*}[t]
	\centering
	\subfigure[Gradient Diversity]{\includegraphics[width=0.28\linewidth]{fig/gradient_diversity.pdf}}
	\subfigure[Largest Batch Size]{\includegraphics[width=0.28\linewidth]{fig/fig2_b.pdf}}
	\subfigure[Diversity vs Batch Size]{\includegraphics[width=0.28\linewidth]{fig/fig2_a.pdf}}
	\vspace{-0.3cm}
	\caption{\small The effect of gradient diversity for linear FCs trained on the synthetic dataset for a regression task. (a) Gradient diversity for different width/depth (b) Largest batch size to converge to loss $10^{-12}$,  within a pre-set number (\ie 14) of epochs. (c) Largest batch size v.s. gradient diversity. }
	\label{fig:gradientdiversity}
\end{figure*}

We first verify whether gradient diversity reflects the amenability to large batch training. 
For each linear FC network with fixed width and depth, we measure its gradient diversity every ten epochs and compute the average. 
Figure \ref{fig:gradientdiversity}(a) shows how the averaged gradient diversity varies as depth/width changes, while
Figure \ref{fig:gradientdiversity}(b) presents the largest batch to converge for each network within a pre-set number of epochs. 
Both of them increase as the width $K$ of the networks increases.
In fact, as shown in Figure \ref{fig:gradientdiversity}(c), the largest batch size that does not
impact the convergence rate grows monotonically w.r.t the gradient diversity. 
This validates our theoretical analysis that gradient diversity can be used to capture 
the amenability to large batch training.
\label{exp-structures}
\begin{figure*}[t]
	\centering
	\subfigure[Synthetic, Linear FC]{\includegraphics[width=0.255\linewidth]{fig/synthesized_gaussian_trend.pdf}}
	\subfigure[MNIST, FC]{\includegraphics[width=0.24\linewidth]{fig/realworld_mnist_fc_trend.pdf}}
	\subfigure[EMNIST, FC]{\includegraphics[width=0.24\linewidth]{fig/realworld_emnist_fc_trend.pdf}}
	\subfigure[Gisette, FC]{\includegraphics[width=0.24\linewidth]{fig/realworld_gisette_fc_trend.pdf}}
	\\
	\subfigure[MNIST,  LeNet]{\includegraphics[width=0.245\linewidth]{fig/realworld_mnist_lenet_trend.pdf}}
	\subfigure[Cifar10, ResNet18, FC]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet18_mlp_trend.pdf}}
	\subfigure[Cifar10, ResNet18, Res]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet18_trend.pdf}}
	\subfigure[Cifar10, ResNet34, Res]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet34_trend.pdf}}
		\vspace{-0.3cm}
	\caption{\small Number of epochs needed to converge to the same loss / accuracy given in Table \ref{Tab:DataStat}. $K$ represents width, and $L$ depth. 
		In (f) We fix the residual blocks of ResNet 18 and only vary the fully-connected parts.
		In (g) and (h), we fix the fully connected layers and vary the residual blocks of ResNet 18 and ResNet 34.
	}
	\label{fig-trend}
\end{figure*}
\begin{figure*}[t]
	\centering
	\subfigure[Synthetic, Linear FC]{\includegraphics[width=0.245\linewidth]{fig/synthesized_gaussian_heatmap.pdf}}
	\subfigure[MNIST, Linear FC]{\includegraphics[width=0.245\linewidth]{fig/realworld_mnist_fc_heatmap.pdf}}
	\subfigure[EMNIST, FC]{\includegraphics[width=0.245\linewidth]{fig/realworld_emnist_fc_heatmap.pdf}}
	\subfigure[Gisette, FC]{\includegraphics[width=0.245\linewidth]{fig/realworld_gisette_fc_heatmap.pdf}}
	\\
	\subfigure[MNIST,  LeNet]{\includegraphics[width=0.245\linewidth]{fig/realworld_mnist_lenet_heatmap.pdf}}
	\subfigure[Cifar10, ResNet18, FC]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet18_mlp_heatmap.pdf}}
	\subfigure[Cifar10, ResNet18, Res]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet18_heatmap.pdf}}
	\subfigure[Cifar10, ResNet34, Res]{\includegraphics[width=0.245\linewidth]{fig/realworld_resnet34_heatmap.pdf}}
	\caption{\small Heatmap on number of epochs
		needed to converge to loss / accuracy defined in Table \ref{Tab:DataStat}. 
		We report the $\log_{10}$ of the epochs for (a) and the real epochs for the others.
		}
	\label{fig-heatmaps}
\end{figure*}

Next, we study the number of epochs needed to converge when different batch sizes are used for real-world datasets.
First, for almost all network architectures, there exists a batch size threshold, such that using a batch size larger than this, requires more epochs for convergence, consistent with the observations in \cite{GradientDiversity2018AISTAT}.
For example, in Figure \ref{fig-trend}(b), when the batch size is smaller than 256, the FC network 
with width $K=17$ and depth $L=10$ needs a small number (2 to 3) of epochs to converge.
But when the batch size becomes larger than $256$, the number of epochs necessary for convergence increases significantly, \eg it takes 50 epochs to converge when batch size is $4096$. Moreover, we observe that this the threshold increases as width increases.
Again as shown in Figure \ref{fig-trend}(b), the batch-size threshold for the FC network with $L=10$ is 256, but goes up to 1024 with $L=1$.
Furthermore, when using the same large batch size, wider networks tend to require fewer epochs to converge than the deeper ones.
In Figure \ref{fig-trend}(c), for instance, using the same batch size of 4096, the required epochs to converge decreases from 211 to 9 as width $K$ increases from 17 to 21. 
Those trends are similar for all FC networks we used in the experiments.

When it comes to ResNets and LeNet, the trends are not always as sharp.
This is expected since our theoretical analysis does not cover such cases, but the main trend  can still be observed.
For example, as shown in Figures \ref{fig-trend}(e) and  \ref{fig-trend}(f), for a fixed batch size, increasing the width almost always leads to a decrease in number of epochs for convergence. 
Figure \ref{fig-heatmaps}, depicts the exact number of epochs to converge for each network architecture, and plots them as a heatmap.
It is interesting to see that for ResNet, there is a small fraction of cases where increase of depth can also reduce the number of epochs for convergence.


In many practical applications, only a reasonable and limited number of data passes is performed due to time and resources constraints. Thus,
we also study how the structure of a network affects the largest possible batch size to converge within a fixed number of epochs/data passes to a pre-specified accuracy. 
As shown in Figure~\ref{fig-largest-batchsizes}, neural networks with larger width $K$ usually allow much larger batch sizes to converge within a small, pre-set number of total epochs. 
This is especially beneficial in the scenarios of large-scale distributed learning, since increasing the batch size can result in more speedup gains due to a reduction in the total amount of communication.
Finally, we should note that the largest batch size differs among different networks, as well as different datasets. This is because gradient diversity is both data-dependent and model-dependent.   

\begin{figure*}[t]
	\centering
	\subfigure[Synthetic, Linear FC]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_synthesized.pdf}}
	\subfigure[MNIST, FC]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_mnist_fc.pdf}}
	\subfigure[EMNIST, FC]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_emnist_fc.pdf}}
	\subfigure[Gisette, FC]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_gisette_fc.pdf}}
	\\
	\subfigure[MNIST on LeNet]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_mnist_lenet.pdf}}
	\subfigure[Cifar10, ResNet18, FC]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_cifar10_resnet.pdf}}
	\subfigure[Cifar10, ResNet18, Res]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_cifar10_resnet_cnn.pdf}}
	\subfigure[Cifar10, ResNet34, Res]{\includegraphics[width=0.245\linewidth]{fig/threshold_batchsize_realworld_cifar10_resnet34_cnn.pdf}}
	\caption{\small Largest possible batch size to converge within a fixed number of epochs. 
		}
	\label{fig-largest-batchsizes}
\end{figure*}
{"config": "arxiv", "file": "1806.03791/Experiment.tex"}