\section{Experiments}\label{sec_experiments}
We compare our AdaS algorithm to several adaptive and non-adaptive optimizers in the context of image classification. In particular, we implement AdaS with SGD with momentum, four adaptive methods i.e. AdaGrad \cite{duchi2011}, RMSProp \cite{goodfellow2016deep}, AdaM \cite{kingma2014adam}, AdaBound \cite{luo2018adaptive}, and two non-adaptive momentum SGDs guided by scheduled learning techniques i.e. OneCyleLR (also known as the super-convergence method) \cite{smith2019super} and SGD with StepLR (step decaying) \cite{goodfellow2016deep}. We further investigate the dependencies of CNN training quality to knowledge gain and mapping conditions and provide useful insights into the usefulness of different optimizers for training different models. For details on ablative studies and a complete set of experiments, please refer to the Appendix-B.

\subsection{Experimental Setup}


We investigate the efficacy of AdaS with respect to variations in the number of deep layers using VGG16 \cite{simonyan2014very} and ResNet34 \cite{he2016deep} and the number of classes using the standard CIFAR-10 and CIFAR-100 datasets \cite{krizhevsky2009learning} for training. The details of pre-processing steps, network implementation and training/testing frameworks are adopted from the CIFAR GitHub repository\footnote{\url{https://github.com/kuangliu/pytorch-cifar}} using PyTorch. We set the initial learning rates of AdaGrad, RMSProp and AdaBound to $\eta_0=\{1\text{e-}2, 3\text{e-}4, 1\text{e-}3\}$ per their suggested default values. We further followed the suggested tuning in \cite{wilson2017marginal} for AdaM ($\eta_0=3\text{e-}4$ for VGG16 and $\eta_0=1\text{e-}3$ for ResNet34) and SGD-StepLR ($\eta_0=1\text{e-}1$ dropping half magnitude every $25$ epochs) to achieve the best performance. For SGD-1CycleLR we set $50$ epochs for the whole cycle and found the best configuration ($\eta_0=3\text{e-}2$ for VGG16 and $\eta_0=4\text{e-}2$ for ResNet34). To configure the best initial learning rate for AdaS, we performed a dense grid search and found the values for VGG16 and ResNet34 to be $\eta_0=\{5\text{e-}3, 3\text{e-}2\}$. Despite the differences in optimal values that are independently obtained for each network, the optimizer performance is fairly robust relative to changes in these values. Each model is trained for $250$ epochs in $5$ independent runs and average test accuracy and training losses are reported. The mini-batch size is also set to $|\Omega_k|=128$.

\begin{figure}[htp]
	\centerline{
		\subfigure[\tiny{Test Acc--VGG16--CIFAR10}]{\includegraphics[width=.24\textwidth]{test_accuracy_comparison_CIFAR10_VGG16.png}}
		\subfigure[\tiny{Test Acc--ResNet34--CIFAR10}]{\includegraphics[width=.24\textwidth]{test_accuracy_comparison_CIFAR10_ResNet34.png}}
		\subfigure[\tiny{Test Acc--VGG16--CIFAR100}]{\includegraphics[width=.24\textwidth]{test_accuracy_comparison_CIFAR100_VGG16.png}}
		\subfigure[\tiny{Test Acc--ResNet34--CIFAR100}]{\includegraphics[width=.24\textwidth]{test_accuracy_comparison_CIFAR100_ResNet34.png}}
	}
	\centerline{
		\subfigure[\tiny{Train Loss--VGG16--CIFAR10}]{\includegraphics[width=.24\textwidth]{training_loss_comparison_CIFAR10_VGG16.png}}
		\subfigure[\tiny{Train Loss--ResNet34--CIFAR10}]{\includegraphics[width=.24\textwidth]{training_loss_comparison_CIFAR10_ResNet34.png}}
		\subfigure[\tiny{Train Loss--VGG16--CIFAR100}]{\includegraphics[width=.24\textwidth]{training_loss_comparison_CIFAR100_VGG16.png}}
		\subfigure[\tiny{Train Loss--ResNet34--CIFAR100}]{\includegraphics[width=.24\textwidth]{training_loss_comparison_CIFAR100_ResNet34.png}}
	}
	\caption{Training performance using different optimizers across two different datasets (i.e. CIFAR10 and CIFAR100) and two different CNNs (i.e. VGG16 and ResNet34)}
	\label{fig_comparison_all_methods}
\end{figure}

\subsection{Image Classification Problem}
We first empirically evaluate the effect of the gain-factor $\beta$ on AdaS convergence by defining eight different grid values (i.e. $\beta\in\{0.8, 0.825,\cdots,0.975\}$). The trade-off between the selection of different values of $\beta$ is demonstrated in Figure \ref{fig_comparison_all_methods} (complete ablation study is provided in Appendix-B). Here, lower $\beta$ translates to faster convergence, whereas setting it to higher values yields better final performance--at the cost of requiring more epochs for training. The performance comparison of optimizers is also overlaid in the same figure, where AdaS (with lower $\beta$) surpasses all adaptive and non-adaptive methods by a large margin in both test accuracy and training loss during the initial stages of training (i.e. $\text{epoch}<50$), where as SGD-StepLR and AdaS (with higher $\beta$) eventually overtake the other methods with more training epochs. Furthermore, AdaGrad, RMSProp, AdaM, and AdaBound all achieve similar or sometimes even lower training losses compared to AdaS (including the two non-adaptive methods), but attain lower test accuracies. Such controversial results were also reported in \cite{wilson2017marginal} where adaptive optimizers generalize worse compared to non-adaptive methods. In retrospect, we claim here that AdaS solves this issue by generalizing better than other adaptive optimizers.

We further provide quantitative results on the convergence of all optimizers trained on ResNet34 in Table \ref{tab_convergence_test_accuracy} with a fixed number of training epochs. The rank consistency of AdaS (using two different gain factors of low $\beta=0.85$ and high $\beta=0.95$ values) over other optimizers is evident. For instance, AdaS$^{\beta=0.850}$ gains $3.63\%$ test accuracy (with half confidence interval) over the second best optimizer AdaM on CIFAR-100 trained with $25$ epochs.

\begin{table}[htp]
  \tiny
  \caption{Image classification performance (test accuracy) of ResNet34 on CIFAR-10 and CIFAR-100 with fixed budget epochs. Four adaptive (AdaGrad, RMSProp, AdaM, AdaS) and one non-adaptive (SGD-StepLR) optimizers are deployed for comparison.}
  \label{tab_convergence_test_accuracy}
  \centering  
  \begin{threeparttable}
    \tiny
    \begin{tabular*}{\textwidth}
      {p{0.1cm}p{0.2cm}p{1.7cm}p{1.7cm}p{1.7cm}p{1.7cm}p{1.7cm}p{1.7cm}}    
      \toprule      
      
      & {Epoch} & {AdaGrad} & {RMSProp} & {AdaM} & {SGD-StepLR} & {AdaS$^{\beta=0.850}$} & {AdaS$^{\beta=0.950}$}\\
			\toprule
      
			\multirow{7}{*}{\rotatebox[origin=c]{90}{CIFAR-10}} 
			& $25$ & $0.8859\pm{0.47\%}$ & $0.8916\pm{0.71\%}$ & ${\bf 0.8957\pm{0.73\%}}$ & $0.8325\pm{2.79\%}$ & ${\bf 0.9136\pm{0.23\%}}$ & $0.8611\pm{1.67\%}$\\
			& $50$ & $0.9017\pm{0.61\%}$ & $0.9086\pm{0.58\%}$ & $0.9154\pm{0.35\%}$ & $0.8653\pm{1.67\%}$ & ${\bf 0.9370\pm{0.13\%}}$ & ${\bf 0.9209\pm{0.52\%}}$\\
			& $75$ & $0.9103\pm{0.18\%}$ & $0.9139\pm{0.78\%}$ & $0.9211\pm{0.26\%}$ & $0.9067\pm{0.38\%}$ & ${\bf 0.9372\pm{0.14\%}}$ & ${\bf 0.9472\pm{0.20\%}}$\\
			& $100$ & $0.9109\pm{0.25\%}$ & $0.9159\pm{0.77\%}$ & $0.9271\pm{0.27\%}$ & $0.9225\pm{0.29\%}$ & ${\bf 0.9372\pm{0.12\%}}$ & ${\bf 0.9510\pm{0.18\%}}$\\
			& $150$ & $0.9134\pm{0.37\%}$ & $0.9269\pm{0.33\%}$ & $0.9307\pm{0.33\%}$ & ${\bf 0.9464\pm{0.09\%}}$ & $0.9370\pm{0.13\%}$ & ${\bf 0.9510\pm{0.11\%}}$\\
			& $200$ & $0.9140\pm{0.15\%}$ & $0.9287\pm{0.30\%}$ & $0.9317\pm{0.30\%}$ & ${\bf 0.9544\pm{0.10\%}}$ & $0.9368\pm{0.11\%}$ & ${\bf 0.9508\pm{0.15\%}}$\\
			& $250$ & $0.9149\pm{0.24\%}$ & $0.9290\pm{0.29\%}$ & $0.9322\pm{0.36\%}$ & ${\bf 0.9543\pm{0.08\%}}$ & $0.9370\pm{0.17\%}$ & ${\bf 0.9516\pm{0.12\%}}$\\
			\toprule
			\multirow{7}{*}{\rotatebox[origin=c]{90}{CIFAR-100}} 
			& $25$ & $0.6221\pm{0.70\%}$ & $0.6341\pm{1.14\%}$ & ${\bf 0.6653\pm{0.46\%}}$ & $0.5545\pm{1.45\%}$ & ${\bf 0.7016\pm{0.27\%}}$ & $0.5981\pm{1.55\%}$\\
			& $50$ & $0.6515\pm{0.27\%}$ & $0.6769\pm{0.62\%}$ & $0.6866\pm{0.46\%}$ & $0.6217\pm{1.68\%}$ & ${\bf 0.7479\pm{0.23\%}}$ & ${\bf 0.7123\pm{0.51\%}}$\\
			& $75$ & $0.6618\pm{0.38\%}$ & $0.6837\pm{0.50\%}$ & $0.6975\pm{0.49\%}$ & $0.6611\pm{1.79\%}$ & ${\bf 0.7491\pm{0.26\%}}$ & ${\bf 0.7714\pm{0.30\%}}$\\
			& $100$ & $0.6658\pm{0.38\%}$ & $0.6928\pm{0.29\%}$ & $0.6978\pm{0.27\%}$ & $0.6878\pm{0.97\%}$ & ${\bf 0.7494\pm{0.26\%}}$ & ${\bf 0.7752\pm{0.31\%}}$\\
			& $150$ & $0.6691\pm{0.31\%}$ & $0.6996\pm{0.48\%}$ & $0.7045\pm{0.42\%}$ & ${\bf 0.7740\pm{0.46\%}}$ & $0.7481\pm{0.22\%}$ & ${\bf 0.7768\pm{0.32\%}}$\\
			& $200$ & $0.6697\pm{0.25\%}$ & $0.7039\pm{0.50\%}$ & $0.7061\pm{0.33\%}$ & ${\bf 0.7763\pm{0.42\%}}$ & $0.7484\pm{0.29\%}$ & ${\bf 0.7765\pm{0.22\%}}$\\
			& $250$ & $0.6702\pm{0.23\%}$ & $0.7025\pm{0.29\%}$ & $0.7111\pm{0.37\%}$ & ${\bf 0.7765\pm{0.32\%}}$ & $0.7483\pm{0.21\%}$ & ${\bf 0.7760\pm{0.22\%}}$\\
      \bottomrule
    \end{tabular*}
  \end{threeparttable}
\end{table}

\begin{figure}[htp]	
	\centerline{
		\subfigure[\tiny{AdaGrad}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaGrad.png}}
		\subfigure[\tiny{RMSProp}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_RMSProp.png}}
		\subfigure[\tiny{AdaM}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaM.png}}
		\subfigure[\tiny{AdaBound}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaBound.png}}
		\subfigure[\tiny{SGD-StepLR}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_SGD_StepLR.png}}
		\subfigure[\tiny{SGD-1CycleLR}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_SGD_OneCycleLR.png}}
	}
	\centerline{
		\subfigure[\tiny{AdaS$^{\beta=0.800}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0800.png}}
		\subfigure[\tiny{AdaS$^{\beta=0.850}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0850.png}}
		\subfigure[\tiny{AdaS$^{\beta=0.900}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0900.png}}
		\subfigure[\tiny{AdaS$^{\beta=0.925}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0925.png}}
		\subfigure[\tiny{AdaS$^{\beta=0.950}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0950.png}}
		\subfigure[\tiny{AdaS$^{\beta=0.975}$}]{\includegraphics[width=.16\textwidth]{knowledge_gain_vs_mapping_condition_CIFAR10_ResNet34_AdaS_beta_0975.png}}
	}
	\caption{Evolution of knowledge gain versus mapping condition across different training epochs using ResNet34 on CIFAR10. The transition of color shades correspond to different convolution blocks. The transparency of scatter plots corresponds to the epoch convergence--the higher transparency is inversely related to the training epoch. For complete results of different optimizers and models, please refer to Appendix-B.}
	\label{fig_mapping_condition}
\end{figure}

\subsection{Dependence of the Quality of Network Training to Knowledge Gain}

Both concepts of knowledge gain $G$ and mapping condition $\kappa$ can be used to prob within an intermediate layer of a CNN and quantify the quality of training with respect to different parameter settings. Such quantization do not require test or evaluation data where one can directly measure the  \textit{``expected performance''} of the network throughout the training updates. Our first observation here is made by linking the knowledge gain measure to the relative success of each method in the test accuracy performance. For instance, by raising the gain factor $\beta$ in AdaS, the deeper layers of CNN eventually gain further knowledge as shown in Figure \ref{fig_mapping_condition}. This directly affects the success in test performance results. Also, deploying different optimizers yields different behavior in knowledge gain obtained in different layers of the CNN. Table \ref{tab_acc_loss_G_kappa} lists all four numerical measurements of Test Accuracy, Training Loss, Knowledge Gain, and Mapping Condition for different optimizers. Note the rank order correlation of knowledge gain and test accuracy. Although for both RMSProp and AdaM the knowledge gains are high, however, the Mapping Conditions are also high which deteriorates the overall performance of the network.

\begin{table}[htp]
  \tiny
  \caption{Performance of ResNet34 on CIFAR10 dataset reported at final training epoch for each optimizer. The average scores are reported for $G$ and $\kappa$ across all convolutional blocks.}
  \label{tab_acc_loss_G_kappa}
  \centering  
  \begin{threeparttable}
    \tiny
    \begin{tabular*}{\textwidth}
      {p{2cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}p{0.65cm}}    
      \toprule      
      
      & \rothead{AdaGrad} & \rothead{RMSProp} & \rothead{AdaM} & \rothead{AdaBound} & \rothead{SGD-1CycleLR} & \rothead{SGD-StepLR} & \rothead{AdaS$^{\beta=0.80}$} & \rothead{AdaS$^{\beta=0.85}$} & \rothead{AdaS$^{\beta=0.90}$} & \rothead{AdaS$^{\beta=0.95}$} \\      
      \toprule
      
      Test Accuracy  & $0.9150$ & $0.9290$ & $0.9322$ & $0.9274$ & $0.9413$ & $0.9543$ & $0.9302$ & $0.9370$ & $0.9446$ & $0.9516$ \\ \hline
      
			Training Loss & $1.25\text{e-}3$ & $6.35\text{e-}3$ & $5.66\text{e-}3$ & $2.30\text{e-}3$ & $4.90\text{e-}3$ & $9.70\text{e-}4$ & $6.10\text{e-}3$ & $2.33\text{e-}3$ & $1.27\text{e-}3$ & $8.69\text{e-}4$  \\ \hline
      
			Knowledge Gain ($G$)  & $0.1790$ & $0.3011$ & $0.2973$ & $0.1288$ & $0.2965$ & $0.3180$ & $0.2063$ & $0.2467$ & $0.2791$ & $0.2938$ \\ \hline
      
			Mapping Condition ($\kappa$)  & $7.482$ & $18.124$ & $18.484$ & $4.833$ & $7.478$ & $10.086$ & $5.524$ & $6.661$ & $7.839$ & $8.783$ \\
      
      \bottomrule\\
    \end{tabular*}
  \end{threeparttable}
\end{table}

Our second observation here is made by studying the effect of mapping condition and how it relates to the possible lack of generalizability of each optimizer. Although adaptive optimizers (e.g. RMSProp and AdaM) yield lower training loss, they over-fit perturbing features (mainly caused by incomplete second order statistic measure e.g. diagonal Hessian approximation) and accordingly hamper their generalization \cite{wilson2017marginal}. We suspect such unwanted phenomena is related to the mapping condition within CNN layers. In fact, a mixture of both  average $\kappa$ and average $G$ can help to better realize how well each optimizer can be generalized for training/testing evaluations.

We conclude by identifying that an ideal optimizer would yield $G \rightarrow 1$ and $\kappa \rightarrow 1$ across all layers within a CNN. We highlight that increases in $\kappa$ correlates to greater disentanglement between intermediate input and output layers, hampering the flow of information. Further, we identify that increases in knowledge gain strengthen the carriage of information through the network which enables greater performance.
{"config": "arxiv", "file": "2006.06587/experiments.tex"}