\section{Sensitivity of \adagrad/ and \amsgrad/}
\label{sec:sensitivity}
In this section, we empirically demonstrate that the commonly-used constant step-size variants of \adagrad/ and \amsgrad/ are sensitive to their choice of step-size, even for convex problems satisfying interpolation.
We use their \pytorch/ implementations~\citep{paszke2017automatic} on a binary classification task with (unregularized) logistic regression. Following the protocol of~\citet{meng2019fast}, we generate a linearly-separable dataset with $n\m!=\m!10^3$ examples, ensuring interpolation is satisfied, and $d\m!=\m!20$ features with varying margins. For \adagrad/ and \amsgrad/ with a batch-size of $100$, we show the training loss for a grid of step-sizes in the $[10^{3}, 10^{-3}]$ range along with their default step-size in \pytorch/. 
In~\cref{fig:syn}, we observe a large variance across step-sizes and poor performance of the default step-size. The best performing variant of \adagrad/ and \amsgrad/ has a step-size of order $10^2$, which is very different from the default step-size used in practice. The above experiment demonstrates the inability of these methods to adapt to the properties of the function, questioning their robustness. In the next section, we design techniques to enable both \adagrad/ and \amsgrad/ to adapt to the 

objective's smoothness.
{"config": "arxiv", "file": "2006.06835/Sensitivity.tex"}