Comment on Stochastic Polyak Step-Size: Performance of ALI-G
  This is a short note on the performance of the ALI-G algorithm (Berrada et
al., 2020) as reported in (Loizou et al., 2021). ALI-G (Berrada et al., 2020)
and SPS (Loizou et al., 2021) are both adaptations of the Polyak step-size to
optimize machine learning models that can interpolate the training data. The
main algorithmic differences are that (1) SPS employs a multiplicative constant
in the denominator of the learning-rate while ALI-G uses an additive constant,
and (2) SPS uses an iteration-dependent maximal learning-rate while ALI-G uses
a constant one. There are also differences in the analysis provided by the two
works, with less restrictive assumptions proposed in (Loizou et al., 2021). In
their experiments, (Loizou et al., 2021) did not use momentum for ALI-G (which
is a standard part of the algorithm) or standard hyper-parameter tuning (for
e.g. learning-rate and regularization). Hence this note as a reference for the
improved performance that ALI-G can obtain with well-chosen hyper-parameters.
In particular, we show that when training a ResNet-34 on CIFAR-10 and
CIFAR-100, the performance of ALI-G can reach respectively 93.5% (+6%) and 76%
(+8%) with a very small amount of tuning. Thus ALI-G remains a very competitive
method for training interpolating neural networks.
