Using Mode Connectivity for Loss Landscape Analysis
  Mode connectivity is a recently introduced frame- work that empirically
establishes the connected- ness of minima by finding a high accuracy curve
between two independently trained models. To investigate the limits of this
setup, we examine the efficacy of this technique in extreme cases where the
input models are trained or initialized differently. We find that the procedure
is resilient to such changes. Given this finding, we propose using the
framework for analyzing loss surfaces and training trajectories more generally,
and in this direction, study SGD with cosine annealing and restarts (SGDR). We
report that while SGDR moves over barriers in its trajectory, propositions
claiming that it converges to and escapes from multiple local minima are not
substantiated by our empirical results.
