Understanding the Impact of Model Incoherence on Convergence of
  Incremental SGD with Random Reshuffle
  Although SGD with random reshuffle has been widely-used in machine learning
applications, there is a limited understanding of how model characteristics
affect the convergence of the algorithm. In this work, we introduce model
incoherence to characterize the diversity of model characteristics and study
its impact on convergence of SGD with random reshuffle under weak strong
convexity. Specifically, minimizer incoherence measures the discrepancy between
the global minimizers of a sample loss and those of the total loss and affects
the convergence error of SGD with random reshuffle. In particular, we show that
the variable sequence generated by SGD with random reshuffle converges to a
certain global minimizer of the total loss under full minimizer coherence. The
other curvature incoherence measures the quality of condition numbers of the
sample losses and determines the convergence rate of SGD. With model
incoherence, our results show that SGD has a faster convergence rate and
smaller convergence error under random reshuffle than those under random
sampling, and hence provide justifications to the superior practical
performance of SGD with random reshuffle.
