Traversing the noise of dynamic mini-batch sub-sampled loss functions: A
  visual guide
  Mini-batch sub-sampling in neural network training is unavoidable, due to
growing data demands, memory-limited computational resources such as graphical
processing units (GPUs), and the dynamics of on-line learning. In this study we
specifically distinguish between static mini-batch sub-sampled loss functions,
where mini-batches are intermittently fixed during training, resulting in
smooth but biased loss functions; and the dynamic sub-sampling equivalent,
where new mini-batches are sampled at every loss evaluation, trading bias for
variance in sampling induced discontinuities. These render automated
optimization strategies such as minimization line searches ineffective, since
critical points may not exist and function minimizers find spurious,
discontinuity induced minima.
  This paper suggests recasting the optimization problem to find stochastic
non-negative associated gradient projection points (SNN-GPPs). We demonstrate
that the SNN-GPP optimality criterion is less susceptible to sub-sampling
induced discontinuities than critical points or minimizers. We conduct a visual
investigation, comparing local minimum and SNN-GPP optimality criteria in the
loss functions of a simple neural network training problem for a variety of
popular activation functions. Since SNN-GPPs better approximate the location of
true optima, particularly when using smooth activation functions with high
curvature characteristics, we postulate that line searches locating SNN-GPPs
can contribute significantly to automating neural network training
