GradVis: Visualization and Second Order Analysis of Optimization
  Surfaces during the Training of Deep Neural Networks
  Current training methods for deep neural networks boil down to very high
dimensional and non-convex optimization problems which are usually solved by a
wide range of stochastic gradient descent methods. While these approaches tend
to work in practice, there are still many gaps in the theoretical understanding
of key aspects like convergence and generalization guarantees, which are
induced by the properties of the optimization surface (loss landscape). In
order to gain deeper insights, a number of recent publications proposed methods
to visualize and analyze the optimization surfaces. However, the computational
cost of these methods are very high, making it hardly possible to use them on
larger networks.
  In this paper, we present the GradVis Toolbox, an open source library for
efficient and scalable visualization and analysis of deep neural network loss
landscapes in Tensorflow and PyTorch. Introducing more efficient mathematical
formulations and a novel parallelization scheme, GradVis allows to plot 2d and
3d projections of optimization surfaces and trajectories, as well as high
resolution second order gradient information for large networks.
