Predicting the success of Gradient Descent for a particular
  Dataset-Architecture-Initialization (DAI)
  Despite their massive success, training successful deep neural networks still
largely relies on experimentally choosing an architecture, hyper-parameters,
initialization, and training mechanism. In this work, we focus on determining
the success of standard gradient descent method for training deep neural
networks on a specified dataset, architecture, and initialization (DAI)
combination. Through extensive systematic experiments, we show that the
evolution of singular values of the matrix obtained from the hidden layers of a
DNN can aid in determining the success of gradient descent technique to train a
DAI, even in the absence of validation labels in the supervised learning
paradigm. This phenomenon can facilitate early give-up, stopping the training
of neural networks which are predicted to not generalize well, early in the
training process. Our experimentation across multiple datasets, architectures,
and initializations reveals that the proposed scores can more accurately
predict the success of a DAI than simply relying on the validation accuracy at
earlier epochs to make a judgment.
