Adaptive Optimization with Examplewise Gradients
  We propose a new, more general approach to the design of stochastic
gradient-based optimization methods for machine learning. In this new
framework, optimizers assume access to a batch of gradient estimates per
iteration, rather than a single estimate. This better reflects the information
that is actually available in typical machine learning setups. To demonstrate
the usefulness of this generalized approach, we develop Eve, an adaptation of
the Adam optimizer which uses examplewise gradients to obtain more accurate
second-moment estimates. We provide preliminary experiments, without
hyperparameter tuning, which show that the new optimizer slightly outperforms
Adam on a small scale benchmark and performs the same or worse on larger scale
benchmarks. Further work is needed to refine the algorithm and tune
hyperparameters.
