Online Deep Learning: Learning Deep Neural Networks on the Fly
  Deep Neural Networks (DNNs) are typically trained by backpropagation in a
batch learning setting, which requires the entire training data to be made
available prior to the learning task. This is not scalable for many real-world
scenarios where new data arrives sequentially in a stream form. We aim to
address an open challenge of "Online Deep Learning" (ODL) for learning DNNs on
the fly in an online setting. Unlike traditional online learning that often
optimizes some convex objective function with respect to a shallow model (e.g.,
a linear/kernel-based hypothesis), ODL is significantly more challenging since
the optimization of the DNN objective function is non-convex, and regular
backpropagation does not work well in practice, especially for online learning
settings. In this paper, we present a new online deep learning framework that
attempts to tackle the challenges by learning DNN models of adaptive depth from
a sequence of training data in an online learning setting. In particular, we
propose a novel Hedge Backpropagation (HBP) method for online updating the
parameters of DNN effectively, and validate the efficacy of our method on
large-scale data sets, including both stationary and concept drifting
scenarios.
