Event-Based Control for Online Training of Neural Networks
  Convolutional Neural Network (CNN) has become the most used method for image
classification tasks. During its training the learning rate and the gradient
are two key factors to tune for influencing the convergence speed of the model.
Usual learning rate strategies are time-based i.e. monotonous decay over time.
Recent state-of-the-art techniques focus on adaptive gradient algorithms i.e.
Adam and its versions. In this paper we consider an online learning scenario
and we propose two Event-Based control loops to adjust the learning rate of a
classical algorithm E (Exponential)/PD (Proportional Derivative)-Control. The
first Event-Based control loop will be implemented to prevent sudden drop of
the learning rate when the model is approaching the optimum. The second
Event-Based control loop will decide, based on the learning speed, when to
switch to the next data batch. Experimental evaluationis provided using two
state-of-the-art machine learning image datasets (CIFAR-10 and CIFAR-100).
Results show the Event-Based E/PD is better than the original algorithm (higher
final accuracy, lower final loss value), and the Double-Event-BasedE/PD can
accelerate the training process, save up to 67% training time compared to
state-of-the-art algorithms and even result in better performance.
