Dynamic learning rate using Mutual Information
  This paper demonstrates dynamic hyper-parameter setting, for deep neural
network training, using Mutual Information (MI). The specific hyper-parameter
studied in this paper is the learning rate. MI between the output layer and
true outcomes is used to dynamically set the learning rate of the network
through the training cycle; the idea is also extended to layer-wise setting of
learning rate. Two approaches are demonstrated - tracking relative change in
mutual information and, additionally tracking its value relative to a reference
measure. The paper does not attempt to recommend a specific learning rate
policy. Experiments demonstrate that mutual information may be effectively used
to dynamically set learning rate and achieve competitive to better outcomes in
competitive to better time.
