Deep Gated Networks: A framework to understand training and
  generalisation in deep learning
  Understanding the role of (stochastic) gradient descent (SGD) in the training
and generalisation of deep neural networks (DNNs) with ReLU activation has been
the object study in the recent past. In this paper, we make use of deep gated
networks (DGNs) as a framework to obtain insights about DNNs with ReLU
activation. In DGNs, a single neuronal unit has two components namely the
pre-activation input (equal to the inner product the weights of the layer and
the previous layer outputs), and a gating value which belongs to $[0,1]$ and
the output of the neuronal unit is equal to the multiplication of
pre-activation input and the gating value. The standard DNN with ReLU
activation, is a special case of the DGNs, wherein the gating value is $1/0$
based on whether or not the pre-activation input is positive or negative. We
theoretically analyse and experiment with several variants of DGNs, each
variant suited to understand a particular aspect of either training or
generalisation in DNNs with ReLU activation. Our theory throws light on two
questions namely i) why increasing depth till a point helps in training and ii)
why increasing depth beyond a point hurts training? We also present
experimental evidence to show that gate adaptation, i.e., the change of gating
value through the course of training is key for generalisation.
