MaxGain: Regularisation of Neural Networks by Constraining Activation
  Magnitudes
  Effective regularisation of neural networks is essential to combat
overfitting due to the large number of parameters involved. We present an
empirical analogue to the Lipschitz constant of a feed-forward neural network,
which we refer to as the maximum gain. We hypothesise that constraining the
gain of a network will have a regularising effect, similar to how constraining
the Lipschitz constant of a network has been shown to improve generalisation. A
simple algorithm is provided that involves rescaling the weight matrix of each
layer after each parameter update. We conduct a series of studies on common
benchmark datasets, and also a novel dataset that we introduce to enable easier
significance testing for experiments using convolutional networks. Performance
on these datasets compares favourably with other common regularisation
techniques.
