Distance-Based Regularisation of Deep Networks for Fine-Tuning
  We investigate approaches to regularisation during fine-tuning of deep neural
networks. First we provide a neural network generalisation bound based on
Rademacher complexity that uses the distance the weights have moved from their
initial values. This bound has no direct dependence on the number of weights
and compares favourably to other bounds when applied to convolutional networks.
Our bound is highly relevant for fine-tuning, because providing a network with
a good initialisation based on transfer learning means that learning can modify
the weights less, and hence achieve tighter generalisation. Inspired by this,
we develop a simple yet effective fine-tuning algorithm that constrains the
hypothesis class to a small sphere centred on the initial pre-trained weights,
thus obtaining provably better generalisation performance than conventional
transfer learning. Empirical evaluation shows that our algorithm works well,
corroborating our theoretical results. It outperforms both state of the art
fine-tuning competitors, and penalty-based alternatives that we show do not
directly constrain the radius of the search space.
