GraphConnect: A Regularization Framework for Neural Networks
  Deep neural networks have proved very successful in domains where large
training sets are available, but when the number of training samples is small,
their performance suffers from overfitting. Prior methods of reducing
overfitting such as weight decay, Dropout and DropConnect are data-independent.
This paper proposes a new method, GraphConnect, that is data-dependent, and is
motivated by the observation that data of interest lie close to a manifold. The
new method encourages the relationships between the learned decisions to
resemble a graph representing the manifold structure. Essentially GraphConnect
is designed to learn attributes that are present in data samples in contrast to
weight decay, Dropout and DropConnect which are simply designed to make it more
difficult to fit to random error or noise. Empirical Rademacher complexity is
used to connect the generalization error of the neural network to spectral
properties of the graph learned from the input data. This framework is used to
show that GraphConnect is superior to weight decay. Experimental results on
several benchmark datasets validate the theoretical analysis, and show that
when the number of training samples is small, GraphConnect is able to
significantly improve performance over weight decay.
