Fiedler Regularization: Learning Neural Networks with Graph Sparsity
  We introduce a novel regularization approach for deep learning that
incorporates and respects the underlying graphical structure of the neural
network. Existing regularization methods often focus on dropping/penalizing
weights in a global manner that ignores the connectivity structure of the
neural network. We propose to use the Fiedler value of the neural network's
underlying graph as a tool for regularization. We provide theoretical support
for this approach via spectral graph theory. We list several useful properties
of the Fiedler value that makes it suitable in regularization. We provide an
approximate, variational approach for fast computation in practical training of
neural networks. We provide bounds on such approximations. We provide an
alternative but equivalent formulation of this framework in the form of a
structurally weighted L1 penalty, thus linking our approach to sparsity
induction. We performed experiments on datasets that compare Fiedler
regularization with traditional regularization methods such as dropout and
weight decay. Results demonstrate the efficacy of Fiedler regularization.
