Neural Network Regularization via Robust Weight Factorization
  Regularization is essential when training large neural networks. As deep
neural networks can be mathematically interpreted as universal function
approximators, they are effective at memorizing sampling noise in the training
data. This results in poor generalization to unseen data. Therefore, it is no
surprise that a new regularization technique, Dropout, was partially
responsible for the now-ubiquitous winning entry to ImageNet 2012 by the
University of Toronto. Currently, Dropout (and related methods such as
DropConnect) are the most effective means of regularizing large neural
networks. These amount to efficiently visiting a large number of related models
at training time, while aggregating them to a single predictor at test time.
The proposed FaMe model aims to apply a similar strategy, yet learns a
factorization of each weight matrix such that the factors are robust to noise.
