An Analysis of Dropout for Matrix Factorization
  Dropout is a simple yet effective algorithm for regularizing neural networks
by randomly dropping out units through Bernoulli multiplicative noise, and for
some restricted problem classes, such as linear or logistic regression, several
theoretical studies have demonstrated the equivalence between dropout and a
fully deterministic optimization problem with data-dependent Tikhonov
regularization. This work presents a theoretical analysis of dropout for matrix
factorization, where Bernoulli random variables are used to drop a factor,
thereby attempting to control the size of the factorization. While recent work
has demonstrated the empirical effectiveness of dropout for matrix
factorization, a theoretical understanding of the regularization properties of
dropout in this context remains elusive. This work demonstrates the equivalence
between dropout and a fully deterministic model for matrix factorization in
which the factors are regularized by the sum of the product of the norms of the
columns. While the resulting regularizer is closely related to a variational
form of the nuclear norm, suggesting that dropout may limit the size of the
factorization, we show that it is possible to trivially lower the objective
value by doubling the size of the factorization. We show that this problem is
caused by the use of a fixed dropout rate, which motivates the use of a rate
that increases with the size of the factorization. Synthetic experiments
validate our theoretical findings.
