Implicit Regularization in Tensor Factorization
  Recent efforts to unravel the mystery of implicit regularization in deep
learning have led to a theoretical focus on matrix factorization -- matrix
completion via linear neural network. As a step further towards practical deep
learning, we provide the first theoretical analysis of implicit regularization
in tensor factorization -- tensor completion via certain type of non-linear
neural network. We circumvent the notorious difficulty of tensor problems by
adopting a dynamical systems perspective, and characterizing the evolution
induced by gradient descent. The characterization suggests a form of greedy low
tensor rank search, which we rigorously prove under certain conditions, and
empirically demonstrate under others. Motivated by tensor rank capturing the
implicit regularization of a non-linear neural network, we empirically explore
it as a measure of complexity, and find that it captures the essence of
datasets on which neural networks generalize. This leads us to believe that
tensor rank may pave way to explaining both implicit regularization in deep
learning, and the properties of real-world data translating this implicit
regularization to generalization.
