Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures
  Model-based methods and deep neural networks have both been tremendously
successful paradigms in machine learning. In model-based methods, problem
domain knowledge can be built into the constraints of the model, typically at
the expense of difficulties during inference. In contrast, deterministic deep
neural networks are constructed in such a way that inference is
straightforward, but their architectures are generic and it is unclear how to
incorporate knowledge. This work aims to obtain the advantages of both
approaches. To do so, we start with a model-based approach and an associated
inference algorithm, and \emph{unfold} the inference iterations as layers in a
deep network. Rather than optimizing the original model, we \emph{untie} the
model parameters across layers, in order to create a more powerful network. The
resulting architecture can be trained discriminatively to perform accurate
inference within a fixed network size. We show how this framework allows us to
interpret conventional networks as mean-field inference in Markov random
fields, and to obtain new architectures by instead using belief propagation as
the inference algorithm. We then show its application to a non-negative matrix
factorization model that incorporates the problem-domain knowledge that sound
sources are additive. Deep unfolding of this model yields a new kind of
non-negative deep neural network, that can be trained using a multiplicative
backpropagation-style update algorithm. We present speech enhancement
experiments showing that our approach is competitive with conventional neural
networks despite using far fewer parameters.
