On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional
  Neural Networks
  Parsimonious representations are ubiquitous in modeling and processing
information. Motivated by the recent Multi-Layer Convolutional Sparse Coding
(ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a
multi-layer setting, introducing similar sparse enforcing penalties at
different representation layers in a symbiotic relation between synthesis and
analysis sparse priors. We explore different iterative methods to solve this
new problem in practice, and we propose a new Multi-Layer Iterative Soft
Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show
that these nested first order algorithms converge, in the sense that the
function value of near-fixed points can get arbitrarily close to the solution
of the original problem.
  We further show how these algorithms effectively implement particular
recurrent convolutional neural networks (CNNs) that generalize feed-forward
ones without introducing any parameters. We present and analyze different
architectures resulting unfolding the iterations of the proposed pursuit
algorithms, including a new Learned ML-ISTA, providing a principled way to
construct deep recurrent CNNs. Unlike other similar constructions, these
architectures unfold a global pursuit holistically for the entire network. We
demonstrate the emerging constructions in a supervised learning setting,
consistently improving the performance of classical CNNs while maintaining the
number of parameters constant.
