Multi-Layer Sparse Coding: The Holistic Way
  The recently proposed multi-layer sparse model has raised insightful
connections between sparse representations and convolutional neural networks
(CNN). In its original conception, this model was restricted to a cascade of
convolutional synthesis representations. In this paper, we start by addressing
a more general model, revealing interesting ties to fully connected networks.
We then show that this multi-layer construction admits a brand new
interpretation in a unique symbiosis between synthesis and analysis models:
while the deepest layer indeed provides a synthesis representation, the
mid-layers decompositions provide an analysis counterpart. This new perspective
exposes the suboptimality of previously proposed pursuit approaches, as they do
not fully leverage all the information comprised in the model constraints.
Armed with this understanding, we address fundamental theoretical issues,
revisiting previous analysis and expanding it. Motivated by the limitations of
previous algorithms, we then propose an integrated - holistic - alternative
that estimates all representations in the model simultaneously, and analyze all
these different schemes under stochastic noise assumptions. Inspired by the
synthesis-analysis duality, we further present a Holistic Pursuit algorithm,
which alternates between synthesis and analysis sparse coding steps, eventually
solving for the entire model as a whole, with provable improved performance.
Finally, we present numerical results that demonstrate the practical advantages
of our approach.
