Multiscale Residual Mixture of PCA: Dynamic Dictionaries for Optimal
  Basis Learning
  In this paper we are interested in the problem of learning an over-complete
basis and a methodology such that the reconstruction or inverse problem does
not need optimization. We analyze the optimality of the presented approaches,
their link to popular already known techniques s.a. Artificial Neural
Networks,k-means or Oja's learning rule. Finally, we will see that one approach
to reach the optimal dictionary is a factorial and hierarchical approach. The
derived approach lead to a formulation of a Deep Oja Network. We present
results on different tasks and present the resulting very efficient learning
algorithm which brings a new vision on the training of deep nets. Finally, the
theoretical work shows that deep frameworks are one way to efficiently have
over-complete (combinatorially large) dictionary yet allowing easy
reconstruction. We thus present the Deep Residual Oja Network (DRON). We
demonstrate that a recursive deep approach working on the residuals allow
exponential decrease of the error w.r.t. the depth.
