Stochastic algorithms with descent guarantees for ICA
  Independent component analysis (ICA) is a widespread data exploration
technique, where observed signals are modeled as linear mixtures of independent
components. From a machine learning point of view, it amounts to a matrix
factorization problem with a statistical independence criterion. Infomax is one
of the most used ICA algorithms. It is based on a loss function which is a
non-convex log-likelihood. We develop a new majorization-minimization framework
adapted to this loss function. We derive an online algorithm for the streaming
setting, and an incremental algorithm for the finite sum setting, with the
following benefits. First, unlike most algorithms found in the literature, the
proposed methods do not rely on any critical hyper-parameter like a step size,
nor do they require a line-search technique. Second, the algorithm for the
finite sum setting, although stochastic, guarantees a decrease of the loss
function at each iteration. Experiments demonstrate progress on the
state-of-the-art for large scale datasets, without the necessity for any manual
parameter tuning.
