Unsupervised Feature Extraction by Time-Contrastive Learning and
  Nonlinear ICA
  Nonlinear independent component analysis (ICA) provides an appealing
framework for unsupervised feature learning, but the models proposed so far are
not identifiable. Here, we first propose a new intuitive principle of
unsupervised deep learning from time series which uses the nonstationary
structure of the data. Our learning principle, time-contrastive learning (TCL),
finds a representation which allows optimal discrimination of time segments
(windows). Surprisingly, we show how TCL can be related to a nonlinear ICA
model, when ICA is redefined to include temporal nonstationarities. In
particular, we show that TCL combined with linear ICA estimates the nonlinear
ICA model up to point-wise transformations of the sources, and this solution is
unique --- thus providing the first identifiability result for nonlinear ICA
which is rigorous, constructive, as well as very general.
