Unsupervised learning of disentangled representations in deep restricted
  kernel machines with orthogonality constraints
  We introduce Constr-DRKM, a deep kernel method for the unsupervised learning
of disentangled data representations. We propose augmenting the original deep
restricted kernel machine formulation for kernel PCA by orthogonality
constraints on the latent variables to promote disentanglement and to make it
possible to carry out optimization without first defining a stabilized
objective. After illustrating an end-to-end training procedure based on a
quadratic penalty optimization algorithm with warm start, we quantitatively
evaluate the proposed method's effectiveness in disentangled feature learning.
We demonstrate on four benchmark datasets that this approach performs similarly
overall to $\beta$-VAE on a number of disentanglement metrics when few training
points are available, while being less sensitive to randomness and
hyperparameter selection than $\beta$-VAE. We also present a deterministic
initialization of Constr-DRKM's training algorithm that significantly improves
the reproducibility of the results. Finally, we empirically evaluate and
discuss the role of the number of layers in the proposed methodology, examining
the influence of each principal component in every layer and showing that
components in lower layers act as local feature detectors capturing the broad
trends of the data distribution, while components in deeper layers use the
representation learned by previous layers and more accurately reproduce
higher-level features.
