InfoNCE is a variational autoencoder
  We show that a popular self-supervised learning method, InfoNCE, is a special
case of a new family of unsupervised learning methods, the self-supervised
variational autoencoder (SSVAE). SSVAEs circumvent the usual VAE requirement to
reconstruct the data by using a carefully chosen implicit decoder. The InfoNCE
objective was motivated as a simplified parametric mutual information
estimator. Under one choice of prior, the SSVAE objective (i.e. the ELBO) is
exactly equal to the mutual information (up to constants). Under an alternative
choice of prior, the SSVAE objective is exactly equal to the simplified
parametric mutual information estimator used in InfoNCE (up to constants).
Importantly, the use of simplified parametric mutual information estimators is
believed to be critical to obtain good high-level representations, and the
SSVAE framework naturally provides a principled justification for using prior
information to choose these estimators.
