Understanding Deep Contrastive Learning via Coordinate-wise Optimization
  We show that Contrastive Learning (CL) under a broad family of loss functions
(including InfoNCE) has a unified formulation of coordinate-wise optimization
on the network parameter $\boldsymbol{\theta}$ and pairwise importance
$\alpha$, where the \emph{max player} $\boldsymbol{\theta}$ learns
representation for contrastiveness, and the \emph{min player} $\alpha$ puts
more weights on pairs of distinct samples that share similar representations.
The resulting formulation, called $\alpha$-CL, unifies not only various
existing contrastive losses, which differ by how sample-pair importance
$\alpha$ is constructed, but also is able to extrapolate to give novel
contrastive losses beyond popular ones, opening a new avenue of contrastive
loss design. These novel losses yield comparable (or better) performance on
CIFAR10 and STL-10 than classic InfoNCE. Furthermore, we also analyze the max
player in detail: we prove that with fixed $\alpha$, max player is equivalent
to Principal Component Analysis (PCA) for deep linear network, and almost all
local minima are global and rank-1, recovering optimal PCA solutions. Finally,
we extend our analysis on max player to 2-layer ReLU networks, showing that its
fixed points can have higher ranks.
