Learning ABCs: Approximate Bijective Correspondence for isolating
  factors of variation with weak supervision
  Representational learning forms the backbone of most deep learning
applications, and the value of a learned representation is intimately tied to
its information content regarding different factors of variation. Finding good
representations depends on the nature of supervision and the learning
algorithm. We propose a novel algorithm that utilizes a weak form of
supervision where the data is partitioned into sets according to certain
inactive (common) factors of variation which are invariant across elements of
each set. Our key insight is that by seeking correspondence between elements of
different sets, we learn strong representations that exclude the inactive
factors of variation and isolate the active factors that vary within all sets.
As a consequence of focusing on the active factors, our method can leverage a
mix of set-supervised and wholly unsupervised data, which can even belong to a
different domain. We tackle the challenging problem of synthetic-to-real object
pose transfer, without pose annotations on anything, by isolating pose
information which generalizes to the category level and across the
synthetic/real domain gap. The method can also boost performance in supervised
settings, by strengthening intermediate representations, as well as operate in
practically attainable scenarios with set-supervised natural images, where
quantity is limited and nuisance factors of variation are more plentiful.
