Learning to cluster in order to transfer across domains and tasks
  This paper introduces a novel method to perform transfer learning across
domains and tasks, formulating it as a problem of learning to cluster. The key
insight is that, in addition to features, we can transfer similarity
information and this is sufficient to learn a similarity function and
clustering network to perform both domain adaptation and cross-task transfer
learning. We begin by reducing categorical information to pairwise constraints,
which only considers whether two instances belong to the same class or not.
This similarity is category-agnostic and can be learned from data in the source
domain using a similarity network. We then present two novel approaches for
performing transfer learning using this similarity function. First, for
unsupervised domain adaptation, we design a new loss function to regularize
classification with a constrained clustering loss, hence learning a clustering
network with the transferred similarity metric generating the training inputs.
Second, for cross-task learning (i.e., unsupervised clustering with unseen
categories), we propose a framework to reconstruct and estimate the number of
semantic clusters, again using the clustering network. Since the similarity
network is noisy, the key is to use a robust clustering algorithm, and we show
that our formulation is more robust than the alternative constrained and
unconstrained clustering approaches. Using this method, we first show state of
the art results for the challenging cross-task problem, applied on Omniglot and
ImageNet. Our results show that we can reconstruct semantic clusters with high
accuracy. We then evaluate the performance of cross-domain transfer using
images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both
datasets. Our approach doesn't explicitly deal with domain discrepancy. If we
combine with a domain adaptation loss, it shows further improvement.
