SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval
  We tackle the cross-modal retrieval problem, where the training is only
supervised by the relevant multi-modal pairs in the data. The contrastive
learning is the most popular approach for this task. However, its sampling
complexity for learning is quadratic in the number of training data points.
Moreover, it makes potentially wrong assumption that the instances in different
pairs are automatically irrelevant. To address these issues, we propose a novel
loss function that is based on self-labeling of the unknown classes.
Specifically, we aim to predict class labels of the data instances in each
modality, and assign those labels to the corresponding instances in the other
modality (i.e., swapping the pseudo labels). With these swapped labels, we
learn the data embedding for each modality using the supervised cross-entropy
loss, hence leading to linear sampling complexity. We also maintain the queues
for storing the embeddings of the latest batches, for which clustering
assignment and embedding learning are done at the same time in an online
fashion. This removes computational overhead of injecting intermittent epochs
of entire training data sweep for offline clustering. We tested our approach on
several real-world cross-modal retrieval problems, including text-based video
retrieval, sketch-based image retrieval, and image-text retrieval, and for all
these tasks our method achieves significant performance improvement over the
contrastive learning.
