Graph Optimal Transport for Cross-Domain Alignment
  Cross-domain alignment between two sets of entities (e.g., objects in an
image, words in a sentence) is fundamental to both computer vision and natural
language processing. Existing methods mainly focus on designing advanced
attention mechanisms to simulate soft alignment, with no training signals to
explicitly encourage alignment. The learned attention matrices are also dense
and lacks interpretability. We propose Graph Optimal Transport (GOT), a
principled framework that germinates from recent advances in Optimal Transport
(OT). In GOT, cross-domain alignment is formulated as a graph matching problem,
by representing entities into a dynamically-constructed graph. Two types of OT
distances are considered: (i) Wasserstein distance (WD) for node (entity)
matching; and (ii) Gromov-Wasserstein distance (GWD) for edge (structure)
matching. Both WD and GWD can be incorporated into existing neural network
models, effectively acting as a drop-in regularizer. The inferred transport
plan also yields sparse and self-normalized alignment, enhancing the
interpretability of the learned model. Experiments show consistent
outperformance of GOT over baselines across a wide range of tasks, including
image-text retrieval, visual question answering, image captioning, machine
translation, and text summarization.
