Transferable Representation Learning in Vision-and-Language Navigation
  Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require
machine agents to interpret natural language instructions and learn to act in
visually realistic environments to achieve navigation goals. The overall task
requires competence in several perception problems: successful agents combine
spatio-temporal, vision and language understanding to produce appropriate
action sequences. Our approach adapts pre-trained vision and language
representations to relevant in-domain tasks making them more effective for VLN.
Specifically, the representations are adapted to solve both a cross-modal
sequence alignment and sequence coherence task. In the sequence alignment task,
the model determines whether an instruction corresponds to a sequence of visual
frames. In the sequence coherence task, the model determines whether the
perceptual sequences are predictive sequentially in the instruction-conditioned
latent space. By transferring the domain-adapted representations, we improve
competitive agents in R2R as measured by the success rate weighted by path
length (SPL) metric.
