Embodied Multimodal Multitask Learning
  Recent efforts on training visual navigation agents conditioned on language
using deep reinforcement learning have been successful in learning policies for
different multimodal tasks, such as semantic goal navigation and embodied
question answering. In this paper, we propose a multitask model capable of
jointly learning these multimodal tasks, and transferring knowledge of words
and their grounding in visual objects across the tasks. The proposed model uses
a novel Dual-Attention unit to disentangle the knowledge of words in the
textual representations and visual concepts in the visual representations, and
align them with each other. This disentangled task-invariant alignment of
representations facilitates grounding and knowledge transfer across both tasks.
We show that the proposed model outperforms a range of baselines on both tasks
in simulated 3D environments. We also show that this disentanglement of
representations makes our model modular, interpretable, and allows for transfer
to instructions containing new words by leveraging object detectors.
