Simple but Effective: CLIP Embeddings for Embodied AI
  Contrastive language image pretraining (CLIP) encoders have been shown to be
beneficial for a range of visual tasks from classification and detection to
captioning and image manipulation. We investigate the effectiveness of CLIP
visual backbones for Embodied AI tasks. We build incredibly simple baselines,
named EmbCLIP, with no task specific architectures, inductive biases (such as
the use of semantic maps), auxiliary tasks during training, or depth maps --
yet we find that our improved baselines perform very well across a range of
tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav leaderboard by a huge
margin of 20 pts (Success Rate). It tops the iTHOR 1-Phase Rearrangement
leaderboard, beating the next best submission, which employs Active Neural
Mapping, and more than doubling the % Fixed Strict metric (0.08 to 0.17). It
also beats the winners of the 2021 Habitat ObjectNav Challenge, which employ
auxiliary tasks, depth maps, and human demonstrations, and those of the 2019
Habitat PointNav Challenge. We evaluate the ability of CLIP's visual
representations at capturing semantic information about input observations --
primitives that are useful for navigation-heavy embodied tasks -- and find that
CLIP's representations encode these primitives more effectively than
ImageNet-pretrained backbones. Finally, we extend one of our baselines,
producing an agent capable of zero-shot object navigation that can navigate to
objects that were not used as targets during training. Our code and models are
available at https://github.com/allenai/embodied-clip
