Trajectory Prediction with Linguistic Representations
  Language allows humans to build mental models that interpret what is
happening around them resulting in more accurate long-term predictions. We
present a novel trajectory prediction model that uses linguistic intermediate
representations to forecast trajectories, and is trained using trajectory
samples with partially-annotated captions. The model learns the meaning of each
of the words without direct per-word supervision. At inference time, it
generates a linguistic description of trajectories which captures maneuvers and
interactions over an extended time interval. This generated description is used
to refine predictions of the trajectories of multiple agents. We train and
validate our model on the Argoverse dataset, and demonstrate improved accuracy
results in trajectory prediction. In addition, our model is more interpretable:
it presents part of its reasoning in plain language as captions, which can aid
model development and can aid in building confidence in the model before
deploying it.
