You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory
  Prediction
  Predicting the future trajectory of a moving agent can be easy when the past
trajectory continues smoothly but is challenging when complex interactions with
other agents are involved. Recent deep learning approaches for trajectory
prediction show promising performance and partially attribute this to
successful reasoning about agent-agent interactions. However, it remains
unclear which features such black-box models actually learn to use for making
predictions. This paper proposes a procedure that quantifies the contributions
of different cues to model performance based on a variant of Shapley values.
Applying this procedure to state-of-the-art trajectory prediction methods on
standard benchmark datasets shows that they are, in fact, unable to reason
about interactions. Instead, the past trajectory of the target is the only
feature used for predicting its future. For a task with richer social
interaction patterns, on the other hand, the tested models do pick up such
interactions to a certain extent, as quantified by our feature attribution
method. We discuss the limits of the proposed method and its links to causality
