Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time
  Series Forecasting
  Autonomous driving gained huge traction in recent years, due to its potential
to change the way we commute. Much effort has been put into trying to estimate
the state of a vehicle. Meanwhile, learning to forecast the state of a vehicle
ahead introduces new capabilities, such as predicting dangerous situations.
Moreover, forecasting brings new supervision opportunities by learning to
predict richer a context, expressed by multiple horizons. Intuitively, a video
stream originated from a front-facing camera is necessary because it encodes
information about the upcoming road. Besides, historical traces of the
vehicle's states give more context. In this paper, we tackle multi-horizon
forecasting of vehicle states by fusing the two modalities. We design and
experiment with 3 end-to-end architectures that exploit 3D convolutions for
visual features extraction and 1D convolutions for features extraction from
speed and steering angle traces. To demonstrate the effectiveness of our
method, we perform extensive experiments on two publicly available real-world
datasets, Comma2k19 and the Udacity challenge. We show that we are able to
forecast a vehicle's state to various horizons, while outperforming the current
state-of-the-art results on the related task of driving state estimation. We
examine the contribution of vision features, and find that a model fed with
vision features achieves an error that is 56.6% and 66.9% of the error of a
model that doesn't use those features, on the Udacity and Comma2k19 datasets
respectively.
