AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition
  Multi-modal learning, which focuses on utilizing various modalities to
improve the performance of a model, is widely used in video recognition. While
traditional multi-modal learning offers excellent recognition results, its
computational expense limits its impact for many real-world applications. In
this paper, we propose an adaptive multi-modal learning framework, called
AdaMML, that selects on-the-fly the optimal modalities for each segment
conditioned on the input for efficient video recognition. Specifically, given a
video segment, a multi-modal policy network is used to decide what modalities
should be used for processing by the recognition model, with the goal of
improving both accuracy and efficiency. We efficiently train the policy network
jointly with the recognition model using standard back-propagation. Extensive
experiments on four challenging diverse datasets demonstrate that our proposed
adaptive approach yields 35%-55% reduction in computation when compared to the
traditional baseline that simply uses all the modalities irrespective of the
input, while also achieving consistent improvements in accuracy over the
state-of-the-art methods.
