Single-Layer Vision Transformers for More Accurate Early Exits with Less
  Overhead
  Deploying deep learning models in time-critical applications with limited
computational resources, for instance in edge computing systems and IoT
networks, is a challenging task that often relies on dynamic inference methods
such as early exiting. In this paper, we introduce a novel architecture for
early exiting based on the vision transformer architecture, as well as a
fine-tuning strategy that significantly increase the accuracy of early exit
branches compared to conventional approaches while introducing less overhead.
Through extensive experiments on image and audio classification as well as
audiovisual crowd counting, we show that our method works for both
classification and regression problems, and in both single- and multi-modal
settings. Additionally, we introduce a novel method for integrating audio and
visual modalities within early exits in audiovisual data analysis, that can
lead to a more fine-grained dynamic inference.
