Software/Hardware Co-design for Multi-modal Multi-task Learning in
  Autonomous Systems
  Optimizing the quality of result (QoR) and the quality of service (QoS) of
AI-empowered autonomous systems simultaneously is very challenging. First,
there are multiple input sources, e.g., multi-modal data from different
sensors, requiring diverse data preprocessing, sensor fusion, and feature
aggregation. Second, there are multiple tasks that require various AI models to
run simultaneously, e.g., perception, localization, and control. Third, the
computing and control system is heterogeneous, composed of hardware components
with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated
accelerators. Therefore, autonomous systems essentially require multi-modal
multi-task (MMMT) learning which must be aware of hardware performance and
implementation strategies. While MMMT learning has been attracting intensive
research interests, its applications in autonomous systems are still
underexplored. In this paper, we first discuss the opportunities of applying
MMMT techniques in autonomous systems and then discuss the unique challenges
that must be solved. In addition, we discuss the necessity and opportunities of
MMMT model and hardware co-design, which is critical for autonomous systems
especially with power/resource-limited or heterogeneous platforms. We formulate
the MMMT model and heterogeneous hardware implementation co-design as a
differentiable optimization problem, with the objective of improving the
solution quality and reducing the overall power consumption and critical path
latency. We advocate for further explorations of MMMT in autonomous systems and
software/hardware co-design solutions.
