TinyM$^2$Net: A Flexible System Algorithm Co-designed Multimodal
  Learning Framework for Tiny Devices
  With the emergence of Artificial Intelligence (AI), new attention has been
given to implement AI algorithms on resource constrained tiny devices to expand
the application domain of IoT. Multimodal Learning has recently become very
popular with the classification task due to its impressive performance for both
image and audio event classification. This paper presents TinyM$^2$Net -- a
flexible system algorithm co-designed multimodal learning framework for
resource constrained tiny devices. The framework was designed to be evaluated
on two different case-studies: COVID-19 detection from multimodal audio
recordings and battle field object detection from multimodal images and audios.
In order to compress the model to implement on tiny devices, substantial
network architecture optimization and mixed precision quantization were
performed (mixed 8-bit and 4-bit). TinyM$^2$Net shows that even a tiny
multimodal learning model can improve the classification performance than that
of any unimodal frameworks. The most compressed TinyM$^2$Net achieves 88.4%
COVID-19 detection accuracy (14.5% improvement from unimodal base model) and
96.8% battle field object detection accuracy (3.9% improvement from unimodal
base model). Finally, we test our TinyM$^2$Net models on a Raspberry Pi 4 to
see how they perform when deployed to a resource constrained tiny device.
