Multimodal Federated Learning on IoT Data
  Federated learning is proposed as an alternative to centralized machine
learning since its client-server structure provides better privacy protection
and scalability in real-world applications. In many applications, such as smart
homes with Internet-of-Things (IoT) devices, local data on clients are
generated from different modalities such as sensory, visual, and audio data.
Existing federated learning systems only work on local data from a single
modality, which limits the scalability of the systems.
  In this paper, we propose a multimodal and semi-supervised federated learning
framework that trains autoencoders to extract shared or correlated
representations from different local data modalities on clients. In addition,
we propose a multimodal FedAvg algorithm to aggregate local autoencoders
trained on different data modalities. We use the learned global autoencoder for
a downstream classification task with the help of auxiliary labelled data on
the server. We empirically evaluate our framework on different modalities
including sensory data, depth camera videos, and RGB camera videos. Our
experimental results demonstrate that introducing data from multiple modalities
into federated learning can improve its classification performance. In
addition, we can use labelled data from only one modality for supervised
learning on the server and apply the learned model to testing data from other
modalities to achieve decent F1 scores (e.g., with the best performance being
higher than 60%), especially when combining contributions from both unimodal
clients and multimodal clients.
