Complexity-aware Adaptive Training and Inference for Edge-Cloud
  Distributed AI Systems
  The ubiquitous use of IoT and machine learning applications is creating large
amounts of data that require accurate and real-time processing. Although
edge-based smart data processing can be enabled by deploying pretrained models,
the energy and memory constraints of edge devices necessitate distributed deep
learning between the edge and the cloud for complex data. In this paper, we
propose a distributed AI system to exploit both the edge and the cloud for
training and inference. We propose a new architecture, MEANet, with a main
block, an extension block, and an adaptive block for the edge. The inference
process can terminate at either the main block, the extension block, or the
cloud. The MEANet is trained to categorize inputs into easy/hard/complex
classes. The main block identifies instances of easy/hard classes and
classifies easy classes with high confidence. Only data with high probabilities
of belonging to hard classes would be sent to the extension block for
prediction. Further, only if the neural network at the edge shows low
confidence in the prediction, the instance is considered complex and sent to
the cloud for further processing. The training technique lends to the majority
of inference on edge devices while going to the cloud only for a small set of
complex jobs, as determined by the edge. The performance of the proposed system
is evaluated via extensive experiments using modified models of ResNets and
MobileNetV2 on CIFAR-100 and ImageNet datasets. The results show that the
proposed distributed model has improved accuracy and energy consumption,
indicating its capacity to adapt.
