Conditionally Deep Hybrid Neural Networks Across Edge and Cloud
  The pervasiveness of "Internet-of-Things" in our daily life has led to a
recent surge in fog computing, encompassing a collaboration of cloud computing
and edge intelligence. To that effect, deep learning has been a major driving
force towards enabling such intelligent systems. However, growing model sizes
in deep learning pose a significant challenge towards deployment in
resource-constrained edge devices. Moreover, in a distributed intelligence
environment, efficient workload distribution is necessary between edge and
cloud systems. To address these challenges, we propose a conditionally deep
hybrid neural network for enabling AI-based fog computing. The proposed network
can be deployed in a distributed manner, consisting of quantized layers and
early exits at the edge and full-precision layers on the cloud. During
inference, if an early exit has high confidence in the classification results,
it would allow samples to exit at the edge, and the deeper layers on the cloud
are activated conditionally, which can lead to improved energy efficiency and
inference latency. We perform an extensive design space exploration with the
goal of minimizing energy consumption at the edge while achieving
state-of-the-art classification accuracies on image classification tasks. We
show that with binarized layers at the edge, the proposed conditional hybrid
network can process 65% of inferences at the edge, leading to 5.5x
computational energy reduction with minimal accuracy degradation on CIFAR-10
dataset. For the more complex dataset CIFAR-100, we observe that the proposed
network with 4-bit quantization at the edge achieves 52% early classification
at the edge with 4.8x energy reduction. The analysis gives us insights on
designing efficient hybrid networks which achieve significantly higher energy
efficiency than full-precision networks for edge-cloud based distributed
intelligence systems.
