DEFER: Distributed Edge Inference for Deep Neural Networks
  Modern machine learning tools such as deep neural networks (DNNs) are playing
a revolutionary role in many fields such as natural language processing,
computer vision, and the internet of things. Once they are trained, deep
learning models can be deployed on edge computers to perform classification and
prediction on real-time data for these applications. Particularly for large
models, the limited computational and memory resources on a single edge device
can become the throughput bottleneck for an inference pipeline. To increase
throughput and decrease per-device compute load, we present DEFER (Distributed
Edge inFERence), a framework for distributed edge inference, which partitions
deep neural networks into layers that can be spread across multiple compute
nodes. The architecture consists of a single "dispatcher" node to distribute
DNN partitions and inference data to respective compute nodes. The compute
nodes are connected in a series pattern where each node's computed result is
relayed to the subsequent node. The result is then returned to the Dispatcher.
We quantify the throughput, energy consumption, network payload, and overhead
for our framework under realistic network conditions using the CORE network
emulator. We find that for the ResNet50 model, the inference throughput of
DEFER with 8 compute nodes is 53% higher and per node energy consumption is 63%
lower than single device inference. We further reduce network communication
demands and energy consumption using the ZFP serialization and LZ4 compression
algorithms. We have implemented DEFER in Python using the TensorFlow and Keras
ML libraries, and have released DEFER as an open-source framework to benefit
the research community.
