Cost-effective Machine Learning Inference Offload for Edge Computing
  Computing at the edge is increasingly important since a massive amount of
data is generated. This poses challenges in transporting all that data to the
remote data centers and cloud, where they can be processed and analyzed. On the
other hand, harnessing the edge data is essential for offering data-driven and
machine learning-based applications, if the challenges, such as device
capabilities, connectivity, and heterogeneity can be mitigated. Machine
learning applications are very compute-intensive and require processing of
large amount of data. However, edge devices are often resources-constrained, in
terms of compute resources, power, storage, and network connectivity. Hence,
limiting their potential to run efficiently and accurately state-of-the art
deep neural network (DNN) models, which are becoming larger and more complex.
This paper proposes a novel offloading mechanism by leveraging installed-base
on-premises (edge) computational resources. The proposed mechanism allows the
edge devices to offload heavy and compute-intensive workloads to edge nodes
instead of using remote cloud. Our offloading mechanism has been prototyped and
tested with state-of-the art person and object detection DNN models for mobile
robots and video surveillance applications. The performance shows a significant
gain compared to cloud-based offloading strategies in terms of accuracy and
latency.
