Real-Time Edge Classification: Optimal Offloading under Token Bucket
  Constraints
  To deploy machine learning-based algorithms for real-time applications with
strict latency constraints, we consider an edge-computing setting where a
subset of inputs are offloaded to the edge for processing by an accurate but
resource-intensive model, and the rest are processed only by a less-accurate
model on the device itself. Both models have computational costs that match
available compute resources, and process inputs with low-latency. But
offloading incurs network delays, and to manage these delays to meet
application deadlines, we use a token bucket to constrain the average rate and
burst length of transmissions from the device. We introduce a Markov Decision
Process-based framework to make offload decisions under these constraints,
based on the local model's confidence and the token bucket state, with the goal
of minimizing a specified error measure for the application. Beyond isolated
decisions for individual devices, we also propose approaches to allow multiple
devices connected to the same access switch to share their bursting allocation.
We evaluate and analyze the policies derived using our framework on the
standard ImageNet image classification benchmark.
