MAPLE-Edge: A Runtime Latency Predictor for Edge Devices
  Neural Architecture Search (NAS) has enabled automatic discovery of more
efficient neural network architectures, especially for mobile and embedded
vision applications. Although recent research has proposed ways of quickly
estimating latency on unseen hardware devices with just a few samples, little
focus has been given to the challenges of estimating latency on runtimes using
optimized graphs, such as TensorRT and specifically for edge devices. In this
work, we propose MAPLE-Edge, an edge device-oriented extension of MAPLE, the
state-of-the-art latency predictor for general purpose hardware, where we train
a regression network on architecture-latency pairs in conjunction with a
hardware-runtime descriptor to effectively estimate latency on a diverse pool
of edge devices. Compared to MAPLE, MAPLE-Edge can describe the runtime and
target device platform using a much smaller set of CPU performance counters
that are widely available on all Linux kernels, while still achieving up to
+49.6% accuracy gains against previous state-of-the-art baseline methods on
optimized edge device runtimes, using just 10 measurements from an unseen
target device. We also demonstrate that unlike MAPLE which performs best when
trained on a pool of devices sharing a common runtime, MAPLE-Edge can
effectively generalize across runtimes by applying a trick of normalizing
performance counters by the operator latency, in the measured hardware-runtime
descriptor. Lastly, we show that for runtimes exhibiting lower than desired
accuracy, performance can be boosted by collecting additional samples from the
target device, with an extra 90 samples translating to gains of nearly +40%.
