Benchmarking TinyML Systems: Challenges and Direction
  Recent advancements in ultra-low-power machine learning (TinyML) hardware
promises to unlock an entirely new class of smart applications. However,
continued progress is limited by the lack of a widely accepted benchmark for
these systems. Benchmarking allows us to measure and thereby systematically
compare, evaluate, and improve the performance of systems and is therefore
fundamental to a field reaching maturity. In this position paper, we present
the current landscape of TinyML and discuss the challenges and direction
towards developing a fair and useful hardware benchmark for TinyML workloads.
Furthermore, we present our four benchmarks and discuss our selection
methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf
working group that is comprised of over 30 organizations.
