It's always personal: Using Early Exits for Efficient On-Device CNN
  Personalisation
  On-device machine learning is becoming a reality thanks to the availability
of powerful hardware and model compression techniques. Typically, these models
are pretrained on large GPU clusters and have enough parameters to generalise
across a wide variety of inputs. In this work, we observe that a much smaller,
personalised model can be employed to fit a specific scenario, resulting in
both higher accuracy and faster execution. Nevertheless, on-device training is
extremely challenging, imposing excessive computational and memory requirements
even for flagship smartphones. At the same time, on-device data availability
might be limited and samples are most frequently unlabelled. To this end, we
introduce PersEPhonEE, a framework that attaches early exits on the model and
personalises them on-device. These allow the model to progressively bypass a
larger part of the computation as more personalised data become available.
Moreover, we introduce an efficient on-device algorithm that trains the early
exits in a semi-supervised manner at a fraction of the whole network's
personalisation time. Results show that PersEPhonEE boosts accuracy by up to
15.9% while dropping the training cost by up to 2.2x and inference latency by
2.2-3.2x on average for the same accuracy, depending on the availability of
labels on-device.
