Modeling Accurate Human Activity Recognition for Embedded Devices Using
  Multi-level Distillation
  Human Activity Recognition (HAR) based on IMU sensors is a crucial area in
ubiquitous computing. Because of the trend of deploying AI on IoT devices or
smartphones, more researchers are designing different HAR models for embedded
devices. Deployment of models in embedded devices can help enhance the
efficiency of HAR. We propose a multi-level HAR modeling pipeline called
Stage-Logits-Memory Distillation (SMLDist) for constructing deep convolutional
HAR models with embedded hardware support. SMLDist includes stage distillation,
memory distillation, and logits distillation. Stage distillation constrains the
learning direction of the intermediate features. The teacher model teaches the
student models how to explain and store the inner relationship among
high-dimensional features based on Hopfield networks in memory distillation.
Logits distillation builds logits distilled by a smoothed conditional rule to
preserve the probability distribution and enhance the softer target accuracy.
We compare the accuracy, F1 macro score, and energy cost on embedded platforms
of a MobileNet V3 model built by SMLDist with various state-of-the-art HAR
frameworks. The product model has a good balance with robustness and
efficiency. SMLDist can also compress models with a minor performance loss at
an equal compression ratio to other advanced knowledge distillation methods on
seven public datasets.
