Mind Your Weight(s): A Large-scale Study on Insufficient Machine
  Learning Model Protection in Mobile Apps
  On-device machine learning (ML) is quickly gaining popularity among mobile
apps. It allows offline model inference while preserving user privacy. However,
ML models, considered as core intellectual properties of model owners, are now
stored on billions of untrusted devices and subject to potential thefts. Leaked
models can cause both severe financial loss and security consequences. This
paper presents the first empirical study of ML model protection on mobile
devices. Our study aims to answer three open questions with quantitative
evidence: How widely is model protection used in apps? How robust are existing
model protection techniques? What impacts can (stolen) models incur? To that
end, we built a simple app analysis pipeline and analyzed 46,753 popular apps
collected from the US and Chinese app markets. We identified 1,468 ML apps
spanning all popular app categories. We found that, alarmingly, 41% of ML apps
do not protect their models at all, which can be trivially stolen from app
packages. Even for those apps that use model protection or encryption, we were
able to extract the models from 66% of them via unsophisticated dynamic
analysis techniques. The extracted models are mostly commercial products and
used for face recognition, liveness detection, ID/bank card recognition, and
malware detection. We quantitatively estimated the potential financial and
security impact of a leaked model, which can amount to millions of dollars for
different stakeholders. Our study reveals that on-device models are currently
at high risk of being leaked; attackers are highly motivated to steal such
models. Drawn from our large-scale study, we report our insights into this
emerging security problem and discuss the technical challenges, hoping to
inspire future research on robust and practical model protection for mobile
devices.
