On the Difficulty of Defending Self-Supervised Learning against Model
  Extraction
  Self-Supervised Learning (SSL) is an increasingly popular ML paradigm that
trains models to transform complex inputs into representations without relying
on explicit labels. These representations encode similarity structures that
enable efficient learning of multiple downstream tasks. Recently,
ML-as-a-Service providers have commenced offering trained SSL models over
inference APIs, which transform user inputs into useful representations for a
fee. However, the high cost involved to train these models and their exposure
over APIs both make black-box extraction a realistic security threat. We thus
explore model stealing attacks against SSL. Unlike traditional model extraction
on classifiers that output labels, the victim models here output
representations; these representations are of significantly higher
dimensionality compared to the low-dimensional prediction scores output by
classifiers. We construct several novel attacks and find that approaches that
train directly on a victim's stolen representations are query efficient and
enable high accuracy for downstream models. We then show that existing defenses
against model extraction are inadequate and not easily retrofitted to the
specificities of SSL.
