Thieves on Sesame Street! Model Extraction of BERT-based APIs
  We study the problem of model extraction in natural language processing, in
which an adversary with only query access to a victim model attempts to
reconstruct a local copy of that model. Assuming that both the adversary and
victim model fine-tune a large pretrained language model such as BERT (Devlin
et al. 2019), we show that the adversary does not need any real training data
to successfully mount the attack. In fact, the attacker need not even use
grammatical or semantically meaningful queries: we show that random sequences
of words coupled with task-specific heuristics form effective queries for model
extraction on a diverse set of NLP tasks, including natural language inference
and question answering. Our work thus highlights an exploit only made feasible
by the shift towards transfer learning methods within the NLP community: for a
query budget of a few hundred dollars, an attacker can extract a model that
performs only slightly worse than the victim model. Finally, we study two
defense strategies against model extraction---membership classification and API
watermarking---which while successful against naive adversaries, are
ineffective against more sophisticated ones.
