Safe Distillation Box
  Knowledge distillation (KD) has recently emerged as a powerful strategy to
transfer knowledge from a pre-trained teacher model to a lightweight student,
and has demonstrated its unprecedented success over a wide spectrum of
applications. In spite of the encouraging results, the KD process per se poses
a potential threat to network ownership protection, since the knowledge
contained in network can be effortlessly distilled and hence exposed to a
malicious user. In this paper, we propose a novel framework, termed as Safe
Distillation Box (SDB), that allows us to wrap a pre-trained model in a virtual
box for intellectual property protection. Specifically, SDB preserves the
inference capability of the wrapped model to all users, but precludes KD from
unauthorized users. For authorized users, on the other hand, SDB carries out a
knowledge augmentation scheme to strengthen the KD performances and the results
of the student model. In other words, all users may employ a model in SDB for
inference, but only authorized users get access to KD from the model. The
proposed SDB imposes no constraints over the model architecture, and may
readily serve as a plug-and-play solution to protect the ownership of a
pre-trained network. Experiments across various datasets and architectures
demonstrate that, with SDB, the performance of an unauthorized KD drops
significantly while that of an authorized gets enhanced, demonstrating the
effectiveness of SDB.
