Mixed-Privacy Forgetting in Deep Networks
  We show that the influence of a subset of the training samples can be removed
-- or "forgotten" -- from the weights of a network trained on large-scale image
classification tasks, and we provide strong computable bounds on the amount of
remaining information after forgetting. Inspired by real-world applications of
forgetting techniques, we introduce a novel notion of forgetting in
mixed-privacy setting, where we know that a "core" subset of the training
samples does not need to be forgotten. While this variation of the problem is
conceptually simple, we show that working in this setting significantly
improves the accuracy and guarantees of forgetting methods applied to vision
classification tasks. Moreover, our method allows efficient removal of all
information contained in non-core data by simply setting to zero a subset of
the weights with minimal loss in performance. We achieve these results by
replacing a standard deep network with a suitable linear approximation. With
opportune changes to the network architecture and training procedure, we show
that such linear approximation achieves comparable performance to the original
network and that the forgetting problem becomes quadratic and can be solved
efficiently even for large models. Unlike previous forgetting methods on deep
networks, ours can achieve close to the state-of-the-art accuracy on large
scale vision tasks. In particular, we show that our method allows forgetting
without having to trade off the model accuracy.
