Evaluating Inexact Unlearning Requires Revisiting Forgetting
  Existing methods in inexact unlearning are evaluated by measuring
indistinguishability from models retrained after removing the deletion set. We
argue that achieving indistinguishability is unnecessary and its practical
relaxations are insufficient. We formulate the goal of unlearning as forgetting
all information specific to the deletion set while maintaining high utility and
resource efficiency.
  We introduce a novel test for forgetting called Interclass Confusion (IC).
Despite being a black-box test, IC can investigate whether information from the
deletion set was erased until the early layers of the network. We analyze two
aspects of forgetting: (i) memorization and (ii) property generalization. We
empirically show that two simple unlearning methods, exact-unlearning and
catastrophic-forgetting the final k layers of a network, outperforms prior
unlearning methods when scaled to large deletion sets. Overall, we believe our
formulation and the IC test will guide the design of better unlearning
algorithms.
