ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent
  Training
  Negative flips are errors introduced in a classification system when a legacy
model is replaced with a new one. Existing methods to reduce the negative flip
rate (NFR) either do so at the expense of overall accuracy using model
distillation, or use ensembles, which multiply inference cost prohibitively. We
present a method to train a classification system that achieves paragon
performance in both error rate and NFR, at the inference cost of a single
model. Our method introduces a generalized distillation objective, Logit
Difference Inhibition (LDI), that penalizes changes in the logits between the
new and old model, without forcing them to coincide as in ordinary
distillation. LDI affords the model flexibility to reduce error rate along with
NFR. The method uses a homogeneous ensemble as the reference model for LDI,
hence the name Ensemble LDI, or ELODI. The reference model can then be
substituted with a single model at inference time. The method leverages the
observation that negative flips are typically not close to the decision
boundary, but often exhibit large deviations in the distance among their
logits, which are reduced by ELODI.
