A Note on "Assessing Generalization of SGD via Disagreement"
  Jiang et al. (2021) give empirical evidence that the average test error of
deep neural networks can be estimated via the prediction disagreement of two
separately trained networks. They also provide a theoretical explanation that
this 'Generalization Disagreement Equality' follows from the well-calibrated
nature of deep ensembles under the notion of a proposed 'class-aggregated
calibration'. In this paper we show that the approach suggested might be
impractical because a deep ensemble's calibration deteriorates under
distribution shift, which is exactly when the coupling of test error and
disagreement would be of practical value. We present both theoretical and
experimental evidence, re-deriving the theoretical statements using a simple
Bayesian perspective and show them to be straightforward and more generic: they
apply to any discriminative model -- not only ensembles whose members output
one-hot class predictions. The proposed calibration metrics are also equivalent
to two metrics introduced by Nixon et al. (2019): 'ACE' and 'SCE'.
