Accuracy-Privacy Trade-off in Deep Ensemble: A Membership Inference
  Perspective
  Deep ensemble learning has been shown to improve accuracy by training
multiple neural networks and fusing their outputs. Ensemble learning has also
been used to defend against membership inference attacks that undermine
privacy. In this paper, we empirically demonstrate a trade-off between these
two goals, namely accuracy and privacy (in terms of membership inference
attacks), in deep ensembles. Using a wide range of datasets and model
architectures, we show that the effectiveness of membership inference attacks
also increases when ensembling improves accuracy. To better understand this
trade-off, we study the impact of various factors such as prediction confidence
and agreement between models that constitute the ensemble. Finally, we evaluate
defenses against membership inference attacks based on regularization and
differential privacy. We show that while these defenses can mitigate the
effectiveness of the membership inference attack, they simultaneously degrade
ensemble accuracy. We illustrate similar trade-off in more advanced and
state-of-the-art ensembling techniques, such as snapshot ensembles and
diversified ensemble networks. The source code is available in supplementary
materials.
