Towards Understanding the Impact of Model Size on Differential Private
  Classification
  Differential privacy (DP) is an essential technique for privacy-preserving.
It was found that a large model trained for privacy preserving performs worse
than a smaller model (e.g. ResNet50 performs worse than ResNet18). To better
understand this phenomenon, we study high dimensional DP learning from the
viewpoint of generalization. Theoretically, we show that for the simple
Gaussian model with even small DP noise, if the dimension is large enough, then
the classification error can be as bad as the random guessing. Then we propose
a feature selection method to reduce the size of the model, based on a new
metric which trades off the classification accuracy and privacy preserving.
Experiments on real data support our theoretical results and demonstrate the
advantage of the proposed method.
