PFGE: Parsimonious Fast Geometric Ensembling of DNNs
  Ensemble methods have been widely used to improve the performance of machine
learning methods in terms of generalization, while they are hard to use in deep
learning systems, as training an ensemble of deep neural networks (DNNs) incurs
an extremely higher computational overhead of model training. Recently,
advanced techniques such as fast geometric ensembling (FGE) and snapshot
ensemble have been proposed. These methods can train the model ensembles in the
same time as a single model, thus getting around the hurdle of training time.
However, their memory overhead for test-time inference remains much higher than
single model based methods. Here we propose a parsimonious FGE (PFGE) that
employs a lightweight ensemble of higher-performing DNNs, generated by
successively-performed stochastic weight averaging procedures. Experimental
results across different advanced DNN architectures on benchmark datasets
CIFAR-$\{10,100\}$ and Imagenet, demonstrate that PFGE matches the
state-of-the-art FGE method in terms of the generalization error, yet requires
only 20% memory overhead for test-time inference. Our code is available at
https://github.com/ZJLAB-AMMI/PFGE.
