Boost Neural Networks by Checkpoints
  Training multiple deep neural networks (DNNs) and averaging their outputs is
a simple way to improve the predictive performance. Nevertheless, the
multiplied training cost prevents this ensemble method to be practical and
efficient. Several recent works attempt to save and ensemble the checkpoints of
DNNs, which only requires the same computational cost as training a single
network. However, these methods suffer from either marginal accuracy
improvements due to the low diversity of checkpoints or high risk of divergence
due to the cyclical learning rates they adopted. In this paper, we propose a
novel method to ensemble the checkpoints, where a boosting scheme is utilized
to accelerate model convergence and maximize the checkpoint diversity. We
theoretically prove that it converges by reducing exponential loss. The
empirical evaluation also indicates our proposed ensemble outperforms single
model and existing ensembles in terms of accuracy and efficiency. With the same
training budget, our method achieves 4.16% lower error on Cifar-100 and 6.96%
on Tiny-ImageNet with ResNet-110 architecture. Moreover, the adaptive sample
weights in our method make it an effective solution to address the imbalanced
class distribution. In the experiments, it yields up to 5.02% higher accuracy
over single EfficientNet-B0 on the imbalanced datasets.
