Ensemble-Compression: A New Method for Parallel Training of Deep Neural
  Networks
  Parallelization framework has become a necessity to speed up the training of
deep neural networks (DNN) recently. Such framework typically employs the Model
Average approach, denoted as MA-DNN, in which parallel workers conduct
respective training based on their own local data while the parameters of local
models are periodically communicated and averaged to obtain a global model
which serves as the new start of local models. However, since DNN is a highly
non-convex model, averaging parameters cannot ensure that such global model can
perform better than those local models. To tackle this problem, we introduce a
new parallel training framework called Ensemble-Compression, denoted as EC-DNN.
In this framework, we propose to aggregate the local models by ensemble, i.e.,
averaging the outputs of local models instead of the parameters. As most of
prevalent loss functions are convex to the output of DNN, the performance of
ensemble-based global model is guaranteed to be at least as good as the average
performance of local models. However, a big challenge lies in the explosion of
model size since each round of ensemble can give rise to multiple times size
increment. Thus, we carry out model compression after each ensemble,
specialized by a distillation based method in this paper, to reduce the size of
the global model to be the same as the local ones. Our experimental results
demonstrate the prominent advantage of EC-DNN over MA-DNN in terms of both
accuracy and speedup.
