SuperNet -- An efficient method of neural networks ensembling
  The main flaw of neural network ensembling is that it is exceptionally
demanding computationally, especially, if the individual sub-models are large
neural networks, which must be trained separately. Having in mind that modern
DNNs can be very accurate, they are already the huge ensembles of simple
classifiers, and that one can construct more thrifty compressed neural net of a
similar performance for any ensemble, the idea of designing the expensive
SuperNets can be questionable. The widespread belief that ensembling increases
the prediction time, makes it not attractive and can be the reason that the
main stream of ML research is directed towards developing better loss functions
and learning strategies for more advanced and efficient neural networks. On the
other hand, all these factors make the architectures more complex what may lead
to overfitting and high computational complexity, that is, to the same flaws
for which the highly parametrized SuperNets ensembles are blamed. The goal of
the master thesis is to speed up the execution time required for ensemble
generation. Instead of training K inaccurate sub-models, each of them can
represent various phases of training (representing various local minima of the
loss function) of a single DNN [Huang et al., 2017; Gripov et al., 2018]. Thus,
the computational performance of the SuperNet can be comparable to the maximum
CPU time spent on training its single sub-model, plus usually much shorter CPU
time required for training the SuperNet coupling factors.
