To Ensemble or Not Ensemble: When does End-To-End Training Fail?
  End-to-End training (E2E) is becoming more and more popular to train complex
Deep Network architectures. An interesting question is whether this trend will
continue-are there any clear failure cases for E2E training? We study this
question in depth, for the specific case of E2E training an ensemble of
networks. Our strategy is to blend the gradient smoothly in between two
extremes: from independent training of the networks, up to to full E2E
training. We find clear failure cases, where over-parameterized models cannot
be trained E2E. A surprising result is that the optimum can sometimes lie in
between the two, neither an ensemble or an E2E system. The work also uncovers
links to Dropout, and raises questions around the nature of ensemble diversity
and multi-branch networks.
