Training Deep Networks from Zero to Hero: avoiding pitfalls and going
  beyond
  Training deep neural networks may be challenging in real world data. Using
models as black-boxes, even with transfer learning, can result in poor
generalization or inconclusive results when it comes to small datasets or
specific applications. This tutorial covers the basic steps as well as more
recent options to improve models, in particular, but not restricted to,
supervised learning. It can be particularly useful in datasets that are not as
well-prepared as those in challenges, and also under scarce annotation and/or
small data. We describe basic procedures: as data preparation, optimization and
transfer learning, but also recent architectural choices such as use of
transformer modules, alternative convolutional layers, activation functions,
wide and deep networks, as well as training procedures including as curriculum,
contrastive and self-supervised learning.
