Regularization in ResNet with Stochastic Depth
  Regularization plays a major role in modern deep learning. From classic
techniques such as L1,L2 penalties to other noise-based methods such as
Dropout, regularization often yields better generalization properties by
avoiding overfitting. Recently, Stochastic Depth (SD) has emerged as an
alternative regularization technique for residual neural networks (ResNets) and
has proven to boost the performance of ResNet on many tasks [Huang et al.,
2016]. Despite the recent success of SD, little is known about this technique
from a theoretical perspective. This paper provides a hybrid analysis combining
perturbation analysis and signal propagation to shed light on different
regularization effects of SD. Our analysis allows us to derive principled
guidelines for choosing the survival rates used for training with SD.
