Regularization and Reparameterization Avoid Vanishing Gradients in
  Sigmoid-Type Networks
  Deep learning requires several design choices, such as the nodes' activation
functions and the widths, types, and arrangements of the layers. One
consideration when making these choices is the vanishing-gradient problem,
which is the phenomenon of algorithms getting stuck at suboptimal points due to
small gradients. In this paper, we revisit the vanishing-gradient problem in
the context of sigmoid-type activation. We use mathematical arguments to
highlight two different sources of the phenomenon, namely large individual
parameters and effects across layers, and to illustrate two simple remedies,
namely regularization and rescaling. We then demonstrate the effectiveness of
the two remedies in practice. In view of the vanishing-gradient problem being a
main reason why tanh and other sigmoid-type activation has become much less
popular than relu-type activation, our results bring sigmoid-type activation
back to the table.
