Bidirectionally Self-Normalizing Neural Networks
  The problem of vanishing and exploding gradients has been a long-standing
obstacle that hinders the effective training of neural networks. Despite
various tricks and techniques that have been employed to alleviate the problem
in practice, there still lacks satisfactory theories or provable solutions. In
this paper, we address the problem from the perspective of high-dimensional
probability theory. We provide a rigorous result that shows, under mild
conditions, how the vanishing/exploding gradients problem disappears with high
probability if the neural networks have sufficient width. Our main idea is to
constrain both forward and backward signal propagation in a nonlinear neural
network through a new class of activation functions, namely Gaussian-Poincar\'e
normalized functions, and orthogonal weight matrices. Experiments on both
synthetic and real-world data validate our theory and confirm its effectiveness
on very deep neural networks when applied in practice.
