Principled Training of Neural Networks with Direct Feedback Alignment
  The backpropagation algorithm has long been the canonical training method for
neural networks. Modern paradigms are implicitly optimized for it, and numerous
guidelines exist to ensure its proper use. Recently, synthetic gradients
methods -where the error gradient is only roughly approximated - have garnered
interest. These methods not only better portray how biological brains are
learning, but also open new computational possibilities, such as updating
layers asynchronously. Even so, they have failed to scale past simple tasks
like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to
ill-suited models and practices forbidding such methods from performing to the
best of their abilities. In this work, we focus on direct feedback alignment
and present a set of best practices justified by observations of the alignment
angles. We characterize a bottleneck effect that prevents alignment in narrow
layers, and hypothesize it may explain why feedback alignment methods have yet
to scale to large convolutional networks.
