Convergence Analysis and Implicit Regularization of Feedback Alignment
  for Deep Linear Networks
  We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient
alternative to backpropagation for training neural networks. We provide
convergence guarantees with rates for deep linear networks for both continuous
and discrete dynamics. Additionally, we study incremental learning phenomena
for shallow linear networks. Interestingly, certain specific initializations
imply that negligible components are learned before the principal ones, thus
potentially negatively affecting the effectiveness of such a learning
algorithm; a phenomenon we classify as implicit anti-regularization. We also
provide initialization schemes where the components of the problem are
approximately learned by decreasing order of importance, thus providing a form
of implicit regularization.
