Forward Signal Propagation Learning
  We propose a new learning algorithm for propagating a learning signal and
updating neural network parameters via a forward pass, as an alternative to
backpropagation. In forward signal propagation learning (sigprop), there is
only the forward path for learning and inference, so there are no additional
structural or computational constraints on learning, such as feedback
connectivity, weight transport, or a backward pass, which exist under
backpropagation. Sigprop enables global supervised learning with only a forward
path. This is ideal for parallel training of layers or modules. In biology,
this explains how neurons without feedback connections can still receive a
global learning signal. In hardware, this provides an approach for global
supervised learning without backward connectivity. Sigprop by design has better
compatibility with models of learning in the brain and in hardware than
backpropagation and alternative approaches to relaxing learning constraints. We
also demonstrate that sigprop is more efficient in time and memory than they
are. To further explain the behavior of sigprop, we provide evidence that
sigprop provides useful learning signals in context to backpropagation. To
further support relevance to biological and hardware learning, we use sigprop
to train continuous time neural networks with Hebbian updates and train spiking
neural networks without surrogate functions.
