Trainable back-propagated functional transfer matrices
  Connections between nodes of fully connected neural networks are usually
represented by weight matrices. In this article, functional transfer matrices
are introduced as alternatives to the weight matrices: Instead of using real
weights, a functional transfer matrix uses real functions with trainable
parameters to represent connections between nodes. Multiple functional transfer
matrices are then stacked together with bias vectors and activations to form
deep functional transfer neural networks. These neural networks can be trained
within the framework of back-propagation, based on a revision of the delta
rules and the error transmission rule for functional connections. In
experiments, it is demonstrated that the revised rules can be used to train a
range of functional connections: 20 different functions are applied to neural
networks with up to 10 hidden layers, and most of them gain high test
accuracies on the MNIST database. It is also demonstrated that a functional
transfer matrix with a memory function can roughly memorise a non-cyclical
sequence of 400 digits.
