PredProp: Bidirectional Stochastic Optimization with Precision Weighted
  Predictive Coding
  We present PredProp, a method for bidirectional, parallel and local
optimisation of weights, activities and precision in neural networks. PredProp
jointly addresses inference and learning, scales learning rates dynamically and
weights gradients by the curvature of the loss function by optimizing
prediction error precision. PredProp optimizes network parameters with
Stochastic Gradient Descent and error forward propagation based strictly on
prediction errors and variables locally available to each layer. Neighboring
layers optimise shared activity variables so that prediction errors can
propagate forward in the network, while predictions propagate backwards. This
process minimises the negative Free Energy, or evidence lower bound of the
entire network. We show that networks trained with PredProp resemble gradient
based predictive coding when the number of weights between neighboring activity
variables is one. In contrast to related work, PredProp generalizes towards
backward connections of arbitrary depth and optimizes precision for any deep
network architecture. Due to the analogy between prediction error precision and
the Fisher information for each layer, PredProp implements a form of Natural
Gradient Descent. When optimizing DNN models, layer-wise PredProp renders the
model a bidirectional predictive coding network. Alternatively DNNs can
parameterize the weights between two activity variables. We evaluate PredProp
for dense DNNs on simple inference, learning and combined tasks. We show that,
without an explicit sampling step in the network, PredProp implements a form of
variational inference that allows to learn disentangled embeddings from low
amounts of data and leave evaluation on more complex tasks and datasets to
future work.
