Finding online neural update rules by learning to remember
  We investigate learning of the online local update rules for neural
activations (bodies) and weights (synapses) from scratch. We represent the
states of each weight and activation by small vectors, and parameterize their
updates using (meta-) neural networks. Different neuron types are represented
by different embedding vectors which allows the same two functions to be used
for all neurons. Instead of training directly for the objective using evolution
or long term back-propagation, as is commonly done in similar systems, we
motivate and study a different objective: That of remembering past snippets of
experience. We explain how this objective relates to standard back-propagation
training and other forms of learning. We train for this objective using short
term back-propagation and analyze the performance as a function of both the
different network types and the difficulty of the problem. We find that this
analysis gives interesting insights onto what constitutes a learning rule. We
also discuss how such system could form a natural substrate for addressing
topics such as episodic memories, meta-learning and auxiliary objectives.
