Neural Function Modules with Sparse Arguments: A Dynamic Approach to
  Integrating Information across Layers
  Feed-forward neural networks consist of a sequence of layers, in which each
layer performs some processing on the information from the previous layer. A
downside to this approach is that each layer (or module, as multiple modules
can operate in parallel) is tasked with processing the entire hidden state,
rather than a particular part of the state which is most relevant for that
module. Methods which only operate on a small number of input variables are an
essential part of most programming languages, and they allow for improved
modularity and code re-usability. Our proposed method, Neural Function Modules
(NFM), aims to introduce the same structural capability into deep learning.
Most of the work in the context of feed-forward networks combining top-down and
bottom-up feedback is limited to classification problems. The key contribution
of our work is to combine attention, sparsity, top-down and bottom-up feedback,
in a flexible algorithm which, as we show, improves the results in standard
classification, out-of-domain generalization, generative modeling, and learning
representations in the context of reinforcement learning.
