Representation of linguistic form and function in recurrent neural
  networks
  We present novel methods for analyzing the activation patterns of RNNs from a
linguistic point of view and explore the types of linguistic structure they
learn. As a case study, we use a multi-task gated recurrent network
architecture consisting of two parallel pathways with shared word embeddings
trained on predicting the representations of the visual scene corresponding to
an input sentence, and predicting the next word in the same sentence. Based on
our proposed method to estimate the amount of contribution of individual tokens
in the input to the final prediction of the networks we show that the image
prediction pathway: a) is sensitive to the information structure of the
sentence b) pays selective attention to lexical categories and grammatical
functions that carry semantic information c) learns to treat the same input
token differently depending on its grammatical functions in the sentence. In
contrast the language model is comparatively more sensitive to words with a
syntactic function. Furthermore, we propose methods to ex- plore the function
of individual hidden units in RNNs and show that the two pathways of the
architecture in our case study contain specialized units tuned to patterns
informative for the task, some of which can carry activations to later time
steps to encode long-term dependencies.
