TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
  In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based
language model designed to directly capture the global semantic meaning
relating words in a document via latent topics. Because of their sequential
nature, RNNs are good at capturing the local structure of a word sequence -
both semantic and syntactic - but might face difficulty remembering long-range
dependencies. Intuitively, these long-range dependencies are of semantic
nature. In contrast, latent topic models are able to capture the global
underlying semantic structure of a document but do not account for word
ordering. The proposed TopicRNN model integrates the merits of RNNs and latent
topic models: it captures local (syntactic) dependencies using an RNN and
global (semantic) dependencies using latent topics. Unlike previous work on
contextual RNN language modeling, our model is learned end-to-end. Empirical
results on word prediction show that TopicRNN outperforms existing contextual
RNN baselines. In addition, TopicRNN can be used as an unsupervised feature
extractor for documents. We do this for sentiment analysis on the IMDB movie
review dataset and report an error rate of $6.28\%$. This is comparable to the
state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally,
TopicRNN also yields sensible topics, making it a useful alternative to
document models such as latent Dirichlet allocation.
