Read, Highlight and Summarize: A Hierarchical Neural Semantic
  Encoder-based Approach
  Traditional sequence-to-sequence (seq2seq) models and other variations of the
attention-mechanism such as hierarchical attention have been applied to the
text summarization problem. Though there is a hierarchy in the way humans use
language by forming paragraphs from sentences and sentences from words,
hierarchical models have usually not worked that much better than their
traditional seq2seq counterparts. This effect is mainly because either the
hierarchical attention mechanisms are too sparse using hard attention or noisy
using soft attention. In this paper, we propose a method based on extracting
the highlights of a document; a key concept that is conveyed in a few
sentences. In a typical text summarization dataset consisting of documents that
are 800 tokens in length (average), capturing long-term dependencies is very
important, e.g., the last sentence can be grouped with the first sentence of a
document to form a summary. LSTMs (Long Short-Term Memory) proved useful for
machine translation. However, they often fail to capture long-term dependencies
while modeling long sequences. To address these issues, we have adapted Neural
Semantic Encoders (NSE) to text summarization, a class of memory-augmented
neural networks by improving its functionalities and proposed a novel
hierarchical NSE that outperforms similar previous models significantly. The
quality of summarization was improved by augmenting linguistic factors, namely
lemma, and Part-of-Speech (PoS) tags, to each word in the dataset for improved
vocabulary coverage and generalization. The hierarchical NSE model on factored
dataset outperformed the state-of-the-art by nearly 4 ROUGE points. We further
designed and used the first GPU-based self-critical Reinforcement Learning
model.
