A Deep Memory-based Architecture for Sequence-to-Sequence Learning
  We propose DEEPMEMORY, a novel deep architecture for sequence-to-sequence
learning, which performs the task through a series of nonlinear transformations
from the representation of the input sequence (e.g., a Chinese sentence) to the
final output sequence (e.g., translation to English). Inspired by the recently
proposed Neural Turing Machine (Graves et al., 2014), we store the intermediate
representations in stacked layers of memories, and use read-write operations on
the memories to realize the nonlinear transformations between the
representations. The types of transformations are designed in advance but the
parameters are learned from data. Through layer-by-layer transformations,
DEEPMEMORY can model complicated relations between sequences necessary for
applications such as machine translation between distant languages. The
architecture can be trained with normal back-propagation on sequenceto-sequence
data, and the learning can be easily scaled up to a large corpus. DEEPMEMORY is
broad enough to subsume the state-of-the-art neural translation model in
(Bahdanau et al., 2015) as its special case, while significantly improving upon
the model with its deeper architecture. Remarkably, DEEPMEMORY, being purely
neural network-based, can achieve performance comparable to the traditional
phrase-based machine translation system Moses with a small vocabulary and a
modest parameter size.
