A Simple Recurrent Unit with Reduced Tensor Product Representations
  idely used recurrent units, including Long-short Term Memory (LSTM) and the
Gated Recurrent Unit (GRU), perform well on natural language tasks, but their
ability to learn structured representations is still questionable. Exploiting
reduced Tensor Product Representations (TPRs) --- distributed representations
of symbolic structure in which vector-embedded symbols are bound to
vector-embedded structural positions --- we propose the TPRU, a simple
recurrent unit that, at each time step, explicitly executes structural-role
binding and unbinding operations to incorporate structural information into
learning. A gradient analysis of our proposed TPRU is conducted to support our
model design, and its performance on multiple datasets shows the effectiveness
of our design choices. Furthermore, observations on a linguistically grounded
study demonstrate the interpretability of our TPRU.
