The Neural State Pushdown Automata
  In order to learn complex grammars, recurrent neural networks (RNNs) require
sufficient computational resources to ensure correct grammar recognition. A
widely-used approach to expand model capacity would be to couple an RNN to an
external memory stack. Here, we introduce a "neural state" pushdown automaton
(NSPDA), which consists of a digital stack, instead of an analog one, that is
coupled to a neural network state machine. We empirically show its
effectiveness in recognizing various context-free grammars (CFGs). First, we
develop the underlying mechanics of the proposed higher order recurrent network
and its manipulation of a stack as well as how to stably program its underlying
pushdown automaton (PDA) to achieve desired finite-state network dynamics.
Next, we introduce a noise regularization scheme for higher-order (tensor)
networks, to our knowledge the first of its kind, and design an algorithm for
improved incremental learning. Finally, we design a method for inserting
grammar rules into a NSPDA and empirically show that this prior knowledge
improves its training convergence time by an order of magnitude and, in some
cases, leads to better generalization. The NSPDA is also compared to a
classical analog stack neural network pushdown automaton (NNPDA) as well as a
wide array of first and second-order RNNs with and without external memory,
trained using different learning algorithms. Our results show that, for Dyck(2)
languages, prior rule-based knowledge is critical for optimization convergence
and for ensuring generalization to longer sequences at test time. We observe
that many RNNs with and without memory, but no prior knowledge, fail to
converge and generalize poorly on CFGs.
