Figure 1: RNN architecture with single hidden layer of LSTM cells, unrolled five steps Figure 2: Definition of LSTM cell Inputs: Input The Input quick Input brown Input fox Input jumps Initial State 1 LSTM LSTM state LSTM LSTM state LSTM LSTM state LSTM LSTM state LSTM Final State 1 LSTM Initial State 2 state Output output state Output output state Output output state Output output Final State 2 Output output input input input input input Targets: quick brown fox jumps over cell on the same level and also provides h$_{t}$ as output to the next layer. The variables c, x, h, f, i, o in Fig. 2 are vectors in$_{R}$ $^{m}$; the W and b (weights and biases) are ”trainable” parameters of suitable dimensions. The functions tanh( . . . ) and σ ( . . . ) are applied elementwise, [ . . . ] represents vector concatenation. Equa- tions 1–6 describe m individual LSTM cells, each operating on a single input from $_{R}$. The number of individual cells on each layer, i.e. the dimensionality m , can be freely chosen. The intuition behind these definitions is as follows. Eq. 1 represents the ”forget gate” that determines, based on the inputs x$_{t}$ and h$_{t}$$_{−}$$_{1}$ , which part of the state to forget. Note that the prior state c$_{t}$$_{−}$$_{1}$ is multiplied by f$_{t}$ in Eq. 4 to derive the new 8 f$_{t}$ = σ ( W$_{f}$ · [ h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{f}$ ) (1) i$_{t}$ = σ ( W$_{i}$ · [ h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{i}$ ) (2) ¯ c$_{t}$ = tanh ( W$_{c}$ · [ h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{c}$ ) (3) c$_{t}$ = f$_{t}$ × c$_{t}$$_{−}$$_{1}$ + i$_{t}$ × ¯ c$_{t}$ (4) o$_{t}$ = σ ( W$_{o}$ · [ h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{o}$ ) (5) h$_{t}$ = o$_{t}$ × tanh( c$_{t}$ ) (6)