f$_{t}$ = σ ( W$_{f}$ · [ c$_{t}$$_{−}$$_{1}$, h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{f}$ ) i$_{t}$ = σ ( W$_{i}$ · [ c$_{t}$$_{−}$$_{1}$, h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{i}$ ) o$_{t}$ = σ ( W$_{o}$ · [ c$_{t}$, h$_{t}$$_{−}$$_{1}$, x$_{t}$ ] + b$_{o}$ ) Figure 3: Definition of peepholes for LSTM cells state. With σ ( . . . ) in Eq. 1 yielding values between 0 and 1, some information in c$_{t}$$_{−}$$_{1}$ will, to some degree, be ”forgotten”. Eq. 2 and 3 determine how the inputs x$_{t}$ and h$_{t}$$_{−}$$_{1}$ contribute to the updated cell state. Eq. 2 represents the ”input gate” and determines which values of the state to update; some of the i$_{t}$ will be close to zero, others close to one. Eq. 3 computes new candidate values ¯ c , which are multiplied with i$_{t}$ in Eq. 4 when computing the new state c$_{t}$ . Eq. 4 is the actual state update equation. The left term represents the ”forgetting” of some prior state information while the right term represents the addition of new information to the cell state. Eq. 5 represents the ”output gate”. It determines which values of the cell state are provided as output to the following layer and subsequent unrolled cell. It is multiplied in Eq. 6 with the tanh of the cell state to produce the final output h$_{t}$ . To further improve the performance of LSTM cells, Gers and Schmidhuber (2000) introduce the idea of ”peepholes”, which allow the forget and input gates to look (”peep”) at the prior state c$_{t}$$_{−}$$_{1}$ , and which allows the output gate to also look (”peep”) at the current state c$_{t}$ , with the equations in Fig. 3 replacing those in Fig. 2. While there exist other variants to this basic LSTM cell, Greff et al. (2015) conclude after an experimental evaluation that ”the most commonly used LSTM architecture, performs reasonably well on various datasets and using any of eight possible modifications does not significantly improve the LSTM performance.” 9