title: Stacked LSTM with Multiple Dense Layers After
body: <p><a href="https://i.stack.imgur.com/odNVZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/odNVZ.png" alt="enter image description here"></a></p>
  
  <p>Andrew Ng talks about Deep RNN architecture by stacking recurrent layers on top of each other. However, he notes that these are usually limited to 2 or 3 recurrent layers due to already complex time-dependent calculations in the structure. But he does add that people commonly add "a bunch of deep layers that are not connected horizontally" after these recurrent layers (Shown as blue boxes that extend from a[3]&lt;1>). I am wondering if he is simply talking about stacking Dense layers on top of the recurrent layers, or is it something more complicated? Something like this in Keras:</p>
  
  <pre><code>model = Sequential()
  model.add(keras.layers.LSTM(100, return_sequences=True, batch_input_shape=(32, 1, input_shape), stateful=True))
  model.add(keras.layers.LSTM(100, return_sequences=False, stateful=True))
  model.add(Dense(100, activation='relu'))
  model.add(Dense(100, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  </code></pre>