title: how to create Keras multi LSTM layer using for loop?
body: <p>I'm trying to implement a multi layer <code>LSTM</code> in <strong>Keras</strong> using <strong>for loop</strong> and <a href="https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/19_Hyper-Parameters.ipynb" rel="nofollow noreferrer">this</a> tutorial to be able to optimize the number of layers, which is obviously a hyper-parameter. In the tutorial, the author used <code>skopt</code> for <code>hyper-parameter optimization</code>. I used Functional API to create my model. For simplicity, I changed <code>input_tensor</code>'s shape to arbitrary values.  My model is:</p>
  
  <pre><code>from keras.layers.core import Dense
  from keras.layers import LSTM, Input
  from keras.models import Model
  
  from keras.optimizers import RMSprop
  from keras.initializers import glorot_uniform, glorot_normal, RandomUniform
  
  
  input_tensor = Input(shape=(10, 20))
  
  def create_model(learning_rate, num_lstm_layers, num_lstm_units, activation):
      init = glorot_normal(seed=None)
      init1 = RandomUniform(minval=-0.05, maxval=0.05)
  
      x = Input(shape=(10, 20))
  
  
      for i in range(num_lstm_layers):
  
          name = 'layer_lstm_{0}'.format(i+1)
  
          if( (i==0) and (num_lstm_layers==1) ):
              x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2, 
                      return_sequences=False, kernel_initializer=init, 
                      activation=activation, name=name)(x)
          elif(i != (num_lstm_layers-1) ):
              x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2,  
                      return_sequences=True, kernel_initializer=init, 
                      activation=activation, name=name)(x)
          else:
              x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2,  
                      return_sequences=False, kernel_initializer=init, 
                      activation=activation, name=name)(x)
      x = Dense(1, activation='linear', kernel_initializer= init1)(x)
  
      model = Model(input_tensor, x)
  
      optimizer = RMSprop(lr=learning_rate, rho=0.9, epsilon=None, decay=0.0)
      model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'] )
  
      return model      
  </code></pre>
  
  <p>Whenever I try to fit the model to the data, I encounter this error :</p>
  
  <blockquote>
    <p>ValueError: Initializer for variable layer_lstm_1_14/kernel/ is from
    inside a control-flow construct, such as a loop or conditional. When
    creating a variable inside a loop or conditional, use a lambda as the
    initializer.</p>
  </blockquote>
  
  <p>So far, I've known that somewhere I should add a <code>lambda</code> function or <code>keras Lambda layer</code>. 
  Also, I tested the model in a separate python script, like below:</p>
  
  <pre><code>model = create_model(learning_rate=1e-3, num_lstm_layers=3, num_lstm_units=64, activation='linear')
  </code></pre>
  
  <p>But again it gives me this error:</p>
  
  <blockquote>
    <p>ValueError: Graph disconnected: cannot obtain value for tensor
    Tensor("input_2:0", shape=(?, 10, 20), dtype=float32) at layer
    "input_2". The following previous layers were accessed without issue:
    []</p>
  </blockquote>
  
  <p>I Also tried to create <code>Sequential</code> version of the model. But encountered the same error.</p>
  
  <p><strong>EDIT1:</strong>
  Edited if statement from  <code>if( i==0)</code>: to  <code>if( (i==0) and (num_lstm_layers==1) ):</code> By doing so and making the changes André suggested, you are able to create LSTM models using for loop.</p>