title: How to avoid memory leakage in an autoregressive model within tensorflow
body: <p>Recently, I am training a LSTM with attention mechanism for regressionin tensorflow 2.9 and I met an problem during training with model.fit():</p>
  <p>At the beginning, the training time is okay, like 7s/step. However, it was increasing during the process and after several steps, like 1000, the value might be 50s/step. Here below is a part of the code for my model:</p>
  <pre><code>class AttentionModel(tf.keras.Model):
    def __init__(self, encoder_output_dim, dec_units, dense_dim, batch):
      super().__init__()
      
      self.dense_dim = dense_dim
      self.batch = batch
      
      encoder = Encoder(encoder_output_dim)
      decoder = Decoder(dec_units,dense_dim)
  
      self.encoder = encoder
      self.decoder = decoder
      
    def call(self, inputs):                                   
      
      # Creat a tensor to record the result
      tempt = list()
      
      encoder_output, encoder_state = self.encoder(inputs)    
      
      new_features = np.zeros((self.batch, 1, 1))  
      
      dec_initial_state = encoder_state  
      
      for i in range(6):        
          dec_inputs = DecoderInput(new_features=new_features, enc_output=encoder_output)
          dec_result, dec_state = self.decoder(dec_inputs, dec_initial_state)   
          
          tempt.append(dec_result.logits)
          
          new_features = dec_result.logits
          dec_initial_state = dec_state
          
      result=tf.concat(tempt,1)  
      return result
  </code></pre>
  <p>In the official documents for tf.function, I notice: &quot;Don't rely on Python side effects like object mutation or list appends&quot;.</p>
  <p>Since I use a dynamic python list with append() to record the intermediate variables, I guess each time during training, a new tf.graph was added. Is the reason my training is getting slower and slower?</p>
  <p>Additionally, what should I use instead of python list to avoid this? I have tried with a numpy.zeros matrix but it will lead to another problem:</p>
  <pre><code>tempt = np.zeros(shape=(1,6))
  ...
  for i in range(6):        
          dec_inputs = DecoderInput(new_features=new_features, enc_output=encoder_output)
          dec_result, dec_state = self.decoder(dec_inputs, dec_initial_state)   
          
          tempt[i]=(dec_result.logits)
  ...
  
  Cannot convert a symbolic tf.Tensor (decoder/dense_3/BiasAdd:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported.
  </code></pre>