title: How to force RNN output to become smoother?
body: <p>I am using experimental data with Keras LSTM to model a complicated physical system. The problem is output value tends to change drastically between two points at certain points. All physical systems must show some continuous/smooth behavior. How can I make my output smoother, is there some kind of layer or regularization?</p>
  
  <p>I tried introducing l1-l2 regularization, drop-outs... They help but I could not get good results. What I seek is some kind of layer which limits sudden changes in the values. By the way I work with a rather small amount of data; I am using 2 series to train and validify, 1 to test.</p>
  
  <p>Network structure: I get similar results for 2 LSTM + 1 Dense layer or 1 LSTM + 1 Dense layer. (With/without dropout layers between LSTM and Dense, and some l2 regularization)</p>
  
  <p>The time-series data represents some measurements. Measurements are taken in short intervals, resulting in repeated values time to time. I remove some of the repeated lines as well.(I concatenated them together and then removed the rows with respect to one of the inputs. I tried doing it for several inputs. But as you can understand, I did not remove all the repeated lines with this approach, can this be the source of the problem?)</p>
  
  <p>I use sklearn.StandardScaler or sklearn.MinMaxScaler to normalize the input data, not much difference between the two.</p>
  
  <p>You can see a sample result on test data, which has l2 regularization - <strong>please note the first two peaks in the start.</strong> There is around 20,000 points in the graph and these peaks occur over 3-5 points. In the training set there are some jumps as well but they are far more smooth and spread out. Is there someway to smoothen the output within the neural net, without adding some external filters?</p>
  
  <p><a href="https://i.stack.imgur.com/vXJla.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vXJla.png" alt="enter image description here"></a></p>