title: non-linear neural network regression - quadratic function is not being estimated correctly
body: <p>I have mostly used ANNs for classification and only recently started to try them out for modeling continuous variables. As an exercise I generated a simple set of (x, y) pairs where y = x^2 and tried to train an ANN to learn this quadratic function. </p>
  
  <p><strong>The ANN model:</strong></p>
  
  <p>This ANN has 1 input node (ie. x), 2 hidden layers each with 2 nodes in each layer, and 1 output node. All four hidden nodes use the non-linear tanh activation function and the output node has no activation function (since it is regression). </p>
  
  <p><strong>The Data:</strong></p>
  
  <p>For the training set I randomly generated 100 numbers between (-20, 20) for x and computed y=x^2. For the testing set I randomly generated 100 numbers between (-30, 30) for x and also computed y=x^2. I then transformed all x so that they are centered around 0 and their min and max are approximately around -1.5 and 1.5. I also transformed all y similarly but made their min and max about -0.9 and 0.9. This way, all the data falls within that mid range of the tanh activation function and not way out at the extremes.</p>
  
  <p><strong>The Problem:</strong></p>
  
  <p>After training the ANN in Keras, I am seeing that only the right half of the polynomial function is being learned, and the left half is completely flat. Does anyone have any ideas why this may be happening? I tried playing around with different scaling options, as well as hidden layer specifications but no luck on that left side.</p>
  
  <p>Thanks!</p>
  
  <p>Attached is the code I used for everything and the image shows the plot of the scaled training x vs the predicted y. As you can see, only half of the parabola is recovered.</p>
  
  <pre><code>import numpy as np, pandas as pd
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.wrappers.scikit_learn import KerasRegressor
  from sklearn.preprocessing import StandardScaler
  from sklearn.pipeline import Pipeline
  import matplotlib.pyplot as plt
  
  seed = 10
  n = 100
  X_train = np.random.uniform(-20, 20, n)
  Y_train = X_train ** 2
  X_test = np.random.uniform(-30, 30, n)
  Y_test = X_test ** 2
  
  #### Scale the data
  
  x_cap = max(abs(np.array(list(X_train) + list(X_test))))
  y_cap = max(abs(np.array(list(Y_train) + list(Y_test))))
  x_mean = np.mean(np.array(list(X_train) + list(X_test)))
  y_mean = np.mean(np.array(list(Y_train) + list(Y_test)))
  
  X_train2 = (X_train-x_mean) / x_cap
  X_test2 = (X_test-x_mean) / x_cap
  Y_train2 = (Y_train-y_mean) / y_cap
  Y_test2 = (Y_test-y_mean) / y_cap
  
  X_train2 = X_train2 * (1.5 / max(X_train2))
  Y_train2 = Y_train2 * (0.9 / max(Y_train2))
  
  # define base model
  def baseline_model1():
  # create model
  model1 = Sequential()
  model1.add(Dense(2, input_dim=1, kernel_initializer='normal', activation='tanh'))
  model1.add(Dense(2, input_dim=1, kernel_initializer='normal', activation='tanh'))
  model1.add(Dense(1, kernel_initializer='normal'))
  # Compile model
  model1.compile(loss='mean_squared_error', optimizer='adam')
  return model1
  
  np.random.seed(seed)
  estimator1 = KerasRegressor(build_fn=baseline_model1, epochs=100, batch_size=5, verbose=0)
  
  estimator1.fit(X_train2, Y_train2)
  prediction = estimator1.predict(X_train2)
  plt.scatter(X_train2, prediction)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/UabMe.png" rel="nofollow noreferrer">enter image description here</a></p>