title: Neural-net Regression predicts same value for all test samples
body: <p>My <strong>neural-network regression model predicts one value for all the test samples</strong>. Playing with hyperparameters like epochs, batch_size, number of layers, hidden units, learning rate, etc. only changes the prediction values to a new constant.</p>
  
  <p>For testing, if I test on the training data itself, I get almost accurate results and a RMSE ~ 1.</p>
  
  <p>Note: The task is to predict remaining life of a machine from its run-till-failure time series data. I have used tsfresh library to generate 1045 features from the original time series data with only 24 features.</p>
  
  <p><strong><em>What should be causing this behavior? How should I visualize a neural network model development to make sure things are going in the right direction?</em></strong></p>
  
  <pre><code>print "Shape of training_features is", train_X.shape
  print "Shape of train_labels is", train_Y.shape
  print "Shape of test_features is", test_X.shape
  print "shape of test_labels is", test_Y.shape
  
  input_dim = train_X.shape[1]
  # Function to create model, required for KerasRegressor
  def create_model(h1=50, h2=50, act1='sigmoid', act2='sigmoid', init='he_normal', learn_rate=0.001, momentum=0.1, loss='mean_squared_error'):
      # create model
      model = Sequential()
      model.add(Dense(h1, input_dim=input_dim, init=init, activation=act1))
      model.add(Dense(h2, init=init, activation=act2))
      model.add(Dense(1, init=init))
      # Compile model
      optimizer = SGD(lr=learn_rate, momentum=momentum)
      model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])
      return model
  
  ''' THE REAL THING '''
  # create model
  model = KerasRegressor(build_fn=create_model, verbose=0)
  
  # SCORING FUNCTION
  grid_scorer = make_scorer(mean_squared_error, greater_is_better=False)
  # Grid Search
  batch_size = [8]
  epochs = [500]
  init_mode = ['glorot_uniform']
  learn_rate = [0.0001]
  momentum = [0.1]
  
  hidden_layer_1 = [75]
  activation_1 = ['sigmoid']
  hidden_layer_2 = [15]
  activation_2 = ['sigmoid']
  
  param_grid = dict(batch_size=batch_size, nb_epoch=epochs, init=init_mode, h1=hidden_layer_1, h2=hidden_layer_2, act1 = activation_1, act2=activation_2, learn_rate=learn_rate, momentum=momentum)
  
  print "\n...BEGIN SEARCH..."
  grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=5, scoring=grid_scorer, verbose=1)
  
  print "\nLet's fit the training data..."
  grid_result = grid.fit(train_X, train_Y)
  
  # summarize results
  print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
  means = grid_result.cv_results_['mean_test_score']
  stds = grid_result.cv_results_['std_test_score']
  params = grid_result.cv_results_['params']
  for mean, stdev, param in zip(means, stds, params):
      print("%f (%f) with: %r" % (mean, stdev, param))
  
  predicted = grid.predict(test_X)  
  print "\nPrediction array is\n", predicted
  rmse = numpy.sqrt(((predicted - test_Y) ** 2).mean(axis=0))
  print "Test RMSE is", rmse
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>Shape of training_features is (249, 1045)
  Shape of train_labels is (249,)
  Shape of test_features is (248, 1045)
  shape of test_labels is (248,)
  
  ...BEGIN SEARCH...
  
  Let's fit the training data...
  Fitting 5 folds for each of 1 candidates, totalling 5 fits
  Best: -891.761863 using {'learn_rate': 0.0001, 'h2': 15, 'act1': 'sigmoid', 'act2': 'sigmoid', 'h1': 75, 'batch_size': 8, 'init': 'glorot_uniform', 'nb_epoch': 500, 'momentum': 0.1}
  -891.761863 (347.253351) with: {'learn_rate': 0.0001, 'h2': 15, 'act1': 'sigmoid', 'act2': 'sigmoid', 'h1': 75, 'batch_size': 8, 'init': 'glorot_uniform', 'nb_epoch': 500, 'momentum': 0.1}
  
  Prediction array is
  [ 295.72067261  295.72067261  295.72067261  295.72067261  295.72067261
    295.72067261  295.72067261  ...
                                295.72067261  295.72067261  295.72067261
    295.72067261  295.72067261  295.72067261]
  Test RMSE is 95.0019297411
  </code></pre>