title: Random Forests classifier Over/Underfitting
body: <p>In a given dataset I trained a Random Forest classifier using sklearn package in Python. I used the <code>class_weight=balanced</code> parameter in order to balance the imbalanced classes, i.e their ratio is approximately 70-30.</p>
  
  <p>For a training dataset with 10 features per entry and with 3,2 million entries I got this classification report:</p>
  
  <pre><code>              precision    recall  f1-score   support
  
    0               0.92      0.94      0.93    510310
    1               0.74      0.64      0.69    124374
  
  avg / total       0.88      0.88      0.88    634684
  </code></pre>
  
  <p>In an effort to enhance my classifier (and achieve better accuracy for class 1 which has fewer samples) I balanced the training dataset myself, using pandas random sampling function <code>pd.sample(frac=0.3)</code> which for all the I know shuffles the data before the sampling, I got the classification report below:</p>
  
  <pre><code>              precision    recall  f1-score   support
  
    0               0.89      0.85      0.87      153117
    1               0.86      0.90      0.88      161326
  avg / total       0.87      0.87      0.87      314443
  </code></pre>
  
  <p>My training dataset at this point has 10 features per entry with approximately 1,6 million entries.</p>
  
  <p>An issue rises when I tried it in a test dataset (test dataset consisting of 4 million entries). Then my model's performance dropped significantly as you can see:</p>
  
  <pre><code>              precision    recall  f1-score   support
  
    0               0.56      0.24      0.33   1221872
    1               0.06      0.20      0.09    277578
  
  avg / total       0.47      0.23      0.29   1499450
  </code></pre>
  
  <p>At this point I must add that I know that my dataset's profile is consistent. I mentioned that as inconsistencies between train and test data could have such a negative effect in my model's accuracy.</p>
  
  <p>Moreover I tried fitting a Voting Classifier that consisted of linear, gaussian and tree estimators and again k-fold CV returned amazing results (~90% precision and recall) but when I tested it in an unseen dataset or even a shuffle instance of my dataset I got extremely poor results.</p>
  
  <p>I have some experience with ML classifiers, but I have never encountered anything like this.</p>
  
  <p>The only parameter I have fiddled with is the number of trees in the forest which is why I cannot believe that I overfit my classifier as I get similar results when I use e.g. 60 and 200 trees.</p>
  
  <p>Are RF affected from imbalanced datasets? </p>
  
  <p>Shouldn't I get at least decent results when I balance my classes?</p>
  
  <p>Are RF biased from the training dataset, i.e. given a 50% - 50% distribution in the training dataset does it expect a similar distribution to the test dataset and it cannot perform as expected given a 70-30 test set?</p>
  
  <p>Sorry for the long post. Any help would be appreciated.</p>
  
  <p><strong>Code</strong></p>
  
  <pre><code>X = load_data()
  labels = X['label'].values
  X.drop(['label'], axis=1, inplace=True)
  
  final_X = X.values
  #scaling makes not difference for RF but left it here since I tried
  # a Voting Classifier as well afterwards
  scaler = StandardScaler()
  X_scl = scaler.fit_transform(final_X)
  X_train, X_test, y_train, y_test = train_test_split(X_scl, labels,     
  test_size=0.2, random_state=46)
  
  clf = RandomForestClassifier(n_estimators=100, n_jobs=-1)
  
  clf.fit(X_train, y_train)
  print ("Classifier:", clf)
  print("Detailed classification report:")
  print()
  print("The model is trained on the full development set.")
  print("The scores are computed on the full evaluation set.")
  print()
  y_true, y_pred = y_test, clf.predict(X_test)
  print(classification_report(y_true, y_pred))
  </code></pre>