title: Scikit learn - fit_transform on the test set
body: <p>I am struggling to use Random Forest in Python with Scikit learn. My problem is that I use it for text classification (in 3 classes - positive/negative/neutral) and the features that I extract are mainly words/unigrams, so I need to convert these to numerical features. I found a way to do it with <code>DictVectorizer</code>'s <code>fit_transform</code>:</p>
  
  <pre><code>from sklearn.preprocessing import LabelEncoder
  from sklearn.metrics import classification_report
  from sklearn.feature_extraction import DictVectorizer
  
  vec = DictVectorizer(sparse=False)
  rf = RandomForestClassifier(n_estimators = 100)
  trainFeatures1 = vec.fit_transform(trainFeatures)
  
  # Fit the training data to the training output and create the decision trees
  rf = rf.fit(trainFeatures1.toarray(), LabelEncoder().fit_transform(trainLabels))
  
  testFeatures1 = vec.fit_transform(testFeatures)
  # Take the same decision trees and run on the test data
  Output = rf.score(testFeatures1.toarray(), LabelEncoder().fit_transform(testLabels))
  
  print "accuracy: " + str(Output)
  </code></pre>
  
  <p>My problem is that the <code>fit_transform</code> method is working on the train dataset, which contains around 8000 instances, but when I try to convert my test set to numerical features too, which is around 80000 instances, I get a memory error saying that:</p>
  
  <pre><code>testFeatures1 = vec.fit_transform(testFeatures)
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 143, in fit_transform
  return self.transform(X)
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 251, in transform
  Xa = np.zeros((len(X), len(vocab)), dtype=dtype)
  MemoryError
  </code></pre>
  
  <p>What could possibly cause this and is there any workaround? Many thanks!</p>