title: Which way is right in tf-idf? Fit all then transform train set and test set or fit train set then transform test set
body: <p>1.Fit train set then transform test set
  <a href="https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#sphx-glr-auto-examples-text-plot-document-classification-20newsgroups-py" rel="nofollow noreferrer">scikit-learn provide this example</a></p>
  <pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
  vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
  X_train = vectorizer.fit_transform(data_train.data)
  X_test = vectorizer.transform(data_test.data)
  </code></pre>
  <p>2.Fit all then transform train set and test set which I've seen in many cases</p>
  <pre><code>import numpy as np
  from sklearn.feature_extraction.text import TfidfVectorizer
  vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
  X_all = np.append(train_x, test_x, axis=0)
  vectorizer.fit(X_all)
  X_train = vectorizer.transform(train_x)
  X_test = vectorizer.transform(test_x)
  </code></pre>
  <p>So, I'm confused which way is right and why</p>