title: Are k-means vectors in scikit learn normalized internally or TfidfVectorizer normalization not working?
body: <p>Are the vectors in scikit-learn <code>Kmeans</code> internally normalized to unit L2 norm or is something wrong with <code>TfidfVectorizer</code>? I perform clustering on text data, which I vectorize using TF-IDF vectorizer. The code is too long to copy here, but in essence I vectorize and cluster data from the 20 newsgroups dataset. I instantiate the vectorizer as (un-normalized):</p>
  
  <pre><code>VectorizerUn = TfidfVectorizer(min_df=10,
                                 max_df=0.5,
                                 stop_words='english',
                                 decode_error='ignore')
  </code></pre>
  
  <p>or as (L2-normalized):</p>
  
  <pre><code>VectorizerL2 = TfidfVectorizer(min_df=10,
                                 max_df=0.5,
                                 stop_words='english',
                                 decode_error='ignore',
                                 norm=u'l2')
  </code></pre>
  
  <p>and I instantiate <em>k</em> means using:</p>
  
  <pre><code>km = KMeans(n_clusters=num_clusters, init='random', n_init=1, verbose=0)
  </code></pre>
  
  <p>Then I proceed to split the data for cross-validation, vectorize and fit the train data set (below X in vectorizer stands for 'Un' or 'L2')</p>
  
  <pre><code>Vectorized = VectorizerX.fit_transform(TrainData.data)
  km.fit(Vectorized)
  </code></pre>
  
  <p>and and assign the data to clusters for messages in the train set</p>
  
  <pre><code>new_msg_vec = VectorizerX.transform([new_msg])
  predicted_clust = km_clust.predict(new_msg_vec)[0]
  </code></pre>
  
  <p>where <code>new_msg</code> runs through the messages in the train data. Then I assign clusters to groups (each cluster belongs to the group of the majority of its contents) based on the known group labels of the messages in 20 newsgroups, and use the test data to characterize the performance of the clustering/classification scheme. Here is a plot of performance in classification error vs number of clusters for data vectorized without normalization and with L2 normalization:</p>
  
  <p><img src="https://i.stack.imgur.com/gQBPW.png" alt="classification error vs cluster size for k-means on text"></p>
  
  <p>The error bars are the standard deviation in the classification error after 10 independent runs of the <code>km.fit(Vectorized)</code> step. The two results are basically identical. Other metrics of the clustering (ARI score, AMI score, NMI score) give essentially the same picture. </p>
  
  <p>So, does Kmeans internally normalize the vectors to L2 norm of 1, or does the <code>TfidfVectorizer</code> <code>norm</code> parameter not do what it's supposed to? (I am using scikit-learn 0.14.1)  </p>
  
  <p>Edit: I found out that the issue is likely not with Kmeans. If I vectorize with L1-normalized constrain (setting <code>norm=u'l1'</code> in <code>TfidfVectorizer</code>),  the clustering error increases  from 45% to about 80%. I changed the title to reflect this.</p>