title: Tfidfvectorizer - How can I check out processed tokens?
body: <p>How can I check the strings tokenized inside <code>TfidfVertorizer()</code>?  If I don't pass anything in the arguments, <code>TfidfVertorizer()</code> will tokenize the string with some pre-defined methods. I want to observe how it tokenizes strings so that I can more easily tune my model.</p>
  
  <pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
  corpus = ['This is the first document.',
            'This document is the second document.',
            'And this is the third one.',
            'Is this the first document?']
  vectorizer = TfidfVectorizer()
  X = vectorizer.fit_transform(corpus)
  </code></pre>
  
  <p>I want something like this:</p>
  
  <pre><code>&gt;&gt;&gt;vectorizer.get_processed_tokens()
  [['this', 'is', 'first', 'document'],
   ['this', 'document', 'is', 'second', 'document'],
   ['this', 'is', 'the', 'third', 'one'],
   ['is', 'this', 'the', 'first', 'document']]
  </code></pre>
  
  <p>How can I do this? </p>