title: Python text processing: NLTK and pandas
body: <p>I'm looking for an effective way to construct a Term Document Matrix in Python that can be used together with extra data.</p>
  
  <p>I have some text data with a few other attributes. I would like to run some analyses on the text and I would like to be able to correlate features extracted from text (such as individual word tokens or LDA topics) with the other attributes.</p>
  
  <p>My plan was load the data as a pandas data frame and then each response will represent a document. Unfortunately, I ran into an issue:</p>
  
  <pre><code>import pandas as pd
  import nltk
  
  pd.options.display.max_colwidth = 10000
  
  txt_data = pd.read_csv("data_file.csv",sep="|")
  txt = str(txt_data.comment)
  len(txt)
  Out[7]: 71581 
  
  txt = nltk.word_tokenize(txt)
  txt = nltk.Text(txt)
  txt.count("the")
  Out[10]: 45
  
  txt_lines = []
  f = open("txt_lines_only.txt")
  for line in f:
      txt_lines.append(line)
  
  txt = str(txt_lines)
  len(txt)
  Out[14]: 1668813
  
  txt = nltk.word_tokenize(txt)
  txt = nltk.Text(txt)
  txt.count("the")
  Out[17]: 10086
  </code></pre>
  
  <p>Note that in both cases, text was processed in such a way that only the anything but spaces, letters and ,.?! was removed (for simplicity).</p>
  
  <p>As you can see a pandas field converted into a string returns fewer matches and the length of the string is also shorter.</p>
  
  <p>Is there any way to improve the above code?</p>
  
  <p>Also, <code>str(x)</code> creates 1 big string out of the comments while <code>[str(x) for x in txt_data.comment]</code> creates a list object which cannot be broken into a bag of words. What is the best way to produce a <code>nltk.Text</code> object that will retain document indices? In other words I'm looking for a way to create a Term Document Matrix, R's equivalent of <code>TermDocumentMatrix()</code> from <code>tm</code> package.</p>
  
  <p>Many thanks.</p>