title: How do I pull key words (not most frequent words) out of a corpus using Python and NLTK?
body: <p>I am trying to pull key words out of a text or corpus.  These are not the most frequent words, but the words which are most "about" the text. I have a comparison example and the list I'm generating is very different from the example list.  Could you please give me a pointer to generating a good list of key words which doesn't include low-meaning words like "thou" and "tis"?</p>
  
  <p>I'm using "Romeo and Juliet" as my text. My approach (see Scott &amp; Tribble below) is to compare R&amp;J with Shakespeare's complete plays, and to pull out words with occur significantly more often in R&amp;J compared to the complete plays.  That should weed out frequent words like "the", but in my code it doesn't.</p>
  
  <p>I'm getting lots of words like "thou", "she", and "tis" which don't show up on their list, and I'm NOT getting words like "banished" and "churchyard".  I am getting "romeo", "juliet", "capulet", and "nurse" so I'm at least close to on the right track if not actually on it.</p>
  
  <p>Here's the function which pulls words and percentages out of the texts:</p>
  
  <pre><code>def keywords(corpus, threshold=0):
      """ generates a list of possible keywords and the percentage of 
             occurrences.
            corpus (list): text or collection of texts
            threshold (int): min # of occurrences of word in corpus                    
                target text has threshold 3, ref corp has 0
            return percentKW: list of tuples (word, percent)                         
      """
  
      # get freqDist of corpus as dict. key is word, value = # occurences
      fdist = FreqDist(corpus)
      n = len(corpus)
  
      # create list of tuple of w meeting threshold &amp; sort w/most common first
      t = [(k, v) for k, v in fdist.items() if v &gt;= threshold]
      t = sorted(t, key=lambda tup: tup[1], reverse=True)
  
      # calculate number of total tokens
      n = len(corpus)
  
      # return list of tuples (word, percent word is of total tokens)
      percentKW =[(k, '%.2f'%(100*(v/n))) for k, v in t]
      return percentKW
  </code></pre>
  
  <p>Here's the crucial part of the calling code. targetKW is R&amp;J, and refcorpKWDict is the complete Shakespeare plays.</p>
  
  <pre><code># iterate through text list of tuples
  for w, p in targetKW:
      # for each word, store the percent in KWList
      targetPerc = float(p)
      refcorpPerc = float(refcorpKWDict.get(w, 0))
      # if % in text &gt; % in reference corpus
      if (refcorpPerc or refcorpPerc == 0) and (targetPerc &gt; refcorpPerc):
          diff = float('%.2f'%(targetPerc - refcorpPerc))
          # save result to KWList
          KWList.append((w, targetPerc, refcorpPerc, diff))        
  </code></pre>
  
  <p>Here's what I've tried so far: </p>
  
  <ul>
  <li>Normalized all potential key words to lower case (helped)</li>
  <li>Created custom short lists of key words (text and comparison text).  Seems to work but doesn't tell me anything</li>
  <li>Compared R&amp;J to abridged list of plays, to plays + sonnets, and to Brown corpus (didn't help)</li>
  <li>Checked percentages of potential key words such as "banished".  Percents were a lot lower than expected.  I'm not sure how to interpret that.</li>
  <li>Set minimum length of potential key words to eliminate words such as "ll" and "is" (helped)</li>
  <li>Googled the question. (couldn't find anything)</li>
  </ul>
  
  <p>I'm working with Python 3.5.2 on Windows 10 using IDLE version 3.5.2.</p>
  
  <p>Source:
  In "Natural language processing with Python" (<a href="http://www.nltk.org/book/" rel="nofollow noreferrer">http://www.nltk.org/book/</a>), exercise 4.24 is "Read up on 'keyword linkage' (chapter 5 of (Scott &amp; Tribble, 2006)). Extract keywords from NLTK's Shakespeare Corpus and using the NetworkX package, plot keyword linkage networks."  I am working through this book on my own for professional development at work.  The 2006 book referenced is "Textual patterns: key words and corpus analysis in language education" (especially p.58-60)</p>
  
  <p>Thank you for your time.</p>