title: Algorithm (or C# library) for identifying 'keywords' in a set of messages?
body: <p>I want to build a list of ~6 keywords (or even better: couple word keyphrases) for each message in a message forum.</p>
  
  <ul>
  <li>The primary use of keywords is to replace subject lines in some instances. For example: <em>Message from Terry sent Dec 5, keywords: <strong>norweigan blue, plumage, not dead</em></strong></li>
  <li>In a super ideal world keywords would identify both unique phases, and phrases that cluster the discussion into "topics", i.e. words that are highly relevant to the message in question, and a few other messages in the forum, but not found frequently in the forum as a whole.</li>
  <li>I expect junk phrases to show up, no big deal.</li>
  <li>Can't be too computationally expensive: I need something that can handle several hundred messages in several seconds, as I'll need to re-run this every time a new message comes in.</li>
  </ul>
  
  <p>Anyone know a good C# library for accomplishing this? Maybe there's a way to bend Lucene.NET into providing this sort of info?</p>
  
  <p>Or, failing that, can anyone suggest an algorithm (or set of algos) to read up on? If I'm implementing myself I need something not terribly complex, I can only tackle this if its tractable in about a week. Right now, the best I've found in terms of simple-but-effective is <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf" rel="nofollow">TF-IDF</a>.</p>
  
  <p><strong>UPDATE:</strong> I've uploaded the results of using TF-IDF to select the top 5 keywords from a real dataset here: <a href="http://jsbin.com/oxanoc/2/edit#preview" rel="nofollow">http://jsbin.com/oxanoc/2/edit#preview</a> </p>
  
  <p>The results are mediocre, but not totally useless... maybe with the addition of detecting multi-word phrases, this would be good enough.</p>