title: Word tokenization using python regular expressions
body: <p>I am trying to split strings into lists of "tags" in python.  The splitting should handle strings such as "HappyBirthday" and remove most punctuation but preserve hyphens, and apostrophes. My starting point is:</p>
  
  <pre><code>tags = re.findall("([A-Z]{2,}(?=[A-Z]|$)|[A-Z][a-z]*)|\w+-\w+|[\w']+"
  </code></pre>
  
  <p>I would want to turn this sample data:</p>
  
  <pre><code>Jeff's dog is un-American SomeTimes! BUT NOTAlways
  </code></pre>
  
  <p>Into:</p>
  
  <pre><code>['Jeff's', 'dog', 'is', 'un-American', 'Some', 'Times', 'BUT', 'NOT', 'Always']
  </code></pre>
  
  <p>P.S. I am sorry my description isn't very good. I am not sure how to explain it, and have been mostly unsuccessful with google.  I hope the example illustrates it properly.</p>
  
  <p>Edit: i think i needed to be more precise, so also,</p>
  
  <ol>
  <li>if the word is hypenated and capital, like 'UN-American' will it keep it as one word so output would be  'UN-American'</li>
  <li>if the hyphen has a space on either or both sides, a la 'THIS- is' or 'This - is' it should ignore the hypen and produce ["THIS", "is"] and ["This", "is"] respecticly, </li>
  <li>and simmilarly for an apostrophe if its in the middle of a word like "What'sItCalled" it should produce ["What's","It", "Called"]   </li>
  </ol>