title: Looking for city names using regex in Python/pandas
body: <p>I have a DataFrame in pandas of strings that contains some random string of letters, followed by a potential city and state. However, the spacing is irregular.</p>
  
  <p>For example, the column that I'm interested in parsing looks like</p>
  
  <pre><code>blah burlingtonvt
  blahessex vt
  blah dover vt
  blahnorthchittendenvt
  blah essex junctionvt
  </code></pre>
  
  <p>replacing each "blah" with some string of alphanumeric characters (potentially different for each row) that I want to preserve and with hundreds of thousands of observations and with cities and states all over the continental US. Since the state code always appears at the end, I can extract it easily with a regular expression like</p>
  
  <pre><code>(ny$|tx$|vt$|...)
  </code></pre>
  
  <p>where the '...' is just the rest of the states as you would expect. I then process each subset of the DataFrame for each state separately. I have a list of all possible "mis-spacings" of the cities for a given state that I've made into regular expression. i.e.,</p>
  
  <pre><code>x = '(essex[ ]{0,1}junction|north[ ]{0,1}chittenden|essex|...)'
  </code></pre>
  
  <p>The number of city name variations in this regular expression sometimes runs into the thousands. I then run the following line</p>
  
  <pre><code>df['city'] = df['string'].str.extract(x, expand=True)
  </code></pre>
  
  <p>which returns</p>
  
  <pre><code>  string                  state   city
  1 blah waterbury cenvt    vt      waterbury
  2 blah waterbury vt       vt
  3 blah waterbury vt       vt
  4 blah vt                 vt
  5 blah shelburne vt       vt
  6 blah waterbury vt       vt
  7 blah richmond vt        vt      richmond
  </code></pre>
  
  <p>but if I run, for example,</p>
  
  <pre><code>df['city'] = df['string'].str.extract('(waterbury|shelburne|richmond)', expand=True)
  </code></pre>
  
  <p>The results are as desired. Why would this be the case? I generate the regular expression programmatically so I want to say that this isn't because of a typo. Is there a chance that it's because there're too many things to look for? Is there a better way to do this?</p>
  
  <p>Edit: it turns out that there was an empty string hiding among the thousands of city names that I didn't think would be there.</p>