title: Regex for unicode sentences without spaces using CountVectorizer
body: <p>I hope I don't have to provide an example set.</p>
  
  <p>I have a 2D array where each array contains a set of words from sentences.</p>
  
  <p>I am using a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html" rel="nofollow">CountVectorizer</a> to effectively call <code>fit_transform</code> on the whole 2D array, such that I can build a vocabulary of words.</p>
  
  <p>However, I have sentences like:</p>
  
  <pre><code>u'Besides EU nations , Switzerland also made a high contribution at Rs 171 million LOCATION_SLOT~-nn+nations~-prep_besides nations~-prep_besides+made~prep_at made~prep_at+rs~num rs~num+NUMBER_SLOT'
  </code></pre>
  
  <p>And my current vectorizer is too strict at removing things like <code>~</code> and <code>+</code> as tokens. Whereas I want it to see each word in terms of <code>split()</code> a token in the vocab, i.e. <code>rs~num+NUMBER_SLOT</code> should be a word in itself in the vocab, as should <code>made</code>. At the same time, stopwords like <code>the</code> the <code>a</code> (the normal stopwords set) should be removed. </p>
  
  <p>Current vectorizer: </p>
  
  <pre><code>vectorizer = CountVectorizer(analyzer="word",stop_words=None,tokenizer=None,preprocessor=None,max_features=5000)
  </code></pre>
  
  <p>You can specify a <code>token_pattern</code> but I am not sure which one I could use to achieve my aims. Trying:</p>
  
  <pre><code>token_pattern="[^\s]*"
  </code></pre>
  
  <p>Leads to a vocabulary of:</p>
  
  <pre><code>{u'': 0, u'p~prep_to': 3764, u'de~dobj': 1107, u'wednesday': 4880, ...}
  </code></pre>
  
  <p>Which messes things up as <code>u''</code> is not something I want in my vocabulary.</p>
  
  <p>What is the right token pattern for this type of <code>vocabulary_</code> I want to build?</p>