title: Difficulties in removing characters and white space to tokenize text via Spacy
body: <p>I'm testing the Spacy library, but I'm having trouble cleaning up the sentences (ie removing special characters; punctuation; patterns like [Verse], [Chorus], \n ...) before working with the library.</p>
  <p>I have removed, to some extent, these elements, however, when I perform the tokenization, I notice that there are extra white spaces, in addition to the separation of terms like &quot;it&quot; and &quot;s&quot; (it's).</p>
  <p>Here is my code with some text examples:</p>
  <pre><code>text1 = &quot;[Intro] Well, alright [Chorus] Well, it's 1969, okay? All across the USA It's another year for me and you&quot;
  text2 = &quot;[Verse 1] For fifty years they've been married And they can't wait for their fifty-first to roll around&quot;
  text3 = &quot;Passion that shouts And red with anger I lost myself Through alleys of mysteries I went up and down Like a demented train&quot;
  </code></pre>
  <pre><code>df = pd.DataFrame({'text':[text1, text2, text3]})
  </code></pre>
  <pre><code>replacer ={'\n':' ',&quot;[\[].*?[\]]&quot;: &quot; &quot;,'[!&quot;#%\'()*+,-./:;&lt;=&gt;?@\[\]^_`{|}~1234567890’”“′‘\\\]':&quot; &quot;}
  df['cleanText'] = df['text'].replace(replacer, regex=True)
  df.head()
  </code></pre>
  <pre><code>df['new_col'] = df['cleanText'].apply(lambda x: nlp(x))
  df
  </code></pre>
  <pre><code>#Output:
  result1 = &quot;  Well  alright   Well  it s       okay  All across the USA It s another year for me and you&quot;
  result2 = &quot;  For fifty years they ve been married And they can t wait for their fifty first to roll around&quot;
  result3 = &quot;Passion that shouts And red with anger I lost myself Through alleys of mysteries I went up and down Like a demented train&quot;
  </code></pre>
  <p>When I try to tokenize, I get, for example: (  , Well,  , alright,   , Well,  , it, s,    ...)</p>
  <p>I used the same logic to remove the characters to tokenize via nltk and there it worked. Does anyone know what I might be wrong?</p>