title: spacy aggressive lemmatization and removing unexpected words
body: <p>I am trying to clean some text data.
  fisrt i removed the stop words, then i tried to Lemmatize the text. But words such as nouns are removed</p>
  <p><strong>Sample Data</strong></p>
  <p><a href="https://drive.google.com/file/d/1p9SKWLSVYeNScOCU_pEu7A08jbP-50oZ/view?usp=sharing" rel="nofollow noreferrer">https://drive.google.com/file/d/1p9SKWLSVYeNScOCU_pEu7A08jbP-50oZ/view?usp=sharing</a>
  <strong>udpated Code</strong></p>
  <pre><code># Libraries  
  import spacy
  import pandas as pd
  import gensim
  from gensim.utils import simple_preprocess
  import nltk; nltk.download('stopwords')
  from nltk.corpus import stopwords
  stop_words = stopwords.words('english')
  stop_words.extend(['covid', 'COVID-19', 'coronavirus'])
  
  article= pd.read_csv(&quot;testdata.csv&quot;)
  data = article.title.values.tolist()
  nlp = spacy.load('en_core_web_sm')
  
  def sent_to_words(sentences):
      for sentence in sentences:
        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))  # deacc=True removes punctuations
  
  data_words = list(sent_to_words(data))
  
  def remove_stopwords(texts):
      return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]
  data_words_nostops = remove_stopwords(data_words)
  print (&quot;*** Text  After removing Stop words:   &quot;)
  print(data_words_nostops)
  def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV','PRON']):
      &quot;&quot;&quot;https://spacy.io/api/annotation&quot;&quot;&quot;
      texts_out = []
      for sent in texts:
          doc = nlp(&quot; &quot;.join(sent)) 
          texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
      return texts_out
  data_lemmatized = lemmatization(data_words_nostops, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV','PRON'])
  print (&quot;*** Text  After Lemmatization:   &quot;)
  
  print(data_lemmatized)
  </code></pre>
  <p>The output after removing Stopwords is :</p>
  <blockquote>
  <p>[['qaia', 'flags', 'amman', 'melbourne', 'jetstar', 'flights', 'recovery', 'plan'],<br />
  ['western', 'amman', 'suburb', 'new','nsw', 'ground', 'zero', children],<br />
  ['flight', 'returned', 'amman','qaia', 'staff', 'contract','driving'], ]]</p>
  </blockquote>
  <p>The output after Lematization :</p>
  <blockquote>
  <p>[['flight', 'recovery', 'plan']</p>
  </blockquote>
  <blockquote>
  <p>['suburb', 'ground']</p>
  </blockquote>
  <blockquote>
  <p>['return', 'contract','driving']</p>
  </blockquote>
  <p>on each reacord I do not understand the following :</p>
  <p><strong>-1st reord: why these words are removed: &quot;'qaia', 'flags', 'amman', 'melbourne', 'jetstar'</strong></p>
  <p><strong>-2ed recored: essential words are reomved same as the first reord, Also, I was expecting children to convert to child</strong></p>
  <p><strong>-3ed, &quot;driving&quot; is not converted to &quot;drive&quot;</strong></p>
  <p>I was expecting that words will such as &quot;Amman&quot; will not removed, Also i am expecting the words will be converted from plural to singular. And the verbs will be converted to the infinitive ...</p>
  <p>What i am missing here???
  Thanx in advance</p>