title: How to count frequency of a tokenized column without including [ ]?
body: <pre><code># Loading required python packages and libraries
  import nltk
  import pandas as pd
  import re
  import string
  import pickle
  import random
  import numpy as np
  
  from wordcloud import WordCloud,STOPWORDS
  
  import nltk
  nltk.download('stopwords')
  stopwords = nltk.corpus.stopwords.words('english')
  ps = nltk.PorterStemmer()
  
  
  # Load the csv file using Pandas and print first 5 lines
  data = pd.read_csv(&quot;/content/fsw.csv&quot;)
  data.tokens=data.tokens.astype(str)
  data.head()
  
  text_wordcloud = &quot; &quot;.join(word.lower() for word in data.tokens)
  print ('There are total {} words in text provided.'.format(len(text_wordcloud)))
  </code></pre>
  <p>This is the code that generates the word frequency. As I am trying to count the frequency of a tokenized column. This function is taking opening '[' and closing ']' as frequency too. To be specific it counts the same word it its 'ameen', differently and 'ameen'] differently. How can I make sure that doesn't happen.</p>
  <pre><code>def add_words_to_csv(clean_text): 
  
      word_list = []
      # Break the string into list of words  
      str = clean_text.split()          
      str2 = [] 
    
      # Loop till string values present in list str 
      for i in str:              
    
          # Checking for the duplicacy 
          if i not in str2: 
    
              # Append value in str2 
              str2.append(i)  
                
      for i in range(0, len(str2)): 
          if(str.count(str2[i])&gt;10): 
            word_list.append([str2[i], str.count(str2[i])])
      word_array = np.asarray(word_list)
      df = pd.DataFrame(word_array, columns=['word', 'frequency'])
      df.to_csv('word_count.csv',index=False)
      
  
  freq(text_wordcloud)
  add_words_to_csv(text_wordcloud)
  </code></pre>
  <p><a href="https://i.stack.imgur.com/UseG2.png" rel="nofollow noreferrer">This is the output after sorting the words in A-Z order</a>
  <a href="https://i.stack.imgur.com/bNRXB.png" rel="nofollow noreferrer">This is another output snapshot</a></p>