title: How do I get a vocabulary from columns in a dataframe?
body: <p>I have a rather large dataset stored in a dataframe. So large , in fact, that sorting through the dataset to generate an example dataset has caused my text editor to crash. Because of this, I am providing links to the datasets I'm using:</p>
  
  <p><a href="https://github.com/moonman239/Capstone-project/blob/master/data.zip" rel="nofollow noreferrer">https://github.com/moonman239/Capstone-project/blob/master/data.zip</a></p>
  
  <p>For planning purposes, I need to retrieve the vocabulary of words from the question,article_title and paragraph_context columns.</p>
  
  <p>However, it appears that in the process of splitting and combining the columns, I have inadvertently created some words by mashing together two words end-to-end (e.g: "raised" and "in" become "raisedin" catalans")</p>
  
  <pre><code>### Loading JSON datasets
  
  import json
  import re
  regex = re.compile(r'\W+')
  def readFile(filename):
    with open(filename) as file:
      fields = []
      JSON = json.loads(file.read())
      for article in JSON["data"]:
        articleTitle = article["title"]
        for paragraph in article["paragraphs"]:
          paragraphContext = paragraph["context"]
          for qas in paragraph["qas"]:
            question = qas["question"]
            for answer in qas["answers"]:
              fields.append({"question":question,"answer_text":answer["text"],"answer_start":answer["answer_start"],"paragraph_context":paragraphContext,"article_title":articleTitle})
    fields = pd.DataFrame(fields)
    fields["question"] = fields["question"].str.replace(regex," ")
    assert not (fields["question"].str.contains("catalanswhat").any())
    fields["paragraph_context"] = fields["paragraph_context"].str.replace(regex," ")
    fields["answer_text"] = fields["answer_text"].str.replace(regex," ")
    assert not (fields["answer_text"].str.contains("catalanswhat").any())
    fields["article_title"] = fields["article_title"].str.replace("_"," ")
    assert not (fields["article_title"].str.contains("catalanswhat").any())
    return fields
  # Load training dataset.
  trainingData = readFile("train-v1.1.json")
  
  # Vocabulary functions
  def vocabulary():
    data_frame = trainingData
    data_frame = data_frame.astype("str")
    text_split = pd.concat((data_frame["question"],data_frame["paragraph_context"],data_frame["article_title"]),ignore_index=True)
    text_split = text_split.str.split()
    words = set()
    text_split.apply(words.update)
    return words
  def vocabularySize():
    return len(vocabulary())
  </code></pre>
  
  <p><strong>Alternate code that also fails:</strong></p>
  
  <pre><code>def vocabulary():
    data_frame = trainingData
    data_frame = data_frame.astype("str")
    concat = data_frame["question"].str.cat(sep=" ",others=[data_frame["paragraph_context"],data_frame["article_title"]])
    concat = concat.str.split(" ")
    words = set()
    concat.apply(words.update)
    print(words)
    assert "raisedin" not in words
    return words
  </code></pre>