title: Setting minDF and vocabularysize of CountVectorizerModel in spark?
body: <p>I am writing a spark Scala code using CountVectorizerModel and MinHashLSH library  for finding Jaccard similarity in two sets of string in a datafarme. Before feeding it to the MinHashLSH, it should be converted into vectors. For that I  am using CountVectorizerModel.</p>
  
  <p>The problem is that I am randomly setting the values like setVocabSize() and setMinDF() as shown below in the code. I am getting an error <code>vocabulary size should be &gt; 0. Lower minDF as necessary spark error</code>. </p>
  
  <p>The snippet of dataframe dfB look like below and its size changes according to the input.</p>
  
  <pre><code> +------------+-----------------+
  |         id1|          values1|
  +------------+-----------------+
  |         USA|  [Stat]|
  |     Germany|[Country]|
  | Clever_Hans|  [Horse]|
  |       China|[Country]|
  |       Japan|[Country]|
  |     Halidae| [Family]|
  |Real_Delight|  [Horse]|
  |Bago_(horse)|  [Horse]|
  
  val dfB = hashtoseq1.toDF("id1", "values1")
  
  val cvModel1: CountVectorizerModel = new CountVectorizer()
    .setInputCol("values1")
    .setOutputCol("features1")
    .setVocabSize(100000)
    .setMinDF(1)
    .fit(dfB)
  
  val kt1 = cvModel1.transform(dfB)
  //  kt.show
  val mh1 = new MinHashLSH()
    .setNumHashTables(5)
    .setInputCol("features1")
    .setOutputCol("hashes1")
  
  val model1 = mh1.fit(kt1)
  
  val dffilter1 = model1.approxSimilarityJoin(kt1, kt1, 0.4)
  </code></pre>
  
  <p>The above line code works for the file size less than 4GB. For 17 GB file, its throwing an error. I went through the spark document and folowing links, but did not help me.</p>
  
  <p><a href="https://stackoverflow.com/questions/48086226/spark-mlib-word2vec-error-the-vocabulary-size-should-be-0">Spark MLib Word2Vec Error: The vocabulary size should be > 0</a></p>