title: What is the performance difference between accumulator and collect() in Spark?
body: <p>Accumulator are basically the shared variable in spark to be updated by executors but read by driver only.
  <code>Collect()</code> in spark is to get all the data into the driver from executors.</p>
  
  <p>So, in both when I am get the data ultimately in driver only. so, what is the difference in performance when we use accumulator or <code>collect()</code> to convert a large RDD into a LIST?</p>
  
  <p>Code to convert dataframe to List using accumulator</p>
  
  <pre><code>val queryOutput = spark.sql(query)
  val acc = spark.sparkContext.collectionAccumulator[Map[String,Any]]("JsonCollector")
  val jsonString = queryOutput.foreach(a=&gt;acc.add(convertRowToJSON(a)))
  acc.value.asScala.toList
  
  
  def convertRowToJSON(row: Row): Map[String,Any] = {
      val m = row.getValuesMap(row.schema.fieldNames)
      println(m)
      JSONObject(m).obj
    }
  </code></pre>
  
  <p>Code to convert dataframe to list using collect()</p>
  
  <pre><code>val queryOutput = spark.sql(query)
  queryOutput.toJSON.collectAsList()
  </code></pre>