title: Databricks Exception: Total size of serialized results is bigger than spark.driver.maxResultsSize
body: <p>I'm running a code in Apache Spark on Azure that converts over 3 million XML-files into one CSV-file. I get the following error when I want to do this:</p>
  <p><strong>org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1408098 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)</strong></p>
  <p>I know what the error means in general, but I don't know what it means in my case and I don't understand how to solve this.</p>
  <p>The code is:</p>
  <h1>All XML files are loaded:</h1>
  <pre><code>df = spark.read.format('com.databricks.spark.xml').option(&quot;rowTag&quot;, &quot;ns0:TicketScan&quot;).load('LOCATION/*.xml')
  </code></pre>
  <h1>All loaded files are put into a CSV-file:</h1>
  <pre><code> def saveDfToCsv(df, tsvOutput):
    tmpParquetDir = &quot;dbfs:/tmp/mart1.tmp.csv&quot;
    dbutils.fs.rm(tmpParquetDir, True)
    df.repartition(1).write.format(&quot;com.databricks.spark.csv&quot;).option(&quot;header&quot;, &quot;true&quot;).save(tmpParquetDir)
    src = filter(lambda x: &quot;part-00000&quot; in x.name, dbutils.fs.ls('dbfs:/tmp/mart1.tmp.csv'))[0].path
    dbutils.fs.mv(src, tsvOutput)
  
  saveDfToCsv(df, 'LOCATION/database.csv')
  </code></pre>
  <p>I hope my question is clear enough. If not, please allow me to explain it further.</p>
  <p>I hope someone can help me.</p>
  <p>Best regards.</p>