title: Spark - 54 GB CSV file transform to single JSON in 16 GB RAM single machine
body: <p>I want to take a CSV file and transform into single JSON, I have written and verified the code. I have a CSV file of 54 GB and I want to transform and export this single file into single JSON, I want to take this data in Spark and it will design the JSON using SparkSQL <code>collect_set(struct</code> built-in functions. 
  I am running Spark job in Eclipse IDE in a single machine only. The machine configuration has 16 GB RAM, i5 Processor, 600 GB HDD.</p>
  
  <p>Now when I have been trying to run the spark program it is throwing <code>java.lang.OutOfMemory</code> and insufficient heap size error. I tried to increase the <code>spark.sql.shuffle.partitions</code> value <code>2000</code> to <code>20000</code> but still the job is failing after loading and during the transformation due to the same error I have mentioned.</p>
  
  <p>I don't want to split the single CSV into multiple parts, I want to process this single CSV, how can I achieve that? Need help. Thanks.</p>
  
  <p><strong>Spark Configuration:</strong></p>
  
  <pre><code>    val conf = new SparkConf().setAppName("App10").setMaster("local[*]")
   //   .set("spark.executor.memory", "200g")
        .set("spark.driver.memory", "12g")
        .set("spark.executor.cores", "4")
        .set("spark.driver.cores", "4")
    //  .set("spark.testing.memory", "2147480000")
          .set("spark.sql.shuffle.partitions", "20000")
          .set("spark.driver.maxResultSize", "500g")      
          .set("spark.memory.offHeap.enabled", "true")
          .set("spark.memory.offHeap.size", "200g")
  </code></pre>