title: Save a large Spark Dataframe as a single json file in S3
body: <p>Im trying to save a Spark DataFrame (of more than 20G) to a single json file in Amazon S3, my code to save the dataframe is like this : </p>
  
  <pre><code>dataframe.repartition(1).save("s3n://mybucket/testfile","json")
  </code></pre>
  
  <p>But im getting an error from S3 "Your proposed upload exceeds the maximum allowed size", i know that the maximum file size allowed by Amazon is 5GB.</p>
  
  <p>Is it possible to use S3 multipart upload with Spark? or there is another way to solve this?</p>
  
  <p>Btw i need the data in a single file because another user is going to download it after.</p>
  
  <p>*Im using apache spark 1.3.1 in a 3-node cluster created with the spark-ec2 script.</p>
  
  <p>Thanks a lot</p>
  
  <p>JG</p>