title: How to save a DataFrame as compressed (gzipped) CSV?
body: <p>I use Spark 1.6.0 and Scala.</p>
  
  <p>I want to save a DataFrame as compressed CSV format.</p>
  
  <p>Here is what I have so far (assume I already have <code>df</code> and <code>sc</code> as <code>SparkContext</code>):</p>
  
  <pre><code>//set the conf to the codec I want
  sc.getConf.set("spark.hadoop.mapred.output.compress", "true")
  sc.getConf.set("spark.hadoop.mapred.output.compression.codec", "true")
  sc.getConf.set("spark.hadoop.mapred.output.compression.codec", "org.apache.hadoop.io.compress.GzipCodec")
  sc.getConf.set("spark.hadoop.mapred.output.compression.type", "BLOCK")
  
  df.write
    .format("com.databricks.spark.csv")
    .save(my_directory)
  </code></pre>
  
  <p>The output is not in <code>gz</code> format.</p>