title: Spark 2.2.0 FileOutputCommitter
body: <p>DirectFileOutputCommitter is no longer available in Spark 2.2.0. This means writing to S3 takes insanely long time (3 hours vs 2 mins). I'm able to work around this by setting FileOutputCommitter version to 2 in spark-shell by doing this,</p>
  
  <pre><code>spark-shell --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 
  </code></pre>
  
  <p>same does not work with spark-sql</p>
  
  <pre><code>spark-sql --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 
  </code></pre>
  
  <p>The above command seems to be setting the version=2 but when the query is exeucted it still shows version 1 behaviour. </p>
  
  <p>Two questions,</p>
  
  <p>1) How do I get FileOutputCommitter version 2 behaviour with spark-sql?</p>
  
  <p>2) Is there a way I can still use DirectFileOutputCommitter in spark 2.2.0? [I'm fine with non-zero chance of missing data]</p>
  
  <p>Related items:</p>
  
  <p><a href="https://stackoverflow.com/questions/35209603/spark-1-6-directfileoutputcommitter">Spark 1.6 DirectFileOutputCommitter</a></p>