title: How to optimize writting each row of a spark dataframe as a separate file
body: <p>My dataframe has around 1 million records. Below is my code I am using to write each row of a spark dataframe to a separate file. But it takes hours to complete. Any suggestion to tweak this would be really helpful.</p>
  
  <pre><code>row_count = df.count()
  row_count = 10,000,00
  df1 = df.repartition(row_count)
  df1.rdd.map(lambda row:row[0]).saveAsTextFile(targetfolder)
  </code></pre>