title: write more file than partition number
body: <p>Using pyspark, I am writing a list of distinct emails from a hive table to flat files in HDFS. 
  The constraint is that my files should not contain more than 4.5M lines. </p>
  
  <p>I wrote this code : </p>
  
  <pre class="lang-py prettyprint-override"><code>df = spark.read.table(working_table).select("email").distinct()
  
  final_df = df.withColumn(
      "rnk", 
      (F.row_number().over(Window.orderBy(F.lit(1))) / 4500000).cast("int")
  )
  
  final_df.groupBy("rnk").count().show()
  
  +---+-------+                                                                   
  |rnk|  count|
  +---+-------+
  |  0|4499999|
  |  1|1858773|
  +---+-------+
  
  
  final_df.repartition("rnk").select("email").write.csv(
      working_dir_email, mode="overwrite", compression="none"
  )
  </code></pre>
  
  <p>This works almost fine, except that I always got an additional empty file : </p>
  
  <pre class="lang-sh prettyprint-override"><code>hdfs dfs -ls /tmp/emails
  Found 4 items
  -rw-rw-r--   3 hadoop hadoop          0 2020-02-05 17:59 /tmp/emails/_SUCCESS
  -rw-rw-r--   3 hadoop hadoop          0 2020-02-05 17:59 /tmp/emails/part-00000-f724cf35-5ed8-4b7a-98c2-fc992dca9ad1-c000.csv
  -rw-rw-r--   3 hadoop hadoop  173936230 2020-02-05 17:59 /tmp/emails/part-00043-f724cf35-5ed8-4b7a-98c2-fc992dca9ad1-c000.csv
  -rw-rw-r--   3 hadoop hadoop  292499935 2020-02-05 17:59 /tmp/emails/part-00191-f724cf35-5ed8-4b7a-98c2-fc992dca9ad1-c000.csv
  </code></pre>
  
  <p>Can I avoid this blank file? Is there a more clever way to achieve the result  without the empty file maybe? </p>