title: RDD repartition doesn't seem to shuffle if increase partitions?
body: <p>Spark v2.4</p>
  
  <pre><code>spark.range(5, numPartitions=1).rdd.keys().repartition(7).glom().collect()
  # [[], [], [], [], [], [], [0, 1, 2, 3, 4]]
  
  # 2 partitions initially
  spark.range(5, numPartitions=2).rdd.keys().glom().collect()
  # [[0, 1], [2, 3, 4]]
  
  spark.range(5, numPartitions=2).rdd.keys().repartition(7).glom().collect()
  # [[], [], [2, 3, 4], [], [], [], [0, 1]]
  </code></pre>
  
  <p>It seems to add empty partitions only in this case?</p>
  
  <p>However, the DataFrame repartition does work properly using <code>RoundRobinPartitioning</code>.</p>
  
  <pre><code>spark.range(5, numPartitions=1).repartition(7).rdd.keys().glom().collect()
  # [[0], [2], [4], [1], [], [], [3]]
  
  spark.range(5, numPartitions=2).rdd.keys().glom().collect()
  # [[0, 1], [2, 3, 4]]
  spark.range(5, numPartitions=2).repartition(7).rdd.keys().glom().collect()
  # [[1, 4], [], [], [], [], [3], [0, 2]]
  </code></pre>
  
  <p>What partition methods does RDD <code>repartition</code> use?</p>
  
  <h3>Update</h3>
  
  <p>@user10938362 lead me to a similar question where I found a reply that answers my questions.</p>
  
  <p>Answered by @user11400142</p>
  
  <p>That happens because Spark doesn't shuffle individual elements but rather blocks of data - <a href="https://github.com/apache/spark/blob/8171b156ebf25a582318c71222716b5c52c8bbb3/python/pyspark/rdd.py#L2107" rel="nofollow noreferrer">with minimum batch size equal to 10</a>.</p>
  
  <p>So if you have less elements than that per partition, Spark won't separate content of partitions.</p>
  
  <p>Examples</p>
  
  <pre><code>spark.range(10, numPartitions=1).rdd.keys().repartition(3).glom().collect()
  # [[], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], []]
  spark.range(11, numPartitions=1).rdd.keys().repartition(3).glom().collect()
  # [[], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10]]
  </code></pre>