title: how to sort data in each partition in spark?
body: <p>There is some data:</p>
  
  <pre><code>a   2
  b   2
  c   2
  a   1
  c   3
  a   3
  c   1
  b   3
  b   1
  </code></pre>
  
  <p>when I repartition the data, and <strong>no sort</strong>, code is:</p>
  
  <pre><code>val sc = new SparkContext
  val file = sc.textFile(args(0)).map { a =&gt; {
             val splits = a.split("\t")
             (new MyObject(splits(0), splits(1).toInt),"") } }
             .partitionBy(new MyPartitioner(3)) //.sortByKey()    no sort
  </code></pre>
  
  <p>The result is:</p>
  
  <pre><code>//file:part-00000
  (a  2,)
  (a  1,)
  (a  3,)
  
  //file:part-00001
  (b  2,)
  (b  3,)
  (b  1,)
  
  //file:part-00002
  (c  2,)
  (c  3,)
  (c  1,)
  </code></pre>
  
  <p>when I repartition the data, and <strong>sort</strong>, code is:</p>
  
  <pre><code>val sc = new SparkContext
  val file = sc.textFile(args(0)).map { a =&gt; {
             val splits = a.split("\t")
             (new MyObject(splits(0), splits(1).toInt),"") } }
             .partitionBy(new MyPartitioner(3)).sortByKey() 
  </code></pre>
  
  <p>The result is(it is not I wanted, the sorted data affect original partitions):</p>
  
  <pre><code>//file:part-00000
  (a  1,)
  (a  2,)
  (a  3,)
  (b  1,)
  
  //file:part-00001
  (b  2,)
  (b  3,)
  (c  1,)
  
  //file:part-00002
  (c  2,)
  (c  3,)
  </code></pre>
  
  <p>The result I expect is:</p>
  
  <pre><code>//file:part-00000
  (a  1,)
  (a  2,)
  (a  3,)
  
  //file:part-00001
  (b  1,)
  (b  2,)
  (b  3,)
  
  //file:part-00002
  (c  1,)
  (c  2,)
  (c  3,)
  </code></pre>
  
  <p>Could you help me? Thank you very much!</p>