title: Spark when union a lot of RDD throws stack overflow error
body: <p>When I use "++" to combine a lot of RDDs, I got error stack over flow error.</p>
  
  <p>Spark version 1.3.1
  Environment: yarn-client.  --driver-memory 8G</p>
  
  <p>The number of RDDs is more than 4000. Each RDD is read from a text file with size of 1 GB.</p>
  
  <p>It is generated in this way</p>
  
  <pre><code>val collection = (for (
    path &lt;- files
  ) yield sc.textFile(path)).reduce(_ union _)
  </code></pre>
  
  <p>It works fine when <code>files</code> has small size.
  And there is the error </p>
  
  <p>The error repeats itself. I guess it is a recursion function which is called too many time?</p>
  
  <pre><code> Exception at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
      at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      at scala.collection.AbstractTraversable.map(Traversable.scala:105)
      at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66)
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
      at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      at scala.collection.AbstractTraversable.map(Traversable.scala:105)
      at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
      at scala.Option.getOrElse(Option.scala:120)
    .....
  </code></pre>