title: Apache Spark or Spark-Cassandra-Connector doesnt look like it is reading multiple partitions in parallel?
body: <p>Apache Spark or Spark-Cassandra-Connector doesnt look like it is reading multiple partitions in parallel.</p>
  
  <p>Here is my code using spark-shell</p>
  
  <pre><code>import org.apache.spark.sql._
  import org.apache.spark.sql.types.StringType
  spark.sql("""CREATE TEMPORARY VIEW hello USING org.apache.spark.sql.cassandra OPTIONS (table "hello", keyspace "db", cluster "Test Cluster", pushdown "true")""")
  val df = spark.sql("SELECT test from hello")
  val df2 = df.select(df("test").cast(StringType).as("test"))
  val rdd = df2.rdd.map { case Row(j: String) =&gt; j }
  val df4 = spark.read.json(rdd) // This line takes forever 
  </code></pre>
  
  <p>I have about 700 million rows each row is about 1KB and this line</p>
  
  <p><code>val df4 = spark.read.json(rdd)</code> takes forever as I get the following output.</p>
  
  <p>Stage 1:==========>                                         (4866 + 24) / 25256]</p>
  
  <p>so at this rate it will probably take roughly 3hrs. </p>
  
  <p>I measured the network throughput rate of spark worker nodes using iftop and it is about 75MB/s (Megabytes per second) which is pretty good but I am not sure if it is reading partitions in parallel. Any ideas on how to make it faster?</p>
  
  <p>Here is my DAG.</p>
  
  <p><a href="https://i.stack.imgur.com/Odh8c.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Odh8c.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/XOyl7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XOyl7.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/AVik2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AVik2.png" alt="enter image description here"></a></p>