title: More than one hour to execute pyspark.sql.DataFrame.take(4)
body: <p>I am running spark 1.6 on 3 VMs (i.e. 1x master; 2x slaves) all with 4 cores and 16GB RAM.</p>
  
  <p>I can see the workers registered on spark-master webUI.</p>
  
  <p>I want to retrieve data from my Vertica database to work on it. As I didn't manage to run complex queries I tried dummy queries to understand. We consider here an easy task.</p>
  
  <p>My code is:</p>
  
  <pre class="lang-py prettyprint-override"><code>df = sqlContext.read.format('jdbc').options(url='xxxx', dbtable='xxx', user='xxxx', password='xxxx').load()
  four = df.take(4)
  </code></pre>
  
  <p>And the output is (note: I replace with <code>@IPSLAVE</code> the slave VM IP:Port):</p>
  
  <pre><code>16/03/08 13:50:41 INFO SparkContext: Starting job: take at &lt;stdin&gt;:1
  16/03/08 13:50:41 INFO DAGScheduler: Got job 0 (take at &lt;stdin&gt;:1) with 1 output partitions
  16/03/08 13:50:41 INFO DAGScheduler: Final stage: ResultStage 0 (take at &lt;stdin&gt;:1)
  16/03/08 13:50:41 INFO DAGScheduler: Parents of final stage: List()
  16/03/08 13:50:41 INFO DAGScheduler: Missing parents: List()
  16/03/08 13:50:41 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at take at &lt;stdin&gt;:1), which has no missing parents
  16/03/08 13:50:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.4 KB, free 5.4 KB)
  16/03/08 13:50:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.6 KB, free 7.9 KB)
  16/03/08 13:50:41 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on @IPSLAVE (size: 2.6 KB, free: 511.5 MB)
  16/03/08 13:50:41 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
  16/03/08 13:50:41 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at take at &lt;stdin&gt;:1)
  16/03/08 13:50:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
  16/03/08 13:50:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, @IPSLAVE, partition 0,PROCESS_LOCAL, 1922 bytes)
  16/03/08 13:50:41 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on @IPSLAVE (size: 2.6 KB, free: 511.5 MB)
  16/03/08 15:02:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4299240 ms on @IPSLAVE (1/1)
  16/03/08 15:02:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
  16/03/08 15:02:20 INFO DAGScheduler: ResultStage 0 (take at &lt;stdin&gt;:1) finished in 4299.248 s
  16/03/08 15:02:20 INFO DAGScheduler: Job 0 finished: take at &lt;stdin&gt;:1, took 4299.460581 s
  </code></pre>
  
  <p>As you can see it take a reaaaaaally long time.
  My table is actually quite big (stores around 220 millions lines, 11 fields each) but such a query would be quite instantly executed using "normal" sql (e.g. pyodbc).</p>
  
  <p>I guess I am missunderstanding/missusing Spark, would you have so ideas or advice to make it work better? </p>