title: Converting PySpark DataFrame to Pandas using Apache Arrow
body: <p>I'd like to convert a <code>PySpark</code> DataFrame (<code>pyspark.sql.DataFrame</code>) to <code>Pandas</code> dataframe. There is a builtin method <code>toPandas()</code> which is very inefficient (Please read Wes McKinney'd article about this issue back in Fenruary 2017 <a href="http://wesmckinney.com/blog/pandas-and-apache-arrow/" rel="nofollow noreferrer">here</a> and his calculation in <a href="https://gist.github.com/wesm/0cb5531b1c2e346a0007" rel="nofollow noreferrer">this jupyter notebook</a>). </p>
  
  <p>In the meantime, some efforts have been done to make such a conversion faster. An example, would be Josh's function <a href="https://gist.github.com/joshlk/871d58e01417478176e7" rel="nofollow noreferrer">here</a>. However, that did not help me since I am looking to transfer +1M rows from <code>pysaprk.DataFrame</code> to <code>Pandas</code>, this solution did not work for me.</p>
  
  <p>Luckily, as shown in <a href="https://arrow.apache.org/blog/2017/07/26/spark-arrow/" rel="nofollow noreferrer">this post</a> in July 26th 2017, thanks to authors <a href="https://github.com/wesm" rel="nofollow noreferrer">Wes</a>, <a href="https://github.com/icexelloss" rel="nofollow noreferrer">Li</a> and <a href="https://github.com/holdenk" rel="nofollow noreferrer">Holden</a>, the functionality of <code>toPandas()</code> has been significantly improved thanks to implementation of <code>Apache Arrow</code> in <code>Spark 2.3</code>. That being said, I don't have access to Spark 2.3 (I am using Spark 2.1).</p>
  
  <p>So, my question is how I can use <code>Apache Arrow</code> functionalities to convert <code>pyspark</code> dataframe to <code>Pandas</code> fast for <code>Spark</code> older than 2.1. I think a lot of people are stuck with older versions of <code>Spark</code> and can benefit from this. </p>
  
  <p><strong>Update 1</strong>: I have been suggested to print <code>pyspark</code> to <code>CSV</code> files first and then read <code>CSV</code> files from <code>Pandas</code> powerful <code>read_csv</code> method. I am really hopeful that I will find a way to avoid doing that! </p>
  
  <p><strong>Update 2:</strong> The reasoning behind slowness of <code>toPandas()</code> method and possible <code>out of memory</code> issues are discussed in details in <a href="https://stackoverflow.com/questions/36140493/java-lang-outofmemoryerror-in-pyspark">this discussion</a></p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>dfSpark = spark.sql(sqlQuery)
  df = dfSpark.toPandas() # Very slow / out of memory error
  </code></pre>