title: RuntimeError: Unsupported type in conversion to Arrow: VectorUDT
body: <p>I want to convert a big spark data frame to Pandas with more than 1000000 rows. I tried to convert a spark data Frame to Pandas data frame using the following code:</p>
  
  <pre><code>spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  result.toPandas()
  </code></pre>
  
  <p>But, I got the error:</p>
  
  <pre><code>TypeError                                 Traceback (most recent call last)
  /usr/local/lib/python3.6/dist-packages/pyspark/sql/dataframe.py in toPandas(self)
     1949                 import pyarrow
  -&gt; 1950                 to_arrow_schema(self.schema)
     1951                 tables = self._collectAsArrow()
  
  /usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in to_arrow_schema(schema)
     1650     fields = [pa.field(field.name, to_arrow_type(field.dataType), nullable=field.nullable)
  -&gt; 1651               for field in schema]
     1652     return pa.schema(fields)
  
  /usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in &lt;listcomp&gt;(.0)
     1650     fields = [pa.field(field.name, to_arrow_type(field.dataType), nullable=field.nullable)
  -&gt; 1651               for field in schema]
     1652     return pa.schema(fields)
  
  /usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in to_arrow_type(dt)
     1641     else:
  -&gt; 1642         raise TypeError("Unsupported type in conversion to Arrow: " + str(dt))
     1643     return arrow_type
  
  TypeError: Unsupported type in conversion to Arrow: VectorUDT
  
  During handling of the above exception, another exception occurred:
  
  RuntimeError                              Traceback (most recent call last)
  &lt;ipython-input-138-4e12457ff4d5&gt; in &lt;module&gt;()
        1 spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  ----&gt; 2 result.toPandas()
  
  /usr/local/lib/python3.6/dist-packages/pyspark/sql/dataframe.py in toPandas(self)
     1962                     "'spark.sql.execution.arrow.enabled' is set to true. Please set it to false "
     1963                     "to disable this.")
  -&gt; 1964                 raise RuntimeError("%s\n%s" % (_exception_message(e), msg))
     1965         else:
     1966             pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
  
  RuntimeError: Unsupported type in conversion to Arrow: VectorUDT
  Note: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true. Please set it to false to disable this.
  </code></pre>
  
  <p>It's not working, but if I set arrow to false, it works. But It's so slow... Any idea?</p>