title: Can we use a Pandas function in a Spark DataFrame column ? If so, how?
body: <p>I have a pandas dataframe called "pd_df".</p>
  
  <p>I want to modify it's column , so I do something like this:</p>
  
  <pre><code>    import pandas as pd
  
      pd_df['notification_dt'] = pd.to_datetime(pd_df['notification_dt'], format="%Y-%m-%d")
  </code></pre>
  
  <p>It works.</p>
  
  <p>On the same database, I created a spark dataframe called "spark_df"</p>
  
  <p>I want the same function (pd.to_datatime) on it's column to perform the same operation. So I did this.</p>
  
  <pre><code>    from pyspark.sql.functions import UserDefinedFunction
  
      from pyspark.sql.types import TimestampType
  
      udf = UserDefinedFunction(lambda x: pd.to_datetime(x, format="%Y-%m-%d"), TimestampType())
  
      spark_df2 = spark_df.withColumn("notification_dt1", (udf(spark_df["notification_dt"])))
  </code></pre>
  
  <p>It should work, according to me. However on</p>
  
  <pre><code>   spark_df.show()
  </code></pre>
  
  <p>I encounter the following error after a minute or so:
  <img src="https://i.stack.imgur.com/QXcIi.png" alt="enter image description here"></p>