title: Pyspark: Remove UTF null character from pyspark dataframe
body: <p>I have a pyspark dataframe similar to the following:</p>
  
  <pre><code>df = sql_context.createDataFrame([
    Row(a=3, b=[4,5,6],c=[10,11,12], d='bar', e='utf friendly'),
    Row(a=2, b=[1,2,3],c=[7,8,9], d='foo', e=u'ab\u0000the')
    ])
  </code></pre>
  
  <p>Where one of the values for column <code>e</code> contains the UTF null character <code>\u0000</code>. If I try to load this <code>df</code> into a postgresql database, I get the following error:</p>
  
  <pre><code>ERROR: invalid byte sequence for encoding "UTF8": 0x00 
  </code></pre>
  
  <p>which makes sense. How can I efficiently remove the null character from the pyspark dataframe before loading the data into postgres?</p>
  
  <p>I have tried using some of the <code>pyspark.sql.functions</code> to clean the data first without success. <code>encode</code>, <code>decode</code>, and <code>regex_replace</code> did not work:</p>
  
  <pre><code>df.select(regexp_replace(col('e'), u'\u0000', ''))
  df.select(encode(col('e'), 'UTF-8'))
  df.select(decode(col('e'), 'UTF-8'))
  </code></pre>
  
  <p>Ideally, I would like to clean the entire dataframe without specifying exactly which columns or what the violating character is, since I don't necessarily know this information ahead of time.</p>
  
  <p>I am using a postgres 9.4.9 database with <code>UTF8</code> encoding.</p>