title: Failing to overwrite parquet hive table in pyspark
body: <p>Using: spark 1.5.2, hive 1.2
  I have an external hive table in parquet format. I created a .py script that selects from my_table into a dataframe, does some transforms and then attempts to write back into the original table. </p>
  
  <p>I've tried the following methods: </p>
  
  <ol>
  <li><code>df.write.insertInto('table_name', overwrite='true')</code>. </li>
  </ol>
  
  <p>This throws the following error:   </p>
  
  <blockquote>
    <p>pyspark.sql.utils.AnalysisException: Cannot insert overwrite into table that is also being read from.</p>
  </blockquote>
  
  <ol start="2">
  <li><code>df.write.mode('overwrite').parquet('my_path')</code></li>
  <li><code>df.write.parquet('my_path', mode='overwrite')</code></li>
  <li><code>df.write.save('my_path', format='parquet', mode = 'overwrite')</code></li>
  </ol>
  
  <p>These all seem to throw this error: </p>
  
  <blockquote>
    <p>ERROR Client fs/client/fileclient/cc/client.cc:1802 Thread: 620 Open failed for file /my_path/part-r-00084-9, LookupFid error No such file or directory(2)
            2016-04-26 16:47:17,0942 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:2488 Thread: 620 getBlockInfo failed, Could not open file /my_path/part-r-00084-9
            16/04/26 16:47:17 WARN DAGScheduler: Creating new stage failed due to exception - job: 16</p>
  </blockquote>
  
  <p>**Note that method 1 above works fine if the file format is orc, but throws that error for parquet. </p>
  
  <p>Any suggestions would be greatly appreciated! </p>