title: HDFS / Hadoop api access from pyspark worker
body: <p>I need to read/scan/write files to/from the hdfs from within a pyspark <strong>worker</strong>.  </p>
  
  <p>Note the following api's are not applicable since they run off of the <strong>driver</strong>:</p>
  
  <pre><code>sc.textFile()
  sc.saveAsParquetFile()
  </code></pre>
  
  <p>etc</p>
  
  <p>It would be very much preferable not to involve additional third party libraries (e.g. pyhadoop).</p>
  
  <p>One option is to shell out e.g.</p>
  
  <pre><code> os.system('hdfs dfs -ls %(hdfsPath)s' %locals())
  </code></pre>
  
  <p>But is there a more native pyspark way to achieve this?</p>
  
  <p><strong>UPDATE</strong> This is not a case of broadcasting data because each worker will read different data from hdfs. One of the use cases is reading a few large binary files in each worker (this is clearly not a case for broadcast). Another case is to read "command" file containing instructions. I have successfully used this pattern in native hadoop and in scala spark.</p>