title: How do I write the output of an EMR streaming job to HDFS?
body: <p>I see <a href="https://stackoverflow.com/questions/11169708/where-is-my-aws-emr-reducer-output-for-my-completed-job-should-be-on-s3-but-no">examples</a> of people writing EMR output to HDFS, but I haven't been able to find examples of how it's done.  On top of that, <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UseCase_Streaming.html" rel="nofollow noreferrer">this documentation</a> seems to say that the --output parameter for an EMR streaming job <em>must</em> be an S3 bucket.</p>
  
  <p>When I actually try to run a script (in this case, using python streaming and mrJob), it throws an "Invalid S3 URI" error.</p>
  
  <p>Here's the command:</p>
  
  <pre><code>python my_script.py -r emr \
   --emr-job-flow-id=j-JOBID --conf-path=./mrjob.conf --no-output \
   --output hdfs:///my-output \
   hdfs:///my-input-directory/my-files*.gz
  </code></pre>
  
  <p>And the traceback...</p>
  
  <pre><code>Traceback (most recent call last):
    File "pipes/sampler.py", line 28, in &lt;module&gt;
      SamplerJob.run()
    File "/Library/Python/2.7/site-packages/mrjob/job.py", line 483, in run
      mr_job.execute()
    File "/Library/Python/2.7/site-packages/mrjob/job.py", line 501, in execute
      super(MRJob, self).execute()
    File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 146, in execute
      self.run_job()
    File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 206, in run_job
      with self.make_runner() as runner:
    File "/Library/Python/2.7/site-packages/mrjob/job.py", line 524, in make_runner
      return super(MRJob, self).make_runner()
    File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 161, in make_runner
      return EMRJobRunner(**self.emr_job_runner_kwargs())
    File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 585, in __init__
      self._output_dir = self._check_and_fix_s3_dir(self._output_dir)
    File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 776, in _check_and_fix_s3_dir
      raise ValueError('Invalid S3 URI: %r' % s3_uri)
  ValueError: Invalid S3 URI: 'hdfs:///input/sample'
  </code></pre>
  
  <p>How can I write the output of an EMR streaming job to HDFS? Is it even possible? </p>