title: Python - How to pass a directory as MapReduce input
body: <p>I've written a simple MapReduce example in Python. If the input was a file, for example <code>text</code> file, for running the code we simply use the following pattern: <code>cat &lt;data&gt; | map | sort | reduce</code>, for example in my case it is: <code>cat data | ./mapper.py | sort | ./reducer.py</code> and all things work right.</p>
  
  <p>But I changed my mapper and reducer to read data from a <code>directory</code> that contains <code>.gz</code> files. So I should pass the <code>path of the directory</code> as input. I test the following terminal command <code>cat dat/ | ./mapper.py | sort | ./reducer.py</code> while the directory containing data is <code>dat/</code>, but I faced error: </p>
  
  <pre><code>cat: dat/: Is a directory
  Traceback (most recent call last):
    File "./mapper.py", line 9, in &lt;module&gt;
      for filename in glob.glob(sys.stdin + '*.gz'):
  TypeError: unsupported operand type(s) for +: 'file' and 'str'
  </code></pre>
  
  <p>How can I pass a directory as input to Mapreduce in Python?</p>
  
  <p>The following is my code:</p>
  
  <p>mapper.py</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  #import timeit
  import glob
  import gzip
  
  QUALITY = '01459'
  MISSING = '+9999'
  for filename in glob.glob(sys.stdin + '*.gz'):
      f = gzip.open(filename, 'r')
      for line in f:
          val = line.strip()
          (year, temp, q) = (val[15:19], val[87:92], val[92:93])
          if temp != MISSING and q in QUALITY:
              print " %s\t%s" % (year, temp)
  </code></pre>
  
  <p>reducer.py</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  
  max_val = -sys.maxint
  key = ''
  for line in sys.stdin:
      (key, val) = line.strip().split('\t')
      max_val = max(max_val, int(val))
  print "The last IF %s\t%s" % (key, max_val)
  </code></pre>