title: Hadoop: Sending Files or File paths to a map reduce job
body: <p>supposed I had N files to process using hadoop map-reduce, let's assume they are large, well beyond the block size and there are only a few hundred of them. Now I would like to process each of these files, let's assume the word counting example.</p>
  
  <p>My question is: What is the difference between creating a map-reduce job whose input is a text file with the paths to each of the files as opposed to sending each of the files directly to the map function i.e. concatenating all the files and pushing <strong>them into different mappers [EDIT]</strong>.</p>
  
  <p>Are these both valid approaches?
  Are there any drawbacks to them?</p>
  
  <p><strong>Thanks for the prompt answers I've included a detailed description of my problem since my abstraction may have missed a few important topics:</strong></p>
  
  <p>I have N small files on Hadoop HDFS in my application and I just need to process each file. So I am using a map function to apply a python script to each file (actually image [I've already looked at all the hadoop image processing links out there]), I am aware of the small file problem and the typical recommendation is to group the smaller files so we avoid the overhead of moving files around (the basic recommendation using sequence files or creating your own data structures as in the case of the HIPI). </p>
  
  <p><em><strong>This makes me wonder can't we just tell each mapper to look for files that are local to him and operate on those?</em></strong></p>
  
  <p>I haven't found a solution to that issue which is why I was looking at either sending a path to the files to each mapper or the file it self. </p>
  
  <p>Creating a list of path names for each of the collection of images seems to be ok, but as mentioned in the comments I loose the data locality property. </p>
  
  <p>Now when I looked at the hadoop streaming interface it mentions that the different pieces communicate based on stdin and stdout typically used for text files. That's where I get confused, if I am just sending a list of path names this shouldn't be an issue since each mapper would just try to find the collection of images it is assigned. But when I look at the word count example the input is the file which then gets split up across the mappers and <strong>so that's when I was confused as to if I should concatenate images into groups and then send these concatenated groups just like the text document to the different mappers or if I should instead concatenate the images leave them in hadoop HDFS and then just pass the path to them to the mapper</strong>... I hope this makes sense... maybe I'm completely off here...</p>
  
  <p>Thanks again!</p>