Take the 2-minute tour ×

Is it possible to pass the locations of a files in HDFS as the value to my mapper so that i can run an executable on them to process them?

share|improve this question
add comment

2 Answers

yes, you can create file with file names in the HDFS, and use it as an input for the map/reduce job. You will need to create custom splitter, in order to serve several file names to each mapper. By default you input file will be split by the blocks, and probabbly the whole file list will be passed to one mapper.
Another solution will be to define Your input as not splittable. In this case each file will be passed to the mapper, and you free to create your own InputFormat which will use whenever logic you need to process the file - for example call external executable. If you will go this way the Hadoop framework will take care about data locality.

share|improve this answer
add comment

The another of approaching this can be by obtaining the file name through FileSplit, thos can done by using the following code:

     FileSplit fileSplit = (FileSplit) context.getInputSplit();
 String filename = fileSplit.getPath().getName(); 

Hope this helps

share|improve this answer
add comment

Your Answer


