title: Is the number of map tasks determined by the number of nodes in hadoop in case the input is small in size or is part of the hardware idle?
body: <p>I have a basic mapreduce question.
  My input consists of many small files and I have designed a custom CombinedFileInputFormat (which is working properly). </p>
  
  <p>The size of all files together is only like 100 Mb for 20 000 files, but processing an individual file takes a couple of minutes (it's a heavy indexing problem), therefore I want as many map tasks as possible. Will hadoop take care of this or do I have to enforce it and how? In the latter case my first guess would be to manipulate the maximum split size but I am not sure if I am on the right track. Any help greatly appreciated! (suggestions on how to set the split size best in the latter case are also helpful)</p>
  
  <p>Some extra information to be more clear:</p>
  
  <p>There is however another reason I wanted to process multiple files per task and that is that I want to be able to use combiners. The output of a single task only produces unique keys, but between several files there might be a substantial overlap. By processing multiple files with the same map task I can implement a combiner or make use of in-mapper combining. This would definitely limit the amount of IO. The fact is that although a single file has a size of a couple of kilobytes the output of this file is roughly 30 * 10^6 key-value pairs which easily leads to a couple of Gigabytes. 
  I don't think there is another way to allow combining (or in-mapper combining) if you have only one file per maptask?</p>
  
  <p>Regards, Dieter</p>