Take the 2-minute tour ×

If I don't specify the number of mappers, how would the number be determined? Is there a default setting read from a configuration file (such as mapred-site.xml)?

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

Adding more to what Chris added above:

  1. The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps.

  2. The right level of parallelism for maps seems to be around 10-100 maps/node, although this can go upto 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.

  3. You can increased number of Map task by modifying JobConf's conf.setNumMapTasks(int num). Note: This could increase the number of map tasks, but will not set the number below that which Hadoop determines via splitting the input data.

Finally controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the InputFormat determines the number of maps.

Read more: http://wiki.apache.org/hadoop/HowManyMapsAndReduces

share|improve this answer
seems like a direct copy of the hadoop wiki. maybe this answer should be paraphrased or you can make it more specific to the question. –  jayunit100 Nov 6 '12 at 1:36
add comment

It depends on a number of factors:

  • Input format and particular configuration properties for the format
  • for file based input formats (TextInputFormat, SequenceFileInputFormat etc):
    • Number of input files / paths
    • are the files splittable (typically compressed files are not, SequenceFiles are an exception to this)
    • block size of the files

There are probably more, but you hopefully get the idea

share|improve this answer
Thanks Chris. If I specify a number, it isn't necessarily going to be honored by Job tracker? –  kee May 15 '12 at 1:30
With the new (.mapreduce packaged) API, this is completely ignored. The old API may do - look through the source for o.a.h.mapred.FileInputFormat.getSplits() (for the release of hadoop you are using) –  Chris White May 15 '12 at 1:45
add comment

Your Answer


