title: How can I turn off hadoop speculative execution from Java
body: <p>After reading <a href="https://stackoverflow.com/questions/15164886/hadoop-speculative-task-execution">Hadoop speculative task execution</a> I am trying to turn off speculative execution using the new Java api, but it has no effect.</p>
  
  <p>This is my Main class:</p>
  
  <pre><code>public class Main {
  
    public static void main(String[] args) throws Exception {
      Configuration conf = new Configuration();
  
      //old api:
      //conf.setBoolean("mapred.map.tasks.speculative.execution", false);
      //new api:
      conf.setBoolean("mapreduce.map.speculative", false);
  
      int res = ToolRunner.run(conf, new LogParserMapReduce(), args);
      System.exit(res);
    }
  }
  </code></pre>
  
  <p>And my MapReducer starts like this:</p>
  
  <pre><code>@Override
  public int run(String[] args) throws Exception {
      Configuration conf = super.getConf();
  
      /*
       * Instantiate a Job object for your job's configuration.  
       */
      Job job = Job.getInstance(conf);
  </code></pre>
  
  <p>But when I look at the logs I see:</p>
  
  <pre><code>2014-04-24 10:06:21,418 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input paths to process : 16
  2014-04-24 10:06:21,574 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:26
  </code></pre>
  
  <p>If I understand then this means that the speculative execution is still on, otherwise why would there be 26 splits if I only have 16 input files. Am I mistaken?</p>
  
  <p>Note: I believe I use the new api, as I see these warnings in the log:</p>
  
  <pre><code>2014-04-24 10:06:21,590 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
  </code></pre>