lucene/src/main/java/com/twitter/elephantbird/mapreduce/input/LuceneIndexInputFormat.java
@Override
  public List<InputSplit> getSplits(JobContext job) throws IOException, InterruptedException {

    // load settings from job conf
    loadConfig(HadoopCompat.getConfiguration(job));

    // find all the index dirs and create a split for each
    PriorityQueue<LuceneIndexInputSplit> splits = findSplits(HadoopCompat.getConfiguration(job));

    // combine the splits based on maxCombineSplitSize
    List<InputSplit> combinedSplits = combineSplits(splits, maxCombinedIndexSizePerSplit,
      maxNumIndexesPerSplit);

    return combinedSplits;
  }
@Override
  public List<InputSplit> getSplits(JobContext job) throws IOException, InterruptedException {

    // load settings from job conf
    loadConfig(HadoopCompat.getConfiguration(job));

    // find all the index dirs and create a split for each
    PriorityQueue<LuceneIndexInputSplit> splits = findSplits(HadoopCompat.getConfiguration(job));

    // combine the splits based on maxCombineSplitSize
    List<InputSplit> combinedSplits = combineSplits(splits, maxCombinedIndexSizePerSplit,
      maxNumIndexesPerSplit);

    return combinedSplits;
  }
Creates splits with multiple indexes per split
(if they are smaller than maxCombinedIndexSizePerSplit).
It is possible for a split to be larger than maxCombinedIndexSizePerSplit,
if it consists of a single index that is
larger than maxCombinedIndexSizePerSplit.
<p>
All inputPaths will be searched for indexes recursively
<p>
The bin-packing problem of combining splits is solved naively:
<ol>
<li>Sort all indexes by size</li>
<li>Begin packing indexes into splits until adding the next split would cause the split to
exceed maxCombinedIndexSizePerSplit</li>
<li>Begin packing subsequent indexes into the next split, and so on</li>
</ol>