title: Hadoop Pig - Optimizing Word Count
body: <p>In the canonical pig wordcount example, I'm curious how folks approach optimizing the condition where grouping by word could result in a bag with many (many) elements.</p>
  
  <p>For example:</p>
  
  <pre><code>A = load 'input.txt';
  B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word;
  C = group B by word;
  D = foreach C generate COUNT(B), group;
  </code></pre>
  
  <p>In line C, if there is a word, let's say "the", that occurs 1 billion times in the input file, this can result in the reducer hanging for a very long time while processing. What can be done to optimize this?</p>