title: Processing Large Binary Files with Hadoop
body: <p>I know there have been similar posts on here but I can't find one that really has a solid answer.</p>
  
  <p>We have a Hadoop cluster loaded with binary files. These files can range anywhere in size from a few hundred k to hundreds of mb. </p>
  
  <p>We are currently processing these files using a custom record reader that reads the entire contents of the file into each map. From there we extract the appropriate metadata we want a serialize it into JSON. </p>
  
  <p>The problem we are foreseeing is that we might eventually reach a size that our namenode can't handle. There is only so much memory to go around and having a namenode with a couple terabytes of memory seems ridiculous. </p>
  
  <p>Is there a graceful way to process large binary files like this? Especially those which can't be split because we don't know what order the reducer will put them back together? </p>