title: Efficiently generating a document index for a large number of small documents in a large file
body: <p><strong>Goal</strong></p>
  
  <p>I have a very large corpus of the following format:</p>
  
  <pre><code>&lt;entry id=1&gt;
  Some text
  ...
  Some more text
  &lt;/entry&gt;
  
  ...
  
  &lt;entry id=k&gt;
  Some text
  ...
  Some more text
  &lt;/entry&gt;
  </code></pre>
  
  <p>There are tens of millions of entries for this corpus, and more for other corpora I want to deal with.</p>
  
  <p>I want to treat each entry as a separate document and have a mapping from words of the corpus to the list of documents they occur in.</p>
  
  <p><strong>Problem</strong></p>
  
  <p>Ideally, I would just split the file into separate files for each entry and run something like a Lucene indexer over the directory with all the files. However, creating millions and millions of files seems to crash my lab computer.</p>
  
  <p><strong>Question</strong></p>
  
  <p>Is there a relatively simple way of solving this problem? Should I keep all the entries in a single file? How can I track where they are in the file for use in an index? Should I use some other tool than separate files for each entry?</p>
  
  <p>If it's relevant, I do most of my coding in Python, but solutions in another language are welcome.</p>