title: Fast, line-wise "grep -n" equivalent for Unix directory structure
body: <p>I am trying to create a web interface for searching through a large number of huge configuration files (approx 60000 files, each one with a size between 20 KByte to 50 MByte). Those files are also updated frequently (~3 times/day).</p>
  
  <p>Requirements:</p>
  
  <ul>
  <li>Concurrency</li>
  <li>Must identify the line numbers for each matching line</li>
  <li>Good update performance</li>
  </ul>
  
  <p>What I have looked into:</p>
  
  <ul>
  <li><strong>Lucene</strong>: To identify a line number, each line must be stored in a separate Lucene document, each containing two fields (the line number and the line). This makes updates hard/slow.</li>
  <li><strong>SOLR</strong> and <strong>Sphinx</strong>: Both based on Lucene, they have the same problem and do not allow for identifying the line number.</li>
  <li><strong>SQL table with a fulltext index</strong>: Again, no way to show the line number.</li>
  <li><strong>SQL table with each line in a separate row</strong>: Tested this with SQLite or MySQL, and update performance was the worst of all options. Updating a 50 MB document took more than an hour.</li>
  <li><strong>eXist-db</strong>: We converted each text file to XML like this: <code>&lt;xml&gt;&lt;line number="1"&gt;test&lt;/line&gt;...&lt;/xml&gt;</code>. Updates take ~5 minutes, which somewhat works but we are still not happy with it.</li>
  <li><strong>Whoosh</strong> for Python: Pretty much like Lucene. I have implemented a prototype that sort-of works by dropping/re-importing all lines of a given file. Updating a 50MB document takes about 2-3 minutes using this method.</li>
  <li><strong>GNU id utils</strong>: Suggested by sarnold, this is blazingly fast (50MB document is updated in less then 10 seconds on my test machine) and would be perfect if it had pagination and an API.</li>
  </ul>
  
  <p>How would you implement an alternative?</p>