title: Indexing 20M of records with python and mongoDB
body: <p>I would like to mention about my little project, and if I'm on track. I need work with all articles from Medline (<a href="http://www.nlm.nih.gov/bsd/licensee/2011_stats/baseline_doc.html" rel="nofollow">http://www.nlm.nih.gov/bsd/licensee/2011_stats/baseline_doc.html</a>). For those are not familiar with Medline database, I add a little information:</p>
  
  <ul>
  <li>There are approx. 20.000.000 of records (83.4 GB disk space), each has many fields and subfields.</li>
  <li>You can download this DB (with license) in XML format.</li>
  <li>These 20M of records are distributed in 653 files.</li>
  <li>Every file has one MedlineCitationSet, and this a set of records (MedlineCitation's).</li>
  </ul>
  
  <p>I want to process these records and get information such as title, abstract... Then I have thought to index these files (or records) with python and mongodb. And I have one option:</p>
  
  <p>I have created a medline parser and for each record is created a JSON entry for mongoDB, and after indexing by pubmedID. Then I can create a function likes get_abstract('pubmedID'):string.</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li>Is it a good idea? (XML parsing --> JSON --> insert and indexing!)</li>
  <li>Can I use GridFS and get chunks equivalents to records for each file? How?</li>
  <li>Do you know other way?</li>
  </ul>