title: Efficiently loading collection array from csv with pymongo
body: <p>The csv file has several fields, but only two are of interest here: <code>ID</code> and <code>CODE</code>. There can be duplicates ( several rows with the same <code>ID</code> and <code>CODE</code> values. The objective is to get a collection like</p>
  
  <pre><code> {
  "_id" : ObjectId("5001"),
  "CODE" : [ code1 , code2 , code3 ],
  "ID" : "10001"
   }
   {
  "_id" : ObjectId("5002"),
  "CODE" : [ code3 ],
  "ID" : "10002"
   }
   {
  "_id" : ObjectId("5003"),
  "CODE" : [ code2 , code3 ],
  "ID" : "10003"
   }
  </code></pre>
  
  <p>where the <code>CODE</code> arrays contain unique elements, and there are no duplicated <code>ID</code> documents.</p>
  
  <p>The csv file is large ( <code>cut | sort | uniq</code> isn't an option ). My current approach is to create a dict <code>_u</code> of unique <code>ID</code> values, then create an indexed collection as such:</p>
  
  <pre><code> for ID in _u:
       ID_obj = { "ID"   : ID , 
                  "CODE" : [] }
       z = db.my_collection.insert( ID_obj )
  
   z = db.my_collection.ensure_index( "ID" , unique = True )
  </code></pre>
  
  <p>Then I iterate through the rows of the csv file making sure that the <code>CODE</code> array only keeps unique values:</p>
  
  <pre><code> z = db.my_collection.update({ "ID" : ID } , {"$addToSet":{ "CODE" : csv_code_value }}, True )
  </code></pre>
  
  <p>What is the most efficient way to load the csv data into <code>my_collection</code>? </p>
  
  <p>The csv data itself has 00's of columns, but the relevant structure is:</p>
  
  <pre><code> ID,CODE,DESC,DATE
   10003,code2,second description,10/7/2010
   10001,code1,first description,10/1/2010
   10003,code3,third description,10/8/2010
   10003,code3,third description,10/9/2010
   10001,code1,first description,10/2/2010
   10001,code1,first description,10/3/2010
   10001,code2,second description,10/4/2010
   10002,code3,third description,10/6/2010
   10001,code3,third description,10/5/2010
  </code></pre>