title: Efficiently Iterating Over A Large MongoDB Collection
body: <p>I am writing a large data reader that created almost a fifty million documents inside of mongodb, and in counting (+2million a day). Basically in order to store data into mongodb, i need to first check if the record already exist. To do this, I use a combination of Redis and a bloom filter. Because the bloom filter is completely in memory, I need to load it every time the program is started, thus bringing me to my problem. </p>
  
  <p>Currently I use an iterator (mongo cursor) to iterate through every single document stored inside the specific mongo collection. This is an example:</p>
  
  <pre><code>private final static MongoDatabase database = client.getDatabase(Constants.MONGO_SCHEMA_NAME);
  private final static MongoCollection collection = database.getCollection(Constants.MONGO_SITE_COLLECTION_NAME);
  
  public void load(){
      MongoCursor cursor = collection.find().iterator();
      while(cursor.hasNext()){
          Document document = (Document)cursor.next();
          if(document != null){
              Filters.add(document.get(Constants.MONGO_ELEMENT_URL_FIELD_NAME).toString());
          }
      }
  }
  </code></pre>
  
  <p>Where add is a function invoked to add the url to both redis and the bloom filter. I use both redis and a bloom filter because a bloom filter is faster than redis and redis is faster than mongo. This guarantees I never have to search through mongo to see if a document already exist (which is significantly slower than redis from my understanding)</p>
  
  <p>I was wondering if it is possible to "split" a collection temporarily or create multiple cursors to iterate through the collection via parallel processing to speed up the time it takes to load my filters (Redis and Bloom Filter). So instead of reading a collection from A -> Z, I would have ideally 26 cursors to read A->B and B->C and ... and M->N and ... and Y->Z all concurrently. </p>
  
  <p>Thank you! :)</p>