title: What is the correct mongo schema design for a job queue?
body: <p>I would like to implement a job queue in Mongo.  The entire software system is based around Mongo so it seems natural and potentially a good fit.</p>
  
  <p>The <strong>jobs</strong> collection stores each job state as a document.  I imagine this to be an uncapped collection based of my query needs.  The <strong>jobs</strong> documents look like the following:</p>
  
  <pre><code>{
      "_id" : ObjectId("50a6742ee4b0a9a1c2cb4fd4"),
      "type" : "archive_job",
      "state" : 2,
      "priority" : 1,
      "timing" : {
          "submitted": ISODate(...),
          "running": ISODate(...),
          "completed": ISODate(...),
          "failed": null,
          "cancelled": null
      },
      payload: {
         ...job-specific JSON...
      }
  }
  </code></pre>
  
  <p>The typical access patterns for the <strong>jobs</strong> collection will be:</p>
  
  <ul>
  <li>find unprocessed jobs to execute based on <em>type</em>, <em>state</em>, <em>priority</em> and possibly a range query on <em>timing.submitted</em> greater than the previous read time</li>
  <li>find all processed (completed, failed, cancelled) jobs</li>
  <li>find all unprocessed (submitted, running) jobs</li>
  <li>find specific job by _id and retrieve its <em>payload</em> (when state is <em>running</em>)</li>
  </ul>
  
  <p>The bulk of the queries will be to find unprocessed jobs that need execution.  Would it be worth while to move <em>payload</em> to a <strong>jobs_payload</strong> collection so the document size does not vary greatly in the <strong>jobs</strong> collection?</p>
  
  <p>Will the large amount of processed (completed, failed, cancelled), versus unprocessed jobs, eventually increase the working set memory required for the <strong>jobs</strong> collections?  Will the access times for unprocessed jobs to execute be slower even with the right indices?</p>
  
  <p>What are my alternatives and trade-offs I can make with the schema design?</p>