title: Logging insert, update and delete operations in MongoDB
body: <p>I am new to MongoDB. I am looking for a way to log all operations that insert, update, or delete documents for the purposes of maintaining a history of changes. So for instance, I would like to know when a specific field in a specific document was updated, what it was updated to, and what it was before. This is for an ASP.NET C# application using the C# MongoDB driver, so I don't mind whether the solution just involves Mongo itself or C# code.</p>
  
  <p>Example: 
  <code></p>
  
  <pre><code>ID  |Timestamp |Operation|ObjectID |         PrevValue           |          NewValue
  -----------------------------------------------------------------------------------------------
  2153|1411390359|    i    |245245...|            null             |{name: "John Smith", age: 35} 
  2154|1411390471|    u    |245245...|         {age: 35}           |          {age: 36}
  2155|1411390478|    d    |245245...|{name: "John Smith", age: 36}|            null
  </code></pre>
  
  <p></code>
  It doesn't have to follow this exact format, but it should make it easy to track history. </p>
  
  <p>I took a look at the MongoDB oplog, but it doesn't seem well-suited for this kind of work. In order to identify changes to a document for example, it looks like I have to look up two entries (the most recent update entry for the document, and the previous document entry, which may be an update or an insert) then compare each field in the document to check which one(s) were changed and what their original values were. I would rather simply have one entry that contains all this information for the sake of efficiency, in case many queries are needed to examine changes to many documents. I want to make sure using the log is as painless as possible. It would also be ideal if the log was stored in a MongoDB collection with a document for each entry (like with oplog) to make it easy to query. </p>
  
  <p>Is there a feature in MongoDB (other than oplog) that can accomplish something like this? If not, is there a good third party tool that can? Or do I have to implement it manually? </p>
  
  <p><strong>EDIT</strong>: A few more details: </p>
  
  <p>1) Another problem with oplog is that is capped, so it deletes the oldest entries once it runs out of space. I would like to maintain a list of <strong>all</strong> history no matter how old it is. </p>
  
  <p>2) The contents of the log will be displayed in some way to the user, who may request to see the history of changes. This is not just for internal records for DB admins, so it needs to be queryable on demand. </p>
  
  <p><strong>EDIT #2</strong>: I have been brainstorming some solutions to this problem, but all of them still have some flaws, so I would appreciate further input if someone has a better idea. </p>
  
  <p><strong><em>Possible Solution 1: Maintain a separate document for each version (with a version #) and a flag if it has been deleted.</em></strong></p>
  
  <p>Implementation: Add a "version" field to each document. Version 1 is the initial state (represents an insertion), and subsequent versions represent updates, which are placed in separate newly created documents, rather than updating old ones. Version -1 would mean the document is now considered "deleted". Obviously this doesn't involve an actual log file. </p>
  
  <p>Problems: Requires a lot of space and costly to query if I need to load historical change data for many documents at once. Also difficult to determine which specific fields were changed when, unless I also include a "Changes" field from the previous version, which also adds more space. Can also be messy and costly to query through newest versions if older versions need to be searched too (no way to tell which version is the current one). Even with yet another flag saying that it's current, I would then need to make sure the flag gets updated all the time as needed. </p>
  
  <p><strong><em>Possible Solution 2: Maintain revision history for each document as a sub-document field</em></strong></p>
  
  <p>Implementation: Each document has a "RevisionHistory" field that contains each revision with a version number. Easier to look up historical data for individual documents</p>
  
  <p>Problems: Harder to perform history lookup across multiple documents, and any API calls to perform updates are more complicated due to the new field. In particular, I will need to show all changes within the past X hours (X is supplied by user) as part of implementation, which would require scanning all documents if this approach is used. </p>
  
  <p><strong><em>Possible Solution 3: Add a manual Log() function call from C#</em></strong></p>
  
  <p>Implementation: Every time the MongoDB C# Driver API is called with a new operation, have the programmer also add a line calling a special function to handle logging to the log file. </p>
  
  <p>Problems: Dependent on the programmer actually remembering to add this function call manually and to call it properly. One simple mistake causes the logging system to become unreliable. </p>
  
  <p><strong><em>Possible Solution 4: Create a wrapper function for the API operations from C#</em></strong></p>
  
  <p>Implementation: Programmers don't call the API directly, but call a function that handles the work for them. This requires a lot of work, handling every possible type of relevant API call at the wrapper function level, but is consistent and reliable. Good way to abstract away DB access so something doesn't get broken with the wrong low-level API call. Somebody not working on the DAL can just call the wrapper function and the function figures out the details, including logging. </p>
  
  <p>Problems: The wrapper function is more challenging to develop as it needs to account for all possible API calls that modify a document. </p>
  
  <p>Right now I'm leaning towards solution 4. But again, if there is an easier way to do it, I am interested in hearing it. </p>