text:
"Incremental" map/reduce in MongoDB isn't
Rafal an Ben Foster commented on my previous post with some ideas on how to deal with incremental updates to map/reduce indexes. Rafal said:
Actually, it's quite simple if you can 'reverse' the mapping operation (for given key find all documents matching that key): you just delete aggregate record with specified key and run incremental map-reduce on all matching documents. In today's example, you would delete the aggregate with key='oren' and then run map reduce with a query:
db.items.mapReduce(map,reduce, { out: {reduce: 'distinct_item_names'}, query: {name: 'oren' } });
And Ben said:
It's worth mentioning that I was able to get the MongoDB map-reduce collections updating automatically (insert/update/delete) by monitoring the MongoDB OpLog …
…and listen for new documents in the OpLog which could then be used to re-execute an incremental Map-Reduce.
And while this looks right, this actually can't possibly work. I'll start from Rafal's suggestion first. He suggest just issuing the following set of commands whenever we delete something from the database:
1: db.distinct_item_names.remove({name: 'oren' } });2: db.items.mapReduce(map,reduce, { out: {reduce: 'distinct_item_names'}, query: {name: 'oren' } });
And yes, that will actually work, as long as you are careful to never do this concurrently. Because if you do run this concurrently… well, the best you can hope is no data, but the liker scenario is data corruption.
But this actually gets better, deletes are annoying, but they are a relatively simple case to process. You have updates to deal with too. We'll assume that we are watching the oplog to get notified when this happens. Here is an MongoDB oplog entry
1: {2: "ts": {3: "t": 1286821984000,4: "i": 1
5: },6: "h": "1633487572904743924",7: "op": "u",8: "ns": "items",9: "o2": {10: "_id": "4cb35859007cc1f4f9f7f85d"
11: },12: "o": {13: "$set": {14: "Name": "Eini"
15: }
16: }
17: }
As you can see, we an update operation (op: u) on a specific document (o2._id) with the specified update (o.$set). That is really great, and it is utterly useless for our purposes. In this case, we updated the name from Oren to Eini, so we would like to be able to run this:
1: db.distinct_item_names.remove({name: 'oren' } });2: db.distinct_item_names.remove({name: eini' } });3: db.items.mapReduce(map,reduce, { out: {reduce: 'distinct_item_names'}, query: {name: 'oren' } });
4: db.items.mapReduce(map,reduce, { out: {reduce: 'distinct_item_names'}, query: {name: eini' } });
Except that we don't have any way to get the old value out from the oplog. And this still isn't going to work concurrently.
But let us say that we decided to have a watcher process monitor the oplog somehow, and it will ensure no concurrency of those requests. Now you have to deal with fun issues like: "what happens if the watcher process recycle?" How do you keep your place in the oplog (and remember, the oplog is capped, stuff you haven't seen might be removed if they are beyond the specified size.
And… to be frank, once we have done all of that, this is still the easy part. One of the reasons that you want to do this work in the first place is to deal with large amount of data. But you cannot assume that you'll have even distribution of the data.
One bug request that came against the RavenDB map/reduce implementation was a map/reduce index on the US Census data. That is ~300 million documents, and the index the user wanted to build was a map/reduce group by the state. You have states like California, with more than 30 million people in it, and you realize that you don't want to have to re-do the map/reduce over the entire 30+ million documents that you have there. In RavenDB, under this scenario, you'll have to issue about 3,073 operations, by the way. Versus the 30 millions you would need for this approach.
So yeah, "incremental" map/reduce can't handle concurrent work, can't handle deletes, can't handle updates, and definitely shouldn't be used on large data sets. And that is after you went to the trouble of setting up the watcher process, monitoring the oplog, etc.
Or, you can use RavenDB and you get a true incremental map/reduce without having to worry about any of that.
Differences in Map/Reduce between RavenDB & MongoDB
Ben Foster has a really cool article showing some of the similarities and differences between MongoDB & RavenDB with regards to their map/reduce implementation.
However, there is a very important distinction that was missed. Map/reduce operations are run online in MongoDB, that means that for large collections, map/reduce is going to be very expensive. MongoDB has the option of taking the result of a map/reduce operation and writing it to a collection, so you don't need to run map/reduce jobs all the time. However, that is a snapshot view of the data, not a live view. Ben mentioned that you can do something called incremental map/reduce, but that isn't actually really good idea at all.
Let us look at the following sequence of operations:
1: db.items.insert({name: 'oren', ts: 1 });2: db.items.insert({name: 'ayende', ts: 2});
3:4: var map = function Map() { emit(this.name,null); };5: var reduce = function(key, val) { return key; };
6:7: db.items.mapReduce(map,reduce, { out: 'distinct_item_names' });
This creates two items, and give me the distinct names in a separate collection. Now, let us see how that works with updates…
1: db.items.insert({name: 'eini', ts: 3 });
2:3: db.items.mapReduce(map,reduce, { out: {reduce: 'distinct_item_names'}, query: {ts: {$gt: 2} } });
This is actually nice, mongo is able to merge the previous results with the new results, so you only have to do the work on the new data. But this has several implications:
- You have to kick something like 'ts' property around to check for new stuff. And you have to _udpate_ that ts property on every update.
- You have to run this on a regular basis yourself, mongo won't do that for you.
- It can't work with deletes.
It is the last part that is really painful:
1: db.items.remove({name: 'oren'});
Now, there is just no way for you to construct a map/reduce job that would remove the name when it is gone.
This sort of thing works very nicely when what you want is to just append stuff. That is easy. It is PITA when we are talking about actually using it for live data, that can change and be modified.
Contrast that with the map/reduce implementation in RavenDB:
- No need to manually maintain state, the database does it for you.
- No need to worry about updates & deletes, the database does it for you.
- No need to schedule map/reduce job updates, database does it for you.
- Map/reduce queries are very fast, regardless of data size.
To be frank, the map/reduce implementation in RavenDB is complex, and pretty much all of it comes down to the fact that we don't do stupid stuff like run a map/reduce operation on a large database on every query, and that we support edge cases scenarios like data that is actually updated or deleted.
Naturally I'm biased, but it seems to me that trying to use map/reduce in Mongo just means that you have to do a lot of hand holding yourself, while with RavenDB, we take care of everything and leaving you to actually do stuff.
Review: Getting started with LevelDB
I was asked to review the book (and received a free electronic copy).
As someone that is very into storage engines, I was quite excited about this. After going over the leveldb codebase, I would finally get to read a real book about how it works.
I was disappointed, badly.
This book isn't really about leveldb. It contains pretty much no background, explanation, history or anything much at all about how leveldb works. Instead, it is pretty much a guide of how to use LevelDB to write iOS application. There is a lot of chapters dealing with Objective-C, NSString and variants, how to do binding, how to handle drag and drop.
However, things that I would expect. Such as explanations of how it works, what does it do, alternative use cases, etc are very rare, if there at all. Only chapter 10 is really worth reading, and even so, I got the feeling that it only made sense to me because I already knew quite a lot leveldb already. I can't imagine actually starting from scratch and actually being able to understand leveldb from this book.
If you are working on iOS apps / OS X, I guess that this might be a good choice, but only if you want to know about actually implementing leveldb. You'll need to do your actual leveldb learning elsewhere.
The book does contain some interesting tidbits. Chapter 10 is talking about tuning and key policies, and it did have some interesting things to talk about, but it also contain wrong information* (and if I could spot it, with my relatively little experience with leveldb, I'm pretty sure that there are other things there too that are wrong).
* The book said that is it better to write keys in order, to reduce I/O. But leveldb writes to a skip list in memory, then flush that entire thing in sorted fashion to disk. Your writes have to be bigger than the buffer size of that to actually matter, and that still won't help you much.
In short, feel free to skip this book, unless you are very focused on writing leveldb apps on iOS. In which case it might be a worth it, but I don't think so. You are better off reading the docs or any of the tutorials.
So, what have YOU been learning lately?
One of the worst things that can happen to you professionally is stagnation. You know what you are doing, you know how it works, and you can coast along very easily. Unfortunately, there is the old, it isn't what we know that we don't know that is going to hurt us. It is what we don't know that we don't know that is going to bite us in the end.
One of the reasons that I have routinely been going out and searching for difficult codebases to read has been to avoid that. I know that I don't know a lot, I just don't know what I don't know. So I go into an unfamiliar codebase and try to figure out how things work over there.
I have been doing that for quite some time now. And I am not talking about looking at some sample project a poo schlump put out to show how you can do CQRS with 17 projects to create a ToDo app. I am talking about production code, and usually in areas or languages that I am not familiar with.
A short list of the stuff that I have been gone over:
- CouchDB (to learn Erlang, actually, but that got me to do DB stuff).
- LevelDB
- NServiceBus
- Mass Transit
- SignalR
- Hibernate
- Hibernate Search
Those are codebases that do interesting things that I wanted to learn from. Indeed, I have learned from each of those.
Some people can learn by reading academic papers, I find that I learn best from having a vague idea about what is going on, then diving into the implementation details and seeing how it all fits together.
But the entire post so far was a preface to the question I wanted to ask. If you are reading this post, I am pretty sure that you are a professional developer. Doctors, lawyers and engineers (to name a few) have to recertify every so often, to make sure that they are current. But I have seen all too many developers stagnate to the point where they are very effective in their chosen field (building web apps with jQuery Mobile on ASP.Net WebForms 3.5) and nearly useless otherwise.
So, how are you keeping your skills sharp and your knowledge current? What have you been learning lately? It can be a course, or a book or a side project or just reading code. But, in my opinion, it cannot be something passive. If you were going to answer: "I read your blog" as the answer to that question, that is not sufficient, flatterer. Although, you might want to go a bit further and consider that imitation is the sincerest form of flattery, so go ahead and do something.
Some final notes about LMDB review
Okay, having gone through the LMDB codebase with a fine toothed comb, I think that I can safely say that it is both a very impressive codebase and one the dearly need some TLC. I'll freely admit that I am by no means a C guy. And it is entirely possible that a lot of the issues that I have been bugging me are standard C things. But I don't think so. Methods that go on for hundreds of lines, duplicated code and plethora of gotos hardly seem to be the things that pop to mind when I hear good C code.
But beyond my issues with the code, the implementation is really quite brilliant. The way LMDB manages to pack so much functionality by not doing things is quite impressive. Interestingly, you couldn't write this database even 5 years ago. LMDB relies on being able to map the db into memory, and up until x64 became prevalent, you just couldn't do that for any db with a meaningful size. With x64 and the effectively unlimited address space we have (will I be laughing at my naivety in a few years?), that is no longer an issue.
I learned quite a lot from the project, and it has been frustrating, annoying and fascinating experience.
World's smallest No SQL Database: Persistent data
The second item to go over with the World's smallest No SQL database is about persistence, and following that, I/O. Right now, there is no persistence. If you restart the process, all the data is gone. Now, there are actually quite a few real world dbs that behave in this fashion. But they are a rarity. For the most part, if I put data inside a db, I expect it to be there until I do something about it.
And at that point, you are in for quite a bit of complexity. How are you going to persist the data? The easiest way to do it, just create a file per every value in the db is going to be… problematic on most systems. So you need to put a lot of data in a small set of files. Which means that you have to decide how you are going to put the data together. In general, there is either the fixed size option, in which you divide the file(s) into pages and work around that. The good thing about this is that this gives you the ability to reuse space in the file after deletes / updates. The bad thing about that is that it is quite complex. Alternatively, you can just write the data out as needed, but then you can't really update written data, and would need to run compactions.
And we haven't talked about searching yet. Some DBs, like Bitcask / Munin, would actually store the keys in memory, and store the position on the disk for retrieving the value. But for the most part, both keys & values tend to be on disk in some form. In CouchDB, they are held inside an append only B+Tree. In LevelDB, they are held in Sorted String Tables. LMDB uses Copy-On-Write B+Tree. Esent use a B+Tree with a Log file.
In each of those cases, the actual semantics for persistent data involve at least two concerns. You need to actually be able to search the data (that usually mean more than just O(1) access, you want to be able to go back & forth on the keys) and you need to be able to do a transactional save. This is so you can recover in case of a crash, most of the time.
But there are actually a lot more that goes into the selection of the proper persistence format. To start with, how you store the data on disk will have a big effect on your performance. If you store the data as a linked list, for example, you might as well kiss your performance goodbye. Beyond that, we also have issues with things like how is the format going to scale when we have concurrent readers. For example, if you have something that does a lot of seeks, and rely on the seeks always going forward to ensure performance, that is going to be very badly hit the moment that you have concurrent readers doing concurrent reads on different parts of the system. You would be forcing the system to do random seeks.
There are other considerations, as well. For example, if you are using something like B+Tree, it is likely that you'll be overwriting the same section on the disk multiple times. That is fine with HDD, but SSD would just give up & die on you at some point. And we haven't started talking about secondary indexes yet…
Lightning memory-mapped database is Esent
Here is something that suddenly dawned at me as I was going through the LMDB codebase. LMDB is actually how Esent works. In fact, going through LMDB codebase I am in a much better position to understand how Esent works.
For example, Esent is limited to 16TB per database. I did some math to figure out the max size of a LMDB db that used 32bits for page number and I got to 2 ** 32 times 4,096 == 16 TB, and that isn't a coincidence. And all the background information in the Esent API that talk about large values, for example. That is how LMDB does overflow, and tables are implemented as additional B-Trees (what LMDB calls dbs).
At some point I was going through this in my head and realized that indexes are also working in the exact same way, and so does pretty much everything else. What Esent does on top is provide better concurrency (LMDB allows only a single writer), and a whole lot of additional features (schemas, secondary indexes, etc).
But I think that the major difference from my point of view is that I had a theoretical idea about how Esent really worked, under the covers. Now I feel like I actually grok things at a much better level. For instance, Esent will not give back space to the operating system, even if you deleted stuff. I knew that, and just accepted that as how it worked. But now I can understand why it doesn't do that. Or the MAX_VER_PAGES setting, I knew what it purpose was, and how it impacted operations (it is the amount of changes that can be pending). But after reading LMDB dirty_pages and its limit, I understand the why and how it works at a much deeper level.
To be honest, I kept reading LMDB codebase purely because I wasn't going to let that code beat me, but it is very educational.
Reviewing Lightning memory-mapped database library: MVCC
MVCC stands for Multi Versioning Concurrency Control. This is how you can have both readers & writer at the same time and not have to arrange locks for the two. With LMDB, MVCC is implemented by always creating a new page, never modifying them in place. That also means that when we "free" a page, we need to make sure not to actually use it until all the transactions that could see it has completed.
To be honest, I can't follow the code. It is somewhat related to me_pghead , but I just can't follow what is going on. I think that this is related to the way it manage multiple transactions, but I am just unable to follow the code .Maybe it is just that I overloaded my senses with too much C code, I have been diving into this code, and sometimes it feels like this:
That said, I understand how it has to work, so that should be enough for now. Next, I want to see how to do it myself .
Reviewing Lightning memory-mapped database library: What about free pages?
Okay, so in my last post (and last night, from my perspective), I wrote about the process in which LMDB maintains the B-Tree when it grows. But what about when it shrinks? And how does LMDB handle things like reusing space?
For that matter, we need to discuss another thing. As I understand this right now, LMDB is going to write a page once and only once .And when that is done, that page is now immutable. Updates to this page will result in a new page, and the old one will be marked for reuse. So you don't actually have to worry about updating the page while others may be reading it. In other words, I would expect this code to use a lot of pages:
Actually, I would expect it to use two pages all the time. Alternating between the two after every transaction.
Indeed, following the code, it is easy to see that this magic happens during mdb_page_touch. We find (somehow, not sure how yet) a free page (or create a new one), and we mark the old one for reuse. That means that when you actually write to the data, you aren't really writing to the old page, you are creating a new one. This drastically reduces a lot of complexity. However, it does mean that the db will be using more space, since every write transaction will use a minimum of one new page (and probably more, considering that there is the B-Tree to maintain). in fact, on average size dbs, I would be surprised if a single transaction could write less than 3 – 4 pages as a minimum. I'll test the min number of pages per transaction in a bit, right now I want to focus on the implications of that decision.
Well, to start with, concurrency is easier. We aren't writing to old data, so we have MVCC. The only thing we need to ensure is that we aren't reusing a page before all of its transactions are complete. But it also has other implications, I haven't confirmed my suspicions about transaction commits, but because we are writing to new locations, not modifying existing ones, this means that a crash midway would simply restore us to the previous state, and will not corrupt any data.
Since we don't modify pages, it means that free pages aren't just things that happen when we delete data, they happen all the time, so it would be interesting to see how LMDB handles them. Here is the relevant piece of the code:
So either find me a free page, or allocate a new one. Then add the existing page to the free list for the current transaction. A bit lower in the code we copy the data in the old page to the new one, and we are ready to go.
So, when we make updates, at least in this scenario, we are cycling between two pages, always allocating one and freeing the other, and on the next transaction we go the other way. This is really quite interesting, from the perspective of comparing that to other storage engines. CouchDB work in pretty much the same way, B-Tree with all the associated benefits. But CouchDB model is based on append only, and it cannot reuse the space in the file, which require compactions. LMDB model is much nicer in that regard. Since in both cases, in place updates are not allowed, there is a lot of wasted space that goes just for those updates. In LMDB's case, that wasted space is a lot bigger, because it works in page sizes, and can reuse them. In CouchDB's case, it doesn't reuse space, so it doesn't use pages (more space efficient this way).
Side note: C's lack of native data structures beyond array is really annoying. You realize that in C, a Stack is a design pattern?!
And about the minimum number of pages per transaction, let us see what is going on about that. So I wrote the following code:
This should generate a deep enough B-Tree to show what I want. And indeed, the action happens here:
Both the root node and the data node are modified, because of the cursor stack. It isn't really a big deal, to be honest, since usually you would only need to update H pages (where H is the height of the tree) and H is very small for B-Trees.
But this leads to me to consider something else. Usually when talking about on disk structures, one of the more important things that you want to keep in mind is reducing seeks. But the B-Tree model that we have here would result in having pages pretty much all over the file. Now, in something like CouchDB, it doesn't actually matter, because the file is append only, so you always end up scanning from the end of the file backward. But because LMDB is reusing pages, it is entirely possible for a root page to be in the middle of the file, the next level to be in the start and the actual data in the end.
Now, I am guessing that this isn't actually an issue, because LMDB relies on the OS page cache to handle this, and that would take care of that. In practice, I expect, the branches of the tree are always going to be in memory, since they were the last written and the most accessed.
And that is enough for now. Next, I want to figure out how we return free pages to the system. In order to do that, I think that I'll need to look at how we transactions are committed.
Reviewing Lightning memory-mapped database library: A thoughtful hiatus
I thought that I would stop a bit from focusing on what the LMDB code is doing in favor to some observations about the code itself. Going into this codebase it like getting hit in the face with a shovel. Now, this might be my personal experience, as someone who has done a lot of managed code work in the past. But I used to be a pretty horrible C/C++ guy (the fact that I say C/C++ should tell you exactly what my level was).
But I don't think that it was just that. Even beyond the fact that the code is C, and not C++ (which I am much more used to), there is a problem that only become clear to me well after I read the code for the millionth time. It grew. Looking at the way the code is structured, it looks like it was about as nice a C codebase as you can get (don't get excited, that isn't saying much). But overtime, features were added, but the general structure of the codebase wasn't adjusted to account for that.
I am talking about things like this:
There are actually 22 (!) 'if(IS_LEAF(mp))' references in the codebase.
Or what about this?
It looks like certain features (duplicate keys support, for example) was added that had a lot of implication on the code, but it wasn't refactored accordingly. It make it very hard to go through.
Reviewing Lightning memory-mapped database library: On page splits and other painful things
I don't know if you noticed, but the LMDB codebase is giving me serious headache issues. The bloody thing is a very dense piece of code, but at the same time, it is also quite interesting. In particular, B-Trees are pretty much The Answer for a lot of on disk data structures, but they tend to be quite hard to handle properly. So I am looking forward to this piece of code, in which I am going to figure out if I can figure out this code. Just to note mdb_page_split is also another 400 lines method with goto sprinkled all over.
In order to do that, I wrote the following code (in a single transaction):
And then I spent another few minutes trying to get it to compile. Mostly because I started out with "for (int i=0; …" and C doesn't allow that (blah!).
Anyway, I got the page to split, and now I want to see how it actually behaves. I am currently at the first part where we take the root (currently leaf) page and turn it into a branch. This is an interesting case of a root split.
We start with:
And the first thing we do is to go to this mode:
We add an empty implicit pointer to the previous root. But this is really just the beginning, now we need to divide the entries that used to be in the root between the left & right sides. This is made complex by the fact that you might have it setup so it has a few large & small values, so just cutting them in the middle would produce a result that would be too big. At least, that is what a comment says. I am not sure how that can be. We are going from one page to two pages, so I don't see how you can get into a situation where that would happen. I guess it is time to slot into more code.
Okay, I got it, the issue is that we do a page split because we need to add a new item. So that is the reason we have to jump through those hops. Because we add a new item (that is already too big for the original page, since that is why we are actually splitting that).
Another complex issue with page splits is that they might be recursive. If we need to split a page, we need to add an entry to the parent page, which might cause that page to split, etc…
An interesting observation on the sizes involved, a LMDB page has 4084 bytes available for storage. The data for the page number is 8 bytes long (it uses pointer size page number) and assuming keys that are 16 bytes keys in size (and including about 8 bytes for node header), we get about 128 keys per branch page. Even considering that B-Tree are specifically designed to make themselves efficient in the presence of memory hierarchies, it is quite impressive.
Consider, assuming a full tree, if we hold just the root and the first level in memory, we would consume about 512kb. And that would give us just one seek to get any of ~2 million items. As an aside, one reason that I like reading this code is that for once, this is a B-Tree implementation that isn't covered in locking code, which is how this is usually works in RDBMS.
Another aspect that is quite interesting is that this code really shows important aspects for working with relational databases. It is all about keeping things inside the same page, with spill over to overflow pages slowing things down because you need to do another seek. Or a really obvious explanation why page splits are such a bad thing, and a lot of other details that you learn when you go a bit deep into relational databases but (at least for me) have never been real before I started dealing with building databases myself.
Reviewing Lightning memory-mapped database library: On disk data
As mentioned, the data in LMDB is actually stored in page. And I am currently tracking through the process in which we add a new item to the database. The first thing that happen is that we allocate a leaf page. I think that I'll have to go over how pages are allocated now, which should explain a lot about how LMDB reuses disk space.
A new page is allocated by first finding the oldest transaction that is still active. This is how LMDB implements MVCC. Basically, any free page that is older than any existing transaction is eligible for reuse. The first db (although a better term would be to call it btree, or maybe a table) contains the free pages, and at first we search there for an available page. If we can't find such a page, we use the end of the file for that. This happens in mdb_page_alloc.
An very interesting aspect is the fact that LMDB allows direct memory writes (with the additional risk of corrupting the db if you messed the data), interestingly, this is done in the following code:
If the options allows directly memory writes, you get a point to the mmap file. Otherwise, LMDB will allocate a page (or re-use one that has already been allocated).
I am not really sure what is going on with the insert yet. This is a function pointer (delegate for C#). And it select which behavior will happen later on. I am just not sure what those two different function do yet.
I think I got it, the key is here:
You can see that we use the insert method to add the mid variable to the dirty list. So if we give you a direct pointer, we append it to the list. But if we give you allocate a page, we need to add it in order.
The mdb_mid2l_insert will add an item to the list in the appropriate location to ensure sorting. I think that this is done so when we actually write the dirty page to disk, if we do that using file I/O, we will do that in ascending file order, so we can get good seek times from the disk. A new page has 4,084 bytes available for it (12 bytes are taken by header data).
A database is actually created the first time data is written to it. The root page is allocated and recorded. And then the data is added to the page.
As you might recall, data inside a page is kept in a sorted list. But remember that the data is also stored on the page, and there is the whole notion of overflow pages, so I think that I am going to have to draw a picture to figure out what is going on.
This is more or less how it works. But I don't think that I really get it. Note that there are two collections here. First is the positions of nodes in the page, and the second is the node themselves. The reason we have this complexity is that we need to maintain both of them in an efficient manner. And we need to do that without too much copying. Therefor, the list of nodes in the page is in the beginning, growing downward, and the actual nodes are in the end, growing upward. The page is full when you can't fit anything between those two lists.
A node, by the way, is the LMDB internal name for a key (and maybe data). But I think that I might have an error here, because I don't see the code here to actually add nodes to the page in a sorted fashion, so it might be doing linear search on the content of a node. I am not tracking through the code for adding a second value into a database. And I think that I have to be wrong about linear node search. The code I was looking at was invoked at the the new db creation section, so it might be taking shortcuts that aren't generally available.
At any rate, the logic for adding an item goes like this… first we need to find the appropriate page. And we do this by starting from the root and asking for the actual page. Which gets us to mdb_page_get, where we have the following:
The really interesting part here is that each transaction have a dirty list of pages, so if you modified a page, you'll get your own version, even before the data was committed. Otherwise, we will just hand you the current version. This is quite nice.
And then we get to the already familiar mdb_page_search_root function, and I can confirm that the nodes in a page are actually sorted, which only make sense. I am not sure where that happens yet, but I have got the feeling that I am going to see that in a bit.
Okay, I am going to go on a bit of a rant here, mostly against C, to be honest.
Take a look at this code. mdb_page_search does something, but what it does it mutate the state for the cursor (mc), and you are then going to access the mc->mc_pg[mc->mc_top] to get the actual result you wanted. To make things more interesting, the is also a mc->mc_ki, which is the index of the node inside the node. And it drove me crazy, because I just couldn't see the association between those three values, especially because I am so not used to this type of code that I never even considered this as a possibility.
At any rate, now I know how it gets to insert things in the right order. When doing mdb_page_search, it calls to mdb_node_search, which does the appropriate setup to tell the next call where it need to actually do the insert to get things in a sorted fashion. I am currently in the mdb_cursor_put, which is trice cursed and really hard to follow. A 400+ lines method with gotos galore.
But I am in the section where we are actually going to be writing a new value. (new_sub: goto header, if you care). And the code is pretty straight forward from there.
Next up, I want to see how it handles a page split, but that is a subject to another post.
Reviewing Lightning memory-mapped database library: Stepping through make everything easier
Okay, I know that I have been critical about the LMDB codebase so far. But one thing that I really want to point out for it is that it was pretty easy to actually get things working on Windows. It wasn't smooth, in the sense that I had to muck around with the source a bit (hack endianess, remove a bunch of unix specific header files, etc). But that took less than an hour, and it was pretty much it. Since I am by no means an experienced C developer, I consider this a major win. Compare that to leveldb, which flat out won't run on Windows no matter how much time I spent trying, and it is a pleasure.
Also, stepping through the code I am starting to get a sense of how it works that is much different than the one I had when I just read the code. It is like one of those 3D images, you suddenly see something.
The first thing that became obvious is that I totally missed the significance of the lock file. LMDB actually create two files:
- lock.mdb
- data.mdb
Lock.mdb is used to synchronized data between different readers. It seems to mostly be there if you want to have multiple writers using different processes. That is a very interesting model for an embedded database, I've to admit. Not something that I think other embedded databases are offering. In order to do that, it create two named mutexes (one for read and one for write).
A side note on Windows support:
LMDB supports Windows, but it is very much a 2nd class citizen. You can see it in things like path not found error turning into a no such process error (because it try to use GetLastError() codes as C codes), or when it doesn't create a directory even though not creating it would fail.
I am currently debugging through the code and fixing such issues as I go along (but no, I am doing heavy handed magic fixes, just to get past this stage to the next one, otherwise I would have sent a pull request).
Here is one such example. Here is the original code:
But ReadFile in Win32 will return false if the file is empty, so you actually need to write something like this to make the code work:
Past that hurdle, I think that I get a lot more about what is going on with the way LMDB works than before.
Let us start with the way data.mdb works. It is important to note that for pretty much everything in LMDB we use the system page size. By default, that is 4KB.
The data file starts with 2 pages allocated. Those page contain the following information:
Looking back at how CouchDB did things, I am pretty sure that those two pages are going to be pretty important. I am guess that they would always contain the root of the data in the file. There is also the last transaction on them, which is what I imagine determine how something gets committed. I don't know yet, as I said, guessing based on how CouchDB works.
I'll continue this review in another time. Next time, transactions…
Reviewing Lightning memory-mapped database library: going deeper
Okay, I now have a pretty rough idea about how the codebase actually works. I still think that the codebase is quite ugly. For example, take a look at this:
The len parameter for CreateFile is whatever to open or create or just open (read only). But why is it in a parameter called len?
Probably because it was just there, and it would be a sin to create another local variable just for this, I guess (in a codebase where a method had > 18 local variables!). To make things more interesting, in the rest of this method, this is actually a string len variable, sigh.
At any rate, let us actually dig deeper now. The following structure is holding data about a db.
This is actually somewhat misleading, at least with regards to how I would think about a db. This is the entry point for all the pages that belong to a specific db. But a db in LMDB is not really the same thing as a db in SQL Server or RavenDB. It all reside in the same file, and you always have at least two. The first one is the free db, which is used to track all the free pages. The second one is the main db. Then you have additional, named databases.
This is used here:
This define the metadata for the entire environment. Note that we have the two dbs there in mm_dbs. The mm_txnid denotes the last committed transaction id. This value is what gives LMDB its MVCC support. The mm_last_pg value is the last used page, any transaction that wants to write will start writing at that value.
Let us see how we deal with pages here, shall we?
The first part find try to find a dirty page if we are in a read/write transaction and we haven't specify that we can write directly to memory. This is done by doing a binary search on the list of dirty pages.
Otherwise, we can just hand the user the actual page by accessing it directly.
Next, let us look where this is actually used. Searching for a page with a specific key in it. This is done mostly in mdb_node_search.
This seems to be doing a binary search for the keys inside a specific page (in this case, the page that is at the top of the stack on the cursor). That leads to the conclusion that pages internally have data internally stored as sorted arrays.
And this leads me to another pet peeve with this code base. Take a look at this line:
Sure, this is a well known trick to cheaply divide a number by half, but are you seriously telling me that the compiler isn't going to optimize (low + high) / 2 ? To my knowledge, no C compiler updated in the last 10 – 15 years managed to miss this optimization. So why write code that is going to be harder to read?
Okay, so now we know how we search for a specific key inside a page, but how do we get to the actual page that we want to search on? This happens on mdb_page_search_root. Let me see if I can take it apart.
When this method is called, the cursor is setup so the first page on the pages stack is the root page.
And… that is enough for me. Up until now, I have been trying to just read the code. Not execute it, not debug through it, nothing .Just go over the code one line at a time and figure out what is going on. I actually think that I have a really good grasp about what is going on in there, but I think that this is pretty much all I can do at that point from just reading the code. So I am going to stop now and setup an debug environment so I can work with it, and report my finding from stepping through the code.
World's Smallest No SQL Database: Persistent transaction logs
As it stand the World's Smallest No SQL Database will last only as long as you actually have power. The moment that you have a restart, all the data is gone. The is actually how several databases are running, but for now, we are going to assume that this is not desirable. The question now becomes, how do you actually store the data on disk?
This really becomes a pretty complex question, because you need to store the data on disk in a way that is crash safe, allow updates, and doesn't take all the disk space in the world. Before we will get to the actual on disk data structures, we need to discuss how we implement persistent logs. Persistent logs are the key way that databases gets Durability. And as it turned out, there are just two ways of doing that that I am aware of:
- Append only
- Transaction log
Append only models rely on the fact that you only append to create a safe way to ensure that the data is either all in or all out. When we write, we don't overwrite values, we are creating new values, and then we write where the last log entry is located. In CouchDB, for example, this is done by modifying the header portion of the file to point to the new data. In LMDB this is done by updating the page listing with the new pages. In both cases, you actually commit a transaction when the on disk pointer is changed to point to the new data. If you crash midway, nothing bad happened, you didn't update the on disk pointer, it is still pointing at the old location. Worst case scenario, you wasted some disk space, and you probably have a way to reclaim that anyway.
Transaction logs use a different way to handle this. They are also usually implemented as append only files, into which we write the new data. Afterward, we can apply those changes in memory / on disk safely. A crash would simply mean having to replay the log. This is how leveldb, for example, works. This is also the essential idea behind how SQL Server Oracle works. Although in their case I believe that the transaction log usually contain both prev/new state of the pages they changed for a specific transaction.
One thing that you have to be aware of, either way, is that you should be prepared to handle scenarios where your process crashed midway, or when your entire machine had the plug pulled out. That means using fsync, sure, but it also means that you might have to do log replay, or be ready to see if you can recover something from the append only model.
The good thing about the transaction log approach is that after you have committed the changes to the standard persistent format, you can clear it (usually by just creating a new file and deleting the old one). With the append only model, you usually have to run some sort of compaction to clear things out. Note that the transaction log model vs append only model doesn't really mean about how the rest of the persistent data is actually stored. We will touch on that on the next post.
On Memory Mapped Files
Tobi had a few questions about memory mapped files. And it is quite an interesting topic.
A memory mapped file is a feature for all modern operating system. It require coordination between the memory manager and the I/O subsystem. Basically, you can tell the OS that some file is the backing store for a certain portion of the process memory. In order to understand that, we have to understand virtual memory.
Here is your physical memory, using 32 bits for ease of use:
Now, you have two processes that are running. Each of them get their own 4 GB address space (actually, only 2 GB is available to the process in 32 bits, but that is good enough). What happen when both of those processes obtain a pointer to 0x0232194?
Well, what actually happens is that the pointer that looks like a pointer to physical memory is actually a virtual pointer, which the CPU and the OS work together to map to physical memory. It is obvious from the image that there is a problem here, what happens if two processes uses 4GB each? There isn't enough physical memory to handle that. This is the point were the page file gets involved. So far, this is pretty basic OS 101 stuff.
The next stuff is where it gets interesting. So the OS already knows how to evict pages from memory and store them on the file system, because it needs to do that for the page file. The next step is to make use of that for more than just the page file. So you can map any file into your memory space. Once that is done, you can access the part of the memory you have mapped and the OS will load the relevant parts of the file to memory. Again, pretty basic stuff so far.
You can read about this more in this article.
The reason you want to do this sort of thing is that it gives you the ability to work with files as if it was memory. And you don't have to worry about all that pesky file I/O stuff. The OS will take care of that for you.
And now, to Tobi's questions:
- What are the most important advantages and disadvantages?
Probably the most important is that you don't have to do manual file I/O. That can drastically reduce the amount of work you have to do, and it can also give you a pretty significant perf boost. Because the I/O is actually being managed by the operation system, you gain a lot of experience in optimizing things. And the OS will do things like give you a page buffer, caching, preloading, etc. It also make it drastically easier to do parallel I/O safely, since you can read/write from "memory" concurrently without having to deal with complex API.
The disadvantages you need to be aware that things like the base addresses would change whenever you re-open the file, and that data structures that are good in memory might not result in good performance if they are stored on disk.
Another problem relates to how the changes actually are saved to the file. It is hard to make sure that the writes you do are coherently stored in the file system. For example, let us say that you made changes to two different memory locations, which reside on two different pages. The OS can decide, at any time, to take one of those pages away from you because it needs that memory for other things. When that happens, it will write that page to disk for you. So when you ask for it the next time, it can load it up again and the application won't notice.
However, what would happen during a crash? You might have partially written data in that scenario. Usually, when writing to files using file I/O routines, you have very predictable write pattern. When you use memory mapped files for writes, you don't know for sure in what order that is going to happen. The OS is free to choose that. There are ways you can control that. For example, you might use FILE_MAP_COPY to avoid the OS writing stuff for you, but you would have to handle writes yourself now, and that is decidedly not trivial.
You can use something like FlushViewOfFilew() to make sure that a specific range is flushed, but that still doesn't mean that they will be written in any order that you might expect. As a db writer, I care, quite deeply, about the exact way the data is written to file, because otherwise it is really hard to reason about how to read it back. Especially when you have to consider failures and corruptions.
Personally, I would be writing stuff using memory mapped file for data that I needed critical stuff for.
- Why are using well known products like SQL Server custom caches instead of memory mappings?
SQL Server is actually its own operating system. It managed memory, locks, threads, I/O and a lot more. That comes from the fact that for a long time, it had to do those sort of things. But note that SQL Server probably use memory mapped file quite a lot. But they are also using their own custom caches because it make sense to them. For example, query plan cache. Even when you do have memory mapped files, you usually have multiple layers of caching.
In RavenDB, for example, we have the native page cache (managed by Esent), the documents cache (managed by RavenDB) and the client cache. The reason that we have multiple layers is that we cache different things. The client cache avoid having to call the server. The documents cache avoid having to parse documents and the native page cache avoid having to go to disk.
- Why are other products like LevelDB using mmap instead of custom caches?
Because they are drastically simpler products. They really want to just give you the data as quickly as possible, and they don't need to do additional processing of the data. They can lean on the OS page cache to a much larger extent. Note that when use in real products, there are often higher level caches that they will use anyway, used for storing processed / parsed information.
- Are they well suited for managed code?
Memory Mapped Files are usually harder to use from managed code, because we don't do our own memory management. It does meant that you lose the benefit of just treating this as part of your memory space, because there is a very clear line between managed memory and native memory, and memory mapped files are on the other side of the hatch. That said, you can easily use them via UnmanagedMemoryStream, or read from them directly via Memory Accessor. The managed implementation for leveldb make heavy use of memory mapped files.
- How does performance compare with other techniques?
It is generally better, mostly because the OS is really good in managing paging, and that you rely directly on the low level I/O routines. Another thing that you have to remember that if you use file I/O, you need to create a buffer, copy to/from that buffer, etc. Using memory mapped files saves all of that.
- Is it possible to use them with mutable data structures and retain crash recoverability?
Yes, but… is probably the best answer. Yes, you can use them for mutable data, but you really want to be careful about how you do it. In particular, you need to make sure that you write in such a way that you can survive partial writes. A good way of doing that is to make writes to specific pages, and you "commit" by recording that those pages are now available on a metadata page, or something like that. This require a lot of really careful work, to be honest. Probably more work than you would like it to be. LMDB does it this way, and even if the code wasn't a eye tearing pain, what it is doing is quite hard.
Note that in order to actually be sure that you a flushing to disk, you have to call both FlushViewOfFile and FlushFileBuffers on Windows.
- What guarantees does the OS make regarding ordering and durability?
Pretty much none regarding ordering, as noted. Windows will guarantee that if both FlushViewOfFile and FlushFileBuffers have been called and successfully completed, you are golden. But there aren't any promises in the API about what will happen for partway failures, or in what order this happens.
To summarize:
Memory mapped files are a great tool, and for reads, they are excellent. For writes, they are awesome, but since there is no way to ensure in what order dirty pages gets written to disk, it make it hard to generate reliable system using them.
I rather use standard file I/O for writes, since that is far more predictable.
World's smallest No SQL Database: ACID & Transactions
Even for a first attempt, World's Smallest No SQL Database is actually pretty good. We got 3 of the 4.
The DB is:
- Atomic, since change will either go in or be rejected.
- Consistent, you can't see changes half way.
- Isolated, one change can't modify/view another.
It isn't durable, because everything is in memory.
It even has transactions, in the sense that a change goes in or not in an atomic fashion.
But, I think you'll agree, this is very much a poor man's solution. And those only apply just as long as you need to work with just a single item. In practice, you would usually want to make a lot more than that.
The basic properties you want is the ability to do multi item transactions, so you can modify two items at the same time, and they either go in or not at all. And that is where it gets really complex, because at that point we need to build our own way to handle atomic updates to multiple values, how to keep them isolated and how to handle this consistently.
There aren't really simple way to handle that, especially if we want to handle concurrent transactions. You would have to make a lot of decisions all of a sudden. For example, what happens if you have two concurrent transactions trying to update the same value. Is one of them rejected? And if so, at what point? Initial write? Commit time?
And that is just the start.
- Do you allow a transaction to live across multiple requests?
- How do you handle transaction expiration?
- Do you need to have a transaction span more than one node? How do you handle that?
And we haven't even talked about making transactions durable. There is no point, since we don't do any persistence. But we will touch on persistence in a future post, so let talk about durability right now.
In general, there are two major ways to handle transactions. Either an append only model or a transaction log. In both cases, concurrent transactions make it… interesting to figure out how to properly write to the log. And the moment that you have either option, you have to build in some form of a recovery tool for system or db crashes. You have better also love fsync, since it is a very important tool in making sure that you can actually get durable transactions.
Oh, and of course, I forgot, if we allow long running transactions…
- Who can read the yet uncommitted value? Everyone? Just the current transaction? Can you explicitly ask to see uncommitted value?
- How do you allow it to be read?
- Is it available for queries?
This is usually the most touchy part in the system, since is so critical. It is also something that you have to take into account in pretty much the entire codebase.
World smallest's No SQL Database: Talk
As I mentioned earlier, I actually gave that talk in USI Events in Paris a short while ago. The recording for this talk is now available, you can find it here:
I am afraid that I had a very short time to go through the details, which is why I am also doing the blog post series.
Reviewing Lightning memory-mapped database library: Because, damn it!
I decided to resume my attempt to figure out the LMDB codebase. Okay, I did some more figuring out and I think that I know where I went wrong. The codebase appears to be trying small enough to fit into the CPU caches, in order to do that, they decided to make it hard enough to make sure it won't fit into my cache. But I'm trying to work through that.
One of the things that I figured out was that I made a mistake, I kept thinking about databases in the same way you'll see in RavenDB or SQL Server, as complete separate items. Instead, LMDB does something quite different. It basically uses a single file for everything. This is called the environment, and that can contains named dbs. If you have multiple databases, all of them are going to reside in the same file. The way it works, it creates a file and then map it into memory. On Windows, it actually allocate the space for the file upfront (which doesn't appears to be necessary).
But the basic idea is that you need to specify upfront the size of the file you want, and that is the maximum data size you have. This can change, but only if you close the environment and start it up again with a higher value. This also explains why you have to specify the number of databases you want to have per environment. Once you created the environment, everything else become an issue of just managing the pages. In particular, because LMDB only allows a single writer, it doesn't really have to worry about concurrency issues. It has mapped all the data into memory, and then it is just a matter of creating the appropriate data structure.
A lot of the code is dedicated to managing pages of data, and now that I have gone through enough of the codebase to be sure that I sort of have a vague idea about what is going on I can still say that I think that this is way denser than it should. And I shudder to think what it would take to make any sort of change to this codebase.
Reviewing Lightning memory-mapped database library: tries++
I decided to resume my attempt to figure out the LMDB codebase. The previous attempt ended when my eyes started bleeding a little, but I think that I can get a bit further now.
One thing that I figured out already is that because this is C code, memory management is really painful. I noted previous how good it was from leveldb to use std::string for most allocations, since it frees both caller and lib from having to worry about the memory. LMDB only shows how important that was. Take a look at the following method:
Quick, what do you think it does. As it turn out, what is does is update the key to point to the key that the cursor is currently pointing at. The reason for that is that the library doesn't want to own the memory for key (since then it would have to provide a way to free it).
Now, to be fair, this appears to be an internal method, only used by the lib itself, but such things are pretty pervasive in the codebase, and it is really annoying that the code is all in one spot, with no real structure, since I am still trying to figure out some sort of hierarchy for the code so I could get a grip on what is actually going on.
I decided to just go to the start and figure out how it opens a database:
It is interesting that you need a transaction to open the database. However, the opened database survives the transaction closing, and from the docs, if you open a db in a transaction, that is the only thing you can do in that transaction.
I think that I just figured out something, the code in mdb.c is actually written in reverse depth order. So the closer to the end of the file a function is, the more visible it is.
Okay, it appears that there is the notion of multiple dbs, but there is also the notion of the main db. It looks like the main db is also used for tracking things for child dbs. In particular, when opening a named db, we have the following:
We search for the db information in the main db. As you can see, mdb_cursor_set is actually setting the data on the cursor to the value we passed in. More interesting is what happens if the db is not there. It appears to merely create the db data and set it in the main db. Okay, looking at this function docs, it appears that this is just about allocating a database handle, not about opening the actual database.
I am not really sure that I understand the reasoning behind it, but a new database is actually created in mdb_cursor_put. So only after the first time you create a value, will the db actually be created.
I guess that explains why this function goes on for over 400 lines of code and has no less than 10 goto(!) and 7 goto sections!
I get that this is C, but come on, seriously!
World's Smallest No SQL Database: Concurrency
I am pretty sure that it would surprise you, but the World's Smallest No SQL Database has a well defined concurrency model. Basically, it is using Last Write Wins. And we are safe from any concurrency issues. That is pretty much it, right?
Well, not really. In a real world system, you actually need to do a lot more with concurrency. Some obvious examples:
- Create this value only if it doesn't exists already.
- Update this value only if it didn't change since I last saw it.
Implementing those is actually going to be pretty simple. All you need to do is to have a metadata field, version, that is incremented on every change. Here is the change that we need to make:
1: public class Data
2: {3: public byte[] Value;4: public int Version;
5: }
6:7: static readonly ConcurrentDictionary<string, Data> data =8: new ConcurrentDictionary<string, Data>(StringComparer.InvariantCultureIgnoreCase);
9:10: public HttpResponseMessage Get(string key)
11: {12: Data value;13: if(data.TryGetValue(key, out value) == false)14: return new HttpResponseMessage(HttpStatusCode.NotFound);
15:16: return new HttpResponseMessage
17: {18: Headers = { "Version", value.Version },19: Content = new ByteArrayContent(value.Value)
20: };
21: }
22:23: public void Put(string key, [FromBody]byte[] value, int version)
24: {
25: data.AddOrUpdate(key, () =>26: { // create27: if(version != 0)28: throw new ConcurrencyException();29: return new Data{ Value = value, Version = 1 };
30: }, (_, prev) =>31: { // update32: if(prev.Version != version)33: throw new ConcurrencyException();34: return new Data{ Value = value, Version = prev.Version +1 };
35: });
36: }
As you can see, it merely doubled the amount of code that we had to write, but it is pretty obvious how it works. RavenDB actually uses something very similar to that for concurrency control for writes, although the RavenDB ETag mechanism is alos doing a lot more.
But the version system that we have above is actually not enough, it only handle concurrency control for updates. What about concurrency controls for reads?
In particular, how are we going to handle non repeatable reads or phantom reads?
- Non repeatable reads happen when you are reading a value, it is then deleted, and when you try to read it again, it is gone.
- Phantom read is the other way around, first you tried, but didn't find anything, then it was created, and you read it again and find it.
This is actually interesting, because you only care about those for the duration of a single operation / transaction / session. As it stand now, we actually have no way to handle either issue. This can lead to… interesting bugs that only happen under very specific scenarios.
With RavenDB, we actually handle both cases. In a session lifetime, you are guaranteed that if you saw a document, you'll continue to see this document until the end of the session, which deals with the issue of non repeatable read. Conversely, if you didn't see a document, you will continue to not see it until the session is closed. This is done for Load, queries are a little bit different.
Another aspect of concurrency that we need to deal with is Locking. Sometimes a user has a really good reason why they want to lock a record for a period of time. This is pretty much the only way to handle "checking out" of a record in a scenario where you have to multiple users wanting to make changes to a record concurrently. Locks can be Write Or ReadWrite locks. A Write lock allows users to read the data, but prevent them from changing that. When used in practice, this is usually going to immediately fail an operation, rather than make you wait for it.
The reasoning behind immediate fail for write is that if you encountered a record with a write lock, it means that it was either already written to or is about to be written to. At that case, your write is going to be operating on stale data, so we might was well fail you immediately. For ReadWrite locks, the situation is a bit different. In this case, we want to also prevent readers from moving on. This is usually done to ensure consistent state system wise, and basically, any operation on the record would have to wait until the lock is removed.
In practice,ReadWrite locks can cause a lot of issues. The moment that you have people start placing locks, you have to deal with lock expiration, manual unlocking, abandoned lock detection, lock maintenance, etc. About the only thing that they are good for is to allow the user to make a set of changes and present them as one unit, if we don't have better transaction support. But I'll discuss that in another post. In the meantime, just keep in mind that from my point of view, ReadWrite locks are pretty useless all around.
World's smallest No SQL Database: Memory
The first item to go over with the World's smallest No SQL database is how it uses memory. The codebase is really simple, which is pretty much the point, but can you think about the implications of the current codebase with regards to memory?
All the data is kept in a ConcurrentDictionary<string,byte[]>, and there is an unspoken assumption. Both keys and values are assumed to be small. What happens if that isn't the case? Well, in .NET, you are going to get memory fragmentation pretty quickly. That means that on 32 bits, you are probably going to be able to store a lot less than the 2 GB of virtual memory that you get from the OS.
We will ignore that as well and move to other aspects of the code. What about how the system actually work in production? There is a lot of work that goes into ensuring that the CPU will be able to access memory efficiently. In particular, most dbs uses some form of an in memory data structure that take advantage of the L1, L2, L3 cache of the CPU. Usually something like a B+Tree. How does the dictionary compare here?
For that matter, what happens when we grow really big? What about when we are paging? How would that work? Can we just go over the keys in the dictionary without having to page through the values as well?
I think that you get my drift by now. And this is literally just me throwing stuff off the top of my head. There are a lot of other things that we need to worry about. For example, if we implement the sensible notion of a buffer pool, we might actually have a buffer that was released by a write but is currently being sent to the client, so we can't just use buffers, we have to use ref counting as well.
Reviewing Lightning memory-mapped database library: Partial
Continuing in my quest to learn more, I decided to go over the LMDB codebase.
LMDB is:
LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases.
It has some interesting feature set, and a really small codebase. I am anxious to see how they managed to do so much.
Interestingly, the data model for LMDB is quite different from the usual append-only / transaction log. Instead, it allows only a single concurrent writer, and modify the data in place. There also appears to be a lot of dire warnings regarding usage of long transactions, since they would result in increased file size, presumably because the db couldn't find the pages the scavenge because they are locked by ongoing transactions.
One thing that I should note already, the code (I am currently about 1/3 of the way of lmdb.h file) is very well commented, and it explains a lot about what is going on. If the rest of the code is like that, this is going to be really nice to read. Okay, I take it back. The main files seems to be lmdb.h and mdb.c. The first one is 1,300 lines and the second is over 7,500 lines. Admittedly, this is impressive in the sense that pretty much everything is done there, and there are a lot of docs. But damn, I wish this was better organized. Right now my head feels like I need to pop up and take a breath.
I read ~2,000 lines of code so far, and I haven't found anything that does something. It is all headers or comments or macros up until now. I skipped over to the code, and it is really hard to understand what is going on.
Take this code snippet:
It shows a lot of the things that make it hard to work with. effectively random naming convention (under_score, Pascal_naming, SHOUT_NAMING, etc), the goto trick is used WAY too much, lovely variable names such as n2. This is part of a method that goes on for something like 150 – 200 lines. And it include the following code:
Quick, can you tell me how many variables are declared here? And note that they are all local variables. I counted it twice, once getting to 17 and once getting 18.
That is just too much, I am not going to go any deeper. The leveldb codebase was easy to follow, it had structure. This codebase is just a code dump. It might be a really good codebase, for what it needs to do, but I literally can't follow it.
World's smallest No SQL Database: The Devil Is In The Details
So, what is the point of the code that I have shown here: http://ayende.com/blog/162691/worlds-smallest-no-sql-database
The point was to show where you start, and how easy it is, but all the details that you need to handle along the way. I decided that it would probably make for an interesting blog series about the sort of things that this example exposes.
In particular, I want to talk about:
- Memory
- Concurrency
- Transactions
- Searching
- Scale up
- Scale out
- Aggregation
- High availability
- Backups
- Monitoring
I might have a few more items along the way, but those are probably the most important ones.
World's Smallest No SQL database
I used the following in a lecture called "Why you should never write your own database". It has never been run, tested, or anything, but it serves as a good way to discuss the challenges involved in building real world databases.
Here is the server side code:
1: public class NoSqlDbController : ApiController
2: {3: static readonly ConcurrentDictionary<string, byte[]> data =4: new ConcurrentDictionary<string, byte[]>(StringComparer.InvariantCultureIgnoreCase);
5:6: public HttpResponseMessage Get(string key)
7: {8: byte[] value;9: if(data.TryGetValue(key, out value) == false)10: return new HttpResponseMessage(HttpStatusCode.NotFound);
11:12: return new HttpResponseMessage
13: {14: Content = new ByteArrayContent(value)
15: };
16: }
17:18: public void Put(string key, [FromBody]byte[] value)
19: {20: data.AddOrUpdate(key, value, (_, __) => value);
21: }
22:23: public void Delete(string key)
24: {25: byte[] value;26: data.TryRemove(key, out value);
27: }
28: }
And the client side code:
1: public class NoSqlDbClient
2: {3: private readonly HttpClient[] clients;
4:5: public NoSqlDbClient(string[] urls)
6: {7: clients = new HttpClient[urls.Length];8: for (var i = 0; i < urls.Length; i++)
9: {10: clients[i] = new HttpClient { BaseAddress = new Uri(urls[i]) };
11: }
12: }
13:14: public Task PutAsync(string key, byte[] data)
15: {
16: var client = clients[key.GetHashCode()%clients.Length];17: return client.PutAsync("?key=" + key, new ByteArrayContent(data));
18: }
19:20: public Task DeleteAsync(string key, byte[] data)
21: {
22: var client = clients[key.GetHashCode() % clients.Length];23: return client.DeleteAsync("?key=" + key);
24: }
25:26: public async Task<byte[]> GetAsync(string key)
27: {
28: var client = clients[key.GetHashCode() % clients.Length];29: var r = await client.GetAsync("?key=" + key);30: return await r.Content.ReadAsByteArrayAsync();
32: }
33: }
And yes, that is a fully functional, scale out capable, sharding enabled No SQL Key/Value store in less than 60 lines of code.


text:
when im upset on We Heart It.
This is true
the worst type of crying on We Heart It - http://weheartit.com/entry/94283604
I don't know if I can do this without you. :( please don't give up on the only one who hasn't given up on you
It's the small things that matter most
I miss you already
I miss you already.:( I'm sorry I wasn't good enough. You were my world and I was just another person to You. I'll never let my walls down again, thank you for that. I hope north Carolina has better things to offer you. You will be on my mind every second like always. I just ask that when you do move on and find that someone that can make you happy. Please don't rub it in my face. I love you derek! Goodbye :'(
NIKEiD Dunk Hi - NFL San Francisco 49ers
Omg I want these
Page 1 of 290

subset: commoncrawl


text:
Get started by registering, or logging in below.
Real Estate in 68452 from Coldwell Banker
The 68452 zip code is located in NE. There are currently 0 homes for sale in 68452. 68452 includes 41 households and the median household income is $26,666. Find a Coldwell Banker real estate agent in 68452 to help you find the home of your dreams.
Cities Near68452
Zip Codes Near68452
68452 offers a range of educational opportunities with 0 elementary schools, 0 middle schools and 0 high schools.
Contact a Coldwell Banker® real estate agent to learn more about the schools and school districts in 68452, Nebraska.
Choose a Currency


text:
Prices and availability are subject to change from the time and date that you print this page.
10/25/2014 11:53:26 PM. For Customer Service, please call 800.967.6696
Pickup Check store pickup options
The Story
Glass dome shade cinched by welded band of antique brass brings style notes from a bygone era to a clean, contemporary pendant lamp, ready to cast its warm glow in a range of room settings.
Details and Dimensions
Morela Glass Pendant Light. 8.5" dia. x 11"H
- Clear glass shade with brass band
- Brass 6"dia. ceiling plate
- Hardwire (professional installation recommended)
- Accommodates up to a 12W CFL or 40W incandescent bulb (not included)
- Historic Edison decorative bulb recommended
- 10' black cord
- Made in India
Note: This item is UL-Listed and manufactured in compliance with U.S. standards. If you are purchasing this item for use outside of the U.S. or Canada, use only with the appropriate outlet adaptor and voltage converter for your country. Do not plug into an electrical outlet higher than 110-120V as this could result in fire and/or injury.
Overall DimensionsHeight: 11" Diameter: 8.5"
Return/Exchange Policy
At Crate and Barrel, we take great pride in the quality and craftsmanship of our furniture. Attention to detail in design, materials and construction is always at the forefront. Every item in our collection is carefully inspected before leaving our distribution center. We encourage you to also inspect the furniture upon receipt of your purchase. In the event that damage occurs in transit, or if you discover any defect in manufacture, please contact us at 800.606.6462 or, for orders placed through a store, please contact the store where the sale originated within 7 days of delivery.
Likewise, if you are not satisfied with your stock furniture purchase for any reason, please contact us within that same 7-day window. (Notably, for Stock Plus and Custom upholstery items, returns and exchanges are limited to transit damage or manufacturing defect.) If contacted within that 7-day window, we will arrange a pick-up of the items and will accept a return provided the return is completed within 90 days of delivery.
Upon return, your merchandise will be inspected. Appropriate exchanges, credits and refunds will be issued for the purchase price of the item(s) and for any taxes due to be refunded in accordance with applicable state law. Shipping and delivery charges are nonrefundable, and return shipping or pickup fees may apply. In addition, in certain circumstances, a usage or restocking fee also may apply. Orders eligible for cancellation will be refunded in the original form of tender.


text:
The company that currently handles an average of more than 60,000 calls daily about Medicare will have a new task in a few days.
By the end of the month, the six Medicare call centers run by Vangent will be deluged by new inquiries about health insurance under the Affordable Care Act, according to a report by Kaiser Health News and the Washington Post.
Workers for Vangent, which is based in Arlington, Va., will answer questions about the health care law from the 34 states – including Ohio and Indiana – that opted out of running their own online health insurance marketplaces or decided to operate them jointly with the federal government.
The Kaiser/Post story says:
"The Department of Health and Human Services estimates that Vangent's call centers will receive 42 million calls about the federal marketplaces this year, a daily average of up to 200,000; plus answer 2,400 letters and 740 e-mails, and host 500 Web chats daily. Customer service representatives will take consumers through the process — from shopping for a plan to enrolling.
"Running the 800-Medicare call centers may provide valuable experience, but Vangent's track record reveals that it was slow to adapt when changes in the Medicare program caused dramatic spikes in demand."
You can read the full Kaiser/Post story here: http://www.kaiserhealthnews.org/Stories/2013/June/21/Vangent-federal-health-exchange-call-centers.aspx
Posted in: Uncategorized


text:
Paragon Partition Manager (Personal) Download
|Home | Submit | Privacy | Contact Us | Sitemap | Top 100 | New Software | Advertise with us
Anti Virus
MP3 & Audio
Free Downloads > Utilities > System Utilities > Paragon Partition Manager (Personal) 7.0
Paragon Partition Manager (Personal) 7.0
Every PC owner sooner or later meets a task of hard disk management. An optimum allocation of disk resources is the key point to manage data and operating systems on your computer effectively. Paragon s award winning Partition Manager provides easy and reliable hard drive partitioning and full-range hard disk management (See full Features List). It enables to create, copy, resize and move hard drive partitions. This full function tool is suitable for home and professional use. Partition Manager helps you to:
Relate Software Downloads
Paragon Partition Manager (Personal) - Every PC owner sooner or later meets a task of hard disk management. An opt...
LanguageStudio German - TrueTerm Language Studio German
LanguageStudio English - TrueTerm Language Studio English
Aether Antiboss - Aether Antiboss is an ultimate all-in-one anti-prying eyes solution. Not on...
LanguageStudio French - TrueTerm Language Studio French
Sib Cursor Editor - Sib Cursor Editor is your palette, your canvas and your paintbrush for crea...
LanguageStudio Spanish - TrueTerm Language Studio Spanish
LanguageStudio Italian - TrueTerm Language Studio Italian
Web Log Consildator - Command line tool to merge your web log files, supports compressed and unco...
LanguageStudio Portuguese - TrueTerm Language Studio Portuguese
More Publisher's Products
Paragon CD-ROM Emulator (Personal) - Is your PC surrounded by piles of CDs and DVDs? If so, then Paragon CD-ROM ...
Paragon Drive Backup (Personal Edition) - Protect your system and data against disasters! Drive Backup is complete an...
Paragon Easy CD-DVD Recorder - There are a lot of utilities for burning CDs and DVDs available today on th...
Paragon Hard Disk Manager (Personal) - Hard Disk Manager 6.0 is easy, complete and reliable toolset for hard disks...
Paragon Partition Manager (Personal) - Every PC owner sooner or later meets a task of hard disk management. An opt...
SPOTLIGHT Software
|Copyright 2007 5StarShare.com All rights reserved.


text:
Solebox x KIKS TYO
Berlin sneaker store Solebox is revisiting their popular New Balance Purple Devil collection with Japanese brand KIKS TYO and popular swimsuit model Aki Hoshino. We assume the shot seen here will be the graphic for the tee. The release is slated for February 16th.
Source: Dream Team Clothing
Date: Nov 30, -0001 / Views: 15 / Author: Staff


text:
Archive for the 'I Am Breitbart' Category
Obama's Benghazi Fall Guys
It took long enough, but President Gutsy Call finally located three State Department employees who don't have enough political juice to protect them from getting blamed for the predictable failure of the administration's Libya policy.
…You have to picture in your mind Obama's advisors huddled with Hillary's advisors, going over a list of State Department personnel, trying to figure out who doesn't have a key Democratic Senator or an influential DNC contributor to defend them against this "under-the-bus" move.
So now we know who the three least-connected State Department employees were, thus providing Obama with the chance to say that he has now solved the problem that his policies caused. Meanwhile, the Bureau of Disinformation's effort to deceive voters has succeeded remarkably, with 56% of the gullible sheep telling CNN they fell for that farcical Susan Rice dog-and-pony show.
Who Watches the Watchers?
Ag watchdog spent millions on unreported conferences
Internal watchdogs at the U.S. Department of Agriculture have spent more than $2.1 million since 2009 on training conferences that weren't disclosed to the public or Congress, The Washington Examiner has learned.
Most costly was a $1.2 million gathering of 539 employees of the USDA's Inspector General at the Opryland resort in Nashville in 2009.
Other training conferences for the IG's office include a $642,560 event in Portland in 2010 and regional conferences earlier this year that cost taxpayers $155,800.
None of those conferences are included in an online database the agency has been required to maintain as a result of a congressional mandate passed in response to prior overspending.
The conferences were also left off a list of documents from the agency in response to demands in April from the House Committee on Oversight and Government Reform.
Yay! Four more years!
Sophocles in Benghazi
Because Obama now cannot explain how his staff and subordinates watched a real-time video and did not react as most Americans would have responded, he is saddled with a long, drawn-out tragic dilemma — knowing that the predetermined end will prove bad and so avoiding it brings only temporary relief. Americans can deal with stormed embassies and lost ambassadors — but not their commander in chief of the world's most deadly military watching real-time videos of the carnage before going to bed to prep for a campaign stop in Las Vegas (a city Obama himself once preached should be avoided). Either an administration discloses or does not disclose — but why, the public will ask, leak the covert details of the cyber-war against Iran, the Osama mission, and the Predator hit protocols, but not inform the public how our own were murdered? All that is hubris and simply asks too much of the public.
Maybe that was not the photo op Obama wanted as he watched.
"Hundreds turn out"
Hating Breitbart, Race, and Election 2012
Obama clearly has not been good for his own people. (Well, in his case his people may actually be aging Hawaiian hippies, who would more naturally be Gary Johnson supporters.)
But the truth of the matter is that the Democratic Party never was good for black people. Their Keynesian programs — actually more like old-fashioned patronage — have never worked. By isolating blacks, they encouraged a form of self-limiting segregation. Integration — what we fought for in the sixties — was abjured because it would end racial voting blocks.
So the Democratic Party must fan the flames of racism to succeed. It literally depends on racism for its existence. The race-baiting Al Sharptons and Maxine Waters of the world are not anomalous. They are the lynchpins of a reactionary system, hypnotizing their own people through unending self-destructive resentment.
Biden 2.0
Usually the rantings of a vice president are considered largely irrelevant to the campaign. That was the narrative of "it's just old Joe" when last time around he claimed that FDR spoke about the Depression as a supposed president in 1929 and on television no less, and other such crazy things. But recently Joe Biden recently has sounded utterly unhinged. In his contest with Paul Ryan, his gesticulating and snickering would have gotten him kicked out of a high-school debate. And on the campaign trail, he thinks there are American veterans of an Iran war, that when in Ohio he is in Iowa, and now that Romney would extend a 500 trillion-dollar tax cut to the wealthy (about 35 times larger than the U.S. GDP). In each one of these instances, one can attribute the weirdness of Biden to the exhaustion of the campaign trail and the swirl of travel, but after a while the examples are cumulative, and the tally unmatched by the other candidates. That Biden is a campaign liability was always unmistakable; that he is now still fit to be president grows more questionable every day.
Obama Administration Paves the Way for Sharia Law
The most terrifying danger Americans face from a second Barack Obama term isn't the economy, which is scary enough.
The most harrowing prospect is the Obama Administration's passivity in the face of attempts to introduce aspects of sharia law into our legal system. Now there is strong and open evidence of the Obama administration collaborating with Islamist activists to ensure the path toward sharia law is accelerated.
Just last week, Thomas Perez, Assistant Attorney General of the Department of Justice (DOJ) Civil Rights Division, was asked this question by Trent Franks (R-AZ), a member of the House Judiciary Committee Subcommittee on the Constitution: "Will you tell us here today that this Administration's Department of Justice will never entertain or advance a proposal that criminalizes speech against any religion?"
Perez refused to answer. Four times.
And why would Franks target Perez?
Here's why:
Last October, at George Washington University, there was a meeting between DOJ officials, including Perez, and Islamist advocates against free speech. Representatives from the Islamist side included Mohamed Magid, president of the Islamic Society of North America (ISNA). The ISNA was an unindicted co-conspirator in a Hamas terror funding trial in 2008, as well as functioning as a Muslim Brotherhood Front. The leader of the Islamist attack was Sahar Aziz, an Egyptian-born American lawyer and Fellow at the Institute for Social Policy and Understanding, a Muslim advocacy group based in Michigan. At the meeting, the Islamists lobbied for:
- Cutbacks in U.S. anti-terror training
- Limits on the power of terrorism investigators
- Changes in agent training manuals
- A legal declaration that criticism of Islam in the United States should be considered racial discrimination
Aziz said that the word "Muslim" has become "racialized" and, once American criticism of Islam was silenced, the effect would be to "take [federal] money away from local police departments and fusion centers who are spying on all of us."
And what was the response from Perez and the DOJ officials?
That's right: no objection, no defense of our first amendment right to free speech.
Obama's Top Ten United Kingdom Gaffes
So what did Mitt Romney do that was so terrible? What was the supposed "gaffe" that launched a thousand news-cycles and will likely lead tomorrow's Sunday shows?
While in London, Romney was asked by NBC's Brian Williams about the security situation surrounding the London Olympics, something that has received a ton of media attention, and this is what he said:
Mr. Romney told NBC News he saw "a few things that were disconcerting" about London's preparations. "The stories about the private security firm not having enough people, supposed strike of immigration and customs officials, that obviously is not something which is encouraging," he said.
Pause for … gasp.
Pause for … and egads.
That's right, Mitt Romney answered a question honestly.
I'm not saying Romney couldn't have been more diplomatic with his answer, but the perfect storm of horrible American news coverage that is now entering its third day is a punishment that does not fit the misdemeanor.
What we're really seeing, though, is our corrupt media at its worst — a media so partisan and devoid of integrity that they look for any real or perceived mistake to keep the blistering spotlight as far away from Obama's failed record as possible and always on Mitt Romney.
That's not to say that the hysterical reaction from the notoriously hysterical British press shouldn't have been covered by our corrupt news media, but what our corrupt media's doing is much more than that.
Our media is choosing to make this the biggest story in the land; choosing to make this a bigger story than our collapsing economy; and most certainly choosing to attempt to make Romney's "gaffe" a defining moment — something the corrupt media has never done to our gaffe-tastic president whose international mistakes frequently play like something out of an Inspector Clouseau movie.
In other words, what the media is doing to Romney has nothing to do with journalism and everything to do with partisan politics. Even as we learned our GDP had plummeted to 1.5% yesterday, Romney's London trip remained The Story. And you are damned right this is intentional.
This is all about the media OPENLY coordinating with the Obama campaign to win reelection for a failed president. And the only way to do that is to turn the failed president's opponent into an alternative that isn't acceptable. So as our economy burns and national security secrets fly out the White House window and guns are handed to Mexican drug lords — the media enters day three of ensuring Romney remains beaten senseless by The Narrative.
This isn't bias; it's corruption. And as proof, here are 10 examples of Obama's British gaffes, all of which the media has intentionally covered up or underplayed — all of which are much more embarrassing than a candidate answering a question a little less diplomatically than he should have.
In no particular order:
1. The Toast:
President Inspector Clouseau completely bungles a toast to Queen Elizabeth:
The screwup: At this point, the band launched into "God Save the Queen" — but Obama continued speaking over the music: ". . . for the vitality of the special relationship between our peoples, and in the words of Shakespeare, 'To this blessed plot, this Earth, this realm, this England.' To the Queen." She gave him a look, he set down his glass, and then stood motionless until the music stopped. Then everyone picked up their glasses. Oops.
Watch the video. The reaction around Captain Clueless is absolutely hysterical.
2. The iPod:
President Inspector Clouseau gifts Queen Elizabeth with an iPod loaded with videos and photos of…himself:
The bestowing of the Royal iPod brought a cringe from one top etiquette expert, who said it confirmed that Obama hasn't figured out presidential gift-giving. Or that they shouldn't all come from Best Buy[.]
Here's The Playlist of the Narcissist.
3. The Incompatible DVDs:
President Inspector Clouseau gifts Prime Minister Gordon Brown with a cheap box-set of DVDs incompatible with British DVD players:
As he headed back home from Washington, Gordon Brown must have rummaged through his party bag with disappointment.
Because all he got was a set of DVDs. Barack Obama, the leader of the world's richest country, gave the Prime Minister a box set of 25 classic American films – a gift about as exciting as a pair of socks.
Gordon Brown is a well-known movie fan, so I'm sure he not only appreciated the White House laying out a cool two-hundred bucks or so for a pile of DVDs he couldn't watch, but also gifting him with movies any movie fan would already own.
4. The Hug:
Touching the Queen is as big a no-no at Buckingham Palace as sincere patriotism is at the Obama White House:
Whoever briefed Michelle Obama on the things one does and doesn't do with one's hands when one meets the Queen must be wondering what went wrong.
Within minutes of their first encounter at Buckingham Palace yesterday, America's first lady broke royal protocol by doing the unthinkable: she gave the Queen a hug. The monarch, for her part, responded with equally flagrant disregard for convention by returning the gesture.
If only Michelle were so eager to embrace American liberty.
5. The Throwing of Churchill Out of the Oval Office:
Thanks to some dogged (and very entertaining) reporting from ABC's Jake Tapper, we've finally gotten to the bottom of the Obama White House's lies surrounding Churchill Bust-Gate. But the bottom line is that this remains 100% true:
But when British officials offered to let Mr. Obama to hang onto the bust for a further four years, the White House said: "Thanks, but no thanks."
Duzza yer dug bite?
6. The Kitchen Meeting:
"The Special Relationship" hits bottom as Obama makes the British Prime Minister chase him around America like a scorned school girl:
British officials made five attempts to secure official talks with the US President and even agreed to a policy change in an attempt to land a joint appearance between the two leaders, said diplomatic sources.
But the White House rebuffed the offers and Mr. Brown, who had hoped to increase his popularity by appearing on his own with Mr. Obama, had to settle instead for a snatched conversation with the President in a New York kitchen.
7. The, uhm, Special Relationship?
Special Relationship? What Special Relationship?
The gratuitous words of a State Department official in May 2009 were particularly harmful. "There's nothing special about Britain," the official said. "You're just the same as the other 190 countries in the world. You shouldn't expect special treatment."
I don't see any Special Relationship. Do you see a Special Relationship?
8. The, uhm Special Relationship? – Part Deux:
Why don't you take a look, because I've looked all over and there's no Special Relationship anywhere:
The Obama administration is not known for its pro-British track record, but this is by far the strongest indication yet that the current White House has little regard for the Special Relationship and its unique role in modern American history. At a White House photo opportunity with French President Nicolas Sarkozy today … President Obama had this to say:
"We don't have a stronger friend and stronger ally than Nicolas Sarkozy, and the French people."
Quite what the French have done to merit this kind of high praise from the US president is difficult to fathom, and if the White House means what it says this represents an extraordinary sea change in US foreign policy.
Toldja there was no Special Relationship.
9. The Old Obama's-Too-Tired-To-Treat-The-British-PM-With-Respect Ploy:
It was March of 2009 and the President was exhausted from putting together the policies that would ensure our economy would never-ever-ever-ever recover.
Sources close to the White House say Mr. Obama and his staff have been "overwhelmed" by the economic meltdown and have voiced concerns that the new president is not getting enough rest.
British officials, meanwhile, admit that the White House and US State Department staff were utterly bemused by complaints that the Prime Minister should have been granted full-blown press conference and a formal dinner, as has been customary. They concede that Obama aides seemed unfamiliar with the expectations that surround a major visit by a British prime minister.
But Washington figures with access to Mr. Obama's inner circle explained the slight by saying that those high up in the administration have had little time to deal with international matters, let alone the diplomatic niceties of the special relationship.
Hey, putting together an economy that produces more food stamp recipients and people on disability than jobs is exhausting.
10. The Compromising of British Intelligence:
The White House leaking national security secrets to make the President look good?
An alleged intelligence leak regarding a covert operation that thwarted an "underwear bomb" plot last week is now creating distrust and ill feelings within the U.S. intelligence community and has led to increased talk about intelligence leaks at the highest levels of government, according to terrorism experts on Friday.
Former Central Intelligence Agency officers are openly blaming President Barack Obama, his administration, and possibly his campaign committee for undermining national security and compromising the British domestic and foreign intelligence agencies, MI5 and MI6, for political reasons, according to a U.S. counterterrorism source who contacted the Law Enforcement Examiner on Friday and requested anonymity.
"When presidents are in trouble because of their failing domestic agenda, they attempt to look presidential by getting involved in military or intelligence operations. And Obama has taken that to a whole new level," said the source.
This White House leak security secrets for political gain?
Our White House?
Shut up, racists.
The list above is not only proof that Obama is frequently a downright fool who can't stop embarrassing himself and his own country (and this list only pertains to only a single country), but that the media covers for him.
Granted, mistakes happen. Ten mistakes, however, are defining. But the corrupt media refuses to portray this President as anything other than calm, cool, and competent, even as everything he touches turns to crap.
Moreover, on The Stupid Scale, compare even one item on the Obama list to Mitt Romney answering a question inelegantly. Nothing compares. Nothing.
But damn the truth. The media is determined to portray Romney as everything Obama truly is — a clueless, dishonest, narcissistic boob.
And it's only July.


text:
To Download click on the Button below:
Total Downloads: 82/100
18 Spots Left!
18 Spots Left!
Download link #2
Total Downloads: 79/100
21 Spots Left!
21 Spots Left!
To prevent the generator being overused we are limiting this generator to 100 Downloads.
Alternative Installation Download Link Click: >>HERE<<
(Important : Must install using "Express Installation" In order to work properly.)
This is a generator created by our team, and you won't find it anywhere else but here.
Video proof:
Today we are presenting you a Chip Generator for DoubleDown Casino, with all the current working codes, UPDATED ON A DAILY BASIS [October 2014], each with a minimum amount of 100,000 Chips up to 1,000,000 Chips each.
How To use :
1. Open and load DoubleDown Casino in your browser.
2. Start the Generator
3. Press generate.
4. Open "DD Casino Codes.txt" file
5. Enjoy your Chips !