text:
On Building a Stupidly Fast Graph Database
It's pretty clear to computer science geeks that Directed Edge is supposed to be doing groovy things with graphs. In fact our recommendation engine, and some of the things that are unique about our approach to recommendations, are built on our super-fast graph database. When we went live yesterday with the latest version of our recommendations web services, another, much bigger thing happened behind the scenes for us: we cut over to the new version of our graph database.
Every time that Directed Edge gets mentioned in nerdier circles we get a lot of questions about this fabled graph-engine, so we thought we'd indulge our techie friends with some background info.
When we first decided to build the Directed Edge engine, we'd built some in-memory and RDF-store based prototypes to start hashing out the recommendations algorithms, but the RDF stores couldn't keep up performance-wise and staying memory-based obviously wasn't an option for persistent data. So we tried porting things over to SQL-based databases … and got a working version, but it was so painfully slow that we knew we needed to keep looking. The initial version of our web services were built on simply serializing data objects to disk and reading records from index files that we loaded up into memory. This worked and was both fast and memory efficient, but ignored most of the harder problems of a true database.
What are those performance characteristics, exactly? Well, it's fuzzy where the cut-off is, but we're now able to import a graph with 2.5 million nodes and 60 million edges in 12 minutes on a MacBook. Query times for finding related items on a similarly large graph with warm disk-buffers average under 5 ms.
With the latest version of our web services and database, we've gone from having a graph-store to having a proper graph database. We now use straight memory-mapped I/O, disk-based linear-hashing, have mechanisms to address data fragmentation, thread-safe read / write locking and logging / log replaying.
Before we jump into the computer-science-y core of what we're doing, I'll throw down some nerd-cred, and give some perspective to the design decisions made. The design was done by me, Directed Edge's CTO, and I've been looking at algorithms for graph-based similarity for around 5 years now. During the earlier half of the decade I did a series of talks on "contextual webs" to our crew in the Linux world. While I have no academic background in database design, I've been doing low level bit-wrangling for most of the last decade and Directed Edge's data backend is the third key-value store that I've written from the ground up. So, what I'm trying to say is: some of these ideas may be original, many are not, some fly in the face of traditional database design which is likely some mix of novel and naive. Be that as it may, the closest that we've gotten to our performance using off the shelf databases was with BDB with 5% of the throughput of our database.
So, on to geekery. Let's start at the top level of abstraction and work our way down. The impatient may feel free to turn the page at any time since the juicer bits are in the lower levels. Â We've tried to bold the highlights.
The Store: Items, Links, References, Tags and Whathaveyou.
Sitting up on top of the architecture is a C++ class called "Store". Like the rest of the things that sit below the web services (which are in Java), it's written in C++. This is the API that we use to deal with a database. To our recommendations code, this is the database.
Things are pretty simple there, and the API is flexible enough that it's let us swap out the implementation below it to compare other backends. Basically it's this:
- The store is a big collection of items, which can be anything — a user, a product, a web page.
- Items are identified by a string unique identifier which maps to an integer index.
- We can look up an item's index by name really quickly.
- Items have links and references, which are basically a list of things they're connected to and things that connect to them. These are symmetrical — so creating a link from item A to item B, creates a reference from item B to item A.
- Items can also have tags, e.g. "user", "product", "musician", "film" and so on. Tags are non-exclusive, so an item can have as many tags as appropriate.
- Links and references can have weights of 1 to 10 or be unweighted. We use that for ratings. So a rating is just a link from user A to product B with weight of 1 to 10.
- We can also store arbitrary other fields along with these like "text". Really, the only limitations on what can be associated with an item are what we choose to expose in the API here. The lower levels are flexible.
So that's the top level. That's what makes things look like a nice, clean graph architecture without mucking with all of the lower-level storage details. At this level everything's thread-safe too, so the map-reduce pattern that we use when generating recommendations doesn't leave a trail of destruction and SIGBADTHINGSs in its wake.
Things Start Looking Databasey: Rows and Columns
Like traditional databases, our database is organized into rows and columns. An item is just a row. Unlike most databases, there's only one set of rows (only a small lie, which we'll ignore for convenience), and the primary things contained in rows are lists.
There are really no huge surprises here:
- A database has a list of columns, a stack of deleted rows (to recycle indexes) and a row count.
- Adding a row to the database adds a row to all columns.
- There's a flag for if a column is hashed or not, which enables fast lookups and requires unique data values.
- There's a basic column type that just stores byte arrays and a bunch built on top of it for storing things like link and reference vectors, string lists, strings and so on.
- Any field in an item can be empty and all of our column data types have sane values for empty contents, which are returned if the data pointer for a row is set to null.
Column Locking
We've got two types of locks that we use, one surprisingly boring, one a little more creative.
The most important fields in the Directed Edge database are the links and references. They convey things like "which pages are linked on this website" or "which products has this customer bought" or "which news items has this user clicked on". Those lists are of the type IndexVector within our database and have their own special kind of lock.
The rest of the columns use normal read-write locks. We use an array of 8 read-write locks per column. When lock request comes in it is hashed by shifting the index by three bits and then issues a lock on the column. We found this in practice to be faster than keeping a table or tree of locks since maintaining that tree would require a single read-write lock for the tree maintenance and contention for that lock was rather high. Non-IndexVector columns (strings, string lists, and binary blobs, mostly) copy the data when they hold the read lock and release it immediately thereafter, so the locks are held only momentarily.
So, to repeat that — using a simple array of locks per column significantly outperformed keeping a list of locked items, even if it means that items share their locks with many other items.
We'll revisit IndexVector locks again in a little while.
Memory-Mapped I/O … and a Disk Based Malloc?
We're up to my second lie now. See, I promised that I'd walk through the layers of abstraction from the top down and here we've gone ahead and skipped to the end. To understand the data structures that are built on top of the files, it helps to know where that data lives and how we manage it.
We experimented with a few ways of doing I/O, but the one that seemed to consistently win, as well as make life a better place, was doing straight memory mapped I/O. For the non-system programmers among you, UNIX systems allows for files to be mapped to memory addresses. So if we've got an open file called "foo" and we call mmap on its descriptor, then we're able to access the contents of "foo" directly based on the address returned by mmap — we can treat the file just like it's an array of characters in memory.
This has a few advantages. One is that it lets the operating system effectively handle paging in data from the disk. UNIX also provides some handy functions like madvise() and msync() so that you can give the operating system hints on how to do that paging and when to write things back out to disk.
There's also no overhead for accessing disk data since it's mapped directly from the file into the system's address space. Usually when you use things like fread() or fwrite() all of the data has to be copied into the kernel's space, then over to the process's space, then you can do stuff with it. By using mmap we avoid all of that copying mess and the overhead of switching between kernel and user space within syscalls, which is often significant.
There's a good section on advanced I/O in O'Reilly's Linux System Programming, which I stumbled across last week and gives a good introduction to mmap and other I/O strategies.
So, that's what mmap does. How do we use it?
Well, one of the typical problems in deciding how to order disk records is how to manage disk fragmentation. As it turns out, that's one of the problems that memory managers face when trying to decide how to allocate chunks of the system heap. So, we thought, since we're treating the files we use as memory anyway, why not put two and two together?
What we ended up with is an implementation in our memory-mapped file class of malloc, realloc, calloc and free — the classic functions for memory management in C. We patterned our allocation with some modifications for our applications on the algorithms used in glibc's malloc. There's a good, if slightly stale, introduction to them here.
Having those in place lets us allocate blocks of our files just like we're allocating memory and handles file fragmentation rather elegantly. It also has the added bonus of letting us translate normal in-memory algorithms directly to disk based algorithms.
This structure lets us put arbitrary data into a file and gives us back an offset to point to where it's at, which is exactly what we do for things like arrays of indexes and strings and so on, but we still need some way of referencing that data to make use of it in our column types.
File-based Data Structures: Stack, Vector, Linear Hash
Most of our data structures are pretty similar to their in-memory based counterparts. The similarity was in fact one of the reasons that we chose the mmap approach. We have three basic data structures that we use on disk to reference the blobs of data that are inserted and removed with malloc, realloc and friends.
When we're writing these records to disk, we choose whether or not they get their own file by whether or not they can make use of sequential access. Notably, vectors almost always get their own file because they're accessed sequentially and only expand at the end. It'd be a pain in the derriere if we had a 50 MB vector and there was no room immediately after it in the file, so we had to free that whole 50 MB chunk and move it to the end of the file leaving a hole where it had been before. So vectors get to live a solitary life off in their own files. Hashes and stacks on the other hand don't have that problem, their elements tend to be of smaller sizes and more scattered, so they can be mixed in with data without worrying about them fragmenting the file too much as they insert and remove elements.
Stacks are simple and don't merit much mention. We use it to keep track of deleted records so that we can recycle their indexes the next time we create a new row. Yawn.
Vectors, on the other hand, are some of the meat-and-potatoes of the Directed Edge workhorse: the things we use to keep track of offsets in data files. These are different from the similarly named IndexVector. In Directed Edge lingo, an Index corresponds to a unique item with fields and stuff, and an Offset just points to a place in a data file. These vectors are used for Offsets. Actually, these vectors are a template, so they can be used for any data type of any fixed size, but they're most important for offsets.
This is another point that we break from typical database design theory. In a typical database you'd look up a record by checking for it in a B-tree and then go off to find the data pointer for its record, which might have a continuation record at the end that you have to look more stuff up in … and so on. Our lookups are constant time. We hash the string identifier to get an Index and then use that Index to find the appropriate Offsets for its data. These vectors are sequential on disk, rather than using continuation tables or something similar, which makes constant time lookups possible.
Note: Like all things in the real world, "constant time" really means, "constant time in the parts we see, under cooperative conditions, if the operating system doesn't have to do anything like figure out where that disk record actually is, so probably not really constant time, but still about as fast as you can get."
Linear hashing is another important component of our lookup scheme. Linear hashing is a special sort of hash table that basically does dynamic expansion rather than having to reallocate the hash-slot array every time the data gets too big and futz around with reorganizing things. We also having a mechanism for letting our hash-table cooperate with our data files so that it handles inserting and removing data from them in addition to keeping track of where they are.
IndexVector The Great and Its Team of Sneaky Locks
So, now we finally return to the IndexVector. Since you've probably dozed off a couple times in this pile of wordiness, I'll remind you of its function — the IndexVector is what keeps track of links between items. "Bob purchased The Beegees Greatest Hits" is a link between Bob and said album. Bob has an index and the album has an index. In Bob's list of "links", which is an IndexVector, we'll put the value of the Beegee's Greatest Hits.
If you're doing traversal heavy graph-applications, like, say, finding related content, how you deal with those links is your where the rubber hits the road. Most of your performance lies in accessing these links.
So, since we're using mmaped I/O we have no syscall overhead for grabbing this stuff from the disk. We also have no copy that must take place to get the data from kernel space to user space. Two points for us. But what we want to do is to Try Real Hard to make sure that we don't copy it at all in the common case.
So our IndexVector class is really just a pointer to a place on the disk where it can read the data from and then it casts that data into indexes when we do things like someIndexVector[4]. Simple, right? Yeah, mostly. But there's a catch.
You see, IndexVectors are sometimes changed. Like, if Bob buys another album, we've got to update that vector. To make things worse, sometimes they get bigger and get moved to a whole other part of the file where there's room for it now. Unlike our other data types (strings, binary blobs and all that) we access IndexVectors a lot, so we don't want to just copy the data every time there's an access. (We tried, slowed down the whole engine by 40%.) So, we get sneaky.
You see, most of the time we can go about just willy-nilly accessing the IndexVector from its disk position and assume that nothing's changing behind its back. We have a similar array of mutexes (8 of them, to be precise) for our IndexVectors. When a read request comes in, an IndexVector adds itself to a list of active readers and goes about reading stuff from the disk.
Now, the magic happens when a write lock comes in. We don't want our writes or our reads to block on these Very Important Pieces Of Data, so what the write lock does is first copy the old data from that record into memory and then go through the list of active readers and do a swaperoo on the data pointer. The readers are now pointing to an in-memory copy of the data, rather than the disk-based version. It can then go on writing and moving and generally doing whatever it wants with the disk based copy. The lock also keeps a copy of the pre-write version of the data in memory so that all readers that come in and try to get a lock on that record get the in-memory version until the write is done.
This leaves us with somewhat more lax consistency than one would expect from an ACID-compliant database, but for our applications that's fine. Like, if our recommendations-fu has missed the item that you rated a couple milliseconds ago when computing product recommendations for the next page, no planes will crash and no babies will be harmed. On the plus side this means that the only case where a read or write on IndexVectors will have to wait is in the case of two threads trying to write to the same item.
Them's the real guts of our database, but I'll leave you with a couple provisos for extra credit:
Every Bit Is Sacred
Conventional wisdom in modern programming is not to get hung up on using a few bytes here or there and just get on with writing your code. And it's right.
But when you're working on low-level, high-performance I/O, every bit is sacred. The reason is that every bit is extra data is extra page misses and extra page misses means more waiting on the big spinning magnet that is the bottleneck in almost everything that touches it. These things also come out of the woodwork when you're multiplying every extraneous bit by several million. If we were to increase the amount of space used for one of our indexes by one byte storing the wikipedia link structure would take 114 more megabytes.
So we get a little trigger happy with our friendly neighborhood binary operators. We combine weights and indexes into the same field using the upper 4 bits of the item index to store the weight. Our malloc implementation uses 8-byte alignment, so we use the extra 3 bits at the bottom end of of the address for flags. By thinking up clever ways of putting the squeeze on data we have databases that are 40% smaller, which means a lot less page misses and a much faster database.
What About Matrices? I Though Recommender Systems Were Based On Linear Algebra.
The difference between a weighted graph and a sparse matrix is mostly one of visualization rather than of data. We conceptualize a matrix to be something different than a graph — a matrix is a huge field of rows and columns and a graph a sprawling series of interconnections, but in fact, a sparse matrix and a weighted graph are almost the same thing and easily convertible when viewed from a data-perspective.
So, if we're thinking of a user's product ratings as a sparse vector that maps to each product, we can easily model that as a series of weighted edges from the user to each product. In fact, it'd be difficult to produce a tighter format for storing a sparse matrix than a list of connections since we're able to avoid things like pointers to the next used vector dimension and munge the weights into the same integer value that we use to represent a graph edge.
Beyond that, a graph model allows for us to do some more wild clustering than you'd typically get if you're imagining the ratings set as a matrix. We can look at things that are not just one, but a couple levels out in the connection structure to figure out how things are related. This is one of the mechanisms that we use to mitigate the "cold start" problem that is endemic to typical collaborative filtering algorithms.
Al fine
I'm tired of writing, you're tired of reading. If you've got questions or have The Killer Application in mind for such a beast, feel free to drop us a line via comments or mail or ping us on Twitter.


text:
Item number: 88005
Status: sold out!
Scale: 1/72
About the model:
* true to scale diecast model made from metal, smaller parts made from plastic
* fully assembled
* rotating wheels with flexible track (unique for 1/72 diecast models)
* rotating turret and elevatable gun
* historically accurate painted and marked for 7th Pz.Div.
* including protective case for display
* package comes with open window display
* limited production!
About the original:
Around 200 Panzer IV F2 were produced and converted from F1 from March 1942 to July 1942. It had the 8 wheel suspension like its predecessors and was armed with the new long barreled 7,5cm KwK40 L/43 and two MG34. The crew consisted of 5 soldiers.
The Panzer IV Ausf.F2 is distuguishable from the new and unique single baffle muzzle brake and the L/43 gun instead of the longer L/48 seen in later versions.
The engine was the 265HP Maybach HL120TRM which allowed a maximum speed of 42km/h on street. The Panzer IV Ausf.F2 was 6.62m long, 2.88m wide and 2.68m high. It weighted 23 tons.
The 7.Pz.Div. was raised in Oct.1939 and fought 1940 in France, 1941/42 on the Eastern Front and from June 1942 on back in France. From 1943 on the Division was again on the eastern front and fought in Charkow, Belgorod, Tarnopol, Minsk, Western Prussia and surrendered to British Troops near Schwerin in May 1945.

subset: commoncrawl


text:
Small firms appear to have taken the recent hike in VAT in their stride.
The standard rate of VAT climbed from 17.5 per cent to 20 per cent on 4 January amid concerns at the effect it would have on smaller traders.
However, a new survey, conducted by Intuit, a financial software provider, found that a majority of businesses polled (67 per cent) had experienced no adverse impact as a result of the change.
Four out of ten businesses (39 per cent) had opted to shoulder the additional cost of the increase rather than pass it on to their customers.
Taking into account not only absorbing costs but also the extra administrative effort required to manage the tax shift and any loss in income, 70 per cent of respondents nevertheless reported that the new rate of VAT had dented their revenue by no more than £350.
Only 46 per cent said that they had invested more than five hours work in preparing for the changeover.
Asked if they believed that the VAT hike was the best tax increase for dealing with the budget deficit, almost half (44 per cent) agreed.
Just 13 per cent voted for a rise in income tax as a more effective measure.
Pernille Bruun-Jensen, managing director of Intuit, said: "It's reassuring to know that many small business owners have been able to cope with the recent VAT increase despite widespread predictions of doom and gloom.
"As this is the third VAT rate change in less than three years, those business owners who have been around for a while may feel that they have had enough practice to cope with the changes.
"Small Business Britain is a huge driver for the UK's economic recovery, and so it's important that the government continues to listen to and address their needs in order to tackle the national deficit."


text:
Limited Woodland Camo & Black 1LoveIE Raincross New Era 59FIFTY Fitted Cap
Product image 1
Product image 2
Regular price $55.00
Celebrate Riverside, California with this custom 1LoveIE Riverside Raincross New Era 59Fifty fitted cap. With a black crown visor and button, and marvelous gold front logo, this hat comes complete with the signature Riverside City patch. Also included is the year Riverside was founded, embroidered in the back of the hat and of course, the world famous new era flag embroidered on the right side. This hat will turn heads. What a perfect way to show your pride for the city that first embraced us with open arms, Riverside.
- 100% Polyester
- Embroidered New Era Flag Left Side
- California Riverside Patch Right Side
- 1870 Flat Embroidery Back Of Hat
- Limited


text:
The moment I saw this winter project, I knew a lighted garland wrapped around the front door surround would bring all of the winter elements together. A garland attached to that limestone surround some 25 feet off the ground that would describe the entrance to this home is the stuff of fairy tales. I told my client it would take a lot of people and equipment to make this happen. There would be no ladder tall enough and safe enough for an installation at this height. Expecting her to say no, she said yes. Given her go ahead, Templeton Building Company was kind enough to send out four sections of scaffolding, and two of their people to set screws in the mortar over the doorway, and down the sides. Four of my crew met the two from Templeton early one morning.
I was not especially happy about the process. The scaffolding was on wheels, for starters. Really? It got built skyward, section by section, until it was high enough to provide a place to work. Would I want to be up there working? No. Was I worried about each and every one of them climbing this expanded ladder to the sky-yes I was. It only took Matt 40 minutes to set the screws that would hold both the magnolia garland, and the light garland. So far, so good. My crew on the ground delivered the materials to those people who were aloft.
Once the screws were in place, LaBelle made his way up the scaffolding, hauling one end of the magnolia garland. David LaBelle is a leading fabricator for my company, the Branch Studio, and he has no fear of heights, or scaffolding. I cannot imagine how that can be, but he seemed relaxed about it, and I knew he would be entirely capable of wiring on the garlands. As the scaffolding was but 7 feet wide, we would have to install one side of the garland at a time, and then move the scaffolding to the other side.
The top surface of the top layer of scaffolding was only one board wide. LaBelle was not in the least bit concerned. The best part of all of my companies are the people who make things happen. Our group of 20 something has varying skills. The variety of work we are able to do depends on this.
Once the first sections of garland came down the right and left side, I knew the big effort to set lighted garland over this two story doorway was a good idea. The final sections of garland could be attached from a ladder.
Once the magnolia garland was secured, the light garland went up next. The garland is 50 feet long overall, and is comprised of a 100 foot strand of LED C-9 lights, and a 100 foot strand of Lumineo mini LED lights, each folded in half, and twisted together. Frequent zip ties keep the twist from unraveling.
LaBelle was ready to take this light garland up high. The LED lights represent a vast improvement in exterior lighting. This garland is amazingly light weight. LED's now have a quality of light very similar to warm incandescent lighting. They consume very little power, and are economical to run. But the best feature for us is the fact that the light covers are plastic – not glass. They are shatterproof. So no harm if you drop them, or bang them against a brick wall. When first introduced, the big bulb LED lights were very expensive. 1 25′ strand would run 27.00. That same strand is less than 10.00 now.
The lighted magnolia garland over this door is beautiful – day and night. The backs of the leaves echo the color of the brick and wood architectural elements.
From the drive court, the scale of every winter element looks appropriate. From the ground plane to the sky, this winter landscape has a lot going on. That lighted magnolia garland better than two stories up over the front door does not look too big or overly elaborate. It is fairly simple, just sized to fit.
From the far side of the drive court, some rare late day December sun describes a landscape and garden ready for winter. This winter landscape has volume, structure, texture, mass, -and most importantly, lighting. A routine winter day in Michigan is gray.
Once the December light fades, there is a garden to be seen of a different sort. Light your own winter landscape as you wish. I would suggest that any effort you make to light the landscape is well worth the effort.


text:
Couple Indicted In Slaying Of Texas Prosecutors
KAUFMAN, Texas (AP) — A former North Texas justice of the peace and his wife have been indicted on capital murder charges in the slayings of three people, including two prosecutors.
A local grand jury indicted Eric and Kim Williams on Thursday.
The couple is accused of killing Kaufman County Assistant District Attorney Mark Hasse, District Attorney Mike McLelland and McLelland's wife, Cynthia. Hasse was shot outside the county courthouse in January, while the McLellands were killed in their home in March.
Prosecutors allege Eric Williams was the gunman in both cases.
The two were arrested in April for what prosecutors allege was a meticulous plot to avenge Eric Williams' conviction for stealing county computer monitors in 2012.
Eric Williams is being held on a $23 million bond. His wife's bond is $10 million.
Copyright 2013 The Associated Press.


text:
Post written by Leo Babauta.
This fantastic quote summarizes something that I've been trying to focus on recently in my daily life:
"Whatever the tasks, do them slowly
with ease,
in mindfulness,
so not do any tasks with the goal
of getting them over with.
Resolve to each job in a relaxed way,
with all your attention."– Thich Nhat Hanh, Zen Master
In our daily lives, we often rush through tasks, trying to get them done, trying to finish as much as we can each day, speeding along in our cars to our next destination, rushing to do what we need to do there, and then leaving so that we can speed to our next destination.
Unfortunately, it's often not until we get to our final destination that we realize what madness this all is.
At the end of the day, we're often exhausted and stressed out from the grind and the chaos and the busy-ness of the day. We don't have time for what's important to us, for what we really want to be doing, for spending time with loved ones, for doing things we're passionate about.
And yet, it doesn't have to be that way. It's possible to live a simpler life, one where you enjoy each activity, where you are present in everything (or most things) you do, where you are content rather than rushing to finish things.
If that appeals to you, let's take a look at some suggestions for living a simple, peaceful, content life:
- What's important. First, take a step back and think about what's important to you. What do you really want to be doing, who do you want to spend your time with, what do you want to accomplish with your work? Make a short list of 4-5 things for your life, 4-5 people you want to spend time with, 4-5 things you'd like to accomplish at work.
- Examine your commitments. A big part of the problem is that our lives are way too full. We can't possibly do everything we have committed to doing, and we certainly can't enjoy it if we're trying to do everything. Accept that you can't do everything, know that you want to do what's important to you, and try to eliminate the commitments that aren't as important. See this article for more.
- Do less each day. Don't fill your day up with things to do. You will end up rushing to do them all. If you normally try (and fail) to do 7-10 things, do 3 important ones instead (with 3 more smaller items to do if you get those three done). This will give you time to do what you need to do, and not rush.
- Leave space between tasks or appointments. Another mistake is trying to schedule things back-to-back. This leaves no cushion in case things take longer than we planned (which they always do), and it also gives us a feeling of being rushed and stressed throughout the day. Instead, leave a good-sized gap between your appointments or tasks, allowing you to focus more on each one, and have a transition time between them.
- Eliminate as much as possible from your to-do list. You can't do everything on your to-do list. Even if you could, more things will come up. As much as you can, simplify your to-do list down to the essentials. This allows you to rush less and focus more on what's important. See this article for more.
- Now, slow down and enjoy every task. This is the most important tip in this article. Read it twice. Whatever you're doing, whether it's a work task or taking a shower or brushing your teeth or cooking dinner or driving to work, slow down. Try to enjoy whatever you're doing. Try to pay attention, instead of thinking about other things. Be in the moment. This isn't easy, as you will often forget. But find a way to remind yourself. Unless the task involves actual pain, there isn't anything that can't be enjoyable if you give it the proper attention.
- Single-task. This is kind of a mantra of mine, as I talk about how to single-task all the time. But it's an important point for me, and for this article. Do one thing at a time, and do it well.
- Eat slower. This is just a more specific application of Tip #6, but it's something we do every day, so it deserves special attention. See this article for more.
- Drive slower. Another application of the same principle, driving is something we do that's often mindless and rushed. Instead, slow down and enjoy the journey. See this article for more.
- Eliminate stress. Find the stressors in your life, and find ways to eliminate them. See this article for more.
- How and why to slow down. This is such an important point, that I'm going to point you to two other articles on this: here and here.
- Create time for solitude. In addition to slowing down and enjoying the tasks we do, and doing less of them, it's also important to just have some time to yourself. Read this article for more.
- Do nothing. Sometimes, it's good to forget about doing things, and do nothing. Here's more.
- Sprinkle simple pleasures throughout your day. Knowing what your simple pleasures are, and putting a few of them in each day, can go a long way to making life more enjoyable. Here's a list of 75 simple pleasures.
- Practice being present. You can practice being in the moment at any time during the day. Here's how.
- Find inspirations. Learn from the best. Here are 5 inspirations for being in the moment.
- Make frugality an enjoyable thing too. Instead of delayed gratification, try enjoying life now while saving for later.


text:
Literate 4 Life: Librarian Branding Project
corset pic by L. Marie
do you want mad librarian cred when
hanging out in the stacks with your homies?
are you Literate4Life and proud of it?
are you ready, willing, and able to show your love - your devotion - your RESPECT! to our gang of book loving library militants?
Check out kick@$$ Librarian X aka jazzmodeus aka Jason Puckett - he's the Librarian for Communication and Educational Technologies at GSU in Atlanta and he's inked & proud!
(photo from jazzmodeus flickr creative commons tattoo by Ron Hendon of Midnight Iguana Tattooing, Athens GA.)
join the Published, Branded & Proud by participating in the Project Brand Yourself a Librarian Week this summer at ALA DC.
Brainchild of Uber cool 8bitlibrary blog founders Justin The Librarian and Librarian JP. So, whether you're going to ALA in DC this summer...you can meet-up and get inked there then go to the appreciation & participation party...
or if you're going to ISTE SIGMS & going to ISTE10 Denver (like me!) ...you can go here
Local business results for tattoo parlors near Denver, CO ...OR i think the AWEsome Bearded JP suggested creating a Flickr group so that wherever you are... if you're branded you can join the tribe and...show it off with pride!
And seriously? In this day and age where school librarians are fighting to show the importance and relevancy of their program, information technology, and 21st century learning strategies in school libraries -- and facing some serious federal cutbacks - the time for advocacy is NOW! Now is the time to step up our game!
or as Joyce Valenza said in her recent Neverending Search Blog....
"All our programs need to deliver high quality, 21st century, learner-centered service and instruction embracing the clear mission of ensuring that our students and staff are effective users [and producers*] of ideas and information.That means thinking out of the box, being visible, connecting with our kids, being relevant and cutting edge, and keeping the faith with our fellow librarians...to show...there is no shame in our game!
Funding decisions will be made at the state and local level.
We have to make sure that our SEAs understand how school library information programs improve outcomes for students.....We must make ourselves visible, relevant, integral to the educational missions of our schools.
We had better start making ourselves known in a positive way to principals, district administrators, superintendents, headmasters, and local legislators, or we will be out of a job."
from Justin the Librarian & 8bitlibrary blog:
"It's really simple. If you're up for it, let's all get tattoos to show our support for libraries and librarians. We are who we are. We are librarians! Let's show the world how proud we are!
Let's do this at the ALA Annual Conference (June 24-29). If you've got ideas on a tattoo parlor, leave a comment. I'll gladly be in charge of setting up an appointments for everyone (we should all go together).
1. DOESN'T IT HURT? Honestly, it's not that bad. I myself am very scared of needles but I don't mind getting tattoos. The worst part is the healing. WHY? It itches…
2. BUT IT'S FOREVER! Be proud to be a librarian! Do you think you'll be leaving this profession? See, you won't.
3. I'VE ALWAYS WANTED ONE…and this is a great chance to finally get one! Think about it! Who else can say that they got their tattoo with a bunch of librarians?"
Read the whole Project Brand posting to get more information how you can join in!
Librarians in Texas have done it!
Oh and for ideas (or for those who are too chicken
Dag! they're out...you can get them from here! Stupid.com
Photo from Product Page ( $7.95 )
now for me....being a Steampunk grrl i think i might have to get a top hatted Cthulhu or a cephalopod underglass like this guy above...from my factory in New Babbage! Or else something very tasteful and victorian like just the type below.....here's also some other tatoos from the pack. (and yes, i do have them!) Whatever you choose to do...do it with passion!


text:
Time::HiRes - High resolution alarm, sleep, gettimeofday, interval timers
use Time::HiRes qw( usleep ualarm gettimeofday tv_interval nanosleep
clock_gettime clock_getres clock_nanosleep clock
stat lstat utime);
usleep ($microseconds);
nanosleep ($nanoseconds);
ualarm ($microseconds);
ualarm ($microseconds, $interval_microseconds);
$t0 = [gettimeofday];
($seconds, $microseconds) = gettimeofday;
$elapsed = tv_interval ( $t0, [$seconds, $microseconds]);
$elapsed = tv_interval ( $t0, [gettimeofday]);
$elapsed = tv_interval ( $t0 );
use Time::HiRes qw ( time alarm sleep );
$now_fractions = time;
sleep ($floating_seconds);
alarm ($floating_seconds);
alarm ($floating_seconds, $floating_interval);
use Time::HiRes qw( setitimer getitimer );
setitimer ($which, $floating_seconds, $floating_interval );
getitimer ($which);
use Time::HiRes qw( clock_gettime clock_getres clock_nanosleep
$realtime = clock_gettime(CLOCK_REALTIME);
$resolution = clock_getres(CLOCK_REALTIME);
clock_nanosleep(CLOCK_REALTIME, 1.5e9);
clock_nanosleep(CLOCK_REALTIME, time()*1e9 + 10e9, TIMER_ABSTIME);
my $ticktock = clock();
use Time::HiRes qw( stat lstat );
my @stat = stat("file");
my @stat = stat(FH);
my @stat = lstat("file");
use Time::HiRes qw( utime );
utime $floating_seconds, $floating_seconds, file...;
The Time::HiRes
module implements a Perl interface to the usleep
, nanosleep
, ualarm
, gettimeofday
, and setitimer
system calls, in other words, high resolution time and timers. See the "EXAMPLES" section below and the test scripts for usage; see your system documentation for the description of the underlying nanosleep
or usleep
, ualarm
, gettimeofday
, and setitimer
If your system lacks gettimeofday()
or an emulation of it you don't get gettimeofday()
or the one-argument form of tv_interval()
. If your system lacks all of nanosleep()
, usleep()
, select()
, and poll
, you don't get Time::HiRes::usleep()
, Time::HiRes::nanosleep()
, or Time::HiRes::sleep()
. If your system lacks both ualarm()
and setitimer()
you don't get Time::HiRes::ualarm()
or Time::HiRes::alarm()
If you try to import an unimplemented function in the use
statement it will fail at compile time.
If your subsecond sleeping is implemented with nanosleep()
instead of usleep()
, you can mix subsecond sleeping with signals since nanosleep()
does not use signals. This, however, is not portable, and you should first check for the truth value of &Time::HiRes::d_nanosleep
to see whether you have nanosleep, and then carefully read your nanosleep()
C API documentation for any peculiarities.
If you are using nanosleep
for something else than mixing sleeping with signals, give some thought to whether Perl is the tool you should be using for work requiring nanosecond accuracies.
Remember that unless you are working on a hard realtime system, any clocks and timers will be imprecise, especially so if you are working in a pre-emptive multiuser system. Understand the difference between wallclock time and process time (in UNIX-like systems the sum of user and system times). Any attempt to sleep for X seconds will most probably end up sleeping more than that, but don't be surprised if you end up sleeping slightly less.
The following functions can be imported from this module. No functions are exported by default.
In array context returns a two-element array with the seconds and microseconds since the epoch. In scalar context returns floating seconds like Time::HiRes::time()
(see below).
Sleeps for the number of microseconds (millionths of a second) specified. Returns the number of microseconds actually slept. Can sleep for more than one second, unlike the usleep
system call. Can also sleep for zero seconds, which often works like a thread yield. See also Time::HiRes::usleep()
, Time::HiRes::sleep()
, and Time::HiRes::clock_nanosleep()
Do not expect usleep() to be exact down to one microsecond.
Sleeps for the number of nanoseconds (1e9ths of a second) specified. Returns the number of nanoseconds actually slept (accurate only to microseconds, the nearest thousand of them). Can sleep for more than one second. Can also sleep for zero seconds, which often works like a thread yield. See also Time::HiRes::sleep()
, Time::HiRes::usleep()
, and Time::HiRes::clock_nanosleep()
Do not expect nanosleep() to be exact down to one nanosecond. Getting even accuracy of one thousand nanoseconds is good.
Issues a ualarm
call; the $interval_useconds
is optional and will be zero if unspecified, resulting in alarm
-like behaviour.
Returns the remaining time in the alarm in microseconds, or undef
if an error occurred.
ualarm(0) will cancel an outstanding ualarm().
Note that the interaction between alarms and sleeps is unspecified.
tv_interval ( $ref_to_gettimeofday [, $ref_to_later_gettimeofday] )
Returns the floating seconds between the two times, which should have been returned by gettimeofday()
. If the second argument is omitted, then the current time is used.
Returns a floating seconds since the epoch. This function can be imported, resulting in a nice drop-in replacement for the time
provided with core Perl; see the "EXAMPLES" below.
NOTE 1: This higher resolution timer can return values either less or more than the core time()
, depending on whether your platform rounds the higher resolution timer values up, down, or to the nearest second to get the core time()
, but naturally the difference should be never more than half a second. See also "clock_getres", if available in your system.
NOTE 2: Since Sunday, September 9th, 2001 at 01:46:40 AM GMT, when the time()
seconds since epoch rolled over to 1_000_000_000, the default floating point format of Perl and the seconds since epoch have conspired to produce an apparent bug: if you print the value of Time::HiRes::time()
you seem to be getting only five decimals, not six as promised (microseconds). Not to worry, the microseconds are there (assuming your platform supports such granularity in the first place). What is going on is that the default floating point format of Perl only outputs 15 digits. In this case that means ten digits before the decimal separator and five after. To see the microseconds you can use either printf
with "%.6f"
, or the gettimeofday()
function in list context, which will give you the seconds and microseconds as two separate values.
Sleeps for the specified amount of seconds. Returns the number of seconds actually slept (a floating point value). This function can be imported, resulting in a nice drop-in replacement for the sleep
provided with perl, see the "EXAMPLES" below.
Note that the interaction between alarms and sleeps is unspecified.
The SIGALRM
signal is sent after the specified number of seconds. Implemented using setitimer()
if available, ualarm()
if not. The $interval_floating_seconds
argument is optional and will be zero if unspecified, resulting in alarm()
-like behaviour. This function can be imported, resulting in a nice drop-in replacement for the alarm
provided with perl, see the "EXAMPLES" below.
Returns the remaining time in the alarm in seconds, or undef
if an error occurred.
NOTE 1: With some combinations of operating systems and Perl releases SIGALRM
restarts select()
, instead of interrupting it. This means that an alarm()
followed by a select()
may together take the sum of the times specified for the alarm()
and the select()
, not just the time of the alarm()
Note that the interaction between alarms and sleeps is unspecified.
Start up an interval timer: after a certain time, a signal ($which) arrives, and more signals may keep arriving at certain intervals. To disable an "itimer", use $floating_seconds
of zero. If the $interval_floating_seconds
is set to zero (or unspecified), the timer is disabled after the next delivered signal.
Use of interval timers may interfere with alarm()
, sleep()
, and usleep()
. In standard-speak the "interaction is unspecified", which means that anything may happen: it may work, it may not.
In scalar context, the remaining time in the timer is returned.
In list context, both the remaining time and the interval are returned.
There are usually three or four interval timers (signals) available: the $which
can be ITIMER_REAL
, or ITIMER_REALPROF
. Note that which ones are available depends: true UNIX platforms usually have the first three, but only Solaris seems to have ITIMER_REALPROF
(which is used to profile multithreaded programs). Win32 unfortunately does not have interval timers.
results in alarm()
-like behaviour. Time is counted in real time; that is, wallclock time. SIGALRM
is delivered when the timer expires.
counts time in (process) virtual time; that is, only when the process is running. In multiprocessor/user/CPU systems this may be more or less than real or wallclock time. (This time is also known as the user time.) SIGVTALRM
is delivered when the timer expires.
counts time when either the process virtual time or when the operating system is running on behalf of the process (such as I/O). (This time is also known as the system time.) (The sum of user time and system time is known as the CPU time.) SIGPROF
is delivered when the timer expires. SIGPROF
can interrupt system calls.
The semantics of interval timers for multithreaded programs are system-specific, and some systems may support additional interval timers. For example, it is unspecified which thread gets the signals. See your setitimer()
Return the remaining time in the interval timer specified by $which
In scalar context, the remaining time is returned.
In list context, both the remaining time and the interval are returned. The interval is always what you put in using setitimer()
Return as seconds the current value of the POSIX high resolution timer specified by $which
. All implementations that support POSIX high resolution timers are supposed to support at least the $which
value of CLOCK_REALTIME
, which is supposed to return results close to the results of gettimeofday
, or the number of seconds since 00:00:00:00 January 1, 1970 Greenwich Mean Time (GMT). Do not assume that CLOCK_REALTIME is zero, it might be one, or something else. Another potentially useful (but not available everywhere) value is CLOCK_MONOTONIC
, which guarantees a monotonically increasing time value (unlike time() or gettimeofday(), which can be adjusted). See your system documentation for other possibly supported values.
Return as seconds the resolution of the POSIX high resolution timer specified by $which
. All implementations that support POSIX high resolution timers are supposed to support at least the $which
value of CLOCK_REALTIME
, see "clock_gettime".
NOTE: the resolution returned may be highly optimistic. Even if the resolution is high (a small number), all it means is that you'll be able to specify the arguments to clock_gettime() and clock_nanosleep() with that resolution. The system might not actually be able to measure events at that resolution, and the various overheads and the overall system load are certain to affect any timings.
Sleeps for the number of nanoseconds (1e9ths of a second) specified. Returns the number of nanoseconds actually slept. The $which is the "clock id", as with clock_gettime() and clock_getres(). The flags default to zero but TIMER_ABSTIME
can specified (must be exported explicitly) which means that $nanoseconds
is not a time interval (as is the default) but instead an absolute time. Can sleep for more than one second. Can also sleep for zero seconds, which often works like a thread yield. See also Time::HiRes::sleep()
, Time::HiRes::usleep()
, and Time::HiRes::nanosleep()
Do not expect clock_nanosleep() to be exact down to one nanosecond. Getting even accuracy of one thousand nanoseconds is good.
Return as seconds the process time (user + system time) spent by the process since the first call to clock() (the definition is not "since the start of the process", though if you are lucky these times may be quite close to each other, depending on the system). What this means is that you probably need to store the result of your first call to clock(), and subtract that value from the following results of clock().
The time returned also includes the process times of the terminated child processes for which wait() has been executed. This value is somewhat like the second value returned by the times() of core Perl, but not necessarily identical. Note that due to backward compatibility limitations the returned value may wrap around at about 2147 seconds or at about 36 minutes.
As "stat" in perlfunc or "lstat" in perlfunc but with the access/modify/change file timestamps in subsecond resolution, if the operating system and the filesystem both support such timestamps. To override the standard stat():
use Time::HiRes qw(stat);
Test for the value of &Time::HiRes::d_hires_stat to find out whether the operating system supports subsecond file timestamps: a value larger than zero means yes. There are unfortunately no easy ways to find out whether the filesystem supports such timestamps. UNIX filesystems often do; NTFS does; FAT doesn't (FAT timestamp granularity is two seconds).
A zero return value of &Time::HiRes::d_hires_stat means that Time::HiRes::stat is a no-op passthrough for CORE::stat() (and likewise for lstat), and therefore the timestamps will stay integers. The same thing will happen if the filesystem does not do subsecond timestamps, even if the &Time::HiRes::d_hires_stat is non-zero.
In any case do not expect nanosecond resolution, or even a microsecond resolution. Also note that the modify/access timestamps might have different resolutions, and that they need not be synchronized, e.g. if the operations are
stat # t1
stat # t2
the access time stamp from t2 need not be greater-than the modify time stamp from t1: it may be equal or less.
As "utime" in perlfunc but with the ability to set the access/modify file timestamps in subsecond resolution, if the operating system and the filesystem, and the mount options of the filesystem, all support such timestamps.
To override the standard utime():
use Time::HiRes qw(utime);
Test for the value of &Time::HiRes::d_hires_utime to find out whether the operating system supports setting subsecond file timestamps.
As with CORE::utime(), passing undef as both the atime and mtime will call the syscall with a NULL argument.
The actual achievable subsecond resolution depends on the combination of the operating system and the filesystem.
Modifying the timestamps may not be possible at all: for example, the noatime
filesystem mount option may prohibit you from changing the access time timestamp.
Returns the number of files successfully changed.
use Time::HiRes qw(usleep ualarm gettimeofday tv_interval);
$microseconds = 750_000;
# signal alarm in 2.5s & every .1s thereafter
ualarm(2_500_000, 100_000);
# cancel that ualarm
# get seconds and microseconds since the epoch
($s, $usec) = gettimeofday();
# measure elapsed time
# (could also do by subtracting 2 gettimeofday return values)
$t0 = [gettimeofday];
# do bunch of stuff here
$t1 = [gettimeofday];
# do more stuff here
$t0_t1 = tv_interval $t0, $t1;
$elapsed = tv_interval ($t0, [gettimeofday]);
$elapsed = tv_interval ($t0); # equivalent code
# replacements for time, alarm and sleep that know about
# floating seconds
use Time::HiRes;
$now_fractions = Time::HiRes::time;
Time::HiRes::sleep (2.5);
Time::HiRes::alarm (10.6666666);
use Time::HiRes qw ( time alarm sleep );
$now_fractions = time;
sleep (2.5);
alarm (10.6666666);
# Arm an interval timer to go off first at 10 seconds and
# after that every 2.5 seconds, in process virtual time
use Time::HiRes qw ( setitimer ITIMER_VIRTUAL time );
$SIG{VTALRM} = sub { print time, "\n" };
setitimer(ITIMER_VIRTUAL, 10, 2.5);
use Time::HiRes qw( clock_gettime clock_getres CLOCK_REALTIME );
# Read the POSIX high resolution timer.
my $high = clock_gettime(CLOCK_REALTIME);
# But how accurate we can be, really?
my $reso = clock_getres(CLOCK_REALTIME);
use Time::HiRes qw( clock_nanosleep TIMER_ABSTIME );
clock_nanosleep(CLOCK_REALTIME, 1e6);
clock_nanosleep(CLOCK_REALTIME, 2e9, TIMER_ABSTIME);
use Time::HiRes qw( clock );
my $clock0 = clock();
... # Do something.
my $clock1 = clock();
my $clockd = $clock1 - $clock0;
use Time::HiRes qw( stat );
my ($atime, $mtime, $ctime) = (stat("istics"))[8, 9, 10];
In addition to the perl API described above, a C API is available for extension writers. The following C functions are available in the modglobal hash:
name C prototype
--------------- ----------------------
Time::NVtime NV (*)()
Time::U2time void (*)(pTHX_ UV ret[2])
Both functions return equivalent information (like gettimeofday
) but with different representations. The names NVtime
and U2time
were selected mainly because they are operating system independent. (gettimeofday
is Unix-centric, though some platforms like Win32 and VMS have emulations for it.)
Here is an example of using NVtime
from C:
NV (*myNVtime)(); /* Returns -1 on failure. */
SV **svp = hv_fetchs(PL_modglobal, "Time::NVtime", 0);
if (!svp) croak("Time::HiRes is required");
if (!SvIOK(*svp)) croak("Time::NVtime isn't a function pointer");
myNVtime = INT2PTR(NV(*)(), SvIV(*svp));
printf("The current time is: %" NVff "\n", (*myNVtime)());
In ualarm() you tried to use number of microseconds or interval (also in microseconds) more than 1_000_000 and setitimer() is not available in your system to emulate that case.
You tried to use a negative time argument.
Something went horribly wrong-- the number of microseconds that cannot become negative just became negative. Maybe your compiler is broken?
In some platforms it is not possible to get an alarm with subsecond resolution and later than one second.
Some calls simply aren't available, real or emulated, on every platform.
Notice that the core time()
maybe rounding rather than truncating. What this means is that the core time()
may be reporting the time as one second later than gettimeofday()
and Time::HiRes::time()
Adjusting the system clock (either manually or by services like ntp) may cause problems, especially for long running programs that assume a monotonously increasing time (note that all platforms do not adjust time as gracefully as UNIX ntp does). For example in Win32 (and derived platforms like Cygwin and MinGW) the Time::HiRes::time() may temporarily drift off from the system clock (and the original time()) by up to 0.5 seconds. Time::HiRes will notice this eventually and recalibrate. Note that since Time::HiRes 1.77 the clock_gettime(CLOCK_MONOTONIC) might help in this (in case your system supports CLOCK_MONOTONIC).
Some systems have APIs but not implementations: for example QNX and Haiku have the interval timer APIs but not the functionality.
In pre-Sierra macOS (pre-10.12, OS X) clock_getres(), clock_gettime() and clock_nanosleep() are emulated using the Mach timers; as a side effect of being emulated the CLOCK_REALTIME and CLOCK_MONOTONIC are the same timer.
gnukfreebsd seems to have non-functional futimens() and utimensat() (at least as of 10.1): therefore the hires utime() does not work.
Perl modules BSD::Resource, Time::TAI64.
Your system documentation for clock
, clock_gettime
, clock_getres
, clock_nanosleep
, clock_settime
, getitimer
, gettimeofday
, setitimer
, sleep
, stat
, ualarm
D. Wegscheid <firstname.lastname@example.com> R. Schertler <email@example.com> J. Hietaniemi <email@example.com> G. Aas <firstname.lastname@example.com>
Copyright (c) 1996-2002 Douglas E. Wegscheid. All rights reserved.
Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Jarkko Hietaniemi. All rights reserved.
Copyright (C) 2011, 2012, 2013 Andrew Main (Zefram) <email@example.com>
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.