text:
Table of Contents
Chapter Two - Intermediate Stuff
|top prev next
In Chapter One we took ØMQ for a drive, with some basic examples of the main ØMQ patterns: request-reply, publish-subscribe, and pipeline. In this chapter we're going to get our hands dirty and start to learn how to use these tools in real programs.
We'll cover:
- How to create and work with ØMQ sockets.
- How to send and receive messages on sockets.
- How to build your apps around ØMQ's asynchronous I/O model.
- How to handle multiple sockets in one thread.
- How to handle fatal and non-fatal errors properly.
- How to handle interrupt signals like Ctrl-C.
- How to shutdown a ØMQ application cleanly.
- How to check a ØMQ application for memory leaks.
- How to send and receive multipart messages.
- How to forward messages across networks.
- How to build a simple message queuing broker.
- How to write multithreaded applications with ØMQ.
- How to use ØMQ to signal between threads.
- How to use ØMQ to coordinate a network of nodes.
- How to create and use message envelopes for publish-subscribe.
- Using the high-water mark (HWM) to protect against memory overflows.
The Zen of Zero
|top prev next
The Ø in ØMQ is all about tradeoffs. On the one hand this strange name lowers ØMQ's visibility on Google and Twitter. On the other hand it annoys the heck out of some Danish folk who write us things like "ØMG røtfl", and "Ø is not a funny looking zero!" and "Rødgrød med Fløde!", which is apparently an insult that means "may your neighbours be the direct descendants of Grendel!" Seems like a fair trade.
Originally the zero in ØMQ was meant as "zero broker" and (as close to) "zero latency" (as possible). In the meantime it has come to cover different goals: zero administration, zero cost, zero waste. More generally, "zero" refers to the culture of minimalism that permeates the project. We add power by removing complexity rather than exposing new functionality.
The Socket API
|top prev next
To be perfectly honest, ØMQ does a kind of switch-and-bait on you. Which we don't apologize for, it's for your own good and hurts us more than it hurts you. It presents a familiar BSD socket API but that hides a bunch of message-processing machines that will slowly fix your world-view about how to design and write distributed software.
Sockets are the de-facto standard API for network programming, as well as being useful for stopping your eyes from falling onto your cheeks. One thing that makes ØMQ especially tasty to developers is that it uses a standard socket API. Kudos to Martin Sustrik for pulling this idea off. It turns "Message Oriented Middleware", a phrase guaranteed to send the whole room off to Catatonia, into "Extra Spicy Sockets!" which leaves us with a strange craving for pizza, and a desire to know more.
Like a nice pepperoni pizza, ØMQ sockets are easy to digest. Sockets have a life in four parts, just like BSD sockets:
- Creating and destroying sockets, which go together to form a karmic circle of socket life (see zmq_socket(3), zmq_close(3)).
- Configuring sockets by setting options on them and checking them if necessary (see zmq_setsockopt(3), zmq_getsockopt(3)).
- Plugging sockets onto the network topology by creating ØMQ connections to and from them (see zmq_bind(3), zmq_connect(3)).
- Using the sockets to carry data by writing and receiving messages on them (see zmq_send(3), zmq_recv(3)).
Which looks like this, in C:
static void *
worker_thread (void *arg) {
void *context = arg;
void *worker = zmq_socket (context, ZMQ_REP);
assert (worker);
int rc;
rc = zmq_connect (worker, "ipc://worker");
assert (rc == 0);
void *broadcast = zmq_socket (context, ZMQ_PUB);
assert (broadcast);
rc = zmq_bind (broadcast, "ipc://publish");
assert (rc == 0);
while (1) {
char *part1 = s_recv (worker);
char *part2 = s_recv (worker);
printf ("Worker got [%s][%s]\n", part1, part2);
s_sendmore (broadcast, "msg");
s_sendmore (broadcast, part1);
s_send (broadcast, part2);
free (part1);
free (part2);
s_send (worker, "OK");
return NULL;
Note that sockets are always void pointers, and messages (which we'll come to very soon) are structures. So in C you pass sockets as-such, but you pass addresses of messages in all functions that work with messages, like zmq_send(3) and zmq_recv(3). As a mnemonic, realize that "in ØMQ all your sockets are belong to us", but messages are things you actually own in your code.
Creating, destroying, and configuring sockets works as you'd expect for any object. But remember that ØMQ is an asynchronous, elastic fabric. This has some impact on how we plug sockets into the network topology, and how we use the sockets after that.
Plugging Sockets Into the Topology
|top prev next
To create a connection between two nodes you use zmq_bind(3) in one node, and zmq_connect(3) in the other. As a general rule of thumb, the node which does zmq_bind(3) is a "server", sitting on a well-known network address, and the node which does zmq_connect(3) is a "client", with unknown or arbitrary network addresses. Thus we say that we "bind a socket to an endpoint" and "connect a socket to an endpoint", the endpoint being that well-known network address.
ØMQ connections are somewhat different from old-fashioned TCP connections. The main notable differences are:
- They go across an arbitrary transport (inproc, ipc, tcp, pgm or epgm). See zmq_inproc(7), zmq_ipc(7), zmq_tcp(7), zmq_pgm(7), and zmq_epgm(7).
- They exist when a client does zmq_connect(3) to an endpoint, whether or not a server has already done zmq_bind(3) to that endpoint.
- They are asynchronous, and have queues that magically exist where and when needed.
- They may express a certain "messaging pattern", according to the type of socket used at each end.
- One socket may have many outgoing and many incoming connections.
- There is no zmq_accept() method. When a socket is bound to an endpoint it automatically starts accepting connections.
- Your application code cannot work with these connections directly; they are encapsulated under the socket.
Many architectures follow some kind of client-server model, where the server is the component that is most stable, and the clients are the components that are most dynamic, i.e. they come and go the most. There are sometimes issues of addressing: servers will be visible to clients, but not necessarily vice-versa. So mostly it's obvious which node should be doing zmq_bind(3) (the server) and which should be doing zmq_connect(3) (the client). It also depends on the kind of sockets you're using, with some exceptions for unusual network architectures. We'll look at socket types later.
Now, imagine we start the client before we start the server. In traditional networking we get a big red Fail flag. But ØMQ lets us start and stop pieces arbitrarily. As soon as the client node does zmq_connect(3) the connection exists and that node can start to write messages to the socket. At some stage (hopefully before messages queue up so much that they start to get discarded, or the client blocks), the server comes alive, does a zmq_bind(3) and ØMQ starts to deliver messages.
A server node can bind to many endpoints and it can do this using a single socket. This means it will accept connections across different transports:
zmq_bind (socket, "tcp://*:5555");
zmq_bind (socket, "tcp://*:9999");
zmq_bind (socket, "ipc://myserver.ipc");
You cannot bind to the same endpoint twice, that will cause an exception.
Each time a client node does a zmq_connect(3) to any of these endpoints, the server node's socket gets another connection. There is no inherent limit to how many connections a socket can have. A client node can also connect to many endpoints using a single socket.
In most cases, which node acts as client, and which as server, is about network topology rather than message flow. However, there are cases (resending when connections are broken) where the same socket type will behave differently if it's a server or if it's a client.
What this means is that you should always think in terms of "servers" as stable parts of your topology, with more-or-less fixed endpoint addresses, and "clients" as dynamic parts that come and go. Then, design your application around this model. The chances that it will "just work" are much better like that.
Sockets have types. The socket type defines the semantics of the socket, its policies for routing messages inwards and outwards, queuing, etc. You can connect certain types of socket together, e.g. a publisher socket and a subscriber socket. Sockets work together in "messaging patterns". We'll look at this in more detail later.
It's the ability to connect sockets in these different ways that gives ØMQ its basic power as a message queuing system. There are layers on top of this, such as devices and topic routing, which we'll get to later. But essentially, with ØMQ you define your network architecture by plugging pieces together like a child's construction toy.
Using Sockets to Carry Data
|top prev next
To send and receive messages you use the zmq_send(3) and zmq_recv(3) methods. The names are conventional but ØMQ's I/O model is different enough from the TCP model that you will need time to get your head around it.
Figure 10 - TCP sockets are 1 to 1
Let's look at the main differences between TCP sockets and ØMQ sockets when it comes to carrying data:
- ØMQ sockets carry messages, rather than bytes (as in TCP) or frames (as in UDP). A message is a length-specified blob of binary data. We'll come to messages shortly, their design is optimized for performance and thus somewhat tricky to understand.
- ØMQ sockets do their I/O in a background thread. This means that messages arrive in a local input queue, and are sent from a local output queue, no matter what your application is busy doing. These are configurable memory queues, by the way.
- ØMQ sockets can, depending on the socket type, be connected to (or from, it's the same) many other sockets. Where TCP emulates a one-to-one phone call, ØMQ implements one-to-many (like a radio broadcast), many-to-many (like a post office), many-to-one (like a mail box), and even one-to-one.
- ØMQ sockets can send to many endpoints (creating a fan-out model), or receive from many endpoints (creating a fan-in model).
Figure 11 - ØMQ Sockets are N to N
So writing a message to a socket may send the message to one or many other places at once, and conversely, one socket will collect messages from all connections sending messages to it. The zmq_recv(3) method uses a fair-queuing algorithm so each sender gets an even chance.
The zmq_send(3) method does not actually send the message to the socket connection(s). It queues the message so that the I/O thread can send it asynchronously. It does not block except in some exception cases. So the message is not necessarily sent when zmq_send(3) returns to your application. If you created a message using zmq_msg_init_data(3) you cannot reuse the data or free it, otherwise the I/O thread will rapidly find itself writing overwritten or unallocated garbage. This is a common mistake for beginners. We'll see a little later how to properly work with messages.
Unicast Transports
|top prev next
ØMQ provides a set of unicast transports (inproc, ipc, and tcp) and multicast transports (epgm, pgm). Multicast is an advanced technique that we'll come to later. Don't even start using it unless you know that your fanout ratios will make 1-to-N unicast impossible.
For most common cases, use tcp, which is a disconnected TCP transport. It is elastic, portable, and fast enough for most cases. We call this 'disconnected' because ØMQ's tcp transport doesn't require that the endpoint exists before you connect to it. Clients and servers can connect and bind at any time, can go and come back, and it remains transparent to applications.
The inter-process transport, ipc, is like tcp except that it is abstracted from the LAN, so you don't need to specify IP addresses or domain names. This makes it better for some purposes, and we use it quite often in the examples in this book. ØMQ's ipc transport is disconnected, like tcp. It has one limitation: it does not work on Windows. This may be fixed in future versions of ØMQ. By convention we use endpoint names with an ".ipc" extension to avoid potential conflict with other file names. On UNIX systems, if you use ipc endpoints you need to create these with appropriate permissions otherwise they may not be shareable between processes running under different user ids. You must also make sure all processes can access the files, e.g. by running in the same working directory.
The inter-thread transport, inproc, is a connected signaling transport. It is much faster than tcp or ipc. This transport has a specific limitation compared to ipc and tcp: you must do bind before connect. This is something future versions of ØMQ may fix, but at present this defines you use inproc sockets. We create and bind one socket, start the child threads, which create and connect the other sockets.
ØMQ is Not a Neutral Carrier
|top prev next
A common question that newcomers to ØMQ ask (it's one I asked myself) is something like, "how do I write a XYZ server in ØMQ?" For example, "how do I write an HTTP server in ØMQ?"
The implication is that if we use normal sockets to carry HTTP requests and responses, we should be able to use ØMQ sockets to do the same, only much faster and better.
Sadly the answer is "this is not how it works". ØMQ is not a neutral carrier, it imposes a framing on the transport protocols it uses. This framing is not compatible with existing protocols, which tend to use their own framing. For example, compare an HTTP request, and a ØMQ request, both over TCP/IP.
Figure 12 - HTTP On the Wire
Where the HTTP request uses CR-LF as its simplest framing delimiter, and ØMQ uses a length-specified frame.
Figure 13 - ØMQ On the Wire
So you could write a HTTP-like protocol using ØMQ, using for example the request-reply socket pattern. But it would not be HTTP.
There is however a good answer to the question, "How can I make profitable use of ØMQ when making my new XYZ server?" You need to implement whatever protocol you want to speak in any case, but you can connect that protocol server (which can be extremely thin) to a ØMQ backend that does the real work. The beautiful part here is that you can then extend your backend with code in any language, running locally or remotely, as you wish. Zed Shaw's Mongrel2 web server is a great example of such an architecture.
I/O Threads
|top prev next
We said that ØMQ does I/O in a background thread. One I/O thread (for all sockets) is sufficient for all but the most extreme applications. This is the magic '1' that we use when creating a context, meaning "use one I/O thread":
typedef struct {
void *socket; // ZeroMQ socket to poll on
int fd; // OR, native file handle to poll on
short events; // Events to poll on
short revents; // Events returned after poll
} zmq_pollitem_t;
There is a major difference between a ØMQ application and a conventional networked application, which is that you don't create one socket per connection. One socket handles all incoming and outgoing connections for a particular point of work. E.g. when you publish to a thousand subscribers, it's via one socket. When you distribute work among twenty services, it's via one socket. When you collect data from a thousand web applications, it's via one socket.
This has a fundamental impact on how you write applications. A traditional networked application has one process or one thread per remote connection, and that process or thread handles one socket. ØMQ lets you collapse this entire structure into a single thread, and then break it up as necessary for scaling.
Core Messaging Patterns
|top prev next
Underneath the brown paper wrapping of ØMQ's socket API lies the world of messaging patterns. If you have a background in enterprise messaging, these will be vaguely familiar. But to most ØMQ newcomers they are a surprise, we're so used to the TCP paradigm where a socket represents another node.
Let's recap briefly what ØMQ does for you. It delivers blobs of data (messages) to nodes, quickly and efficiently. You can map nodes to threads, processes, or boxes. It gives your applications a single socket API to work with, no matter what the actual transport (like in-process, inter-process, TCP, or multicast). It automatically reconnects to peers as they come and go. It queues messages at both sender and receiver, as needed. It manages these queues carefully to ensure processes don't run out of memory, overflowing to disk when appropriate. It handles socket errors. It does all I/O in background threads. It uses lock-free techniques for talking between nodes, so there are never locks, waits, semaphores, or deadlocks.
But cutting through that, it routes and queues messages according to precise recipes called patterns. It is these patterns that provide ØMQ's intelligence. They encapsulate our hard-earned experience of the best ways to distribute data and work. ØMQ's patterns are hard-coded but future versions may allow user-definable patterns.
ØMQ patterns are implemented by pairs of sockets with matching types. In other words, to understand ØMQ patterns you need to understand socket types and how they work together. Mostly this just takes learning, there is little that is obvious at this level.
The built-in core ØMQ patterns are:
- Request-reply, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern.
- Publish-subscribe, which connects a set of publishers to a set of subscribers. This is a data distribution pattern.
- Pipeline, connects nodes in a fan-out / fan-in pattern that can have multiple steps, and loops. This is a parallel task distribution and collection pattern.
We looked at each of these in the first chapter. There's one more pattern that people tend to try to use when they still think of ØMQ in terms of traditional TCP sockets:
- Exclusive pair, which connects two sockets in an exclusive pair. This is a low-level pattern for specific, advanced use-cases. We'll see an example at the end of this chapter.
The zmq_socket(3) man page is fairly clear about the patterns, it's worth reading several times until it starts to make sense. We'll look at each pattern and the use-cases it covers.
These are the socket combinations that are valid for a connect-bind pair (either side can bind):
- PUB and SUB
- REQ and REP
- REQ and ROUTER
- DEALER and REP
- DEALER and ROUTER
- DEALER and DEALER
- ROUTER and ROUTER
- PUSH and PULL
- PAIR and PAIR
Any other combination will produce undocumented and unreliable results and future versions of ØMQ will probably return errors if you try them. You can and will of course bridge other socket types via code, i.e. read from one socket type and write to another.
High-level Messaging Patterns
|top prev next
These four core patterns are cooked-in to ØMQ. They are part of the ØMQ API, implemented in the core C++ library, and guaranteed to be available in all fine retail stores. If one day the Linux kernel includes ØMQ, for example, these patterns would be there.
On top, we add high-level patterns. We build these high-level patterns on top of ØMQ and implement them in whatever language we're using for our application. They are not part of the core library, do not come with the ØMQ package, and exist in their own space, as part of the ØMQ community.
One of the things we aim to provide you with this guide are a set of such high-level patterns, both small (how to handle messages sanely) to large (how to make a reliable publish-subscribe architecture).
Working with Messages
|top prev next
On the wire, ØMQ messages are blobs of any size from zero upwards, fitting in memory. You do your own serialization using Google Protocol Buffers, XDR, JSON, or whatever else your applications need to speak. It's wise to choose a data representation that is portable and fast, but you can make your own decisions about trade-offs.
In memory, ØMQ messages are zmq_msg_t structures (or classes depending on your language). Here are the basic ground rules for using ØMQ messages in C:
- You create and pass around zmq_msg_t objects, not blocks of data.
- To read a message you use zmq_msg_init(3) to create an empty message, and then you pass that to zmq_recv(3).
- To write a message from new data, you use zmq_msg_init_size(3) to create a message and at the same time allocate a block of data of some size. You then fill that data using memcpy[3], and pass the message to zmq_send(3).
- To release (not destroy) a message you call zmq_msg_close(3). This drops a reference, and eventually ØMQ will destroy the message.
- To access the message content you use zmq_msg_data(3). To know how much data the message contains, use zmq_msg_size(3).
- Do not use zmq_msg_move(3), zmq_msg_copy(3), or zmq_msg_init_data(3) unless you read the man pages and know precisely why you need these.
Here is a typical chunk of code working with messages, which should be familiar if you have been paying attention. This is from the zhelpers.h file we use in all the examples:
typedef struct {
void *socket; // 0MQ socket to poll on
int fd; // OR, native file handle to poll on
short events; // Events to poll on
short revents; // Events returned after poll
} zmq_pollitem_t;
You can easily extend this code to send and receive blobs of arbitrary length.
Note than when you have passed a message to zmq_send(3), ØMQ will clear the message, i.e. set the size to zero. You cannot send the same message twice, and you cannot access the message data after sending it.
If you want to send the same message more than once, create a second message, initialize it using zmq_msg_init(3) and then use zmq_msg_copy(3) to create a copy of the first message. This does not copy the data but the reference. You can then send the message twice (or more, if you create more copies) and the message will only be finally destroyed when the last copy is sent or closed.
ØMQ also supports multipart messages, which let you handle a list of blobs as a single message. This is widely used in real applications and we'll look at that later in this chapter and in Chapter Three.
Some other things that are worth knowing about messages:
- ØMQ sends and receives them atomically, i.e. you get a whole message, or you don't get it at all.
- ØMQ does not send a message right away but at some indeterminate later time.
- You can send zero-length messages, e.g. for sending a signal from one thread to another.
- A message must fit in memory. If you want to send files of arbitrary sizes, you should break them into pieces and send each piece as a separate message.
- You must call zmq_msg_close(3) when finished with a message, in languages that don't automatically destroy objects when a scope closes.
And to be necessarily repetitive, do not use zmq_msg_init_data(3), yet. This is a zero-copy method and guaranteed to create trouble for you. There are far more important things to learn about ØMQ before you start to worry about shaving off microseconds.
Handling Multiple Sockets
|top prev next
In all the examples so far, the main loop of most examples has been:
- wait for message on socket
- process message
- repeat
What if we want to read from multiple sockets at the same time? The simplest way is to connect one socket to multiple endpoints and get ØMQ to do the fanin for us. This is legal if the remote endpoints are in the same pattern but it would be illegal to e.g. connect a PULL socket to a PUB endpoint. Fun, but illegal. If you start mixing patterns you break future scalability.
The right way is to use zmq_poll(3). An even better way might be to wrap zmq_poll(3) in a framework that turns it into a nice event-driven reactor, but it's significantly more work than we want to cover here.
Let's start with a dirty hack, partly for the fun of not doing it right, but mainly because it lets me show you how to do non-blocking socket reads. Here is a simple example of reading from two sockets using non-blocking reads. This rather confused program acts both as a subscriber to weather updates, and a worker for parallel tasks:
// Reading from multiple sockets
// This version uses a simple recv loop
#include "zhelpers.h"
int main (void)
// Connect to task ventilator
void *context = zmq_ctx_new ();
void *receiver = zmq_socket (context, ZMQ_PULL);
zmq_connect (receiver, "tcp://localhost:5557");
// Connect to weather server
void *subscriber = zmq_socket (context, ZMQ_SUB);
zmq_connect (subscriber, "tcp://localhost:5556");
zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, "10001 ", 6);
// Process messages from both sockets
// We prioritize traffic from the task ventilator
while (1) {
char msg [256];
while (1) {
int size = zmq_recv (receiver, msg, 255, ZMQ_DONTWAIT);
if (size != -1) {
// Process task
while (1) {
int size = zmq_recv (subscriber, msg, 255, ZMQ_DONTWAIT);
if (size != -1) {
// Process weather update
// No activity, so sleep for 1 msec
s_sleep (1);
zmq_close (receiver);
zmq_close (subscriber);
zmq_ctx_destroy (context);
return 0;
msreader.c: Multiple socket reader
The cost of this approach is some additional latency on the first message (the sleep at the end of the loop, when there are no waiting messages to process). This would be a problem in applications where sub-millisecond latency was vital. Also, you need to check the documentation for nanosleep() or whatever function you use to make sure it does not busy-loop.
You can treat the sockets fairly by reading first from one, then the second rather than prioritizing them as we did in this example. This is called "fair-queuing", something that ØMQ does automatically when one socket receives messages from more than one source.
Now let's see the same little senseless application done right, using zmq_poll(3):
// Reading from multiple sockets
// This version uses zmq_poll()
#include "zhelpers.h"
int main (void)
// Connect to task ventilator
void *context = zmq_ctx_new ();
void *receiver = zmq_socket (context, ZMQ_PULL);
zmq_connect (receiver, "tcp://localhost:5557");
// Connect to weather server
void *subscriber = zmq_socket (context, ZMQ_SUB);
zmq_connect (subscriber, "tcp://localhost:5556");
zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, "10001 ", 6);
// Process messages from both sockets
while (1) {
char msg [256];
zmq_pollitem_t items [] = {
{ receiver, 0, ZMQ_POLLIN, 0 },
{ subscriber, 0, ZMQ_POLLIN, 0 }
zmq_poll (items, 2, -1);
if (items [0].revents & ZMQ_POLLIN) {
int size = zmq_recv (receiver, msg, 255, 0);
if (size != -1) {
// Process task
if (items [1].revents & ZMQ_POLLIN) {
int size = zmq_recv (subscriber, msg, 255, 0);
if (size != -1) {
// Process weather update
zmq_close (subscriber);
zmq_ctx_destroy (context);
return 0;
mspoller.c: Multiple socket poller
Handling Errors and ETERM
|top prev next
ØMQ's error handling philosophy is a mix of fail-fast and resilience. Processes, we believe, should be as vulnerable as possible to internal errors, and as robust as possible against external attacks and errors. To give an analogy, a living cell will self-destruct if it detects a single internal error, yet it will resist attack from the outside by all means possible. Assertions, which pepper the ØMQ code, are absolutely vital to robust code, they just have to be on the right side of the cellular wall. And there should be such a wall. If it is unclear whether a fault is internal or external, that is a design flaw that needs to be fixed.
In C, assertions stop the application immediately with an error. In other languages you may get exceptions or halts.
When ØMQ detects an external fault it returns an error to the calling code. In some rare cases it drops messages silently, if there is no obvious strategy for recovering from the error. In a few places ØMQ still asserts on external faults, but these are considered bugs.
In most of the C examples we've seen so far there's been no error handling. Real code should do error handling on every single ØMQ call. If you're using a language binding other than C, the binding may handle errors for you. In C you do need to do this yourself. There are some simple rules, starting with POSIX conventions:
- Methods that create objects will return NULL in case they fail.
- Other methods will return 0 on success and other values (mostly -1) on an exceptional condition (usually failure).
- The error code is provided in errno or zmq_errno(3).
- A descriptive error text for logging is provided by zmq_strerror(3).
There are two main exceptional conditions that you may want to handle as non-fatal:
- When a thread calls zmq_recv(3) with the NOBLOCK option and there is no waiting data. ØMQ will return -1 and set errno to EAGAIN.
- When a thread calls zmq_term(3) and other threads are doing blocking work. The zmq_term(3) call closes the context and all blocking calls exit with -1, and errno set to ETERM.
What this boils down to is that in most cases you can use assertions on ØMQ calls, like this, in C:
typedef struct {
void *socket; // 0MQ socket to poll on
int fd; // OR, native file handle to poll on
short events; // Events to poll on
short revents; // Events returned after poll
} zmq_pollitem_t;
In the first version of this code I put the assert() call around the function. Not a good idea, since an optimized build will turn all assert() macros to null and happily wallop those functions. Use a return code, and assert the return code.
Let's see how to shut down a process cleanly. We'll take the parallel pipeline example from the previous section. If we've started a whole lot of workers in the background, we now want to kill them when the batch is finished. Let's do this by sending a kill message to the workers. The best place to do this is the sink, since it really knows when the batch is done.
How do we connect the sink to the workers? The PUSH/PULL sockets are one-way only. The standard ØMQ answer is: create a new socket flow for each type of problem you need to solve. We'll use a publish-subscribe model to send kill messages to the workers:
- The sink creates a PUB socket on a new endpoint.
- Workers bind their input socket to this endpoint.
- When the sink detects the end of the batch it sends a kill to its PUB socket.
- When a worker detects this kill message, it exits.
It doesn't take much new code in the sink:
// Receive 0MQ string from socket and convert into C string
static char *
s_recv (void *socket) {
zmq_msg_t message;
zmq_msg_init (&message);
int size = zmq_msg_recv (&message, socket, 0);
if (size == -1)
return NULL;
char *string = malloc (size + 1);
memcpy (string, zmq_msg_data (&message), size);
zmq_msg_close (&message);
string [size] = 0;
return (string);
// Convert C string to 0MQ string and send to socket
static int
s_send (void *socket, char *string) {
zmq_msg_t message;
zmq_msg_init_size (&message, strlen (string));
memcpy (zmq_msg_data (&message), string, strlen (string));
int size = zmq_msg_send (&message, socket, 0);
zmq_msg_close (&message);
return (size);
Figure 14 - Parallel Pipeline with Kill Signaling
Here is the worker process, which manages two sockets (a PULL socket getting tasks, and a SUB socket getting control commands) using the zmq_poll(3) technique we saw earlier:
// Task worker - design 2
// Adds pub-sub flow to receive and respond to kill signal
#include "zhelpers.h"
int main (void)
// Socket to receive messages on
void *context = zmq_ctx_new ();
void *receiver = zmq_socket (context, ZMQ_PULL);
zmq_connect (receiver, "tcp://localhost:5557");
// Socket to send messages to
void *sender = zmq_socket (context, ZMQ_PUSH);
zmq_connect (sender, "tcp://localhost:5558");
// Socket for control input
void *controller = zmq_socket (context, ZMQ_SUB);
zmq_connect (controller, "tcp://localhost:5559");
zmq_setsockopt (controller, ZMQ_SUBSCRIBE, "", 0);
// Process messages from either socket
while (1) {
zmq_pollitem_t items [] = {
{ receiver, 0, ZMQ_POLLIN, 0 },
{ controller, 0, ZMQ_POLLIN, 0 }
zmq_poll (items, 2, -1);
if (items [0].revents & ZMQ_POLLIN) {
char *string = s_recv (receiver);
printf ("%s.", string); // Show progress
fflush (stdout);
s_sleep (atoi (string)); // Do the work
free (string);
s_send (sender, ""); // Send results to sink
// Any waiting controller command acts as 'KILL'
if (items [1].revents & ZMQ_POLLIN)
break; // Exit loop
zmq_close (receiver);
zmq_close (sender);
zmq_close (controller);
zmq_ctx_destroy (context);
return 0;
taskwork2.c: Parallel task worker with kill signaling
Here is the modified sink application. When it's finished collecting results it broadcasts a KILL message to all workers:
// Task sink - design 2
// Adds pub-sub flow to send kill signal to workers
#include "zhelpers.h"
int main (void)
// Socket to receive messages on
void *context = zmq_ctx_new ();
void *receiver = zmq_socket (context, ZMQ_PULL);
zmq_bind (receiver, "tcp://*:5558");
// Socket for worker control
void *controller = zmq_socket (context, ZMQ_PUB);
zmq_bind (controller, "tcp://*:5559");
// Wait for start of batch
char *string = s_recv (receiver);
free (string);
// Start our clock now
int64_t start_time = s_clock ();
// Process 100 confirmations
int task_nbr;
for (task_nbr = 0; task_nbr < 100; task_nbr++) {
char *string = s_recv (receiver);
free (string);
if (task_nbr % 10 == 0)
printf (":");
printf (".");
fflush (stdout);
printf ("Total elapsed time: %d msec\n",
(int) (s_clock () - start_time));
// Send kill signal to workers
s_send (controller, "KILL");
zmq_close (receiver);
zmq_close (controller);
zmq_ctx_destroy (context);
return 0;
tasksink2.c: Parallel task sink with kill signaling
Handling Interrupt Signals
|top prev next
Realistic applications need to shutdown cleanly when interrupted with Ctrl-C or another signal such as SIGTERM. By default, these simply kill the process, meaning messages won't be flushed, files won't be closed cleanly, etc.
Here is how we handle a signal in various languages:
// Shows how to handle Ctrl-C
#include <stdlib.h>
#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <fcntl.h>
#include <zmq.h>
// Signal handling
// Create a self-pipe and call s_catch_signals(pipe's writefd) in your application
// at startup, and then exit your main loop if your pipe contains any data.
// Works especially well with zmq_poll.
#define S_NOTIFY_MSG " "
#define S_ERROR_MSG "Error while writing to self-pipe.\n"
static int s_fd;
static void s_signal_handler (int signal_value)
int rc = write (s_fd, S_NOTIFY_MSG, sizeof(S_NOTIFY_MSG));
if (rc != sizeof(S_NOTIFY_MSG)) {
write (STDOUT_FILENO, S_ERROR_MSG, sizeof(S_ERROR_MSG)-1);
static void s_catch_signals (int fd)
s_fd = fd;
struct sigaction action;
action.sa_handler = s_signal_handler;
// Doesn't matter if SA_RESTART set because self-pipe will wake up zmq_poll
// But setting to 0 will allow zmq_read to be interrupted.
action.sa_flags = 0;
sigemptyset (&action.sa_mask);
sigaction (SIGINT, &action, NULL);
sigaction (SIGTERM, &action, NULL);
int main (void)
int rc;
void *context = zmq_ctx_new ();
void *socket = zmq_socket (context, ZMQ_REP);
zmq_bind (socket, "tcp://*:5555");
int pipefds[2];
rc = pipe(pipefds);
if (rc != 0) {
perror("Creating self-pipe");
for (int i = 0; i < 2; i++) {
int flags = fcntl(pipefds, F_GETFL, 0);
if (flags < 0) {
perror ("fcntl(F_GETFL)");
rc = fcntl (pipefds, F_SETFL, flags | O_NONBLOCK);
if (rc != 0) {
perror ("fcntl(F_SETFL)");
s_catch_signals (pipefds[1]);
zmq_pollitem_t items [] = {
{ 0, pipefds[0], ZMQ_POLLIN, 0 },
{ socket, 0, ZMQ_POLLIN, 0 }
while (1) {
rc = zmq_poll (items, 2, -1);
if (rc == 0) {
} else if (rc < 0) {
if (errno == EINTR) { continue; }
// Signal pipe FD
if (items [0].revents & ZMQ_POLLIN) {
char buffer [1];
read (pipefds[0], buffer, 1); // clear notifying byte
printf ("W: interrupt received, killing server…\n");
// Read socket
if (items [1].revents & ZMQ_POLLIN) {
char buffer [255];
// Use non-blocking so we can continue to check self-pipe via zmq_poll
rc = zmq_recv (socket, buffer, 255, ZMQ_DONTWAIT);
if (rc < 0) {
if (errno == EAGAIN) { continue; }
if (errno == EINTR) { continue; }
printf ("W: recv\n");
// Now send message back.
// …
printf ("W: cleaning up\n");
zmq_close (socket);
zmq_ctx_destroy (context);
return 0;
interrupt.c: Handling Ctrl-C cleanly
The program provides s_catch_signals(), which traps Ctrl-C (SIGINT) and SIGTERM. When either of these signals arrive, the s_catch_signals() handler sets the global variable s_interrupted. Your application will not die automatically, you have to now explicitly check for an interrupt, and handle it properly. Here's how:
- Call s_catch_signals() (copy this from interrupt.c) at the start of your main code. This sets-up the signal handling.
- If your code is blocking in zmq_recv(3), zmq_poll(3), or zmq_send(3), when a signal arrives, the call will return with EINTR.
- Wrappers like s_recv() return NULL if they are interrupted.
- So, your application checks for an EINTR return code, a NULL return, and/or s_interrupted.
Here is a typical code fragment:
s_catch_signals ();
client = zmq_socket (...);
while (!s_interrupted) {
char *message = s_recv (client);
if (!message)
break; // Ctrl-C used
zmq_close (client);
If you call s_catch_signals() and don't test for interrupts, the your application will become immune to Ctrl-C and SIGTERM, which may be useful, but is usually not.
Detecting Memory Leaks
|top prev next
Any long-running application has to manage memory correctly, or eventually it'll use up all available memory and crash. If you use a language that handles this automatically for you, congratulations. If you program in C or C++ or any other language where you're responsible for memory management, here's a short tutorial on using valgrind, which among other things will report on any leaks your programs have.
- To install valgrind, e.g. on Ubuntu or Debian: sudo apt-get install valgrind.
- By default, ØMQ will cause valgrind to complain a lot. To remove these warnings, create a file valgrind.supp that contains this:
- Fix your applications to exit cleanly after Ctrl-C. For any application that exits by itself, that's not needed, but for long-running applications (like devices), this is essential, otherwise valgrind will complain about all currently allocated memory.
- Build your application with -DDEBUG, if it's not your default setting. That ensures valgrind can tell you exactly where memory is being leaked.
- Finally, run valgrind thus:
valgrind --tool=memcheck --leak-check=full --suppressions=valgrind.supp someprog
And after fixing any errors it reported, you should get the pleasant message:
==30536== ERROR SUMMARY: 0 errors from 0 contexts...
Multipart Messages
|top prev next
ØMQ lets us compose a message out of several frames, giving us a 'multipart message'. Realistic applications use multipart messages heavily, especially to make "envelopes". We'll look at them later. What we'll learn now is simply how to safely (but blindly) read and write multipart messages because otherwise the devices we write won't work with applications that use multipart messages.
When you work with multipart messages, each part is a zmq_msg item. E.g. if you are sending a message with five parts, you must construct, send, and destroy five zmq_msg items. You can do this in advance (and store the zmq_msg items in an array or structure), or as you send them, one by one.
Here is how we send the frames in a multipart message (we receive each frame into a message object):
void *control = zmq_socket (context, ZMQ_PUB);
zmq_bind (control, "tcp://*:5559");
// Send kill signal to workers
zmq_msg_init_data (&message, "KILL", 5);
zmq_msg_send (control, &message, 0);
zmq_msg_close (&message);
Here is how we receive and process all the parts in a message, be it single part or multipart:
zmq_msg_send (socket, &message, ZMQ_SNDMORE);
zmq_msg_send (socket, &message, ZMQ_SNDMORE);
zmq_msg_send (socket, &message, 0);
Some things to know about multipart messages:
- When you send a multipart message, the first part (and all following parts) are only sent when you send the final part.
- If you are using zmq_poll(3), when you receive the first part of a message, all the rest has also arrived.
- You will receive all parts of a message, or none at all.
- Each part of a message is a separate zmq_msg item.
- You will receive all parts of a message whether or not you check the RCVMORE option.
- On sending, ØMQ queues message frames in memory until the last is received, then sends them all.
- There is no way to cancel a partially sent message, except by closing the socket.
Intermediates and Devices
|top prev next
Any connected set hits a complexity curve as the number of set members increases. A small number of members can all know about each other but as the set gets larger, the cost to each member of knowing all other interesting members grows linearly, and the overall cost of connecting members is factorial. The solution is to break sets into smaller ones, and use intermediates to connect the sets.
This pattern is extremely common in the real world and is why our societies and economies are filled with intermediaries who have no other real function than to reduce the complexity and scaling costs of larger networks. Intermediaries are typically called wholesalers, distributors, managers, etc.
A ØMQ network like any cannot grow beyond a certain size without needing intermediaries. In ØMQ, we call these "devices". When we use ØMQ we usually start building our applications as a set of nodes on a network with the nodes talking to each other, without intermediaries.
Figure 15 - Small-scale ØMQ Application
And then we extend the application across a wider network, placing devices in specific places and scaling up the number of nodes.
Figure 16 - Larger-scale ØMQ Application
ØMQ devices generally connect a set of 'frontend' sockets to a set of 'backend' sockets, though there are no strict design rules. They ideally run with no state, so that it becomes possible to stretch applications over as many intermediates as needed. You can run them as threads within a process, or as stand-alone processes. ØMQ provides some very basic devices but you will in practice develop your own.
ØMQ devices can do intermediation of addresses, services, queues, or any other abstraction you care to define above the message and socket layers. Different messaging patterns have different complexity issues and need different kinds of intermediation. For example, request-reply works well with queue and service abstractions, while publish-subscribe works well with streams or topics.
What's interesting about ØMQ as compared to traditional centralized brokers is that you can place devices precisely where you need them, and they can do the optimal intermediation.
A Publish-Subscribe Proxy Server
|top prev next
It is a common requirement to extend a publish-subscribe architecture over more than one network segment or transport. Perhaps there are a group of subscribers sitting at a remote location. Perhaps we want to publish to local subscribers via multicast, and to remote subscribers via TCP.
We're going to write a simple proxy server that sits in between a publisher and a set of subscribers, bridging two networks. This is perhaps the simplest case of a useful device. The device has two sockets, a frontend facing the internal network, where the weather server is sitting, and a backend facing subscribers on the external network. It subscribes to the weather service on the frontend socket, and republishes its data on the backend socket:
// Weather proxy device
#include "zhelpers.h"
int main (void)
void *context = zmq_ctx_new ();
// This is where the weather server sits
void *frontend = zmq_socket (context, ZMQ_XSUB);
zmq_connect (frontend, "tcp://192.168.55.210:5556");
// This is our public endpoint for subscribers
void *backend = zmq_socket (context, ZMQ_XPUB);
zmq_bind (backend, "tcp://10.1.1.0:8100");
// Run the proxy until the user interrupts us
zmq_proxy (frontend, backend, NULL);
zmq_close (frontend);
zmq_close (backend);
zmq_ctx_destroy (context);
return 0;
wuproxy.c: Weather update proxy
We call this a proxy because it acts as a subscriber to publishers, and acts as a publisher to subscribers. That means you can slot this device into an existing network without affecting it (of course the new subscribers need to know to speak to the proxy).
Figure 17 - Forwarder Proxy Device
Note that this application is multipart safe. It correctly detects multipart messages and sends them as it reads them. If we did not set the SNDMORE option on outgoing multipart data, the final recipient would get a corrupted message. You should always make your devices multipart safe so that there is no risk they will corrupt the data they switch.
A Request-Reply Broker
|top prev next
Let's explore how to solve a problem of scale by writing a little message queuing broker in ØMQ. We'll look at the request-reply pattern for this case.
In the Hello World client-server application we have one client that talks to one service. However in real cases we usually need to allow multiple services as well as multiple clients. This lets us scale up the power of the service (many threads or processes or boxes rather than just one). The only constraint is that services must be stateless, all state being in the request or in some shared storage such as a database.
There are two ways to connect multiple clients to multiple servers. The brute-force way is to connect each client socket to multiple service endpoints. One client socket can connect to multiple service sockets, and the REQ socket will then load-balance requests among these services. Let's say you connect a client socket to three service endpoints, A, B, and C. The client makes requests R1, R2, R3, R4. R1 and R4 go to service A, R2 goes to B, and R3 goes to service C.
Figure 18 - Load-balancing of Requests
This design lets you add more clients cheaply. You can also add more services. Each client will load-balance its requests to the services. But each client has to know the service topology. If you have 100 clients and then you decide to add three more services, you need to reconfigure and restart 100 clients in order for the clients to know about the three new services.
That's clearly not the kind of thing we want to be doing at 3am when our supercomputing cluster has run out of resources and we desperately need to add a couple of hundred new service nodes. Too many stable pieces are like liquid concrete: knowledge is distributed and the more stable pieces you have, the more effort it is to change the topology. What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add and remove services or clients at any time without touching any other part of the topology.
So we'll write a little message queuing broker that gives us this flexibility. The broker binds to two endpoints, a frontend for clients and a backend for services. It then uses zmq_poll(3) to monitor these two sockets for activity and when it has some, it shuttles messages between its two sockets. It doesn't actually manage any queues explicitly — ØMQ does that automatically on each socket.
When you use REQ to talk to REP you get a strictly synchronous request-reply dialog. The client sends a request, the service reads the request and sends a reply. The client then reads the reply. If either the client or the service try to do anything else (e.g. sending two requests in a row without waiting for a response) they will get an error.
But our broker has to be non-blocking. Obviously we can use zmq_poll(3) to wait for activity on either socket, but we can't use REP and REQ.
Luckily there are two sockets called DEALER and ROUTER that let you do non-blocking request-response. These sockets used to be called XREQ and XREP, and you may see these names in old code. The old names suggested that XREQ was an "extended REQ" and XREP was an "extended REP" but that's inaccurate. You'll see in Chapter Three how DEALER and ROUTER sockets let you build all kinds of asynchronous request-reply flows.
Now, we're just going to see how DEALER and ROUTER let us extend REQ-REP across a device, that is, our little broker.
In this simple stretched request-reply pattern, REQ talks to ROUTER and DEALER talks to REP. In between the DEALER and ROUTER we have to have code (like our broker) that pulls messages off the one socket and shoves them onto the other.
Figure 19 - Extended Request-reply
The request-reply broker binds to two endpoints, one for clients to connect to (the frontend socket) and one for services to connect to (the backend). To test this broker, you will want to change your services so they connect to the backend socket. Here are a client and service that show what I mean:
// Hello World client
// Connects REQ socket to tcp://localhost:5559
// Sends "Hello" to server, expects "World" back
#include "zhelpers.h"
int main (void)
void *context = zmq_ctx_new ();
// Socket to talk to server
void *requester = zmq_socket (context, ZMQ_REQ);
zmq_connect (requester, "tcp://localhost:5559");
int request_nbr;
for (request_nbr = 0; request_nbr != 10; request_nbr++) {
s_send (requester, "Hello");
char *string = s_recv (requester);
printf ("Received reply %d [%s]\n", request_nbr, string);
free (string);
zmq_close (requester);
zmq_ctx_destroy (context);
return 0;
rrclient.c: Request-reply client
Here is the service:
// Hello World server
// Connects REP socket to tcp://*:5560
// Expects "Hello" from client, replies with "World"
#include "zhelpers.h"
int main (void)
void *context = zmq_init (1);
// Socket to talk to clients
void *responder = zmq_socket (context, ZMQ_REP);
zmq_connect (responder, "tcp://localhost:5560");
while (1) {
// Wait for next request from client
char *string = s_recv (responder);
printf ("Received request: [%s]\n", string);
free (string);
// Do some 'work'
sleep (1);
// Send reply back to client
s_send (responder, "World");
// We never get here but clean up anyhow
zmq_close (responder);
zmq_term (context);
return 0;
rrserver.c: Request-reply service
And here is the broker. You will see that it's multipart safe:
// Simple request-reply broker
#include "zhelpers.h"
int main (void)
// Prepare our context and sockets
void *context = zmq_ctx_new ();
void *frontend = zmq_socket (context, ZMQ_ROUTER);
void *backend = zmq_socket (context, ZMQ_DEALER);
zmq_bind (frontend, "tcp://*:5559");
zmq_bind (backend, "tcp://*:5560");
// Initialize poll set
zmq_pollitem_t items [] = {
{ frontend, 0, ZMQ_POLLIN, 0 },
{ backend, 0, ZMQ_POLLIN, 0 }
// Switch messages between sockets
while (1) {
zmq_msg_t message;
zmq_poll (items, 2, -1);
if (items [0].revents & ZMQ_POLLIN) {
while (1) {
// Process all parts of the message
zmq_msg_init (&message);
zmq_msg_recv (&message, frontend, 0);
int more = zmq_msg_more (&message);
zmq_msg_send (&message, backend, more? ZMQ_SNDMORE: 0);
zmq_msg_close (&message);
if (!more)
break; // Last message part
if (items [1].revents & ZMQ_POLLIN) {
while (1) {
// Process all parts of the message
zmq_msg_init (&message);
zmq_msg_recv (&message, backend, 0);
int more = zmq_msg_more (&message);
zmq_msg_send (&message, frontend, more? ZMQ_SNDMORE: 0);
zmq_msg_close (&message);
if (!more)
break; // Last message part
// We never get here, but clean up anyhow
zmq_close (frontend);
zmq_close (backend);
zmq_ctx_destroy (context);
return 0;
rrbroker.c: Request-reply broker
Using a request-reply broker makes your client-server architectures easier to scale since clients don't see services, and services don't see clients. The only stable node is the device in the middle.
Figure 20 - Request-reply Broker
Built-in Devices
|top prev next
ØMQ provides some built-in devices, though most advanced users write their own devices. The built-in devices are:
- QUEUE, which is like the request-reply broker.
- FORWARDER, which is like the pub-sub proxy server.
- STREAMER, which is like FORWARDER but for pipeline flows.
To start a device, you call zmq_device(3) and pass it two sockets, one for the frontend and one for the backend:
while (1) {
zmq_msg_t message;
zmq_msg_init (&message);
zmq_msg_recv (socket, &message, 0);
// Process the message frame
zmq_msg_close (&message);
int64_t more;
size_t more_size = sizeof (more);
zmq_getsockopt (socket, ZMQ_RCVMORE, &more, &more_size);
if (!more)
break; // Last message frame
Which if you start a QUEUE device is exactly like plugging the main body of the request-reply broker into your code at that spot. You need to create the sockets, bind or connect them, and possibly configure them, before calling zmq_device(3). It is trivial to do. Here is the request-reply broker re-written to call QUEUE and rebadged as an expensive-sounding "message queue" (people have charged houses for code that did less):
// Simple message queuing broker
// Same as request-reply broker but using shared queue proxy
#include "zhelpers.h"
int main (void)
void *context = zmq_ctx_new ();
// Socket facing clients
void *frontend = zmq_socket (context, ZMQ_ROUTER);
int rc = zmq_bind (frontend, "tcp://*:5559");
assert (rc == 0);
// Socket facing services
void *backend = zmq_socket (context, ZMQ_DEALER);
rc = zmq_bind (backend, "tcp://*:5560");
assert (rc == 0);
// Start the proxy
zmq_proxy (frontend, backend, NULL);
// We never get here…
zmq_close (frontend);
zmq_close (backend);
zmq_ctx_destroy (context);
return 0;
msgqueue.c: Message queue broker
The built-in devices do proper error handling, whereas the examples we have shown don't. Since you can configure the sockets as you need to, before starting the device, it's worth using the built-in devices when you can.
If you're like most ØMQ users, at this stage your mind is starting to think, "what kind of evil stuff can I do if I plug random socket types into devices?" The short answer is: don't do it. You can mix socket types but the results are going to be weird. So stick to using ROUTER/DEALER for queue devices, SUB/PUB for forwarders and PULL/PUSH for streamers.
When you start to need other combinations, it's time to write your own devices.
Multithreading with ØMQ
|top prev next
ØMQ is perhaps the nicest way ever to write multithreaded (MT) applications. Whereas as ØMQ sockets require some readjustment if you are used to traditional sockets, ØMQ multithreading will take everything you know about writing MT applications, throw it into a heap in the garden, pour gasoline over it, and set it alight. It's a rare book that deserves burning, but most books on concurrent programming do.
To make utterly perfect MT programs (and I mean that literally) we don't need mutexes, locks, or any other form of inter-thread communication except messages sent across ØMQ sockets.
By "perfect" MT programs I mean code that's easy to write and understand, that works with one technology in any language and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns.
If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing. If there's one lesson we've learned from 30+ years of concurrent programming it is: just don't share state. It's like two drunkards trying to share a beer. It doesn't matter if they're good buddies. Sooner or later they're going to get into a fight. And the more drunkards you add to the pavement, the more they fight each other over the beer. The tragic majority of MT applications look like drunken bar fights.
The list of weird problems that you need to fight as you write classic shared-state MT code would be hilarious if it didn't translate directly into stress and risk, as code that seems to work suddenly fails under pressure. Here is a list of "11 Likely Problems In Your Multithreaded Code" from a large firm with world-beating experience in buggy code: forgotten synchronization, incorrect granularity, read and write tearing, lock-free reordering, lock convoys, two-step dance, and priority inversion.
Yeah, we also counted seven, not eleven. That's not the point though. The point is, do you really want that code running the power grid or stock market to start getting two-step lock convoys at 3pm on a busy Thursday? Who cares what the terms actually mean. This is not what turned us on to programming, fighting ever more complex side-effects with ever more complex hacks.
Some widely used metaphors, despite being the basis for billion-dollar industries, are fundamentally broken, and shared state concurrency is one of them. Code that wants to scale without limit does it like the Internet does, by sending messages and sharing nothing except a common contempt for broken programming metaphors.
You should follow some rules to write happy multithreaded code with ØMQ:
- You MUST NOT access the same data from multiple threads. Using classic MT techniques like mutexes are an anti-pattern in ØMQ applications. The only exception to this is a ØMQ context object, which is threadsafe.
- You MUST create a ØMQ context for your process, and pass that to all threads that you want to connect via inproc sockets.
- You MAY treat threads as separate tasks, with their own context, but these threads cannot communicate over inproc. However they will be easier to break into standalone processes afterwards.
- You MUST NOT share ØMQ sockets between threads. ØMQ sockets are not threadsafe. Technically it's possible to do this, but it demands semaphores, locks, or mutexes. This will make your application slow and fragile. The only place where it's remotely sane to share sockets between threads are in language bindings that need to do magic like garbage collection on sockets.
If you need to start more than one device in an application, for example, you will want to run each in their own thread. It is easy to make the error of creating the device sockets in one thread, and then passing the sockets to the device in another thread. This may appear to work but will fail randomly. Remember: Do not use or close sockets except in the thread that created them.
If you follow these rules, you can quite easily split threads into separate processes, when you need to. Application logic can sit in threads, processes, boxes: whatever your scale needs.
ØMQ uses native OS threads rather than virtual "green" threads. The advantage is that you don't need to learn any new threading API, and that ØMQ threads map cleanly to your operating system. You can use standard tools like Intel's ThreadChecker to see what your application is doing. The disadvantages are that your code, when it for instance starts new threads, won't be portable, and that if you have a huge number of threads (thousands), some operating systems will get stressed.
Let's see how this works in practice. We'll turn our old Hello World server into something more capable. The original server was a single thread. If the work per request is low, that's fine: one ØMQ thread can run at full speed on a CPU core, with no waits, doing an awful lot of work. But realistic servers have to do non-trivial work per request. A single core may not be enough when 10,000 clients hit the server all at once. So a realistic server must start multiple worker threads. It then accepts requests as fast as it can, and distributes these to its worker threads. The worker threads grind through the work, and eventually send their replies back.
You can of course do all this using a queue device and external worker processes, but often it's easier to start one process that gobbles up sixteen cores, than sixteen processes, each gobbling up one core. Further, running workers as threads will cut out a network hop, latency, and network traffic.
The MT version of the Hello World service basically collapses the queue device and workers into a single process:
// Multithreaded Hello World server
#include "zhelpers.h"
#include <pthread.h>
#include <unistd.h>
static void *
worker_routine (void *context) {
// Socket to talk to dispatcher
void *receiver = zmq_socket (context, ZMQ_REP);
zmq_connect (receiver, "inproc://workers");
while (1) {
char *string = s_recv (receiver);
printf ("Received request: [%s]\n", string);
free (string);
// Do some 'work'
sleep (1);
// Send reply back to client
s_send (receiver, "World");
zmq_close (receiver);
return NULL;
int main (void)
void *context = zmq_ctx_new ();
// Socket to talk to clients
void *clients = zmq_socket (context, ZMQ_ROUTER);
zmq_bind (clients, "tcp://*:5555");
// Socket to talk to workers
void *workers = zmq_socket (context, ZMQ_DEALER);
zmq_bind (workers, "inproc://workers");
// Launch pool of worker threads
int thread_nbr;
for (thread_nbr = 0; thread_nbr < 5; thread_nbr++) {
pthread_t worker;
pthread_create (&worker, NULL, worker_routine, context);
// Connect work threads to client threads via a queue proxy
zmq_proxy (clients, workers, NULL);
// We never get here, but clean up anyhow
zmq_close (clients);
zmq_close (workers);
zmq_ctx_destroy (context);
return 0;
mtserver.c: Multithreaded service
All the code should be recognizable to you by now. How it works:
- The server starts a set of worker threads. Each worker thread creates a REP socket and then processes requests on this socket. Worker threads are just like single-threaded servers. The only differences are the transport (inproc instead of tcp), and the bind-connect direction.
- The server creates a ROUTER socket to talk to clients and binds this to its external interface (over tcp).
- The server creates a DEALER socket to talk to the workers and binds this to its internal interface (over inproc).
- The server starts a QUEUE device that connects the two sockets. The QUEUE device keeps a single queue for incoming requests, and distributes those out to workers. It also routes replies back to their origin.
Note that creating threads is not portable in most programming languages. The POSIX library is pthreads, but on Windows you have to use a different API. We'll see in Chapter Three how to wrap this in a portable API.
Here the 'work' is just a one-second pause. We could do anything in the workers, including talking to other nodes. This is what the MT server looks like in terms of ØMQ sockets and nodes. Note how the request-reply chain is REQ-ROUTER-queue-DEALER-REP.
Figure 21 - Multithreaded Server
Signaling between Threads
|top prev next
When you start making multithreaded applications with ØMQ, you'll hit the question of how to coordinate your threads. Though you might be tempted to insert 'sleep' statements, or use multithreading techniques such as semaphores or mutexes, the only mechanism that you should use are ØMQ messages. Remember the story of The Drunkards and the Beer Bottle.
Let's make three threads that signal each other when they are ready. In this example we use PAIR sockets over the inproc transport:
// Multithreaded relay
#include "zhelpers.h"
#include <pthread.h>
static void *
step1 (void *context) {
// Connect to step2 and tell it we're ready
void *xmitter = zmq_socket (context, ZMQ_PAIR);
zmq_connect (xmitter, "inproc://step2");
printf ("Step 1 ready, signaling step 2\n");
s_send (xmitter, "READY");
zmq_close (xmitter);
return NULL;
static void *
step2 (void *context) {
// Bind inproc socket before starting step1
void *receiver = zmq_socket (context, ZMQ_PAIR);
zmq_bind (receiver, "inproc://step2");
pthread_t thread;
pthread_create (&thread, NULL, step1, context);
// Wait for signal and pass it on
char *string = s_recv (receiver);
free (string);
zmq_close (receiver);
// Connect to step3 and tell it we're ready
void *xmitter = zmq_socket (context, ZMQ_PAIR);
zmq_connect (xmitter, "inproc://step3");
printf ("Step 2 ready, signaling step 3\n");
s_send (xmitter, "READY");
zmq_close (xmitter);
return NULL;
int main (void)
void *context = zmq_ctx_new ();
// Bind inproc socket before starting step2
void *receiver = zmq_socket (context, ZMQ_PAIR);
zmq_bind (receiver, "inproc://step3");
pthread_t thread;
pthread_create (&thread, NULL, step2, context);
// Wait for signal
char *string = s_recv (receiver);
free (string);
zmq_close (receiver);
printf ("Test successful!\n");
zmq_ctx_destroy (context);
return 0;
mtrelay.c: Multithreaded relay
Figure 22 - The Relay Race
This is a classic pattern for multithreading with ØMQ:
- Two threads communicate over inproc, using a shared context.
- The parent thread creates one socket, binds it to an inproc:// endpoint, and then starts the child thread, passing the context to it.
- The child thread creates the second socket, connects it to that inproc:// endpoint, and then signals to the parent thread that it's ready.
Note that multithreading code using this pattern is not scalable out to processes. If you use inproc and socket pairs, you are building a tightly-bound application. Do this when low latency is really vital. For all normal apps, use one context per thread, and ipc or tcp. Then you can easily break your threads out to separate processes, or boxes, as needed.
This is the first time we've shown an example using PAIR sockets. Why use PAIR? Other socket combinations might seem to work but they all have side-effects that could interfere with signaling:
- You can use PUSH for the sender and PULL for the receiver. This looks simple and will work, but remember that PUSH will load-balance messages to all available receivers. If you by accident start two receivers (e.g. you already have one running and you start a second), you'll "lose" half of your signals. PAIR has the advantage of refusing more than one connection, the pair is exclusive.
- You can use DEALER for the sender and ROUTER for the receiver. ROUTER however wraps your message in an "envelope", meaning your zero-size signal turns into a multipart message. If you don't care about the data, and treat anything as a valid signal, and if you don't read more than once from the socket, that won't matter. If however you decide to send real data, you will suddenly find ROUTER providing you with "wrong" messages. DEALER also load-balances, giving the same risk as PUSH.
- You can use PUB for the sender and SUB for the receiver. This will correctly deliver your messages exactly as you sent them and PUB does not load-balance as PUSH or DEALER do. However you need to configure the subscriber with an empty subscription, which is annoying. Worse, the reliability of the PUB-SUB link is timing dependent and messages can get lost if the SUB socket is connecting while the PUB socket is sending its messages.
For these reasons, PAIR makes the best choice for coordination between pairs of threads.
Node Coordination
|top prev next
When you want to coordinate nodes, PAIR sockets won't work well any more. This is one of the few areas where the strategies for threads and nodes are different. Principally nodes come and go whereas threads are stable. PAIR sockets do not automatically reconnect if the remote node goes away and comes back.
The second significant difference between threads and nodes is that you typically have a fixed number of threads but a more variable number of nodes. Let's take one of our earlier scenarios (the weather server and clients) and use node coordination to ensure that subscribers don't lose data when starting up.
This is how the application will work:
- The publisher knows in advance how many subscribers it expects. This is just a magic number it gets from somewhere.
- The publisher starts up and waits for all subscribers to connect. This is the node coordination part. Each subscriber subscribes and then tells the publisher it's ready via another socket.
- When the publisher has all subscribers connected, it starts to publish data.
In this case we'll use a REQ-REP socket flow to synchronize subscribers and publisher. Here is the publisher:
// Synchronized publisher
#include "zhelpers.h"
#define SUBSCRIBERS_EXPECTED 10 //// We wait for 10 subscribers //
int main (void)
void *context = zmq_ctx_new ();
// Socket to talk to clients
void *publisher = zmq_socket (context, ZMQ_PUB);
int sndhwm = 1100000;
zmq_setsockopt (publisher, ZMQ_SNDHWM, &sndhwm, sizeof (int));
zmq_bind (publisher, "tcp://*:5561");
// Socket to receive signals
void *syncservice = zmq_socket (context, ZMQ_REP);
zmq_bind (syncservice, "tcp://*:5562");
// Get synchronization from subscribers
printf ("Waiting for subscribers\n");
int subscribers = 0;
while (subscribers < SUBSCRIBERS_EXPECTED) {
// - wait for synchronization request
char *string = s_recv (syncservice);
free (string);
// - send synchronization reply
s_send (syncservice, "");
// Now broadcast exactly 1M updates followed by END
printf ("Broadcasting messages\n");
int update_nbr;
for (update_nbr = 0; update_nbr < 1000000; update_nbr++)
s_send (publisher, "Rhubarb");
s_send (publisher, "END");
zmq_close (publisher);
zmq_close (syncservice);
zmq_ctx_destroy (context);
return 0;
syncpub.c: Synchronized publisher
Figure 23 - Pub-Sub Synchronization
And here is the subscriber:
// Synchronized subscriber
#include "zhelpers.h"
#include <unistd.h>
int main (void)
void *context = zmq_ctx_new ();
// First, connect our subscriber socket
void *subscriber = zmq_socket (context, ZMQ_SUB);
zmq_connect (subscriber, "tcp://localhost:5561");
zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, "", 0);
// 0MQ is so fast, we need to wait a while…
sleep (1);
// Second, synchronize with publisher
void *syncclient = zmq_socket (context, ZMQ_REQ);
zmq_connect (syncclient, "tcp://localhost:5562");
// - send a synchronization request
s_send (syncclient, "");
// - wait for synchronization reply
char *string = s_recv (syncclient);
free (string);
// Third, get our updates and report how many we got
int update_nbr = 0;
while (1) {
char *string = s_recv (subscriber);
if (strcmp (string, "END") == 0) {
free (string);
free (string);
printf ("Received %d updates\n", update_nbr);
zmq_close (subscriber);
zmq_close (syncclient);
zmq_ctx_destroy (context);
return 0;
syncsub.c: Synchronized subscriber
This Linux shell script will start ten subscribers and then the publisher:
echo "Starting subscribers..."
for a in 1 2 3 4 5 6 7 8 9 10; do
syncsub &
echo "Starting publisher..."
Which gives us this satisfying output:
Starting subscribers...
Starting publisher...
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
Received 1000000 updates
We can't assume that the SUB connect will be finished by the time the REQ/REP dialog is complete. There are no guarantees that outbound connects will finish in any order whatsoever, if you're using any transport except inproc. So, the example does a brute-force sleep of one second between subscribing, and sending the REQ/REP synchronization.
A more robust model could be:
- Publisher opens PUB socket and starts sending "Hello" messages (not data).
- Subscribers connect SUB socket and when they receive a Hello message they tell the publisher via a REQ/REP socket pair.
- When the publisher has had all the necessary confirmations, it starts to send real data.
Zero Copy
|top prev next
We teased you in Chapter One, when you were still a ØMQ newbie, about zero-copy. If you survived this far, you are probably ready to use zero-copy. However, remember that there are many roads to Hell, and premature optimization is not the most enjoyable nor profitable one, by far. In English, trying to do zero-copy properly while your architecture is not perfect is a waste of time and will make things worse, not better.
ØMQ's message API lets you can send and receive messages directly from and to application buffers without copying data. Given that ØMQ sends messages in the background, zero-copy needs some extra sauce.
To do zero-copy you use zmq_msg_init_data(3) to create a message that refers to a block of data already allocated on the heap with malloc(), and then you pass that to zmq_send(3). When you create the message you also pass a function that ØMQ will call to free the block of data, when it has finished sending the message. This is the simplest example, assuming 'buffer' is a block of 1000 bytes allocated on the heap:
zmq_proxy (frontend, backend, capture);
There is no way to do zero-copy on receive: ØMQ delivers you a buffer that you can store as long as you wish but it will not write data directly into application buffers.
On writing, ØMQ's multipart messages work nicely together with zero-copy. In traditional messaging you need to marshal different buffers together into one buffer that you can send. That means copying data. With ØMQ, you can send multiple buffers coming from different sources as individual message frames. We send each field as a length-delimited frame. To the application it looks like a series of send and recv calls. But internally the multiple parts get written to the network and read back with single system calls, so it's very efficient.
Pub-sub Message Envelopes
|top prev next
We've looked briefly at multipart messages. Let's now look at their main use-case, which is message envelopes. An envelope is a way of safely packaging up data with an address, without touching the data itself.
In the pub-sub pattern, the envelope at least holds the subscription key for filtering but you can also add the sender identity in the envelope.
If you want to use pub-sub envelopes, you make them yourself. It's optional, and in previous pub-sub examples we didn't do this. Using a pub-sub envelope is a little more work for simple cases but it's cleaner especially for real cases, where the key and the data are naturally separate things. It's also faster, if you are writing the data directly from an application buffer.
Here is what a publish-subscribe message with an envelope looks like:
Figure 24 - Pub-sub Envelope with Separate Key
Recall that pub-sub matches messages based on the prefix. Putting the key into a separate frame makes the matching very obvious, since there is no chance an application will accidentally match on part of the data.
Here is a minimalist example of how pub-sub envelopes look in code. This publisher sends messages of two types, A and B. The envelope holds the message type:
// Pubsub envelope publisher
// Note that the zhelpers.h file also provides s_sendmore
#include "zhelpers.h"
#include <unistd.h>
int main (void)
// Prepare our context and publisher
void *context = zmq_ctx_new ();
void *publisher = zmq_socket (context, ZMQ_PUB);
zmq_bind (publisher, "tcp://*:5563");
while (1) {
// Write two messages, each with an envelope and content
s_sendmore (publisher, "A");
s_send (publisher, "We don't want to see this");
s_sendmore (publisher, "B");
s_send (publisher, "We would like to see this");
sleep (1);
// We never get here, but clean up anyhow
zmq_close (publisher);
zmq_ctx_destroy (context);
return 0;
psenvpub.c: Pub-sub envelope publisher
The subscriber only wants messages of type B:
// Pubsub envelope subscriber
#include "zhelpers.h"
int main (void)
// Prepare our context and subscriber
void *context = zmq_ctx_new ();
void *subscriber = zmq_socket (context, ZMQ_SUB);
zmq_connect (subscriber, "tcp://localhost:5563");
zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, "B", 1);
while (1) {
// Read envelope with address
char *address = s_recv (subscriber);
// Read message contents
char *contents = s_recv (subscriber);
printf ("[%s] %s\n", address, contents);
free (address);
free (contents);
// We never get here, but clean up anyhow
zmq_close (subscriber);
zmq_ctx_destroy (context);
[[span style="color:#008000"]]return [[span style="color:#666666"]]0;
psenvsub.c: Pub-sub envelope subscriber
When you run the two programs, the subscriber should show you this:
[B] We would like to see this
[B] We would like to see this
[B] We would like to see this
[B] We would like to see this
This examples shows that the subscription filter rejects or accepts the entire multipart message (key plus data). You won't get part of a multipart message, ever.
If you subscribe to multiple publishers and you want to know their identity so that you can send them data via another socket (and this is a fairly typical use-case), you create a three-part message:
Figure 25 - Pub-sub Envelope with Sender Address
High Water Marks
|top prev next
When you can send messages rapidly from process to process, you soon discover that memory is a precious resource, and one that's trivially filled up. A few seconds delay somewhere in a process can turn into a backlog that blows up a server, unless you understand the problem and take precautions.
The problem is this: if you have process A sending messages to process B, which suddenly gets very busy (garbage collection, CPU overload, whatever), then what happens to the messages that process A wants to send? Some will sit in B's network buffers. Some will sit on the Ethernet wire itself. Some will sit in A's network buffers. And the rest will accumulate in A's memory. If you don't take some precaution, A can easily run out of memory and crash. It is a consistent, classic problem with message brokers.
What are the answers? One is to pass the problem upstream. A is getting the messages from somewhere else. So tell that process, "stop!" And so on. This is called "flow control". It sounds great, but what if you're sending out a Twitter feed? Do you tell the whole world to stop tweeting while B gets its act together?
Flow control works in some cases but in others, the transport layer can't tell the application layer "stop" any more than a subway system can tell a large business, "please keep your staff at work another half an hour, I'm too busy".
The answer for messaging is to set limits on the size of buffers, and then when we reach those limits, take some sensible action. In most cases (not for a subway system, though), the answer is to throw away messages. In a few others, it's to wait.
ØMQ uses the concept of "high water mark" or HWM to define the capacity of its internal pipes. Each connection out of a socket or into a socket has its own pipe, and HWM capacity.
In ØMQ/2.x the HWM was set to infinite by default. In ØMQ/3.x it's set to 1,000 by default, which is more sensible. If you're using ØMQ/2.x you should always set a HWM on your sockets, be it 1,000 to match ØMQ/3.x or another figure that takes into account your message sizes.
The high water mark affects both the transmit and receive buffers of a single socket. Some sockets (PUB, PUSH) only have transmit buffers. Some (SUB, PULL, REQ, REP) only have receive buffers. Some (DEALER, ROUTER, PAIR) have both transmit and receive buffers.
When your socket reaches its high-water mark, it will either block or drop data depending on the socket type. PUB sockets will drop data if they reach their high-water mark, while other socket types will block.
Over the inproc transport, the sender and receiver share the same buffers, so the real HWM is the sum of the HWM set by both sides. This means in effect that if one side does not set a HWM, there is no limit to the buffer size.
A Bare Necessity
|top prev next
ØMQ is like a box of pieces that plug together, the only limitation being your imagination and sobriety.
The scalable elastic architecture you get should be an eye-opener. You might need a coffee or two first. Don't make the mistake I made once and buy exotic German coffee labeled Entkoffeiniert. That does not mean "Delicious". Scalable elastic architectures are not a new idea - flow-based programming and languages like Erlang already worked like this - but ØMQ makes it easier to use than ever before.
As Gonzo Diethelm said, 'My gut feeling is summarized in this sentence: "if ØMQ didn't exist, it would be necessary to invent it". Meaning that I ran into ØMQ after years of brain-background processing, and it made instant sense… ØMQ simply seems to me a "bare necessity" nowadays.'


text:
Click here to get information on Police Clearance Certificate from Asian Countries
Police Clearance Certificate (PCC) In Andhra Pradesh
We provide Fingerprint Services for Police Clearance Certificate (PCC), Visa/Immigration Application in Andhra Pradesh
RealScan Biometric ensure that the immigration visa documentation process is smooth and stress-free for you even as they carefully examine every requirement related to permit documentation, which may be applicable to the specific visa plan of your chosen immigration hotspot. The necessary supporting certificates/documents for visa and immigration purposes will be prepared by the experienced advisers. Our RealScan advisers show you the way on everything. we provide highly professional services to our clients using various modes and techniques of fingerprinting as per the requirement like Ink-Rolled Fingerprinting. We are providing Fingerprinting Services from more than a decade, with our well-equipped and well-qualified team of experts. The fingerprinting service provided by us is strictly as per the rules and regulations, mentioned/defined by the concerned Country/Embassy.
We provide our clients Quality Service, by providing them timely update about the process status, irrespective of the time and distance issues. Working in the field of forensics since a long time and providing our clients with the satisfactory response, we are one of the most experienced agency to provide fingerprint services around the country, with Excellency, Perfection and Highly-affordable services.
We do provide On-Site Fingerprinting Service* to our customers who wish to have the Ink Fingerprinting done at their site instead of coming to our office. The applicant can opt for this service as per their schedule in Andhra Pradesh, India.
Different Kinds of Fingerprinting Services Available in Andhra Pradesh Branch
- Fast fingerprinting for PCC
- Digital fingerprints for PCC
- Fingerprint consultant for visa
- Fingerprint consultant for Visa
- FBI FD 258 card fingerprint expert
- Fingerprint consultant for passports
- RCMP Canada Fingerprint on Form C216C
- Fingerprints for FBI clearance in INDIA
- Fingerprinting for good citizen certificate
- Fingerprinting for visa immigration in India
- FBI fingerprints criminal record check India
- Agency for live scan fingerprint in Andhra Pradesh
- Fingerprinting Services for SAPS-91 South African fingerprint card
Fingerprint Services Available Various Regions in Andhra Pradesh
Nellore, Cuddapah, Karim Nagar, Kurnool, West Godavari, Srikakulam, Anantpur, Adilabad, Chittor, east Godavari, Guntur, Hyderabad Urban, Khammam, Krishna, Mehboobnagar, Medak, Nalgonda, Nizamabad, Prakasam, Ranga Reddy, Vishakapatnam, Vizianagaram, Warangal.
We assist to get PCC from these Countries at Andhra Pradesh
Afghanistan, African Republic, Alabama, Alaska, Albania, Algeria, American Samoa, Andorra, Angola, Anguilla, Antigua and Barbuda, Arab, Argentina, Arizona, Aruba, Australia, Austria, Azerbaijan, Bahamas, Bahrain, Bangladesh, arbados, Belarus, Belgium, Belize, Benin, Bermuda, Bhutan, Bolivia, Bosnia, Botswana, Brazil, British Virgin Islands, Brunei, Bulgaria, Burkina Faso, Burundi, California, Cambodia, Cameroon, Canada, Capo Verde, Caribbean, Cayman Islands, Chad, Chile, China, Colombia, Colorado, Comoros, Connecticut, Cook Islands, Costa Rica, Croatia, Cuba, Curaçao, Cyprus, Czech Republic, Delaware, Democratic Congo, Denmark, Djibouti, Dominica, Dominican Republic, Ecuador, Egypt, El Salvador, Equatorial Guinea, Eritrea, Estonia, Ethiopia, Falkland Islands, Fiji, Finland, Florida, France, French Guiana, French Polynesia, Gabon, Gambia, Georgia, eorgia,Germany, Ghana, Greece, Greenland, Grenada, Guadeloupe, Guam, Guatemala, Guinea, Guinea-Bissau, Guyana, Haiti, Hawaii, Honduras, HongKong, Hungary, Iceland, Idaho, Illinois, India, Indiana, Indonesia, Iran, Ireland, Italy, Ivory Coast, Jamaica, Japan, Jordan, Kansas, Kazakhstan, Kazakhstan, Kentucky, Kenya, Kiribati, Kuwait, Kyrgyzstan, Laos, Latvia, Lebanon, Lesotho, Liberia, Liberland, Libya, Liechtenstein, Lithuania, Louisiana, Lowa, Luxembourg, Macau, Macedonia, Madagascar, Maine, Malawi, Malaysia, Maldives, Mali, Malta, Marshall Islands, Martinique, Maryland, Massachusetts, Mauritania, Mauritius, Mayotte, Mexico, Michigan, Micronesia, Minnesota, Mississippi, Missouri, Moldova, Monaco, Mongolia, Montenegro, Montserrat, Morocco, Mozambique, Myanmar, Namibia, Nauru, Nepal, Netherlands, New Caledonia, New Zealand, Nicaragua, Niger, Nigeria, Niue NZ, Norfolk Island, North Korea, Norway, Oman, Pakistan, Palau, Palestine, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Pitcairn Islands, Poland, Portugal, Puerto Rico, Qatar, Republic of Israel, Republic of Congo, Romania, Rwanda, Saint Barthélemy, Saint Martin, Saint Pierre, Samoa, San Marino, São Tomé, Saudi Arabia, Senegal, Serbia, Seychelles, Sierra Leone, Singapore, Sint Maarten, Slovakia, Solomon Islands, Somalia, South Africa, South Korea, South Sudan, Spain, Sri Lanka, St. Kitts, St. Lucia, St. Vincent, Sudan, Suriname, Swaziland, Sweden, Switzerland, Syria, Taiwan, Tajikistan, Tanzania, Thailand, Timor-Leste, Togo, Tokelau, Tonga, Trinidad and Tobago Tunisia, Turkey, Turkmenistan, Turks and Caicos Islands, Tuvalu, Uganda, United Arab Emirates, United Kingdom, United States, Uruguay, Uzbekistan, Vanuatu, Vatican City, Venezuela, Vietnam, Virgin Islands, Wallis and Futuna, Yemen, Yugoslavia, Zambia, Zimbabwe.
*Terms and Conditions Apply.
[php snippet=1]

subset: commoncrawl


text:
Learning From Aaron Swartz: Content Must Not Be The End Game For Knowledge
from the serious-stuff dept
In the wake of the suicide of Aaron Swartz, there have been many fine tributes to the man and his work. Another growing class of posts that have flowed from this unhappy event are people reflecting on the important lessons he taught them. Here, for instance, is Jeff Jarvis recounting his journey from a fairly traditional position on copyright to one that recognized how the Internet had reshaped that landscape.
Ten years ago, Jarvis appended this "mock copyright notice" to his blog posts:
It's mine, I tell you, mine! All mine! You can't have it because it's mine! You can read it (please); you can quote it (thanks); but I still own it because its mine! I own it and you don't. Nya-nya-nya. So there. COPYRIGHT ... by Jeff Jarvis.
But gradually, under the influence of key thinkers in this area, he came to see things differently:
Lessig and company have taught me that content's value can lie in what it spawns and inspires. Locked away, unseen, unused, not discussed, not linked, it might as well not exist.
The tension between knowledge and content is no mere abstraction. As well as lying behind some of the most problematic sections of SOPA, ACTA and TPP, and the larger war on digital sharing they are part of, it was almost certainly a contributory factor in the death of Aaron Swartz too.
And Aaron Swartz has taught me that content must not be the end game for knowledge. Why does knowledge become an article in a journal -- or that which fills a book or a publication -- except for people to use it? And only when they use it does content become the tool it should be. Not using knowledge is an offense to it. If it cannot fly free beyond the confines of content, knowledge cannot reach its full value through collaboration, correction, inspiration, and use.
Follow me @glynmoody on Twitter or identi.ca, and on Google+
Filed Under: aaron swartz, acta, content, sopa, tpp


text:
A wrongful death lawsuit against the Biltmore Hotel moved forward on Monday when a man whose wife was found dead at the LA landmark during an audition for "America's Got Talent" alleged the hotel destroyed evidence to cover up the slaying of his wife.
The allegation comes as part of a lawsuit filed by Joseph Finley in March 2011.
The body of his wife, Laura, was found in a stairwell at the famed downtown hotel on Oct. 23, 2010.
U.S. & World
The lawsuit claims that a homeless man or "uninvited third party" walked through an unsecured first-floor door and killed Finley's wife while he was sleeping in the couple's sixth-floor room.
Hotel security failed to warn guests about security issues related to remodeling work being done on the first and second floors, court documents said.
The hotel was booked, many there to participate in an audition of "America's Got Talent."
Hotel security found Laura Finley's body at the bottom of the first-floor staircase. The body of the 48-year-old woman was found with no top and wearing pants that were inside out. Her diamond rings were missing, court papers said.
Even though the coroner's office determined she had suffered multiple blunt force injuries and cuts to her head, Biltmore hotel security told police there was no sign of foul play, according to court documents.
The court documents say that hotel security were aware that homeless people and others had committed "assaults with deadly weapons, thefts, and other disturbances," but "failed to take any security measures" to keep guests safe, the complaint alleges.
A door into the hotel that led to the street was propped open by a piece of brick tied to a shoestring and "transients followed guests into their hotel rooms," the complaint alleges.
"Open alcohol bottles, empty food containers, filthy clothing, cigarette butts, wrappers, bottles of urine and dirty blankets" could be seen along the floors that were being remodeled, the complaint alleges.
The couple were staying at the hotel that weekend because Joseph Finley, a 30-year musician, was auditioning for the popular TV show. He didn't report his wife missing -- which was several hours after her body was found -- until after his audition, police said.
Finley was questioned in the case but was released. At the time, police said the death appeared to be accidental.
The LAPD said Monday the case is an open death investigation but they would not release any information, saying it was sealed.
Finley's attorney, Sean E. Macias, declined to comment. Biltmore attorneys did not return a call.
An Los Angeles Superior Court judge ordered a hearing on oral arguments about the evidence for July 19.


text:
Saturday, November 28, 2009
Just finished watching Up with Jamie and Dynisha. I got a little misty eyed a few times. The beginning was boring to Jamie, but touching to me. Once the balloons were inflated Jamie was completely captivated and I had a difficult time making him go to the bathroom because he didn't want to stop watching it even for a moment. I think this is one of the best kids movies that has come out in a long time. I highly recommend it.
Thursday, November 26, 2009
Pulling Thanksgiving together when you are on bed rest is a little bit of a challenge. I must admit I wasn't in bed as much as I should have been. Even so we ended up with our family over without the floors mopped or Jamie dressed up and bathed.
I always enjoy the holidays, but it seems that everyone else is always so unhappy. Usually we have Thanksgiving dinner at my husbands family's home. I decided that this year I wasn't going to do that. I'm tired of listening to everyone complain, complain, complain about the stress and the mess. I absolutely love cooking and having guests. If you don't enjoy it, don't do it out of some sense of obligation in the name of the holiday. Having Thanksgiving on my own terms this year made for a much better holiday, though I was a little stressed because there was so much I was not allowed to do.
Blah Blah Blah by Laura at 11/26/2009 11:23:00 PM 0 Remarks
Classification Reflection
Wednesday, November 25, 2009
4D Ultrasounds
I can't believe we will be 30 weeks tomorrow! We got our first real peak at her little face today. Isn't she beautiful? The tech took a long video where we saw her move and she appears to yawn really big. Her little fingers fluttered about her face and she looks like she tried to put her little foot in her mouth. Unfortunately, when we got home we discovered that the video did not record. I could have cried. Still, I have these beautiful pictures.
Blah Blah Blah by Laura at 11/25/2009 10:14:00 PM 0 Remarks
Classification pregnancy
Wednesday, November 11, 2009
My Anniversary
I'm 27 weeks 6 days and today is my 9th anniversary. We have sure been through our ups and downs over the last 9 years but what a fabulous life we have together. We have a beautiful little boy who brings us lots of joy. We're expecting a beautiful little girl even though we thought we could not have another child. We truly have cause to celebrate.
I had an OB appointment in the afternoon and then we stopped by to visit with my uncle. He gave us money to go out to eat. We went to Red Lobster with Jamie. I had crab alfredo, which is my favorite, but terrible for my blood sugar! Jamie ate about half a dozen cheese biscuits and a little salad. That's about it.
Blah Blah Blah by Laura at 11/11/2009 09:53:00 AM 0 Remarks
Classification love, pregnancy, Reflection
Saturday, November 7, 2009
Some Artwork and Writing
Blah Blah Blah by Laura at 11/07/2009 10:06:00 PM 0 Remarks
Classification Jamie's Development, Just Jamie
Thursday, November 5, 2009
Jamie's First Kindergarten Project
Jamie doesn't get homework so I was surprised and delighted when he came home with a project for us to do together. He had to take a picture of a Turkey and glue it to orange construction paper and disguise it so that it would not be eatedn for Thanksgiving. I believe that nobody in their right minds would eat this monster turkey!
Blah Blah Blah by Laura at 11/05/2009 10:47:00 PM 0 Remarks
Classification crafts, Jamie's Development
Subscribe to: Posts (Atom)


text:
Financing Programs make Walk-In Bath Tubs Available to More Seniors
This is an advertisement. *See site for details. Your privacy is important to us. You are receiving this email because you opted-in to receive it. To stop receiving emails from Walk-in Tub Outlet, please click here.
You can also mail a request to: Walk-in Tub Outlet, 32392 S. Coast Hwy, Suite 200, Laguna Beach, CA 9265292652


text:
Coloring is one of the best developing toys for children up to 10-12 years old.
Now a very large assortment of coloring books for children, but there are exactly those colorings that are popular with most children, namely the alicorn coloring pages.
How to download the alicorn coloring pages?
To successfully download the alicorn coloring pages, you need to save the image to your computer. This can be done simply by clicking on the download button or by right-clicking on the alicorn coloring pages to save the image.
How to print alicorn coloring pages?
To successfully print the alicorn coloring pages, you need to save to the PC first, or you can send the picture to print simply by right-clicking it and then selecting the item to print.