Holograms and Data Science

This will be a quick and different post, recording one of these A-HA moments we sometimes have, before I forget it. It is a random thought that’s got more than 140 characters. A tweet would not do it.

I just finished reading The Black Hole War, by Leonard Susskind (one of the most important physicists alive today). It is a great book if you are interested in modern physics. Here is an excerpt of my review on Goodreads:

(…). For non-physicists like me, this was a fantastic introduction on what we currently know about quantum gravity and its relation with other areas of science. As a bonus, it also (finally) helped me start grasping string theory, and better understand entropy, the event horizon (complementarity, information paradox) and the holographic principle.

My A-HA moment came while the book discussed the Holographic Principle. Coincidence or not, I have been studying a whole bunch of Machine Learning, and things converged. I love when (seemingly) completely different areas of science suddenly converge. It reinforces my belief that there must be something common underlying all of it (call it Grand Unified Theory, or God, or as you please).

Principal Component Analysis and Dimensionality Reduction are the Data Science (Machine Learning) topics in particular that seem to be correlated with the Holographic Principle. They are techniques of removing redundancy in data sets and extracting the minimal data necessary to represent something. It allows you to reduce the number of columns in a database without losing meaningful information, for example. The main technique can be seen as a linear algebra transformation: a projection of the N-dimensional data onto a smaller K-dimensional space.

In a sense, holograms are the same thing: a projection (encoding) of a higher dimensional space onto a smaller dimensional space (3D to 2D, for example). The Holographic Principle states that all information inside a N-dimensional space is contained into its (N-1)-dimensional boundary. For example, all the information inside a volume (3D) is contained (or described) onto its surface/area (2D).

This is fascinating. Holograms are fascinating. It could mean that not all dimensions (e.g.: rows/columns in your database) are necessary to perfectly describe any information, and removing that redundancy is precisely what Dimensionality Reduction (and PCA) tries to do. I wonder if we can use the Holographic Principle to find ways to do Dimensionality Reduction without losing any information (a loss-less compression, if you will).

Linear algebra is another fascinating aspect of all this. Projecting data and extracting meaningful information (compression) always seems to involve the calculation of Eigenvectors and Eigenvalues. Google is constantly calculating Eigenvectors that power your searches. Somehow, Eigenvectors also seem to be central to all of this.

I am sure this is not something I’m inventing or discovering and there are plenty of papers about it out there. I only happen to not have stumbled upon any of them. A quick search tells me that Holography and Dimensionality Reduction is being used in many different areas of science, including genetics and biology. If you know of any of such papers (proving or disproving my random thought), let me know in the comments.


My DockerCon 2014 talk: Thoughts on interoperable containers




interoperable containers





docker run hone/ruby-builder --version 2.1.0p123

# a new ruby VM is now available in a local directory


We are interested in a particular usage for Docker though. Docker to run portable, server side, web applications following the 12factor.net guidelines.



Docker GitHub issues


different groups and me






– darren0, at #docker-dev


1 vs 1M





App developers often want root access because:


kernelspace abuse



LXC, Docker, Security







User Namespaces



Kernel capabilities in user namespaces




Another attack vector is code like this:

if (getuid() == 0) {
  // do root stuff




Just don’t run as root?












image size

Container images, in my experience, have 2-5GB.


Docker solves this problem with layers.








Compare that with what we currently do at Heroku.











Runtime environments don’t need to be just containers. Portable apps (12factor.net) can then be mapped to raw VMs, or even bare metal servers.







$ cat my-portable-app/Dockerfile
FROM heroku/heroku-buildpack-ruby


$ cat heroku/heroku-buildpack-ruby/Dockerfile
# Heroku's base image based on Ubuntu 10.04
FROM heroku/cedar

ADD . /buildpack
ONBUILD RUN /buildpack/bin/compile /app




#!/usr/bin/env make -f

buildpath := .build
buildpackpath := $(buildpath)/pack
buildpackcache := $(buildpath)/cache

build: $(buildpackpath)/bin

    mkdir -p $(buildpath)
    mkdir -p $(buildpackcache)
    curl -O https://codon-buildpacks.s3.amazonaws.com/buildpacks/kr/go.tgz
    mv go.tgz $(buildpath)

$(buildpackpath)/bin: $(buildpackcache)
    mkdir -p $(buildpackpath)




ruby = "https://codon-buildpacks.s3.amazonaws.com/buildpacks/heroku/ruby.tgz"

app_container "myapp" do
  buildpack ruby
  git_url "git@mycompany.com:myapp.git"

define :app_container,
       name: nil,
       buildpack: nil,
       git_url: nil do
  # ...

  execute "#{name} buildpack compile" do





Thank you!

Memory inside Linux containers

Or why don’t free and top work in a Linux container?

Lately at Heroku, we have been trying to find the best way to expose memory usage and limits inside Linux containers. It would be easy to do it in a vendor-specific way: most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others.

An implementation of Linux containers could easily inject one or more of those files inside containers. Here is an hypothetical example of what Heroku, Docker and others could do:

# create a new dyno (container):
$ heroku run bash

# then, inside the dyno:
(dyno) $ cat /sys/fs/cgroup/memory/memory.stat
cache 15582273536
rss 2308546560
mapped_file 275681280
swap 94928896
pgpgin 30203686979
pgpgout 30199319103
# ...

/sys/fs/cgroup/ is the recommended location for cgroup hierarchies, but it is not a standard. If a tool or library is trying to read from it, and be portable across multiple container implementations, it would need to discover the location first by parsing /proc/self/cgroup and /proc/self/mountinfo. Further, /sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location. Thinking about it, /sys/fs/cgroup/self would not be a bad idea.

If we decide to go down that path, I would personally prefer to work with the rest of the Linux containers community first and come up with a standard.

I wish it were that simple.

The problem

Most of the Linux tools providing system resource metrics were created before cgroups even existed (e.g.: free and top, both from procps). They usually read memory metrics from the proc filesystem: /proc/meminfo, /proc/vmstat, /proc/PID/smaps and others.

Unfortunately /proc/meminfo, /proc/vmstat and friends are not containerized. Meaning that they are not cgroup-aware. They will always display memory numbers from the host system (physical or virtual machine) as a whole, which is useless for modern Linux containers (Heroku, Docker, etc.). Processes inside a container can not rely on free, top and others to determine how much memory they have to work with; they are subject to limits imposed by their cgroups and can’t use all the memory available in the host system.

This causes a lot of confusion for users of Linux containers. Why does free say there is 32GB free memory, when the container only allows 512MB to be used?

With the popularization of linux container technologies – Heroku, Docker, LXC (version 1.0 was recently released), CoreOS, lmctfy, systemd and friends – more and more people will face the same problem. It is time to start fixing it.

Why is this important?

Visibility into memory usage is very important. It allows people running applications inside containers to optimize their code and troubleshoot problems: memory leaks, swap space usage, etc.

Some time ago, we shipped log-runtime-metrics at Heroku, as an experimental labs feature. It is not a portable solution though, and does not expose the information inside containers, so that monitoring agents could read it. To make things worse, most of the monitoring agents out there (e.g.: NewRelic?1) rely on information provided by free, /proc/meminfo, etc. That is plain broken inside Linux containers.

On top of that, more and more people have been trying to maximize resource usage inside containers, usually by auto-scaling the number of workers, processes or threads running inside them. This is usually a function of how much memory is available (and/or free) inside the container, and for that do be done programmatically, the information needs to be accessible from inside the container.

More about /proc

In case you wondered, none of the files provided by the cgroup filesystem (/sys/fs/cgroup/memory/memory.*) can be used as a drop-in replacement (i.e.: bind mounted on top of) for /proc/meminfo, or /proc/vmstat. They have different formats and use slightly different names for each metric. Why memory.stat and friends decided to use a format different from what was already being used at /proc/meminfo is beyond my comprehension.

Some of the contents of a /proc filesystem are properly containerized, like the /proc/PID/* and /proc/net/* namespaces, but not all of them. Unfortunately, /proc in general is considered to be a mess. From the excellent “Creating Linux virtual filesystems” article on LWN:

Linus and numerous other kernel developers dislike the ioctl() system call, seeing it as an uncontrolled way of adding new system calls to the kernel. Putting new files into /proc is also discouraged, since that area is seen as being a bit of a mess. Developers who populate their code with ioctl() implementations or /proc files are often encouraged to create a standalone virtual filesystem instead.

I went ahead and started experimenting with that: procg is an alternative proc filesystem that can be mounted inside linux containers. It replaces /proc/meminfo with a version that reads cgroup specific information. My goal was for it to be a drop-in replacement for proc, without requiring any patches to the Linux kernel. Unfortunately, I later found that it was not possible, because none of the functions to read memory statistics from a cgroup (linux/memcontrol.h and mm/memcontrol.c) are public in the kernel. I hope to continue this discussion on LKML soon.

Others have tried similar things modifying the proc filesystem directly, but that is unlikely to be merged to the mainstream kernel if it affects all users of the proc filesystem. It would either need to be a custom filesystem (like procg) or a custom mount option to proc. E.g.:

mount -t proc -o meminfo-from-cgroup none /path/to/container/proc


There is also a group of kernel developers advocating that this would be better served by something outside of the kernel, in userspace, making /proc/meminfo be a virtual file that collects information elsewhere and formats it appropriately.

FUSE can be used to implement a filesystem in userspace to do just that. Libvirt went down that path with its libvirt-lxc driver. There were attempts to integrate a FUSE version of /proc/meminfo into LXC too.

Even though there is a very nice implementation of FUSE in pure Go, and that I am excited with the idea to contribute a plugin/patch to Docker using it, at Heroku we (myself included) have a lot of resistance against using FUSE in production.

This is mainly due to bad past experiences with FUSE filesystems (sshfs, s3fs) and the increased surface area for attacks. My research so far has revealed that the situation may be much better nowadays, and I would even be willing to give it a try if there were not other problems with using fuse to replace the proc filesystem.

I am also not comfortable with making my containers dependent on an userspace daemon that serves FUSE requests. What happens when that daemon crashes? All containers in the box are probably left without access to their /proc/meminfo. Either that, or having to run a different daemon per container. Hundreds of containers in a box would require hundreds of such daemons. Ugh.

/proc is not the only issue: sysinfo

Even if we could find a solution to containerize /proc/meminfo with which everyone is happy, it would not be enough.

Linux also provides the sysinfo(2) syscall, which returns information about system resources (e.g. memory). As with /proc/meminfo, it is not containerized: it always returns metrics for the box as a whole.

I was surprised while testing my proc replacement (procg) that it did not work with Busybox. Later, I discovered that the Busybox’s implementation of free does not use /proc/meminfo. Guess what? It uses sysinfo(2). What else out there could also be using sysinfo(2) and be broken inside containers?

ulimit, setrlimit

On top of cgroup limits, Linux processes are also subject to resource limits applied to them individually, via setrlimit(2).

Both cgroup limits and rlimit apply when memory is being allocated by a process.


Soon, cgroups are going to be managed by systemd. All operations on cgroups are going to be done through API calls to systemd, over DBUS (or a shared library).

That makes me think that systemd could also expose a consistent API for processes to query their available memory.

But until then…


Some kernel developers (and I am starting to agree with them) believe that the best option is an userspace library that processes can use to query their memory usage and available memory.

libmymem would do all the hard work of figuring out where to pull numbers from (/proc vs. cgroup vs. getrlimit(2) vs. systemd, etc.). I am considering starting one. New code could easily benefit from it, but it is unlikely that all existing tools (free, top, etc.) will just switch to it. For now, we might need to encourage people to stop using those tools inside containers.

I hope my unfortunate realization – figuring out how much memory you can use inside a container is harder than it should be – helps people better understand the problem. Please leave a comment below and let me know what you think.

  1. To be fair, I don’t really know what NewRelic is doing these days, I am just using them as an example. They may be reading memory metrics in a different way (maybe aggregating information from proc/*/smaps). Pardon my ignorance. 

Software Development Guidelines

From Merrian-Webster:

guide·line (noun): an indication or outline of policy or conduct

U.S. Dept. of Veterans Affairs:


These definitions are very important. What I am listing here are my guidelines, a set of items that usually drive my particular style of coding and engineering. They are not rules. As someone once wisely pointed out (no references, sorry):

“guidelines account for judgement, rules don’t”

They are also my guidelines. It is ok if they do not fit in your case, I am not trying to describe the definitive way of writing, maintaining and operating software. Most of the items here should be obvious for many people, but I hope this helps you think about your own guidelines and understand more others.

After a quick disclaimer, let us go to the list. There is a lot more I would like to publish someday, but I will try to not make this too long and painful.


There is a big chance that ying-yang is in my blood; my father is Chinese. One of the most important lessons I learned so far is that extremism is bad, everything needs balance.

This is at the top of my list, mainly because it nicely applies to the other items as well. I consider them to be good ideas, but I will not just blindly apply them to every situation. I usually do, unless I can come up with a very good justification not to.

The recursiveness of this item is also beautiful. It means that you can even be extreme (passionate?) about something, as long as you can give a damn good reason for it. You should not be extreme about not being extreme.

Take for example my deep hate for inheritance (in the OO context): there are good reasons for it. But it is not true that I would never use inheritance: the fact that it should be avoided is something to keep in mind, not something to block you from delivering software.

While we are on the topic, I really like how Go approaches inheritance.

Enabling vs. Directing

I took this very important lesson from some good discussions I had with my friend Tiago many years ago, back when we were coworkers in Germany.

Enabling means that something can be used in many different ways. Even in ways that we have not even considered yet. Humans are creative and will come up with different ways of using and applying an enabling idea. Directing however, means something that is designed to be used in a particular way, or an specific action.

A good example is how to design Java annotations. Here is how we could annotate a class to define that its objects need to be saved to a database (i.e. they are persistent):

public class Person {
  // ...

This is an example of a directing implementation, it specifically tells that objects of this class need to be saved. Using that information for anything else would be awkward.

Here is how the Java Persistence API (thanks to Hibernate) defines it:

public class Person {
  // ...

This information could be used by many different components on a system. For instance, a logging component could read and use it to determine that some of its fields need to be filtered from logs (such as passwords), just because persistent entities usually contain sensitive data. A formatting component could use different colors for entity objects. There are all sorts of different uses for that information and creative people will come up with even more.

It does not mean that all of those uses will be correct, but that enables more from your code and makes it potentially more flexible.

This concept applies to almost any decision we need to do related to software development. Think about shipping a new feature, designing a component, planning an api, coding styleguides, people and project management styles, technology (framework?) choices, etc. There is probably a more enabling than directing way of doing them all in each context, which would empower people more.

In practice, it can be very hard. I have experienced many cases where it was really hard to tell if we were directing or enabling people. Keeping this in mind already helps a lot though.

Premature Optimization

This has been discussed a lot already, no need to talk too much about it. We all know that “Premature optimization is the root of all evil”.

However, quoting Albert Einstein:


Do not use premature optimization as an excuse to be lazy, or irresponsible about what you ship. Keep it in mind, but balance it up with your previous experience and feedback from others. There are times when you just know it is going to bite you soon.

Which leads us to the next item…


I have learnt a lot of this over the past years running large production services (both at Locaweb and now at Heroku).

From the beginning, think about how your code is going to run in production. Do you understand the platform (runtime) it is going to run on? Are you confortable in troubleshooting hard problems? While we are on the topic, how are you planning to debug those problems?

Are you collecting metrics of how it is being used and how it services its requests? Even if you do not officially publish a SLA, understanding your service times is invaluable to dig into issues that will come with production load. Remember to track not only mean values, mean alone is useless. Always mix it up with variance, or rather with percentiles. Try to target for high percentiles – 95th and 99th are good targets, depending on what your scale is.

Last but not least: are you confident that you can notice (and be notified of) problems before your customers start complaining about your service on twitter? Metrics and automated monitoring are very important to run production services.


It does not really matter if automated or not. Yes, I said it. I have seen successful software (for whatever that means) both ways. Testing what you ship is just being responsible about it. And that includes having the proper infrastructure to do it: staging environments, gradual rollouts, feature flagging, etc.

Do not get me wrong. Automated testing should be usually preferred, but it is not the ultimate goal. I have seen many “evangelists” speaking hours about how automated testing is important, while the kernel of the operating system they use prefers a more traditional approach with lots of manual (or semi-automated) tests, Q/A testing teams and not many (or even zero) automated tests in its codebase.

Quick note about test (or behavior) driven development: it has more to do with software design methods than with the tests themselves. IMO it is a good practice, which works for me sometimes, but not always in every project, everyday.

Dependency Inversion

I am really proud of what some of my heavy Java development days have taught me.

Designing loosely coupled components is a big one. Dependency Injection, Dependency Inversion Principle and Inversion of Control are all related topics that to me mean a simple thing: design for single (or few) responsibilities.

It means that when you are writing that piece of code (be it a method, function, class, module, script or whatever), focus it on a single responsibility, and keep it small. If it gets to big, break it. If it needs resources to do its job, do not go after the resources it needs there, receive (inject) them instead.

In summary: instead of going after your dependencies, inject them. When a component needs to go after a resource it needs, it usually means that:

  1. The component is now responsible for its dependency lifecycle: this adds more responsibility. Now that it opens that damn connection to your DB (or your queue service), it needs to decide when to close, and needs to know how to do it. Also, what happens when you need to share the same connection in other parts of the system?
  2. The resource needs to be globally accessible: if the component did not create the resource it needs, or if that resource is shared in many parts of the codebase, it needs to be globally available. Hiding the resource creation behind factories does not help when the factory objects themselves need to be globally reachable. I hope I do not need to talk much about how globally accessible things are bad, but it is usually hard to test components using them, and it leads to tighter coupling. Changing that global resource gets harder with more stuff using it.

When you invert control and inject dependencies instead, you have the opportunity to centralize resource management in a single place. That single place centralizes the (single) responsibility of deciding where to inject that resource and how to share it. In more fancy environments, something like this can be called Dependency Injection Framework, but it does not need to be a big bloated framework once you understand the mechanics. In fact, this does not require a framework at all, it is just plain old method/function/constructor invocation with proper parameters.

All in all, let us please not forget about balance. This is not a rule, there are times when what you need is just a goddamn simple function.

Do Not Block The Event Loop

Evented programming (or event-driven programming) is very popular these days and one of the side effects is that some projects will want to be evented without careful consideration.

Going evented or not is one of the big decisions involved in writing code, with big implications. Things blocking the event loop are usually very hard to debug. When you go that route, all code called (including external libraries) needs to be aware of the event loop and not to block it.

There are many advantages though, notably it leads to much more lightweight servers which support much higher concurrency levels. When I am deciding if I should do event-driven code or not, here is what I consider (feel free to add a comment below with your own thoughts):

  • Is it an event-driven runtime/platform? Nodejs, for example, has been designed from the ground up to be evented. Meaning that all libraries and code written to run on it are already aware of the event loop. If the platform was not designed to be evented (like Ruby with EventMachine) much more care must be taken to not call code which will block the event loop. It is hard to control all the code in libraries included in your project. Take that into consideration.
  • Consider evented if the piece of code you are writing is mostly a data multiplexer, meaning that it just takes data from one side and sends it to another, acting like a pipe, distributor, load balancer, or router. This type of component is usually I/O bound.
  • Avoid evented if the piece of code you are writing is CPU bound and does a lot of processing. The chance it will block the event loop is much higher. I have seen projects having to resort to threads (or external processes) to move that part out of the event loop, often leading to spaghetti concurrency very hard to follow. Part evented, part threaded.

There Is Always Something To Learn

As I learn more, I expect this list to change, but I would say that it currently contains the factors influencing my development style the most. It was a very healthy exercise to think about what defines me as a programmer. I hope it is for you too, try it out!

Running Java Web Applications on Heroku Cedar Stack

Update: Heroku now does support Java.





VRaptor on Heroku


    $ gem install heroku
    $ gem install foreman
    $ brew update
    $ brew install leiningen
    $ gem install vraptor


  1. First, we need to create the project skeleton:
    $ vraptor new <project-name> -b mvn
    $ lein new <project-name>
    $ cd <project-name>


    $ wget https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj -P src

    Or if you do not have wget installed:

    $ curl -L https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj > src/heroku_autobuild.clj
    $ curl -L https://gist.github.com/raw/1129081/5790e173307d28a8df256e0aaa5a9fe7757e922f/project.clj > project.clj



    $ lein deps


    web: lein herokujetty

    And test with:

    $ foreman start


    $ rm -rf src/<project-name>
  9. Create a git repository:
    $ git init
    $ git add .
    $ git commit -m "initial import"
    $ heroku create --stack cedar
    $ git push -u heroku master
    $ heroku logs -t
    $ heroku ps:scale web=2
    $ heroku ps:scale web=4
    $ heroku ps:scale web=0
    $ heroku run bash

spring-roo on Heroku




Scala/Lift on Heroku




My Lift application is running on Heroku here.


DSLs: one interface, multiple implementations

One interface, multiple implementations is one of these design concepts I like most. It’s basically polymorphism in its pure form!

Coding a little bit a while ago, I realized how DSLs could reinforce this idea. My example here is a simple implementation for queue consumers, as a simple Ruby internal DSL:

class GitRepositoryCloner < Consumer
  queue "RepositoriesToBeCloned"
  exclusive true

  handle do |message|
    # git clone repository!
    # I'm pretty sure github has something alike

And to enqueue a message, we could do:


GitRepositoryCloner is a simple consumer of the queue named RepositoriesToBeCloned. One of the times I did something similar to this, I needed support for exclusive consumers. Then, my choices were ActiveMQ as the messaging middleware together with the Stomp protocol.

Using them as an example, let’s take a look on a possible implementation for the Consumer class, using the stomp gem:

module ActiveMQ
  class Consumer

    def self.queue(name)
      @queue_name = name

    def self.exclusive(bool)
      @exclusive = bool

    def self.handle(&blk)
      @callback = blk

    def self.listen
      broker = Stomp::Client.new(Config[:broker])
          :'activemq.exclusive' => @exclusive) do |message|

    def self.publish(message)
      broker.publish(@queue_name, message, :persistent => true)


Consumer = ActiveMQ::Consumer

The last line is where we choose the ActiveMQ::Consumer as the default implementation.

The beautiful aspect of this little internal DSL, composed only of three methods (queue, exclusive and handle), is that it defines an interface. Here, I have seen a common misconception from many developers coming from Java, C# and similar languages which have the interface construct. An interface in Object Orientation is composed of all accessible methods of an object. In other words, the interface is the object’s face to the rest of the world. We are not necessarily talking about a Java or C# interface construct.

In this sense, these three methods (queue, exclusive and handle) are the interface of the Consumer internal DSL (or class object, as you wish).

Let’s say for some reason, we would like to switch our messaging infrastructure to something else, like Resque, which Github uses and is awesome. Resque’s documentation says that things are a little bit different for Resque consumers. They must define a @queue class attribute and must have a perform class method.

As we would do with regular Java/C# interfaces, let’s make another implementation respecting the previous contract:

module Resque
  class Consumer

    def self.queue(name)
      @queue = name

    def self.exclusive(bool)
      self.extend(Resque::Plugins::LockTimeout) if bool

    def self.handle(&blk)
      self.send(:define_method, :perform, &@blk)

    def self.listen
      raise "Not ready yet" unless self.respond_to?(:perform)

    def self.publish(message)
      Resque.enqueue(self, message)


There you can see how the implementations differ. The exclusive consumer feature is provided by the LockTimeout plugin. In this case, instead of passing the activemq.exclusive parameter to the connection, we must use the Resque::Plugins::LockTimeout module, as the documentation says. Another key difference is in the message handling process. Instead of passing a handler block to the subscribe method, Resque consumers are required to define a perform method, which we are dynamically creating with some metaprogramming: Class#define_method(name).

Finally, here is how we switch our messaging backend to Resque, without any changes to the consumer classes (GitRepositoryCloner in this example):

Consumer = Resque::Consumer

That’s it: one interface, two (multiple) implementations.

Ruby and dependency injection in a dynamic world

It’s been many years I’ve been teaching Java and advocating dependency injection. It makes me design more loosely coupled modules/classes and generally leads to more extensible code.

But, while programming in Ruby and other dynamic languages, the different mindset always intrigued me. Why the reasons that made me love dependency injection in Java and C# don’t seem to apply to dynamic languages? Why am I not using DI frameworks (or writing simple wiring code as I did many times before) in my Ruby projects?

I’ve been thinking about it for a long time and I don’t believe I’m alone.

DI frameworks are unnecessary. In more rigid environments, they have value. In agile environments like Ruby, not so much. The patterns themselves may still be applicable, but beware of falling into the trap of thinking you need a special tool for everything. Ruby is Play-Doh, remember! Let’s keep it that way.

— Jamis Buck

Even being a huge fan of DI frameworks in the Java/C# world (particularly the lightweights), I completely agree with Jamis Buck (read his post, it’s very good!). This is a recurrent subject in talks with really smart programmers I know and I’ve tried to explain my point many times. I guess it’s time to write it down.

The whole point about Dependency Injection is that it is one of the best ways to achieve Inversion of Control for object dependencies. Take the Carpenter object: his responsibility is to make and repair wooden structures.

class Carpenter {
    public void repair(WoodenStructure structure) {
        // ...
    public WoodenStructure make(Specification spec) {
        // ...

Because of his woodwork, the carpenter often needs a saw (dependency). Everytime the carpenter needs the saw, he may go after it (objects who take care of their dependencies on their own).

class Carpenter {
    public void repair(WoodenStructure structure) {
        PowerSource powerSource = findPowerSourceInsideRoom(this.myRoom);
        Saw saw = new ElectricalSaw(powerSource);

        // ok, now I can *finally* do my real job...

The problem is that saws could be complicated to get, to assemble, or even to build. The saw itself may have some dependencies (PowerSource), which in turn may also have some dependencies… Everyone who needs a saw must know where to find it, and potencially how to assemble it. What if saw technology changes and some would rather to use hydraulic saws? Every object who needs saws must then be changed.

When some change requires you to mess with code everywhere, clearly it’s a smell.

Have you also noticed that the carpenter has spent a lot of time doing stuff which isn’t his actual job? Finding power sources and assembling saws aren’t his responsibilities. His job is to repair wooden structures! (Separation of Concerns)

One of the possible solutions is the Inversion of Control principle. Instead of going after their dependencies, objects receive them somehow. Dependency Injection is the most common way:

class Carpenter {
    private Saw saw;
    public Carpenter(Saw saw) {
        this.saw = saw;
    public void repair(WoodenStructure structure) {
        // I can focus on my job!

Now, the code is more testable. In unit tests where you don’t care about testing saws, but only about testing the carpenter, a mock of the saw can be provided (injected in) to the carpenter.

In the application code, you can also centralize and isolate code which decides what saw implementation to use. In other words, you now may be able to give the responsibility to do wiring to someone else (and only to him), so that objects can focus on their actual responsibilities (again Separation of Concerns). DI framework configuration is a good example of wiring centralized somewhere. If the saw implementation changes somehow you have just one place to modify (Factories help but don’t actually solve the problem – I’ll leave this discussion for another post).

class GuyWithResponsibilityToDoWiring {
    public Saw wireSaw() {
        PowerSource powerSource = getPowerSource();
        return new Saw(powerSource);
    public PowerSource getPowerSource() { ... }

Ok. Plain Old Java Dependency Injection so far. But what about Ruby?

Sure you might do dependency injection in Ruby (and its dynamic friends) exactly the same way as we did in Java. But, it took some time for me to realize that there are other ways of doing Inversion of Control in Ruby. That’s why I never needed a DI framework.

First, let’s use a more realistic example: repositories that need database connections:

class Repository {
    private Connection connection;
    public Repository(Connection connection) {
        this.connection = connection;
    public Something find(Integer id) {
        return this.connection.execute("SELECT ...");

Our problem is that many places need a database connection. Then, we inject the database connection as a dependency in everywhere it is needed. This way, we avoid replicating database-connection-retrieval code in such places and we may keep the wiring code centralized somewhere.

In Ruby, there is another way to isolate and centralize common behavior which is needed in many other places. Modules!

module ConnectionProvider
  def connection
    # open a database connection and return it

This module can now be mixed, or injected in other classes, providing them its functionality (wiring). The “module injection” process is to some degree similar to what we did with dependency injection, because when a dependency is injected, it is providing some functionality to the class too.

Together with Ruby open classes, we may be able to centralize/isolate the injection of this module (perhaps even in the same file it is defined):

# connection_provider.rb

module ConnectionProvider
  def connection
    # open a database connection and return it

# reopening the class to mix the module in
class Repository
  include ConnectionProvider

The effect inside the Repository class is very similar to what we did with dependency injection before. Repositories are able to simply use database connections without worrying about how to get or to build them.

Now, the method that provides database connections exists because the ConnectionProvider module was mixed in this class. This is the same as if the dependency was injected (compare with the Java code for this Repository class):

# repository.rb

class Repository
  def find(id)
    connection.execute("SELECT ...")

The code is also very testable. This is due to the fact that Ruby is a dynamic language and the connection method of Repository objects can be overridden anywhere. Particularly inside tests or specs, the connection method can be easily overridden to return a mock of the database connection.

I see it as a different way of doing Inversion of Control. Of course it has its pros and cons. I can tell that it’s simpler (modules are a feature of the Ruby language), but it may give you headaches if you have a multithreaded application and must use different implementations of database connections in different places/threads, i.e. to inject different implementations of the dependency, depending on where and when it’s being injected.

Opening classes and including modules inside them is a global operation and isn’t thread safe (think of many threads trying to include different versions of the module inside the class). I’m not seeing this issue out there because most of Ruby applications aren’t multithreaded and run on many processes instead; mainly because of Rails.

Regular dependency injection still might have its place in Ruby for these cases where plain modules aren’t enough.

IMO, it’s just much harder to justify. In my experience modules and metaprogramming are usually just enough.