Take the 2-minute tour ×

Google has described a novel framework for distributed processing on Massive Graphs.


I wanted to know if similar to Hadoop (Map-Reduce) are there any open source implementations of this framework?

I am actually in process of writing a Pseudo distributed one using python and multiprocessing module and thus wanted to know if someone else has also tried implementing it. Since public information about this framework is extremely scarce. (A link above and a blog post at Google Research)

share|improve this question
add comment

8 Answers

The main Hadoop project for distributed graph processing is the Hama project. Its still in incubation though.

The project has broken its work into two areas; a matrix package and a graph package.


A better option would be the Apache Giraph project which is based on Google Pregel.

share|improve this answer
Thanks, Hama seems to be using BSP model as well, though its desciption of the algorithm seems to be more conceptually complicated than that of Pregel. Maybe its because the documentation of Pregel is so scarce. –  Akshay Bhat Jun 7 '10 at 5:15
Calling Hama the main Hadoop project for this seems a bit odd as Giraph, and soon, Golden Orb, are specifically focused on Pregel implementations. –  Jakob Homan Nov 4 '11 at 17:14
@Jakob Homan - your correct with regards to Hama in 2012. Back in 2010 at the time of answering neither giraph or GoldenOrb existed. I've updated my answer accordingly. –  Binary Nerd Mar 22 '12 at 4:40
add comment

Yes, a new project called Golden Orb, which is an open-source Pregel implementation written in Java that runs on both HBASE and Cassandra.

It has been submitted to Apache incubator for approval, and Ravel, the company behind Golden Orb, said they are releasing it this month (http://www.raveldata.com/goldenorb/).

See http://www.quora.com/Graph-Databases/What-open-source-graph-databases-support-horizontal-scaling

UPDATE: GraphX is GraphLab2 on Spark implemented by Joey Gonzalez, the creator of GraphLab2.

Spark's unique primitives make GraphX-Pregel the fastest JVM-based Pregel implementation. Spark is written in Scala, but Spark has a Java and Python API.


P.S. There is also Bagel, which was the first cut at Pregel on Spark. It works; however, GraphX will be the way forward.

share|improve this answer
add comment

There is also Signal/Collect a framework written in Scala and now using Akka http://code.google.com/p/signal-collect/


From their website:

In Signal/Collect an algorithm is written from the perspective of vertices and edges. Once a graph has been specified the edges will signal and the vertices will collect. When an edge signals it computes a message based on the state of its source vertex. This message is then sent along the edge to the target vertex of the edge. When a vertex collects it uses the received messages to update its state. These operations happen in parallel all over the graph until all messages have been collected and all vertex states have converged.

Many algorithms have very simple and elegant implementations in Signal/Collect. You find more information about the programming model and features in the project wiki. Please take the time to explore some of the example algorithms below.

share|improve this answer
add comment

I create a framework called Phoebus. It is an implementation of Pregel written in Erlang. Checkout my blog entry for applying Pregel model to path finding as well..

share|improve this answer
add comment

Apache Giraph is currently in Incubator and under very active development, with committers from LinkedIn, Twitter, Facebook and academia looking to bring it up to production scale very quickly. It is pretty directly modeled on Pregel and was originally developed at Yahoo! Research. We're looking for new contributors and have several introductory JIRA issues to help people get started with the project. We'd love to have you get involved.

share|improve this answer
add comment

Two projects from Carnegie Mellon University provide Pregel-style computation on graphs:

The programming model is not exactly same as Pregel, as they are not based on messaging but on modifying the graph (edge, vertex) data directly. Basically, it is easy to emulate Pregel in these framework.

share|improve this answer
add comment

Stanford Students have developed an open Source implementation of Pregel. http://infolab.stanford.edu/gps/

share|improve this answer
add comment

Your Answer


