Take the 2-minute tour ×

I have an application that requires analytics for different level of aggregation, and that's the OLAP workload. I want to update my database pretty frequently as well.

e.g., here is what my update looks like (schema looks like: time, dest, source ip, browser -> visits)

(15:00-1-2-2010, www.stackoverflow.com,, safari) -->  105

(15:00-1-2-2010, www.stackoverflow.com,, firefox) --> 110


(15:00-1-5-2010, www.cnn.com,, firefox) --> 110

And then I want to ask what is the total visit to www.stackoverflow.com from a firefox browser last month.

I understand Vertica system can do this in a relatively cheap way (performance and scalability wise, but not cost-wise probably). I have two questions here.

1) Is there an open-source product that I can build upon to solve this problem? In particular, how well does a Mondrian system work? (scalability, and performance) 2) Is there an HBase or Hypertable base solution (obviously, a naked HBase/Hypertable can't do this) for this? -- but if there is a project based on HBase/Hypertable, scalability probably won't be an issue IMO)?


share|improve this question
What is your expected data volume? 1 million hits/day? 10 million? –  Data Monk Jan 21 '10 at 9:23
add comment

4 Answers

You can download a free edition (the single node edition) of the greenplum database. I haven't tried it myself but I think/guess it is a powerful beast. Read here: http://www.dbms2.com/2009/10/19/greenplum-free-single-node-edition/

Another option is MongoDB, it is fast and free and you can write MapReduce functions with JavaScript to do analytics.

My reputation here is to low to add a hyperlink to mongodb, so you have to google . I can add only one hyper link per post.

share|improve this answer
greenplum is not free –  charlie111 Jan 19 '10 at 6:57
The single node edition is free. –  AABBCCDD Jan 19 '10 at 18:58
To clarify: The Greenplum SNE is "free as in beer". You don't have to pay money for it, but there are licensing restrictions on the size and number of servers you can deploy it on, and the source code is not publicly released. –  goodside Sep 22 '11 at 14:03
add comment

The zohmg project aims to solve this problem using Hadoop and HBase.

share|improve this answer
add comment

Facebook also built Hive on-top of Hadoop. Pretty simple to get going - reasonable query API too.


share|improve this answer
add comment

Is your data model more complex than that? If it isn't you might be beter of just writing custom code for it. Then you can really tune it to your data. Real products have to offer a lot of flexibility, need a lot of complexiy to achieve that, and suffer in speed as a result.

Your question is not clear in one aspect: when you talk about scalable, what do you mean by that? Are you collecting data from lots of sites but only have a limited amount of query users, or do you also have a lot of users? That situation leads to a significantly different model.

share|improve this answer
I don't see data model matters here. The original post wants to find a solution of pre-computation (cuboid lattices) on top of Hbase and etc. You are talking about Mapreduce, which extends what Hive is doing, but essentially it's a batch mode processing –  user279941 Feb 5 '11 at 9:53
No, I'm talking about writing code and using flat files. Depending on the data model and number of entities of course. –  Stephan Eggermont Feb 5 '11 at 9:53
add comment

Your Answer


