What is meta? ×

Are there any scripts available to import the data dump into databases other then SQL Server? I don't have an SQL Server license, and the data dump will probably out grow the SQL Server Express 4 gig limit if it hasn't already.

share|improve this question
fyi it has not outgrown the 4 gig size yet. –  waffles Aug 25 '09 at 22:08

3 Answers 3

I've created a script that uses Groovy and Hibernate to at least in theory, import the data dump into any database Hibernate supports.

I have gotten it to work with MySQL and PostgreSQL but it doesn’t work on HSQLDB, those are the only three I’ve tested it on. The code isn’t optimized and takes about an hour to import all five files, but I plan on improving it when I get a chance. The Git repository can be found on Github, I hope it's useful to others.

share|improve this answer
@Jared, I load it up on SQL in about 8 minutes, including tag splitting and indexing, make sure you have a look through my code for some of the tricks used, granted the fast inserter is not available with hibernate github.com/SamSaffron/So-Slow/tree/master –  waffles Aug 25 '09 at 22:10
I'll take a look at your code when I get a chance. This was actually the first time I've used Hibernate so I'm sure there's room for improvement. –  Jared Aug 25 '09 at 23:21
It looks like the code is gone now. –  carson Oct 22 '09 at 19:06
I found some problems with the code so took it down. I haven't gotten around to fixing it yet. Since there's a ruby script to import the data into mysql it's not a high priority of myne now. –  Jared Nov 18 '09 at 17:53

Here is a windows application that will import to MySQL, MS SQL Server and SQLite:


share|improve this answer

There's been an SQLite version of the data dump for past versions... I'll dig up a couple of URLs.

Here's the June dump - I don't believe July and August are available, unfortunately.

I have some code to convert it into Protocol Buffer files, but that's probably a bit of a niche market :)

There's also the Data Dump Sandbox if you're happy to use a web interface :)

share|improve this answer
No, they're not, I've been meaning to get to them, but I wanted to tweak my import script first. I don't know if it was SQLite or my script, but I was experiencing an exponential slowdown in the number of records I was processing. June took half an hour on my machine, and I fear that July and August would take even longer... –  Kyle Cronin Aug 11 '09 at 19:44
All props to Ian for the Sandbox. It's MySQL, btw. –  Eric Aug 11 '09 at 19:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .