You probably came here after searching for this error. Before you try anything else, check if your /tmp directory exists and if it has the right permissions. If not, create it and do a “chmod 777 /tmp”. MySQL will give you the weirdest errors if it can not use the /tmp folder.

If this does not help, check your /etc/mysql/my.cnf. There could be a line somewhere, stating which directory to use as a temporary directory. Create that directory, or remove/change the line so MySQL uses your /tmp folder instead.

OK, we have it running on a local Ubuntu installation, on Apache and MySQL5 with PHP5. Not a big deal. Now it’s time to download the Wikipedia data, which is licensed in such a way that you are free to use it. In my case, I will be using it for scientific reseach, about which I’ll surely post more in the coming months. The english data of just the current version of all the articles is 4+ gigabytes in compressed bzip2 format. Let’s decompress - you are now left with a whopping 18.2GB xml file. Outch. Importing this xml file, with the importData.php tool from mediawiki, will take lots of time. It starts importing at a rate of 4 pages per second, but this rate will go down to 2.5 per second after about 20,000 pages. After 30,000 pages, it seemed to stabilize at 2,25 pages/sec, so I started to do some math. There are about 15 million pages in total, if I remember correctly. That is 15,000,000 / (2.25 * 60*60*24) = 77 days.

Unfortunately, but to be expected, the rate keeps going down. After 150,000 pages I’m now at 1.7 pages/sec. Maybe this wasn’t such a good idea..

Plan B

I can almost hear you screaming now: ‘thank god, there is a plan B’.

Head over the MWDumper page and download the jar file. Follow the instructions and use the example they give, looking like:

java -jar mwdumper.jar --format=sql:1.5 pages_full.xml.bz2 |
   mysql -u <username> -p <databasename>

This is a lot better already, it will import at 200 to 300 pages/sec. But it gets better. If you remove ALL indexes and auto_increments, the speed goes up beyond 2000 pages per second! Don’t forget to re-add the indexes and auto_increment fields when the import is done.

You can have your own locally installed Wikipedia in about 5 hours or less if your PC is fast. The example PC I used is a relatively dated AMD athlon 3200+ with 1GB of memory and a regular sata disk.

P.S.: you might want to look into MySQLs binary logging. Turning it off or reducing the maximum log size to 1MB will increase performance too!