Hadoop on the Pi

Question

I've seen a couple of articles claiming to build a "super computer" using several Raspberry Pis. I know the processing power of the Raspberry Pi is relatively slow so you're not going to be simulating nuclear physics here, but they are cheap and would provide a good playground for learning the accompanying software that goes along with distributed computing. Has anyone seen any instance of anyone running Hadoop on the Raspberry Pi? Does this seem possible? It seems like a great, cheap entry platform if you just bought a few.

Answer 1

Saw this question a few years ago and I thought I would provide an up-to-date answer for anyone else who may be interested.

A project I did was to create a single node Hadoop node and to test out word count. I wrote up the process here: Yes, you can run Hadoop on a Raspberry Pi and also create a cluster.

The only problem you may face on the Pi2 is memory leaks but otherwise it runs smoothly (but slowly). Functions such as wordcount and map reduce work with small files, and there's now a large enough community running clusters to provide support and answer questions.

Using Raspbian Jessie on a Raspberry Pi 2, to install Hadoop:

cd ~/ 

wget http://apache.mirrors.spacedump.net/hadoop/core/hadoop-2.6.4/hadoop-2.6.4.tar.gz

sudo mkdir /opt

sudo tar -xvzf hadoop-2.6.4.tar.gz -C /opt/

The Hadoop version has changed, so make sure you get the latest version (for security reasons and to try out any new features).

Answer 2

You'd have to find Java for ARM, and with past experience, Hadoop runs best with Oracle/Sun Java instead of OpenJava. I did a quick Google search and found evidence of Oracle Java being available for ARM, but I couldn't find a download link.

However, having run Hadoop on 32 PlayStation 3's (which also have low memory and non-Intel architecture), I can tell you it's rather slow. The Raspberry Pi is even worse because it doesn't have gigabit Ethernet (PS3 does), and communication is always a major issue in the high performance computing world. If you are able to pull it off, it'll be for the sake of novelty or education. Good luck, and let us know how it works.

If you're hellbent on Hadoop + ARM, check out the Parallela platform: http://www.kickstarter.com/projects/adapteva/parallella-a-supercomputer-for-everyone. It has a fancy processor, plus dual-core ARM, 1 GB of RAM, and the all-important gigabit Ethernet. It's not out yet, but should be by the summer for $99. I'm a backer, and it seems they are making great progress.

UPDATE: I found proper Java for ARM: http://jdk8.java.net/fxarmpreview/index.html. It's super new, so I don't know how well it'll work with Hadoop.