Take the 2-minute tour ×
Raspberry Pi Stack Exchange is a question and answer site for users and developers of hardware and software for Raspberry Pi. It's 100% free, no registration required.

There have been a few articles floating around online over the last few years about building a cluster of computers at home - here for example.

The Pi strikes me as a good platform for building / testing this kind of thing due to their low cost; should "generic" guides like this transfer over to the Pi pretty easily, or is there anything specifically I should be aware of when attempting such a project?

share|improve this question
3  
Related forum thread: raspberrypi.org/phpBB3/viewtopic.php?p=18356#p18356 –  finnw Jun 12 '12 at 21:50
    
I've added the tag "bramble" as this is what these are named. Can't give a proper answer beyond "put a bunch together and run hadoop or something like that" - indeed, I've favved the question to watch for answers myself! –  winwaed Jun 12 '12 at 23:48
    
Other Pi users also don't know about the magic word "bramble"; searching the Foundation's forum might have helped but it's often slow to respond so I won't down-vote you. –  mlp Jun 14 '12 at 0:57
4  
University of SouthHampton have produced steps to make a 64 pi cluster (or "supercomputer"): southampton.ac.uk/~sjc/raspberrypi/… –  Alex L Sep 12 '12 at 13:53
add comment

6 Answers 6

up vote 25 down vote accepted

I suggest looking at Dispy - the distributed computation python module.

To run a program on a number of Raspberry Pi's (nodes) from a PC (server - assume IP is 192.168.0.100):

  • Install an operating system on each RasPi

  • Attach each RasPi to your network. Find the IP (if dynamic), or set up static IPs.
    (Let's assume that you have three nodes, and their IPs are 192.168.0.50-52)

  • Set up Python (if not already), install dispy, then run dispynode.py -i 192.168.0.100 on each RasPi. This will tell dispynode to receive job information from the server.

  • On the PC (the server), install dispy, then run the following python code:

#!/usr/bin/env python
import dispy
cluster = dispy.JobCluster('/some/program', nodes=['192.168.0.50', '192.168.0.51', '192.168.0.52'])

You can also replace /some/program with a python function - e.g. compute.

You can also include dependencies such as python objects, modules and files (which dispy will transfer to each node) by adding depends=[ClassA, moduleB, 'file1']

share|improve this answer
7  
I'd love to hear if someone has done this - please leave a comment letting me know if it works! –  Alex L Jun 15 '12 at 9:48
    
In my case (openelec + python 2.7) I get this error "File "./Lib/multiprocessing/__init__.py", line 84, in <module>, ImportError: /usr/lib/python2.7/lib-dynload/_multiprocessing.so: undefined symbol: SemLockType". –  Guido García Mar 13 '13 at 21:40
    
But dispy3-3.6 runs with wheezy + python3 ! 2013-03-13 23:01:30,664 - dispynode - serving 1 cpus at 192.168.1.34:51348. When you launch a task (i.e. /bin/ls) the node receives the task, moves the executable to /tmp, but something goes wrong "Executing ['/tmp/dispy/b7e04cb4a1e144e1/ls'] failed with (<class 'OSError'>, OSError(8, 'Exec format error'), <traceback object at 0x16f2580>)" –  Guido García Mar 13 '13 at 23:45
add comment

You should be aware of the work that has already been done - there's even a name for a cluster of RasPi boxen. The Embedded Linux Wiki says a Bramble is defined as "a Beowulf cluster of Raspberry Pi devices". Raspberry Pi Homebrew has a number of posts about Brambles, and see also the Foundation's own forum.

share|improve this answer
12  
I do not think telling someone to google the answer is good for this site. It is useful to know the name but consider adding some content and reference links to your answer. –  Joe Jun 12 '12 at 23:47
    
Content and links added, @Joe. It would be nice if the downvoters now reassessed their opinions ... –  mlp Jun 14 '12 at 0:59
1  
I would if I actually downvoted you... –  Joe Jun 14 '12 at 3:00
    
I phrased it very carefully to avoid insinuating that you had, Joe. Perhaps they don't realise that votes here can be undone by re-clicking the same button, not just reversed by clicking the opposite button. –  mlp Jun 15 '12 at 0:56
add comment

Some guys at Southampton Uni have put together a cluster and written a detailed overview of their work at http://www.southampton.ac.uk/~sjc/raspberrypi/.

share|improve this answer
1  
Rep whore :P –  Alex L Oct 18 '12 at 8:54
    
Actually, there is a lot to learn from this project using MPI. I am buidling my experiments based on these guy's work. I would recommend marking this answer the correct answer for the question. –  benqus Dec 14 '12 at 17:16
add comment

It is completely possible, but the biggest problem is attainability. It is an idea I would not only think workable, but useful as you could go with the idea of portable parallel computing. As far as specifics, coding languages like FORTRAN and C++ will do best.

Look at beowulf.org for more on cluster computing

share|improve this answer
add comment

This is reply to Guido Garcia's post above regarding 'dispy' - I can't figure out how to reply to his post.

When a program ('/bin/ls') is distributed with 'dispy' for parallel execution, then that program on the client machine is transferred to each of the nodes (to '/tmp'). This is so that the user developed program on the client machine is transferred without having NFS or some shared diretory. This works with binary programs only when nodes and client architectures are compatible. In your case, I am guessing that client architecture is different from that of the remote nodes and a node can't execute the binary '/bin/ls' transferred from the client. If you want to execute /bin/ls on each node, it may easier to write a Python function or program to print the directory (e.g., using os.listdir) and distribute that instead of binary executable.

share|improve this answer
    
You need 50 reputation to leave comments on other peoples answers/questions. Welcome though! –  Impulss Mar 21 '13 at 3:02
add comment

There's also http://pi.interworx.com if you want a full featured control panel with it. They have instructions up on this page on how to replicate, but you'll have to be patient as that subdomain tself is running from a Rasberry Pi cluster. Here's a photo in case it goes down:

http://www.facebook.com/photo.php?fbid=596262440393836&set=a.244167858936631.60071.170159826337435&type=1

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.