1

I have been looking to make a cluster of raspberry pi 4s, and have been pretty interested in it and made good head way. I have seen quite a few guides on the topic as well. I followed this one found here: https://www.instructables.com/id/How-to-Make-a-Raspberry-Pi-SuperComputer/ which suggested MPICH. I followed the guide pretty closely, and to my surprise, my program speeds did not increase for a python program performing matrix multiplication as a test.

So I am taking a few steps back. My concerns are this:

1) It seems MPICH is written in fortran or C and I had to install a python library on top of that. I am concerned the cluster is slower because of the use of multiple languages, and if it is fortran, my guess is that fortran is slow to boot. I generally prefer solutions that are native to one language (here, Python 3). Is there an all python solution for this? My thinking is just to connect to each raspberry pi through ssh in python and create a thread for each raspberry pi, and then use python's thread libraries. Would I need mpich in this case?

2) When I want to run programs on the cluster there is a command (it is listed at the bottom of the guide) mpiexec that essentially makes the program run on the cluster with mpich. I am not sure if I would then do threads in python with that command to work. The guide, and others, made it seem that I just run mpiexec ... python3 foo.py and presto, my program is now faster. My intuition tells me that this command just runs the same program on multiple pies, not actual parallel processing. For two 750 length and height matrix multiplication the time for two raspberry p 4 does not decrease hence this guess. Am I missing something?

3) In addition, to do something like run mpiexec ... python3 test.py, I need to have the file on both raspberry pies. This defeats the purpose of the cluster to me, because having multiple gigabyte csv file on multiple raspberry pies is a waste. I would ideally like like to run SQL in python to make visualizations so to me it makes sense to have something like this: 10 GB data file on pi01, and run a program to divide the work amount let's say 10 pies for speed. This is the correct way to look at it, no?

My father and I are hoping to test data processing on raspberry pi 4's ie SQL and pandas. I think it is an interesting opportunity to learn more about hardware, linux, python, and start learning about parallel processing so any help or pointers are appreciated.

  • MPICH is interesting. (1) mpich.org, (2) en.wikipedia.org/wiki/MPICH (3) MPICHs are used exclusively on 9 of 10 supercomputers (2016), including world’s fastest supercomputer: Taihu Light. (4) Alpha Go used 1,000+ python tensorFlow computers to beat world champion. So I think 4 Rpi4Bs should be a good learning tool for distributed processing using python NN. Ages ago I heard "crowding computing" project when every PC in the world could contribute their idle time to analyze complicated DNA stuff. Can 10+ million Rpi's can do something similar? – tlfong01 Sep 21 at 3:22
  • IF your big database can be split into 4 smaller databases, then you can do parallel searching which should be faster than searching the big data base. For my image classification application. I train one Rpi to classify dog and non dog, another cat and non cat etc. Then I can send an identical image to all 4 Rpis, and the dog Rpi should return good news, and other Rpi's no news. Again this is faster than one Rpi sequentially trying dog = yes/no?, cat = yes/no?, and so on. – tlfong01 Sep 21 at 6:23
  • @tlfong01 thanks for the feedback. I saw that one large governmental science agency has a large supercomputer and so build a large cluster of rasp pies for people to practice on. – David Frick Sep 21 at 13:22
3

If you are going for max speed, then Python is not your friend. What it gains in flexibility it pays for in reduced capability for the compiler to perform substantial optimizations. Certain optimizations are simply not available to Python.

Although I'm not a fan of Fortran specifically, you should also be aware that for matrix multiplication, you almost surely want not only a LAPACK and BLAS library (which are Fortran) but a LAPACK and BLAS that have been tweaked for the hardware that you're using (which will typically be some Fortran and some hardware-specific assembly). For matrix multiplication specifically, there are entire communities that focus just on making these processes as fast as possible, and, if you're really going for speed, you should leverage what they've done.

Separate from the question of the language used, matrix multiplication is not a poster-child case for parallel processing increasing speed. You didn't really provide any sort of baseline problem or benchmarks, but you may find that a "direct" parallelization of a code that's just doing matrix multiplication won't benefit much from going parallel. In particular on a cluster, you'll need to pass messages over the LAN, and the time required for that may more than count against the time you save doing parts of the calculation in parallel. (Details to depend on the algorithms used, the size of the matrices, the speed of the network....)

The MPI libraries (whether you use MPICH or a different implementation) are doing more than you are giving them credit for, I think, because in a typical problem it's not just a question of starting parallel jobs but also keeping them coordinated by passing messages between them. (MPI, of course, stands for message passing interface.) If you've got some highly parallel data processing job where you don't need to coordinate, then you don't need MPI and can accomplish this by starting completely independent jobs on different machines, in any language you like. If you have a job that actually requires coordination, however, you'll need some sort of message passing scheme. You could, of course, reinvent the wheel on that, but MPI is one well-tested and well-supported tool that already exists. Only you can tell if it's a good tool for your job because we don't know what exactly you are trying to accomplish.

Some other misconceptions that you seem to have:

  • There is no such thing as a pure Python approach to this. At the bottom of it all - in some cases deeper than others - your Python is calling libraries written in C or C++. Maybe even some Fortran. There's no apparently principled reason to think that's different just because you don't know where the line is on, say, Pandas, but you happen to know on MPI.
  • Python is a tool, but it's not a catch-all tool for every problem. Based on what you've said so far about what you want to do, it seems like you have a hammer and are trying to make your problem look like a nail rather than choosing a tool suited to the problem.
  • No matter what language you use, when you go to parallel processing (including cluster computing or even multi-threading on a single machine), you typically need to design the parallelization into the program from the start. If you had a program that you originally intended to run in a single thread on a single machine, you will have to at least partially rewrite that to explicitly include how and where you want the parallel sections to run and synchronize. The only exception is a problem that is "embarrassingly parallel".
  • You will certainly need code on each machine that is going to process. How you get the data around is part of the design problem. For example if you have a massive file that they are all supposed to read, you may put that on a common file server and have them request chunks as needed. There are also different approaches that effectively let you send code to the data (e.g. MapReduce or Hadoop), which is a good idea when the code is small but the data is large. Again these are broad, fundamental design choices that you'll need to research and understand.

You used the phrase "taking a few steps back". If you're seriously interested in this type of processing, I suggest you take several more back to get a broader view of what's available. You seem to have locked yourself into a small and probably suboptimal set of technologies available already.

  • Thanks for the answer, I think that has given my some insights on how to move forward. Thanks! – David Frick Sep 21 at 13:21
  • Could you give me a pointer or two in carrying out parallel processing in SQL? – David Frick Sep 21 at 13:36
  • The expression "parallel processing in SQL" doesn't really make sense to me if taken literally. SQL is a database query language. You can hit the db with multiple queries and it will, if it can, usually process them in parallel internally. (Sometimes it cannot, for example if a query locks a table needed by another or you hit it simultaneously with more than it can handle.) I guess what you mean is to have a parallel program that's querying a db. As a user of SQL it's usually a question of having your code that generates and processes the data retrieved from the queries properly parallel. – Brick Sep 21 at 13:41
  • Sorry, I mispoke. I pretty much meant what you saying by focusing on the parallel processing not of SQL but rather processing the code to process the data quicker. However, a question remains. Let's say I have an SQL query with subqueries. Would it not be a lot quicker to run thise subqueries on another raspberry pi (granted the table is not locked) and return the results then complete the original query? – David Frick Sep 21 at 19:42
  • The time for running the queries will be dominated by the resources of the server holding the database. You cannot distribute that unless you're planning to run a lot of redundant servers. What you're asking now is straying from your original question and not really specific to Raspberry Pi. If you have new questions, you should post them separately in smaller chunks, possibly on sister sites more related to your topics of interest. – Brick Sep 22 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.