Can I just create a python thread per raspberry pi for parallel processing?

Question

I have been looking to make a cluster of raspberry pi 4s, and have been pretty interested in it and made good head way. I have seen quite a few guides on the topic as well. I followed this one found here: https://www.instructables.com/id/How-to-Make-a-Raspberry-Pi-SuperComputer/ which suggested MPICH. I followed the guide pretty closely, and to my surprise, my program speeds did not increase for a python program performing matrix multiplication as a test.

So I am taking a few steps back. My concerns are this:

1) It seems MPICH is written in fortran or C and I had to install a python library on top of that. I am concerned the cluster is slower because of the use of multiple languages, and if it is fortran, my guess is that fortran is slow to boot. I generally prefer solutions that are native to one language (here, Python 3). Is there an all python solution for this? My thinking is just to connect to each raspberry pi through ssh in python and create a thread for each raspberry pi, and then use python's thread libraries. Would I need mpich in this case?

2) When I want to run programs on the cluster there is a command (it is listed at the bottom of the guide) mpiexec that essentially makes the program run on the cluster with mpich. I am not sure if I would then do threads in python with that command to work. The guide, and others, made it seem that I just run mpiexec ... python3 foo.py and presto, my program is now faster. My intuition tells me that this command just runs the same program on multiple pies, not actual parallel processing. For two 750 length and height matrix multiplication the time for two raspberry p 4 does not decrease hence this guess. Am I missing something?

3) In addition, to do something like run mpiexec ... python3 test.py, I need to have the file on both raspberry pies. This defeats the purpose of the cluster to me, because having multiple gigabyte csv file on multiple raspberry pies is a waste. I would ideally like like to run SQL in python to make visualizations so to me it makes sense to have something like this: 10 GB data file on pi01, and run a program to divide the work amount let's say 10 pies for speed. This is the correct way to look at it, no?

My father and I are hoping to test data processing on raspberry pi 4's ie SQL and pandas. I think it is an interesting opportunity to learn more about hardware, linux, python, and start learning about parallel processing so any help or pointers are appreciated.

Answer 1

If you are going for max speed, then Python is not your friend. What it gains in flexibility it pays for in reduced capability for the compiler to perform substantial optimizations. Certain optimizations are simply not available to Python.

Although I'm not a fan of Fortran specifically, you should also be aware that for matrix multiplication, you almost surely want not only a LAPACK and BLAS library (which are Fortran) but a LAPACK and BLAS that have been tweaked for the hardware that you're using (which will typically be some Fortran and some hardware-specific assembly). For matrix multiplication specifically, there are entire communities that focus just on making these processes as fast as possible, and, if you're really going for speed, you should leverage what they've done.

Separate from the question of the language used, matrix multiplication is not a poster-child case for parallel processing increasing speed. You didn't really provide any sort of baseline problem or benchmarks, but you may find that a "direct" parallelization of a code that's just doing matrix multiplication won't benefit much from going parallel. In particular on a cluster, you'll need to pass messages over the LAN, and the time required for that may more than count against the time you save doing parts of the calculation in parallel. (Details to depend on the algorithms used, the size of the matrices, the speed of the network....)

The MPI libraries (whether you use MPICH or a different implementation) are doing more than you are giving them credit for, I think, because in a typical problem it's not just a question of starting parallel jobs but also keeping them coordinated by passing messages between them. (MPI, of course, stands for message passing interface.) If you've got some highly parallel data processing job where you don't need to coordinate, then you don't need MPI and can accomplish this by starting completely independent jobs on different machines, in any language you like. If you have a job that actually requires coordination, however, you'll need some sort of message passing scheme. You could, of course, reinvent the wheel on that, but MPI is one well-tested and well-supported tool that already exists. Only you can tell if it's a good tool for your job because we don't know what exactly you are trying to accomplish.

Some other misconceptions that you seem to have:

• There is no such thing as a pure Python approach to this. At the bottom of it all - in some cases deeper than others - your Python is calling libraries written in C or C++. Maybe even some Fortran. There's no apparently principled reason to think that's different just because you don't know where the line is on, say, Pandas, but you happen to know on MPI.
• Python is a tool, but it's not a catch-all tool for every problem. Based on what you've said so far about what you want to do, it seems like you have a hammer and are trying to make your problem look like a nail rather than choosing a tool suited to the problem.
• No matter what language you use, when you go to parallel processing (including cluster computing or even multi-threading on a single machine), you typically need to design the parallelization into the program from the start. If you had a program that you originally intended to run in a single thread on a single machine, you will have to at least partially rewrite that to explicitly include how and where you want the parallel sections to run and synchronize. The only exception is a problem that is "embarrassingly parallel".
• You will certainly need code on each machine that is going to process. How you get the data around is part of the design problem. For example if you have a massive file that they are all supposed to read, you may put that on a common file server and have them request chunks as needed. There are also different approaches that effectively let you send code to the data (e.g. MapReduce or Hadoop), which is a good idea when the code is small but the data is large. Again these are broad, fundamental design choices that you'll need to research and understand.

You used the phrase "taking a few steps back". If you're seriously interested in this type of processing, I suggest you take several more back to get a broader view of what's available. You seem to have locked yourself into a small and probably suboptimal set of technologies available already.