I have been looking to make a cluster of raspberry pi 4s, and have been pretty interested in it and made good head way. I have seen quite a few guides on the topic as well. I followed this one found here: https://www.instructables.com/id/How-to-Make-a-Raspberry-Pi-SuperComputer/ which suggested MPICH. I followed the guide pretty closely, and to my surprise, my program speeds did not increase for a python program performing matrix multiplication as a test.
So I am taking a few steps back. My concerns are this:
1) It seems MPICH is written in fortran or C and I had to install a python library on top of that. I am concerned the cluster is slower because of the use of multiple languages, and if it is fortran, my guess is that fortran is slow to boot. I generally prefer solutions that are native to one language (here, Python 3). Is there an all python solution for this? My thinking is just to connect to each raspberry pi through ssh in python and create a thread for each raspberry pi, and then use python's thread libraries. Would I need mpich in this case?
2) When I want to run programs on the cluster there is a command (it is listed at the bottom of the guide)
mpiexec that essentially makes the program run on the cluster with mpich. I am not sure if I would then do threads in python with that command to work. The guide, and others, made it seem that I just run
mpiexec ... python3 foo.py and presto, my program is now faster. My intuition tells me that this command just runs the same program on multiple pies, not actual parallel processing. For two 750 length and height matrix multiplication the time for two raspberry p 4 does not decrease hence this guess. Am I missing something?
3) In addition, to do something like run
mpiexec ... python3 test.py, I need to have the file on both raspberry pies. This defeats the purpose of the cluster to me, because having multiple gigabyte csv file on multiple raspberry pies is a waste. I would ideally like like to run SQL in python to make visualizations so to me it makes sense to have something like this: 10 GB data file on pi01, and run a program to divide the work amount let's say 10 pies for speed. This is the correct way to look at it, no?
My father and I are hoping to test data processing on raspberry pi 4's ie SQL and pandas. I think it is an interesting opportunity to learn more about hardware, linux, python, and start learning about parallel processing so any help or pointers are appreciated.