2

I have a text processing project that is currently Bash, GNU Parallel and other command-line tools. It's getting moved to a C++ queue type design. It is about 14,000 files each day, plus performing math and making models. I have a 60gb backlog. The format is RAR archives and then inside are CSV files.

I'm trying to decide on an 8 node Raspberry Pi 3B+ cluster (about $500) or buying a used Dual Xeon (12 core) desktop (about $800, HP Z820).

The RPi cluster will help ensure my code scales well. However, I don't quite know how speeds compare.

8 RPi's x 4 cores each = 32 cores 2 x Xeon (12 cores) = 24 cores.

The RPI's are 1.4GHz 64-bit quad-core ARMv8 CPU, 1 GB RAM.

The dual Xeon is 2.5ghz each, 64gb RAM.

How can I accurately think about the cores and speeds of each to decide what to do? This project will get larger within the next year and there will be more files to parse, manipulate, etc.

Improve this question
5
  • It might be useful to know how the 'work' arrives - if its all batch jobs you might find cloud processing is by far the cheapest and scales as far as you want to pay for it. If its a constant processing workload the cloud premium becomes significant. For straight out number crunching I suspect Xeons are far more potent and have a massive memory advantage, if its a truly parallelisable workload a pi cluster can grow and grow as required to meet the demand - that and replacement units are cheap compared to a whole server.
    – tobyd
    Jan 21 '19 at 21:04
  • @tobyd Thanks. I updated the first paragraph. It's basically about 14,000 files each day, math, models.
    – Jasmine
    Jan 21 '19 at 22:05
  • You might want to try Linode or Digital Ocean (or whoever) and put your workload through there before committing to anything else - you might find you can get a vague benchmark of how your code will perform on a 2, 4 or 8 core Xeon for very little outlay. Benchmarking on the Pi is that much harder as there are few (if any) public clusters for rent.
    – tobyd
    Jan 21 '19 at 22:26
  • @tobyd Thanks. I looked at Digital Ocean before posting at it seems that it would cost $80 or so a month at least. After 6 months I could have paid for the whole Pi cluster and most of the Xeon if I chose that path.
    – Jasmine
    Jan 21 '19 at 22:40
  • 2
    Absolutely; if you want constant load cloud is not at all economic - but you can rent one for 3 hours (for <$1), run your code and see how it performs to give you an idea of what an x86 will do and how quickly it'll chew through some work.
    – tobyd
    Jan 21 '19 at 22:42
1

While I like the idea of small devices in a cluster, my experience is that RAM will be an issue in the future. And it is much easier to expand the 64 GB, where as if the 1 GB becomes a problem, then you need to replace all 8 Rpis.

The second reason is that if you have a 64 GB machine then there are tasks that you will be able to do on this machine (maybe some monthly big data analysis of all files?) that would be impossible on eight 1 GB machines.

Improve this answer
1
  • I'm using GNU Parallel to handle all of this currently. A series of queues and I submit the files as jobs to a series of queues. I recognize you as the author :-) Thank you for maintaining GNU Parallel.
    – Jasmine
    Jan 22 '19 at 13:03
1

Just to give you an idea: Dhrystone 2 benchmark score for an RPi 3B+ is about 2500 VAX MIPS in 32-bit and about 3500 VAX MIPS in 64-bit. The same benchmark for a Core i7 around 13000-15000 VAX MIPS, depending on the CPU modification and the compiler used. The test is single-threaded, so let's assume you'll get about the same difference in performance for the same number of cores.

How representative this benchmark is for your project is an open question of course, but with a difference of about 5 times I bet a Xeon will beat your RPi cluster (with a comparable number of cores) hands down.

Improve this answer
2
  • 1
    You misregarded the fact that the RPi Benchmark was measured in MIPS and the Xeon benchmark was in IPS. So Xeon 45 MIPS vs. RPi 3.000 MIPS head scratching. The Xeon processor was testet usind Dhrystone 2. Which seems to be a completely different test. Also this ist just the Xeon's single thread performance. It is not stated if it is the Pi's single tread or multi thread performance. So I don't think the numbers are comparable. Still I expect the Xeon to be multiple orders of magnitde faster than the Pi. It is also easier to manage one workhorse than 100 ponys.
    – kwasmich
    Jan 22 '19 at 10:09
  • @kwasmich Thanks for your catch. I think I found more relevant and comparable tests now. The difference is still huge IMO.
    – Dmitry Grigoryev
    Jan 22 '19 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.