How many blocks can a RPi 2B load per day as a Bitcoin full node?

Question

My Raspi 2B is running a full node over a >20MBit/s internet provider (I subtracted another) and it seems that it only loads about 6 blocks a day which would amount to about 1GB per day.

Is it related to CPU bottleneck?

bitcoin.conf is:

maxconnections=40
maxuploadtarget=5000

# running without swap
dbcache=100
maxorphantx=10
maxmempool=50

Output of vmstat does not indicate any problems:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  2 102396  25776  18044 128556    0    0     5     3    4    3  5  4 56 35  0

This is from top

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                     
    9 root      20   0       0      0      0 S   4,5  0,0 485:44.81 ksoftirqd/0                                                 
14220 bitcoin   20   0  586328 408024   1456 S   3,2 46,4 734:08.12 bitcoin-init 

---

EDIT:

My sd card crashed during compiling libbitcoin, I think. It might have been related to a swapfile I had to enable on external USB-HDD. Probably the file was initialized before the USB was mounted and so it was on the sd card instead of on the HDD.

However, I was able to use this card again by putting on it the backup I made. Works fine so far.

I also set up a second Raspberry, a model 3B now (other one is 2B). Curiously, now both of my RPis run very different values in vmstat than before (here for model 2):

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1      0  31300 164228  90760   56   88    97    84  136   82 15  3 57 24  0

Model 3B:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0  27252   6376 250844   18   36    31    42   48   20 13  1 69 16  0

I had 4GB of swap activated and reduced it to 0 now with swapoff -a (I did not turn off dphys-swapfile yet), free shows 0.

Model 2:

Linux 4.19.58-v7+ (raspberrypi)     05.09.2019  _armv7l_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15,02    0,00    3,33   24,34    0,00   57,31

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           7,66     2,79   19,89    0,46   720,24    61,89    76,87     0,10   33,40   28,89  229,40   2,25   4,59
sda               4,42   112,74   52,78   83,69   241,45   815,69    15,49     0,50    6,67   13,43    2,41   3,64  49,63

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8,70    0,00    3,36   29,21    0,00   58,73

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0,00     0,00    6,40    0,00    89,60     0,00    28,00     0,02    3,09    3,09    0,00   1,25   0,80
sda               0,00   165,60   20,80  196,60   215,20  1460,00    15,41     0,91    7,24   21,73    5,71   4,19  91,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,12    0,00    4,85   27,64    0,00   56,39

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sda               0,00   232,40   16,00  284,00   193,60  2064,80    15,06     0,85    5,19   26,26    4,01   2,85  85,40

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9,71    0,00    3,97   29,44    0,00   56,89

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0,00     0,00    1,60    0,00    37,60     0,00    47,00     0,00    3,75    3,75    0,00   1,25   0,20
sda               0,00   218,00   18,00  263,40   201,60  1924,80    15,11     0,92    5,92   12,91    5,44   3,28  92,20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9,17    0,00    4,05   31,53    0,00   55,25

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sda               0,20   170,60   19,20  206,40   258,40  1520,00    15,77     0,91    7,28   29,88    5,18   4,05  91,40

You are right, %util of my HDDs is very high. But since I am mostly downloading the blockchain I wonder why there is such a high amount of usage of the disk(s). Should it not just save it there? Is there a lot of calculation done with transactions to verify the blocks once they are downloaded? There should just be checked if the hash of the previous block fits the new block's entry. Then the next one can be added.

Btw.: Ethernet/USB on the Model 2 is even lower, more like 10MB/s in my experience.

I also think I am downloading more blocks at the moment than before. I have to wait some days to check it.

Answer 1

In general, I found memory to be the biggest problem with running bitcoin full nodes, but ymmv.

When you are faced with problems like this, vmstat should give you a good idea of what is going on. Running vmstat 5 5 will give you five lines of output with 5 seconds between them.

The columns mean the following:

Procs
    r: The number of processes waiting for run time.
    b: The number of processes in uninterruptible sleep.
Memory
    swpd: the amount of virtual memory used.
    free: the amount of idle memory.
    buff: the amount of memory used as buffers.
    cache: the amount of memory used as cache.
Swap
    si: Amount of memory swapped in from disk (/s).
    so: Amount of memory swapped to disk (/s).
IO
   bi: Blocks received from a block device (blocks/s).
   bo: Blocks sent to a block device (blocks/s).
System
    in: The number of interrupts per second, including the clock.
    cs: The number of context switches per second.
CPU
    These are percentages of total CPU time.
    us: Time spent running user code
    sy: Time spent running system code. 
    id: Time spent idle
    wa: Time spent waiting for IO.
    st: Time stolen from a virtual machine.

As a simplistic, first order, explanation:

• If your si/so columns are high (should be 0 most of the time), you have a memory problem.
• If your bi/bo are high (should be single digit most of the time), in combination with IO wait time, you have too much disk access
• If cs is high (should be below 50), reduce the number of processes that you are running
• If us+sys is high (and id+wa almost zero) you have a CPU problem.

So, no yes or no answer, but you should be able to figure it out yourself this way. The norms I gave are for a Pi, not for large server hardware.

-- edit -- In general, the first line of vmstat is not that important; the next lines show an average over the previous interval.

But her, the wait-IO is alarmingly high. Wait-IO means that the CPU is doing nothing, because there are no runnable processes, but there are processes waiting for their IO.

Please do:

apt-get install sysstat
apt-get install iotop

for the next step of the analysis.

iostat -x 5 5 (part of sysstat) will give you an extended (hence -x) on the use of the disks. On my example Pi, that would give:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.85    0.00    0.65    0.10    0.00   98.40

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.04     0.08    0.06    0.07     2.40     2.31    72.21     0.02  146.15    8.32  259.81   8.74   0.11
sdf               0.00     0.05    0.00    0.06     0.01     5.80   182.63     0.04  587.85   39.83  601.09   6.54   0.04

Look at the last column; I bet your %util on one of your disks is much higher that this. That disk is the problem.

Next step: find which process is the problem. iotop would do the same as top, but then for IO. So now you know what the problem is, and what to do about it.

Well, for that last bit: IO on a Pi 2 is a bit of a problem.

Theoretically, the maximum transfer rate to your SD card is somewhat above 20MB/s, depending also on the quality of your SD card (Ultra is better than Cheapo). Using an external drive will give you some additional room, but the Ehternet adapter is on the USB bus as well. Combined Ethernet and USBdisk should be around 50MB/s, theoretically. So adding an USB disk (even a memory stick) to contain the Bitcoin stuff could give some extra speed, if you put the bitcoin stuff on it.