When booting with rootfs on NFS, where is the best place to put swap file?

Question

I have 6 raspberry pi (model B) units on my home network and, following a tutorial (here: http://youresuchageek.blogspot.com.au/2013/01/raspberrypi-root-over-nfs-share.html?m=1), have configured them to mount rootfs on NFS from my NAS. (they each have their own NFS mounts - not sharing the same)

The rationale behind doing this was to simplify backups of the main root partition, and eliminate the risk of SD card corruption in the case of power failure, since my current setup is a rats nest of cables on top of my printer(!), which occasionally get knocked causing reboots.

The 2nd half of the linked article provides steps to create a swap file using a loop device, and the swapfile itself is also on the NAS. The NAS and pis are all connected to the same gigabit switch.

My question is this: is the swapfile in the best place for performance and stability? Since the pis now only use a boot partition, I could move the swap file back to the SD card, but I don't know if I would be re-exposing myself to data corruption risk. Also, if the swap gets too much use, will it have any adverse affect on the (WD red) disks in the NAS?

Edit following answer from Mark Komarinski:

As it turns out, there is not actually much swap being used - currently only 54k on the busiest pi:

~ $ free total used free shared buffers cached Mem: 496632 479404 17228 0 560 50740 -/+ buffers/cache: 428104 68528 Swap: 1048572 53684 994888

I ran the time tests with some interesting results:

First, creating a file on the NAS:

~ $ time dd if=/dev/urandom of=/tmp/myfile bs=1k count=10k 10240+0 records in 10240+0 records out 10485760 bytes (10 MB) copied, 11.4046 s, 919 kB/s

real 0m11.460s user 0m0.050s sys 0m9.610s

Second, mount the original root partition to /tmp/p2 ~ $ mkdir /tmp/p2 && sudo mount /dev/mmcblk0p2 /tmp/p2/

Third, create a file on the mounted SD card 2nd partition

~ $ time dd if=/dev/urandom of=/tmp/p2/tmp/myfile bs=1k count=10k 10240+0 records in 10240+0 records out 10485760 bytes (10 MB) copied, 10.1706 s, 1.0 MB/s

real 0m10.217s user 0m0.020s sys 0m9.960s

Not much difference. I was expecting the NAS to be much faster, especially since the pi and NAS are on the same switch.

I also tested with creating a 100MB file and the results were more obvious - SD averaged 968 kB/s (1m48s) and the NAS averaged 692 kB/s (2m31s), so the SD card is definitely faster.

Many thanks, Tiksi

Answer 1

If you're concerned about SD corruption, then I'd say leave swap on the NAS. Spinning disk is much better at handling constant read/write/rewrite operations than SD cards and the WD Red series is designed to act in a NAS system. That being said, SD cards are getting better at handling the larger number of IO operations, provided the SD card is of good quality.

The things to look at really are: 1) How much do you need to use swap? 2) What are the differences in performance between SD and NAS?

So let's look at these individually.

1) How much do you need to use swap?

If your Pi has sufficient memory allocated and the applications you're using don't use much memory then you don't need to have a large swap and this entire question becomes moot. Linux likes to use swap when it can, but usually for applications in memory that aren't used for a long time to free space for apps that are frequently used. Consuming way more memory than is available on a Pi will cause you to start shuffling lots of data between swap and main memory which will cause thrashing no matter where swap is located. The solution in this case is to reduce the amount of memory you're using by either moving to a different hardware platform or moving applications to another Pi.

2) What are the differences in performance between SD and NAS?

This can play a bit of a part if you have low to moderate thrashing, say an application that bursts the memory it needs. The faster you can go into and back from swap, the faster the system will respond. Remember that disk IO of any variety is still IO that will cause the system to lag. I'd use a utility like bonnie++ or iozone to measure disk performance between a filesystem on the SD card and a the NFS server. You could also use dd to write a bunch of blocks to the filesystem and time how long it takes for each:

time dd if=/dev/urandom of=myfile bs=1k count=10k

(edit: you can also use /dev/zero, though you're just writing a bunch of 0s rather than random data. This might give better raw timing. YMMV.)

The above will write out a 10MB file and tell you how long it took along with the write speed. The resulting size of 'myfile' is found by multiplying the block size and the count, so this writes 10,240 blocks, each 1k in size which ends up with 10MB. The blocks used for swap are probably going to be that size or smaller so that's probably close to what you'll what you'd see.

SD might be faster because there's less contention for IO to the SD card - nobody else is using it you you get a nice fat pipe to it. The NAS might be faster because you're going over a 100Mbps network to a device specifically designed to offload disk operations.