When I check the performance of SD cards for random write, I can see the performance is quite bad for record size 4 kB (this is not surprising) but then for several cards it even drops for larger record sizes before it increases. I measured the random write performance with iozone v3.430 and tested several flash cards of different manufacturers. This is the iozone command, I used to measure with file size 50 MB:
iozone -RaeI -i 0 -i 1 -i 2 -y 4k -q 1M -s 50m -o -f /tmp/testfile
This are the results with file size 50 MB:
Question: What is the reason that the random write performance with a record size of 8, 16, 32, 64 and 128 kB is slower as with 4 kB record size?
Peter Brittain suggested to test with a larger file size, so I tried it also with file size 500 MB. This are the results:
The overall performance got worse but the phenomenon still occurs.
Partitions are aligned to 4 MB boundaries. File system is ext4 with 4 kB block size. The partition used for the tests starts is mmcblk0p2.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 953.7M 0 loop /mnt/sdb1
mmcblk0 179:0 0 14.9G 0 disk
├─mmcblk0p1 179:1 0 56M 0 part /boot
├─mmcblk0p2 179:2 0 7.8G 0 part /
└─mmcblk0p3 179:3 0 7G 0 part /mnt/mmcblk0p3
$ cat /etc/fstab | grep mmcblk0p2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
$ sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 15.9 GB, 15931539456 bytes
4 heads, 16 sectors/track, 486192 cylinders, total 31116288 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000981cb
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 8192 122879 57344 c W95 FAT32 (LBA)
/dev/mmcblk0p2 122880 16506879 8192000 83 Linux
/dev/mmcblk0p3 16506880 31115263 7304192 83 Linux
$ mount | grep ext4 | grep root
/dev/root on / type ext4 (rw,noatime,data=ordered)
# tune2fs -l /dev/mmcblk0p2 | grep Block
Block count: 2048000
Block size: 4096
Blocks per group: 32768
Update 1: It is clear that the performance for random write especially for small record sizes is significantly lower compared with sequential write. The memory cells of NAND flash storage are grouped to pages and so called erase blocks. Typical page sizes are 4, 8 or 16 kB. Although it is possible for the controller to write single pages, the data cannot be overwritten without being erased first and an erase block is the smallest unit that a NAND flash storage can erase. The erase block size is typically between 128 kB and 2 MB. In modern SD cards, small numbers of erase blocks are combined into larger units of equal size which are called allocation groups or allocation units or segments. The usual segment size is 4 MB. Each write operation on the storage results in a read-modify-write operation for an entire segment. For instance on a SD card with 4 MB segment size, writing 4 kB of data to random locations results in a write amplification factor of 1024. The controllers of SD cards implement a translation layer. For any I/O operation, a translation from virtual to physical address is carried out by the controller. If data inside a segment shall be overwritten, the translation layer remaps the virtual address of the segment to another erased physical address. The old physical segment is marked dirty and queued for an erase. Later, when it is erased, it can be reused. The controllers of SD cards usually cache a single or more segments for increasing the performance of random write operations. If the SD cards stores a root file system, it is beneficial if the controller of the card can cache the segment(s) where the write operation(s) takes place, the segments, which store the metadata for the file system and (if available) the journal of the file system. Consequently, the random write performance of a SD card depends of the erase block size, the segment size and the number of segments, the controller caches. But this all does not explain why the random write performance with a record size of 8, 16, 32, 64 and 128 kB is slower as with 4 kB record size.
Update 2 (answer to myaut): The screenshot of the table is my own work. Currently, I write an article/paper about clusters of single board computers because they are an interesting option for providing resources to student projects and researchers. In this context I also investigated the performance of the CPU, storage and network interface of a single node. I have purchased all tested SD cards. On one of the cards I installed (copied via dd) Raspian Wheezy (version 2014-06-20). After I configured the network settings and installed some additional packages (e.g. iozone), I copied the entire SD card to all other SD cards.
Update 3 (answer to Gabriel Southern): The results are from single runs. The procedure was:
- Insert card into Raspberry Pi Model B
- Boot the system
- Login via SSH
- Start iozone test run
- Halt the system and try with another SD card
Some of the cards I tried several times to double check. There was just little variation. The phenomenon occur all the time except for the two Samsung cards and one Verbatim card.
Update 4: At the moment I try to find a contact to a company which produces NAND flash clontrollers (Samsung, SanDisk, Toshiba...) in order to ask there for a definite answer. SanDisk has a forum. I asked there for an explanation. I also sent a request to the technical support department of Kingston.
Update 5: The erase block size and allocation unit (segment) size are not responsible for the phenomenon. I tested the erase block size of all SD cards with the pritcsd.py tool fist in the internal card reader of a ThinkPad X240 notebook and finally with a Raspberry Pi Model B. For all cards the output is: Erase block size of mmcblk0 is 65536 bytes
. Also the segment size is equal for all tested SD cards. It is 4 MB. This information can be found in the file /sys/class/mmc_host/mmc0/mmc0*/preferred_erase_size
.
It is quite extraordinary in my opinion that all these cards have the same erase block size and segment size. In the meantime I collected the product IDs/item numbers from the packagings of the tested cards. Here they are.
Update 6: The technical support of Kingston wrote me that the controllers of the tested Kingston cards (and most likely of the other cards) are optimized for files of size 4 kB. The exact controller implementation is confidential. The answer from Kingston is the best one I got. SanDisk never responded to my support request and I was unable to finde a contact from Sony, Samsung or Verbatim