19

When I check the performance of SD cards for random write, I can see the performance is quite bad for record size 4 kB (this is not surprising) but then for several cards it even drops for larger record sizes before it increases. I measured the random write performance with iozone v3.430 and tested several flash cards of different manufacturers. This is the iozone command, I used to measure with file size 50 MB:

iozone -RaeI -i 0 -i 1 -i 2 -y 4k -q 1M -s 50m -o -f /tmp/testfile

This are the results with file size 50 MB:

Performance drop of SD cards for random write when tested with iozone and 50 MB file size

Question: What is the reason that the random write performance with a record size of 8, 16, 32, 64 and 128 kB is slower as with 4 kB record size?

Peter Brittain suggested to test with a larger file size, so I tried it also with file size 500 MB. This are the results:

Performance drop of SD cards for random write when tested with iozone and 500 MB file size

The overall performance got worse but the phenomenon still occurs.

Partitions are aligned to 4 MB boundaries. File system is ext4 with 4 kB block size. The partition used for the tests starts is mmcblk0p2.

$ lsblk 
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 953.7M  0 loop /mnt/sdb1
mmcblk0     179:0    0  14.9G  0 disk 
├─mmcblk0p1 179:1    0    56M  0 part /boot
├─mmcblk0p2 179:2    0   7.8G  0 part /
└─mmcblk0p3 179:3    0     7G  0 part /mnt/mmcblk0p3

$ cat /etc/fstab | grep mmcblk0p2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

$ sudo fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 15.9 GB, 15931539456 bytes
4 heads, 16 sectors/track, 486192 cylinders, total 31116288 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000981cb

Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            8192      122879       57344    c  W95 FAT32 (LBA)
/dev/mmcblk0p2          122880    16506879     8192000   83  Linux
/dev/mmcblk0p3        16506880    31115263     7304192   83  Linux

$ mount | grep ext4 | grep root
/dev/root on / type ext4 (rw,noatime,data=ordered)

# tune2fs -l /dev/mmcblk0p2 | grep Block
Block count:              2048000
Block size:               4096
Blocks per group:         32768

Update 1: It is clear that the performance for random write especially for small record sizes is significantly lower compared with sequential write. The memory cells of NAND flash storage are grouped to pages and so called erase blocks. Typical page sizes are 4, 8 or 16 kB. Although it is possible for the controller to write single pages, the data cannot be overwritten without being erased first and an erase block is the smallest unit that a NAND flash storage can erase. The erase block size is typically between 128 kB and 2 MB. In modern SD cards, small numbers of erase blocks are combined into larger units of equal size which are called allocation groups or allocation units or segments. The usual segment size is 4 MB. Each write operation on the storage results in a read-modify-write operation for an entire segment. For instance on a SD card with 4 MB segment size, writing 4 kB of data to random locations results in a write amplification factor of 1024. The controllers of SD cards implement a translation layer. For any I/O operation, a translation from virtual to physical address is carried out by the controller. If data inside a segment shall be overwritten, the translation layer remaps the virtual address of the segment to another erased physical address. The old physical segment is marked dirty and queued for an erase. Later, when it is erased, it can be reused. The controllers of SD cards usually cache a single or more segments for increasing the performance of random write operations. If the SD cards stores a root file system, it is beneficial if the controller of the card can cache the segment(s) where the write operation(s) takes place, the segments, which store the metadata for the file system and (if available) the journal of the file system. Consequently, the random write performance of a SD card depends of the erase block size, the segment size and the number of segments, the controller caches. But this all does not explain why the random write performance with a record size of 8, 16, 32, 64 and 128 kB is slower as with 4 kB record size.

Update 2 (answer to myaut): The screenshot of the table is my own work. Currently, I write an article/paper about clusters of single board computers because they are an interesting option for providing resources to student projects and researchers. In this context I also investigated the performance of the CPU, storage and network interface of a single node. I have purchased all tested SD cards. On one of the cards I installed (copied via dd) Raspian Wheezy (version 2014-06-20). After I configured the network settings and installed some additional packages (e.g. iozone), I copied the entire SD card to all other SD cards.

Update 3 (answer to Gabriel Southern): The results are from single runs. The procedure was:

  1. Insert card into Raspberry Pi Model B
  2. Boot the system
  3. Login via SSH
  4. Start iozone test run
  5. Halt the system and try with another SD card

Some of the cards I tried several times to double check. There was just little variation. The phenomenon occur all the time except for the two Samsung cards and one Verbatim card.

Update 4: At the moment I try to find a contact to a company which produces NAND flash clontrollers (Samsung, SanDisk, Toshiba...) in order to ask there for a definite answer. SanDisk has a forum. I asked there for an explanation. I also sent a request to the technical support department of Kingston.

Update 5: The erase block size and allocation unit (segment) size are not responsible for the phenomenon. I tested the erase block size of all SD cards with the pritcsd.py tool fist in the internal card reader of a ThinkPad X240 notebook and finally with a Raspberry Pi Model B. For all cards the output is: Erase block size of mmcblk0 is 65536 bytes. Also the segment size is equal for all tested SD cards. It is 4 MB. This information can be found in the file /sys/class/mmc_host/mmc0/mmc0*/preferred_erase_size. It is quite extraordinary in my opinion that all these cards have the same erase block size and segment size. In the meantime I collected the product IDs/item numbers from the packagings of the tested cards. Here they are.

Product IDs/item numbers from the packagings of the tested cards

Update 6: The technical support of Kingston wrote me that the controllers of the tested Kingston cards (and most likely of the other cards) are optimized for files of size 4 kB. The exact controller implementation is confidential. The answer from Kingston is the best one I got. SanDisk never responded to my support request and I was unable to finde a contact from Sony, Samsung or Verbatim

Improve this question
13
  • 1
    This is an interesting question. Are the results you reported averages over multiple runs, or just from a single run? I'd be curious to know how much variation there is in the results.
    – Gabriel Southern
    Jun 21, 2015 at 18:31
  • 1
    As a result of logical remapping and wear-levelling, this claim in the question "For instance on a SD card with 4 MB segment size, writing 4 kB of data to random locations results in a write amplification factor of 1024." is false.
    – Ben Voigt
    Jun 21, 2015 at 20:43
  • 1
    In my experience performance testing, you hit all sorts of optimizations and caches at smaller scale tests. In particular, I could believe that manufacturers with slower flash would need these optimizations to handle file system updates efficiently, but can't prove it due to the lack of public documentation for all the controllers. That said, I notice you're only using a 50MB file. Have you tried much larger files (as per "run rules" in iozone.org/docs/IOzone_msword_98.pdf) to try to counter this?
    – Peter Brittain
    Jun 22, 2015 at 10:00
  • 1
    On the assumption that you find no difference, I had a look for any other data on this matter. Looks like Linaro org have done some similar research. In particular, the extra large SLC cache may explain your very fast results. And the FAT32 optimisations would be aimed specifically at small writes.
    – Peter Brittain
    Jun 23, 2015 at 23:23
  • 1
    Yeah - already spotted that issue in the page... I think you might be missing something with the FAT32, though: the cards are optimized "for the access patterns that are observed on FAT32" and not just for FAT32. A typical access pattern on FAT will be to write a new file - which requires the file data to be streamed plus a FAT update. The FAT update will usually involve a small number of blocks. If I were writing a FTL, I'd therefore plan to optimize any writes that were smaller than my page size to help FAT performance.
    – Peter Brittain
    Jun 24, 2015 at 10:14

1 Answer 1

Reset to default
4

SD Cards Cells Structure :

In solid-state electronics, a cell is memory element capable of storing one or many bits of informations, the number of bit per cell depend on the used technology. (SLC / MLC / TLC)

Manufacturers use different technology in flash memory to manage it's structure, most used structure are TLC and MLC because of the cheaper cost related to those technology especially TLC.

This technical information is hard to obtain for SD Cards and USB sticks, manufacturers decided so, regarding other flash like technology, as SSD where this information is almost always provided.

This have a direct impact on the hardware life but also on the speed.

SLC, Single Level Cell (1 bit)

Generally 100000 write erase cycles
Erase time: 1-2.5ms

MLC, Multilevel Cell (2 or more bits)

Anywhere from 3000 to 15000 write erase cycles
Erase time: 2.5-3.5ms

TLC, Triple Level Cell (3 bits)

Anywhere from 1000 to 5000 write/erase cycles
Erase time: 4-5ms

Note :

As the cells could contain 1, 2 or 3 bits in some case your sd card controller chip will need to perform more access cycles depending on the record size and the cell capacity.

Your Samsung cards are probably using an SLC technology or they have a powerful controller chip.

Note 2 :

I tried some tests like you do with partitions ext4 block size 4 kb and 1 kb but without big difference

Improve this answer
2
  • The Samsung cards and the Verbatim card are consumer products and in the last years, SLC memory was not common in such devices. The Samsung cards are MB-MP16D and MB-MS16D and the verbatim card ist article number 44007.
    – Neverland
    Jun 25, 2015 at 13:22
  • Maybe the specifications of some controllers are available. I can open the SD cards and check which controllers these cards contain, but I cannot open the microSD cards. Is there any chance to read out the product id/number of the controllers of SD cards via software?
    – Neverland
    Jun 25, 2015 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.