I tried running Pis on SD cards. Those failed after days. Then, I switched to USB thumb drives. I tested two models only so far (Store 'n' Stay NANO and SanDisk Ultra Fit, 32 GB). They fail within weeks to months, so at least a noticeable improvement over SD cards. But still a major reliability and maintenance headache.
I feel since many people are using Raspberry Pis, including in industrial settings, there must be a way to run Raspberry Pis for years at a time without failure?! I would hope that at least some storage hardware manufacturers produce at least some reliable models?! I assume SSDs would be more reliable, but that seems to undermine the defining small form factor and low price of the Raspberry Pi. While I would be willing to pay extra for reliability, I do not want to give up on the small form factor.
My setup and experience: I run Raspberry Pis continuously, rebooting them every night. They run Raspberry Pi OS, writing logs to RAM only and caching writes for 15 min before writing them to disk. My application writes only a few tens of MB per day and has only relatively light read loads, too. When I say the disks fail, I mean that the Pis usually fail to boot and show random error messages on the boot screen (every time a different error). To me, this looks like disk corruption. As reducing writes to disk to a minimum did not help, I assume the problem is disk corruption from heat. The drives get hot while in use, even though my application does not write (as stated) or read much. Touching a thumb drive in use, or directly after shutting down, hurts.
I understand that most SD cards and USB thumb drives are not designed for reliability and that a small form factor means that it is hard to dissipate heat. Though, I believe that it should be possible to build a reliable drive which does not generate much heat by trading off other specs like I/O performance. I really do not care much about disk space or I/O speed, just reliability.
I searched, not just for hours or days, but weeks for solutions online, to no avail. Which surprises me, given the number of Raspberry Pis sold. Can it be that none of those run for more than a few months at a time?!
Update: I use Raspberry Pi 4, Model B. 8 GB RAM, though I typically only use 2-3 GB RAM. And I use the official Raspberry Pi USB-C power supply. The Pis are operated at room temperature (think in an office) and have air freely flowing all around. The OS is Raspberry Pi OS (Bullseye). So, I feel like my setup is as standard as it gets. Before, I also used Raspberry Pi 3s and experienced the same hardware failures.
All software and firmware updates are applied when I install the Pis. Afterwards, they are left untouched, so none of the issues experienced are from software updates. Also, I do not unplug my Pis without shutting them down properly, first. Concerning regular reboots: I experienced that systems often hung after running for some weeks, so I tried rebooting every day and it seems to help, in the sense that the mean time to failure increased from weeks to months.
Detailed error descriptions: (most incidents are undocumented, unfortunately)
- During Linux startup:
EXT4-fs error (device sda2): ext4_dx_find_entry
critical medium error, dev sda, sector 538368 op 0x0 (READ)
/sbin/init: error while loading shared libraries: libmount.so.1: cannot open shared object file
- During Linux startup:
critical medium error, dev sda, sector 605168
Timed out waiting for device /dev/disk/by-partuuid/8d.....
Dependency failed for /boot
- Logged in to desktop already: My tkinter Python application did not automatically go to fullscreen any more on launch.
- In bootloader (Raspberry Pi icon on top left):
MSD error [02:00] 3.32 000000:02 error 7
FAT read failed @ 119024
- Pi boots into console (tty1) instead of desktop, automatially logged in as 'pi'
- In two more instances that I remember, Linux would boot into the console and ask for login credentials (instead of logging in automatically and booting to desktop).
- During Linux startup: The following error repeats many times, with 3 different inode numbers and variants of (spawn)/plymouthd/systemd-udevd/(ifup)/(tmpfiles) instead of "systemd".
EXT4-fs error (device sda2): __ext4_find_entry:1664: inode #28: comm systemd: reading directory inode lblock 0
- During Linux startup (repeated many time):
EXT4-fs error: (device sda2): __ext4_find_entry:1664: inode #[different numbers]: comm (rsync) reading directory lblock 0
After three reboot attempts, it booted. dmesg shows (repeated many times):
critical medium error, dev sda, sector [different numbers] op 0x0:(READ) flags 0x80700 phys_seg [different numbers] prio class 2
Update 2 (Temperature): I measured the following temperatures (min/median/average/max):
CPU: 56/64/64.6/72 °C
GPU: 56.9/64.7/64.8/72 °C
CPU and GPU are always within 2.7 °C of each other on a given Pi.
It seems that devices installed in the past weeks run about 7 °C cooler than older systems (only two samples, though). Maybe this has to do with a recent Raspberry Pi OS (still running Bullseye) or firmware update?
Those temperatures certainly reflect a good processing load. Though, this picture from an "official" Pi4 thermal testing article suggests that CPU/GPU temperatures should not influence the USB port's temperature by much even under heavy processing load. The picture shows the CPU at a temperature of 64.8 °C while the USB ports are below 35 °C.
I noticed that the two thumb drive models which I used so far either do not list operating temperature (which is the case for 12 of 21 drives I just checked) or have a max. operating temperature of just 35 °C. Most thumb drives which do indicate an operating temperature range seem to have a max. temperature of 70 or 60 °C. Still, the picture linked above suggests that even under heavy load, the CPU/GPU would not heat the USB ports to more than 35 °C, so within the acceptable range for my (and probably any) thumb drives.
Update 3 (error statistics): I have 21 Pis running with SanDisk Ultra Fit thumb drives, which are the ones I'm referring to when I say they get too hot to touch. (Recently, I tried some other models to see if they last longer.) I would say those 21 Pis run about a year on average (the spread is from 6 to 21 months). One Pi is running for like 18 months without a problem while others burned through 4 thumb drives already. For those 21 Pis, I used 40 thumb drives, which is to say 21 are still running, while 19 failed. That means the mean time to failure is about 6 months (since my devices run about a year and used about 2 drives on average). Actually, I reformatted some of the drives and used them 2-3 times before throwing them away, so the actual failure rate is even somewhat higher.
Before, I used 2 or 3 Verbatim Store 'n' Stay NANO and they all failed, I think within weeks. That's why I switched to the SanDisk ones.