0

I tried running Pis on SD cards. Those failed after days. Then, I switched to USB thumb drives. I tested two models only so far (Store 'n' Stay NANO and SanDisk Ultra Fit, 32 GB). They fail within weeks to months, so at least a noticeable improvement over SD cards. But still a major reliability and maintenance headache.

I feel since many people are using Raspberry Pis, including in industrial settings, there must be a way to run Raspberry Pis for years at a time without failure?! I would hope that at least some storage hardware manufacturers produce at least some reliable models?! I assume SSDs would be more reliable, but that seems to undermine the defining small form factor and low price of the Raspberry Pi. While I would be willing to pay extra for reliability, I do not want to give up on the small form factor.

My setup and experience: I run Raspberry Pis continuously, rebooting them every night. They run Raspberry Pi OS, writing logs to RAM only and caching writes for 15 min before writing them to disk. My application writes only a few tens of MB per day and has only relatively light read loads, too. When I say the disks fail, I mean that the Pis usually fail to boot and show random error messages on the boot screen (every time a different error). To me, this looks like disk corruption. As reducing writes to disk to a minimum did not help, I assume the problem is disk corruption from heat. The drives get hot while in use, even though my application does not write (as stated) or read much. Touching a thumb drive in use, or directly after shutting down, hurts.

I understand that most SD cards and USB thumb drives are not designed for reliability and that a small form factor means that it is hard to dissipate heat. Though, I believe that it should be possible to build a reliable drive which does not generate much heat by trading off other specs like I/O performance. I really do not care much about disk space or I/O speed, just reliability.

I searched, not just for hours or days, but weeks for solutions online, to no avail. Which surprises me, given the number of Raspberry Pis sold. Can it be that none of those run for more than a few months at a time?!

Update: I use Raspberry Pi 4, Model B. 8 GB RAM, though I typically only use 2-3 GB RAM. And I use the official Raspberry Pi USB-C power supply. The Pis are operated at room temperature (think in an office) and have air freely flowing all around. The OS is Raspberry Pi OS (Bullseye). So, I feel like my setup is as standard as it gets. Before, I also used Raspberry Pi 3s and experienced the same hardware failures.

All software and firmware updates are applied when I install the Pis. Afterwards, they are left untouched, so none of the issues experienced are from software updates. Also, I do not unplug my Pis without shutting them down properly, first. Concerning regular reboots: I experienced that systems often hung after running for some weeks, so I tried rebooting every day and it seems to help, in the sense that the mean time to failure increased from weeks to months.

Detailed error descriptions: (most incidents are undocumented, unfortunately)

  1. During Linux startup:
    EXT4-fs error (device sda2): ext4_dx_find_entry
    critical medium error, dev sda, sector 538368 op 0x0 (READ)
    /sbin/init: error while loading shared libraries: libmount.so.1: cannot open shared object file
  1. During Linux startup:
    critical medium error, dev sda, sector 605168
    Timed out waiting for device /dev/disk/by-partuuid/8d.....
    Dependency failed for /boot
  1. Logged in to desktop already: My tkinter Python application did not automatically go to fullscreen any more on launch.
  2. In bootloader (Raspberry Pi icon on top left):
    MSD error [02:00] 3.32 000000:02 error 7
    FAT read failed @ 119024
  1. Pi boots into console (tty1) instead of desktop, automatially logged in as 'pi'
  2. In two more instances that I remember, Linux would boot into the console and ask for login credentials (instead of logging in automatically and booting to desktop).
  3. During Linux startup: The following error repeats many times, with 3 different inode numbers and variants of (spawn)/plymouthd/systemd-udevd/(ifup)/(tmpfiles) instead of "systemd".
EXT4-fs error (device sda2): __ext4_find_entry:1664: inode #28: comm systemd: reading directory inode lblock 0
  1. During Linux startup (repeated many time):
EXT4-fs error: (device sda2): __ext4_find_entry:1664: inode #[different numbers]: comm (rsync) reading directory lblock 0

After three reboot attempts, it booted. dmesg shows (repeated many times):

critical medium error, dev sda, sector [different numbers] op 0x0:(READ) flags 0x80700 phys_seg [different numbers] prio class 2

Update 2 (Temperature): I measured the following temperatures (min/median/average/max):
CPU: 56/64/64.6/72 °C
GPU: 56.9/64.7/64.8/72 °C
CPU and GPU are always within 2.7 °C of each other on a given Pi.
It seems that devices installed in the past weeks run about 7 °C cooler than older systems (only two samples, though). Maybe this has to do with a recent Raspberry Pi OS (still running Bullseye) or firmware update?

Those temperatures certainly reflect a good processing load. Though, this picture from an "official" Pi4 thermal testing article suggests that CPU/GPU temperatures should not influence the USB port's temperature by much even under heavy processing load. The picture shows the CPU at a temperature of 64.8 °C while the USB ports are below 35 °C.

I noticed that the two thumb drive models which I used so far either do not list operating temperature (which is the case for 12 of 21 drives I just checked) or have a max. operating temperature of just 35 °C. Most thumb drives which do indicate an operating temperature range seem to have a max. temperature of 70 or 60 °C. Still, the picture linked above suggests that even under heavy load, the CPU/GPU would not heat the USB ports to more than 35 °C, so within the acceptable range for my (and probably any) thumb drives.

Update 3 (error statistics): I have 21 Pis running with SanDisk Ultra Fit thumb drives, which are the ones I'm referring to when I say they get too hot to touch. (Recently, I tried some other models to see if they last longer.) I would say those 21 Pis run about a year on average (the spread is from 6 to 21 months). One Pi is running for like 18 months without a problem while others burned through 4 thumb drives already. For those 21 Pis, I used 40 thumb drives, which is to say 21 are still running, while 19 failed. That means the mean time to failure is about 6 months (since my devices run about a year and used about 2 drives on average). Actually, I reformatted some of the drives and used them 2-3 times before throwing them away, so the actual failure rate is even somewhat higher.

Before, I used 2 or 3 Verbatim Store 'n' Stay NANO and they all failed, I think within weeks. That's why I switched to the SanDisk ones.

3
  • What Pi, OS, power supply?
    – CoderMike
    Commented Nov 15, 2023 at 5:50
  • "Touching a thumb drive in use, or directly after shutting down, hurts." -> This to me implies there's a physical defect/problem with the hardware, although you refer to "Pis" plural, which would be too much coincidence if they are all like this. How many are there and how many of them have this problem?
    – goldilocks
    Commented Nov 16, 2023 at 16:36
  • I mostly used 40 drives of a single model, see Update 3 in my post. So, I think it would rather be a design flaw than a defect. Three weeks ago, I switched to other models, to see if they are better. 2 out of 12 had some I/O error out of the box, but that 's another discussion... And one failed in use, today. Commented Nov 16, 2023 at 17:22

3 Answers 3

1

You are obviously doing something wrong (although you have failed to explain what or provided any diagnostics). You don't even mention what Pi or what OS.

I run several Pi continuously (one running a datalogger has been running for over 3 years) using a SD Card 9 years old. Even with the inevitable power failures it has restarted.

The only SD Card corruption I have experienced is due to operator error or during a software upgrade.

This is the experience of most of the experienced users on this site.

3
  • I updated my question with info on my setup. What other information do you think would help to investigate the problem? Commented Nov 15, 2023 at 13:12
  • I would suggest the actual errors you get on reboot. Perhaps paste the text of dmesg (not a picture) into your question.
    – NomadMaker
    Commented Nov 15, 2023 at 14:23
  • Ok, I did that to the extent that I documented the errors. In most instances, the Pi does not boot so far that I could run any commands like dmesg. I will try to collect dmesg outputs in the future. Commented Nov 15, 2023 at 14:29
1

I have to agree w/ the other answer here in that your experience does not seem to match that of most users... certainly not mine. FWIW, I've got 4 systems (B+, 3B+, Zero W & 4B) that remain essentially powered on & operating for years without an SD card failure (until last night!). But so much for other's experiences...

You mentioned temperature:

The drives get hot while in use, even though my application does not write (as stated) or read much. Touching a thumb drive in use, or directly after shutting down, hurts.

So - you're using thumb drives, presumably plugged into one of the USB ports, and they get so hot that it hurts to touch them. It seems incredible to me that much heat can be generated in - or conducted to - a thumb drive.

With that much heat, I'd be very tempted to invest in some monitors & thermal scopes to try and learn the source of all that heat. Less expensively, you could monitor chip (CPU & GPU) temperatures as follows:

$ cpu=$(</sys/class/thermal/thermal_zone0/temp) && echo "$((cpu/1000)) C" 
41 C
$ # GPU temp: 
$ vcgencmd measure_temp 
temp=40.4'C 

But given that your issue seems to be heat-related, the first solution that comes to mind is to mount a cooling fan(s). There's a handy overlay for that in the device tree:

Name:   gpio-fan  
Info:    Configure a GPIO pin to control a cooling fan.  
Load:    dtoverlay=gpio-fan,<param>=<val>  
Params:  
         gpiopin -              GPIO used to control the fan (default 12)  
         temp    -              Temperature at which the fan switches on,in millicelcius (default 55000).  
         hyst    -              Temperature delta (in millicelcius) below. temp at which the fan will drop to minrpm (default 10000)  

Add the following line to /boot/config.txt to set your fan(s) to turn ON at 55C, and OFF at 45C:

dtoverlay=gpio-fan,gpiopin=12,temp=55000,hyst=10000

Fans will treat the symptoms, but my feeling is that you'll need to dig deeper to find the cause of the excessive heat. Until then, this may resolve your high failure rate on memory cards.

4
  • Thank you for the suggestions. I would really like to find the cause rather than treat the symptoms. Given the two answers so far, it seems like other people can make it work without a fan. So, I would like to do without one, too. I will measure CPU and GPU temperatures and report back. Does that mean the USB port (and SD card holder) act as heat dissipators for the CPU and GPU? Otherwise, those temperatures would probably not be related, as I have fresh air flowing around the Pi. Commented Nov 15, 2023 at 22:53
  • @LiterallyCode: "Does that mean the USB port (and SD card holder) act as heat dissipators for the CPU and GPU?". Yes, they do REF. But it's amazing that they are getting so warm. The CPU & GPU are the primary active (heat generating) components, so you will learn something by monitoring - as explained in answer. But now for a "commercial break": please read this. Points don't pay any bills, but they do make this exercise worth doing.
    – Seamus
    Commented Nov 15, 2023 at 23:15
  • I know about the physics of heat transfer :D. My question is whether the USB ports are connected to the CPU/GPU in a way that they act as heat sinks by design or whether there should only be a minimal/unavoidable heat flow between those parts. Anyway, I updated my post with temperature statistics. (As for the upvote, I do not have the required reputation :-S) Commented Nov 16, 2023 at 15:13
  • @LiterallyCode: No - you don't have the rep for upvotes, but you can accept an answer.
    – Seamus
    Commented Nov 16, 2023 at 19:59
0

Sorry, tl-dr;

i have 2 PIs with runtime well over months; let me share my 2 cents -

  1. get really good power supply. I use a settable DC-DC buck modules from china, that i am able to set PRECISELY to 5,1 - 5,15V. It does not hurt pi - official USBC power supply is 5,1V too, so i guess RPI foundation knows about this problem but is still stubborn to use proper power input solution.
  2. I feed it straight into 5V GPIO and GND. Use both 5V pins, and 2 GND pins. Less connection resistance - less losses on the input. Never in my life had any problem with it.
  3. Disable swap. Search for disable swap - there is one question on web that always pops up (asked by Jeff Geerling) - and there is solution from him as well. Worked for me always. Stresses memory media less.
  4. Whenever possible, use USB keys, i have nothing special, but i trust san disk USB3 ones. I had SD card failures up to a point it was damaged and unreadable (that system was running multiple years too, back then i did not know you can run PI3A+ from usb too. Again - RPI foundation is stubborn to give us proper storage solution. At least PCIE on RPI5 is something, but not a proper solution.
  5. A good low esr capacitor of 4700uF on 5V input rail is helping a little too, but i used it mainly on system with high power usage - that means from 5V rail it was running Rpi, HDMI monitor, SATA drive through USB adapter, few I2C modules etc... Never needed it on others.
  6. In my experience, PI Zero and Zero 2 are more resilient to poor power supply than PI4, but i killed one Zero 2 due to too low input voltage, so CHECK INPUT VOLTAGE! Always stay above 5V no matter what. I had also 5.3V power supply output and found due to piss poor usb cables Pi4 was seeing only 4.9V, since then i refuse to use USB for power input.

I overclock too, and good supply helps with stability of overclock. Good cooling also helps a lot. Cooling for Pi4 is a must in my opinion, ideally active. Not only the CPU itself is heating, but also LAN and USB chip. A little airflow never hurt them.

2
  • Thanks! Are you using Ultra Fit, too or some other SanDisk model? Concerning the power supply, are you saying the official USB-C power supply is not good? I guess the other people who claim to have no problems might use the official power supply, too. So, not sure if my problem is related to the power supply. Commented Nov 22, 2023 at 15:39
  • I use SanDisk Ultra flair and Sandisk Cruzer Ultra FIT. For power supply i mean anything that puts out stable voltage and high enough current for DC/DC buck module. A "power brick" similar as for laptops, lets say 12V or 19V. Thing is ,if i run multiple of PIs close to each other, i need only one power brick instead of multiple USBC official power supplies. Official USBC power supply is ok, but i would not use it for anything more demanding, like added sata drives, sensors, LCD screens. Commented Nov 23, 2023 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.