Raspberry Pi 3B+ freezes randomly

Question

General setup

My RPi is connected to a mouse, keyboard, and monitor. My swap size is set to 1000 MB and I'm not using wifi.

Problem

The RPi freezes randomly, meaning the GUI is frozen, no SSH connection is possible, and Magic + reisub doesn't work, either. Sometimes, this happens when I use Chromium to surf the web, sometimes it reboots while booting.

While the freezes happen rather occasionally "out of nowhere" (as stated above), I can always reproduce the freeze by executing the following python3 script:

from rtlsdr import RtlSdr

# Get a list of detected device serial numbers (str)
serial_numbers = RtlSdr.get_device_serial_addresses()

# assign object to handle the RTL-SDR
sdr = RtlSdr(serial_number=serial_numbers[0])

sample_rate_sdr = sdr.sample_rate = 2.048e6
center_freq_sdr = sdr.center_freq = 100e6

sdr.read_samples(128)

sdr.close()

To be specific, the script is not the only reason for the freezes. However, it makes them reproducable.

More information about the problem

Note, that it only(!) happens for sample reads, that are smaller than 129 samples (argument to read_samples method). The curious thing is, that the Pi won't - in most cases - freeze immediately, but a few seconds after the script has finished executing and I'm already back at the terminal. While I'm running the script, I have no other window open (only lxterminal to run the script; sometimes another terminal with top, sometimes Geany to modify the script, but mostly none and it doesn't make a difference).

Sometimes, between the script excecution and the freeze, I can still type in commands, start applications, and run them for a while (mostly, it will then freeze when I try to save a file in Geany, but not always). Sometimes, I can still switch windows, but can't open or close any. Sometimes the freeze occurs while the script is still running, but mostly not. Sometimes, every command I type in the terminal (and may they be random letters) + Enter, this will give me "Segmentation fault". This doesn't always happen, though. Once I saw kworker/u8+ at 100 % when frozen, but this might not mean anything.

When I do the read_samples call in a loop, the freeze will mostly occur after a few iterations (after fewer iterations if the loop takes longer because of some other calculations I'm doing in between). This seems to be congruent with my observations with the freeze after several seconds in the stripped down minimal version posted above. When having a long loop, I sometimes see python go into a zombie state in 'top' (when I load many modules, I can also see a D state for python when loading modules, but this might be normal as it will leave the D state again after loading the modules). When I run the loop version of the script in jupyter-notebook, I can sometimes see chromium-browser go into a D state in top before freezing.

Sometimes, when I go to the console with Strg + Alt + F1, I can see several kernel messages appear with random numbers, and some messages like Internal error: Oops: 5 [#5] SMP ARM and Unable to handle kernel paging request at virtual address 55555555. Sometimes, several of those messages (that span multiple lines) will appear, but in the very end there will mostly be a message about a kernel panic.

Things I've tried to solve the problem

The first thing, that pops up searching for random freezes, is the suggestion to try an adequate 2.5 A power supply. So I connected a laboratory power supply, that controls the voltage at 5.1 V, but nothing changed about the feezes. At least, the undervoltage warning I'd occasionally get at startup (and only then - no yellow flash in the corner later or a blinking LED) was gone. Now, the power supply shows me a current of about 500 ... 1000 mA when the freeze happens.

Next, I tried underclocking the Pi by adding the following lines to /boot/config.txt. Didn't help.

arm_freq=900
core_freq=250
sdram_freq=450

I noticed, that the Broadcom controller's metal case got pretty hot when freezing during execution of the looped script, so I installed a small fan. Now, the chip stayed nice and cool. However, that didn't do anything to the freezes.

Then I tried to use another RPi 3B+. Needless to say, that this didn't solve the problem, either.

Now my best bet for a cause is the SD card. I've executed e2fsck -cvf /dev/mmcblk0p2 from a linux live system (as indicated in the accepted answer in another thread), but it only showed a few orphaned inodes (repaired that), no bad blocks. I also used dd to copy the SD card to another one and put that into the second RPi, but that didn't help (perhaps, I also copied the possibly corrupted part by doing this?). I really hope, it's not the SD card as it took a lot of time to set everything up and running (including self-compiled kernel modules), but if that's the fault, I at least know why the freezes occur, although I have no idea what could have corrupted the card. I can't remember any power failures.

Here is the extract from /var/log/syslog that was written during a session that eventually froze (this time when I started Mathematica after executing the script with 120 read samples). Let me know if you need any more information.

Answer 1

First thought: How often is the yellow read/write LED on the pi flashing or 'always on' when the freezes occur? I surf with chromium and often have 15 tabs, or more, open. Depending on the sites (the default gmail page uses lots of memory and you should use their basic html page instead), any more tabs, and the Memory is used up (checking it with gnome-system-monitor). As soon as that happens, the pi is swapping things in and out of memory, with the swap on the SDcard, viciously, to complete whatever it is trying to do with the memory fully used. This state will wear out an SDcard sooner than anyone would appreciate. Depending on where and how badly it corrupts, the system can become erratic, or unbootable. So watch that LED activity when freezes or partial freezes begin. If it is always on or almost always on during your typical tasks, you might need to change your habits, or have far less tabs open in chromium, depending on the sites. And clone your SDcard while you still can, maybe after an fsck through another *nix system? Of course, research what fsck might do in an attempted repair, before trusting it -- as it may make things worse in the case of fatal unrepairable corruption or other mangled data. I assume no liability regarding the results. Point? Tasks that max out your pi's RAM may force intensive memory swapping, and a pi can behave very strangely depending on how vicious the swapping. Menu bar indicators may not update properly, or at all. System monitor may not update anymore, or only very sporadically. Some windows may be fine while others will no longer respond or even close... etc.

My pi3B+ sdr project, powered with a 2.5 amp supply, had stability issues until I plugged keyboard, mouse, usb audio out and other 'stuff' into a powered usb hub, and plugged the rtl-sdr directly into one of the pi's usb ports. After plenty reading to select compatible models for the rpi, I now have two rtl-sdr devices that work, one noisier than the other, but they do work, and the pi no longer freezes when they are plugged 'directly' into the pi, unless I ask for far too much sampling resolution that maxes out processor ability and memory. I can not give you a full technical reason, except to say that rtlSDR directly in pi, and all other usb in a powered hub, is the only way I got them to work consistently.

Working in computers for 30 years, you might be amazed at how many hard to diagnosis freeze problems came down to power or cable or both, or brand of memory (even if its specs were correct). One model of expensive computer brand had intermittent problems on two power circuits. Turned out that occasional loads on those circuits elsewhere would cause brownouts just significant enough to affect the computer(s). Adding a line conditioner solved it, with a note to the customer to get a dedicated power circuit to that station location. More details than that, but the point is... plenty rpi problems have already been shown to be power related due to number and type of USB devices connected, and how they are connected. Using a 2.5 amp power source alone may not fully address a power load situation. Try the rtlsdr directly in pi's USB port, and everything else usb in a powered hub. Simple first before going rocket science with examining sd card blocks. And just in case, have spare configured sd cards on hand in case one gets scrambled (if your budget allows). One can get 32 gig cards now for around 9 US dollars. Load one up with your configuration, and then image it so you can make a few more 'ready to go'.

ANother computer was in an office working fine for months. THen I got a call. Freezes or would not boot. They had just re-arranged desks and equipment. The fax machine was now 10 inches away instead of 3 feet away from the computer. Just for fun, I told them, "As a general class HAM operator, I want to try something most simple first". I explained how quickly RF energy increases or decreases when very close to another device, when moved just a foot or two. I moved the fax machine back. The compuiter ran fine. They were in disbelief. I moved it next to the computer again. Freeze or no boot. Solution? Keep the fax machine on the other side of the table. The problem never returned. Now your problem may be corruption on the sd card, but keep thinking simple a bit longer. Then there is device orientation change when I noticed neighborhood mains lines in the back yard. Twist the device 90 degrees to those power lines... problem gone! Sounds too goofy to be true, and it certainly isn't always that simple or odd (fun when it is!), but analyzing drive blocks is usually low on my list, and I'm willing to pay for a good diagnostic utility to do that for me to save many hours, and to be able to use its report as a type of authoritative reference. But think simple changes first as they are usually the quickest and least expensive things to try.

More goofiness... noisy ballast in flourescent light fixture. Put computer in another room with an incandescent bulb. Problem gone. Move it back. Problem back. And it was an RFI shielded computer, but interference found a way in to some component in the system enough to confuse things. A book full of other stories, but the problem isn't always so easy or odd. It is at times drive corruption or software problems or driver incompatibilities, but think simple first.

Depending how much work you give the pi with the sdr, the pi may run hot, very hot. I found that a heat sink alone is not enough. A fan is necessary, even a small cpu fan sold with some rpi cases, directed towards an installed heatsink on the CPU.

Answer 2

Can you try adding

IPQoS cs0 cs0

at the bottom of your /etc/ssh/sshd_config file?

OpenSSH sets the TOS (Type Of Service) field in the IP datagram but my openwrt doesn't really like it, so I forced them to Cs0 Cs0 (best effort, best effort) and solved the instability/freeze SSH issues.

Found it in the sshd_config man page.