Raspberry PI 4 gets stuck after few days of working

Question

Could not find help for this issue in this nor any forums

I have finally integrated system with desired few apps (HA over docker, Node-red and Grafana) and temperature/Power measurements using 1-wire OWFS via DS9490 USB dongle. During this process I've done 100+ SD images/backups/re-installations/steps and finally got it working decently. Now when "resting" and leaving system to run and monitor without touch, it gets totally stuck after 2-4 days. No changes done, only grafana views taken to see house system data.

When it's stuck, I see LAN still blinking, but cannot access via SSH, direct keyboard/HDMI, nor VNC, all which normally works. only power off/on works

Are there some system logs which could be traced to find root cause? I found in /var/log several logs where stuck is seen in timestamps stopping updates until I gave power reset, like daemon, kern and user.log, but nothing suspicious stands out

I have varied between several SDs all the time (mostly 16GB where system taking ~8+GB), took DD backups regularly, I could try this still with another SD, or I have SSD obtained but did not go there yet as have needed so many re-installations the SD works easier, my power sourceis 65W from laptop

It's really annoying after months of study/learn by doing/re-doing etc. as after all pieces in place it fails where intended to provide robust logging system.

Update after 4 days So far I have found slow increase in memory use but cannot pinpoint any SW doing so much of it. I put into a DB to store "free -m" equivalent counter of used memory, and since reboot its gradually grown in ~3 days from 50%->65%.

With top or htop I can see that nodered increased minor 30Mb, Grafana 20Mb, but this 15% increase from unit 3700 MB memory is as much as ~550MB which is tenfold more than top/htop shows increase.

If I compare free -m results after boot until today, I see:

• free memory -570, this matches this 15% less memory,
• used memory increased only 118Mb,
• shared increased 177Mb,
• cache increased 450Mb which seems to be the main change.

before this run, unit stuck previously already at point where sued memory was at 60%, now I am past that in 65%, but still far a away from total memory used.

Update after 5 days Used memory jumped in a day from ~60% to 75%. The cache had took another 400 Mb junk. Then I found these instructions, and was able to free memory back https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/

Before cache cleanup

Clear cache: sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"

result

1GB released and free memory back to level after boot!!!

So will create a cron taskto free regularly, fo sure tryin to catch why this happens but no experience on linux nor SW behaviour as not been doing programming in decades...

Answer 1

Power would be my first concern. Start by trying a new power supply with adequate capacity, and more then what you now have. If this fixes it great. If not determining the problem will be extremely difficult. The first thing I would suspect is software. Try loading another Pi preferably from a different batch and see if it behaves the same. If so it probably a problem with the Pi software or an external device. Add some code to show how much RAM memory is being used. If it is stable that is good, if not and usage is growing you have a memory leak in the code. Also do a CRC (checksum or a simple sum will help) on the SD card every few hours and see if that changes. If that does not help you find the problem section you may have to exclude sections of code for each additional test. If the memory is stable. If this checks and it behaves the same then I would expect some hardware device is failing, try eliminating one at a time. Either remove it from the system and adjust the code or use a new unit. I know this will take time and will be a real pain but I have no known shortcuts to help, hopefully somebody has a better handle on this then I do. You will need access to a lab to get much further into it.