Could not find help for this issue in this nor any forums
I have finally integrated system with desired few apps (HA over docker, Node-red and Grafana) and temperature/Power measurements using 1-wire OWFS via DS9490 USB dongle. During this process I've done 100+ SD images/backups/re-installations/steps and finally got it working decently. Now when "resting" and leaving system to run and monitor without touch, it gets totally stuck after 2-4 days. No changes done, only grafana views taken to see house system data.
When it's stuck, I see LAN still blinking, but cannot access via SSH, direct keyboard/HDMI, nor VNC, all which normally works. only power off/on works
Are there some system logs which could be traced to find root cause? I found in /var/log several logs where stuck is seen in timestamps stopping updates until I gave power reset, like daemon, kern and user.log, but nothing suspicious stands out
I have varied between several SDs all the time (mostly 16GB where system taking ~8+GB), took DD backups regularly, I could try this still with another SD, or I have SSD obtained but did not go there yet as have needed so many re-installations the SD works easier, my power sourceis 65W from laptop
It's really annoying after months of study/learn by doing/re-doing etc. as after all pieces in place it fails where intended to provide robust logging system.
Update after 4 days So far I have found slow increase in memory use but cannot pinpoint any SW doing so much of it. I put into a DB to store "free -m" equivalent counter of used memory, and since reboot its gradually grown in ~3 days from 50%->65%.
With top or htop I can see that nodered increased minor 30Mb, Grafana 20Mb, but this 15% increase from unit 3700 MB memory is as much as ~550MB which is tenfold more than top/htop shows increase.
If I compare free -m results after boot until today, I see:
- free memory -570, this matches this 15% less memory,
- used memory increased only 118Mb,
- shared increased 177Mb,
- cache increased 450Mb which seems to be the main change.
before this run, unit stuck previously already at point where sued memory was at 60%, now I am past that in 65%, but still far a away from total memory used.
Update after 5 days Used memory jumped in a day from ~60% to 75%. The cache had took another 400 Mb junk. Then I found these instructions, and was able to free memory back https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/
Clear cache: sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"
1GB released and free memory back to level after boot!!!
So will create a cron taskto free regularly, fo sure tryin to catch why this happens but no experience on linux nor SW behaviour as not been doing programming in decades...