0

I have some scripts that are running via systemctl. They generate lots of logs.

My raspberry pi crashed, I checked and found out the SD card was full, and there were huge files in /var/log.

I have not set any specific settings for systemd or journalctl to clear the logs.

I read that journalctl logs are cleared automatically, to preserve some disk space.

However, thats not happening in my case. I have a 16GB SD Card, of which max 5 GB is filled normally. All of remaining space was consumed by these logs.

Update:

I had the following two huge files in /var/log:

  1. daemon.log
  2. daemon.log.1

All parameters inside /etc/systemd/journald.conf are commented.

  • should I move this to unix.stackexchange.com? – Dushyant Bangal Mar 4 at 7:04
  • There should be the logrotate package to compress and delete logs based on controls in /etc/logrotate.d change the parms in there and you can get logrotate to be more aggressive at deleting old stuff. – Dougie Mar 4 at 7:55
2

At least on Raspberry Pi Stretch from 11/2018, both journalctl and rsyslog are running. This means that log information is repeated twice. You can safely disable rsyslog, which will stop those from growing, and eliminate the need for logrotate.

journalctl log size is controlled by /etc/systemd/journald.conf. Take a look at 'man journald.conf', especially SystemMaxUse, SystemKeepFree, etc. I havent' used these myself, but you may need to adjust these if /var/log/journal is full of logs as well, although I suspect not, since by default on a Pi the journalctl log is kept in non-permanent storage. If you disable rsyslog, you may want to: cd /var/log; mkdir journal; chmod 755 journal; chmod g+s journal and then restart the system. This will make the journalctl log permanent.

  • I removed /var/log/daemon.log and /var/log/daemon.log.1. There are still some huge files in the same dir, but I kept them for further investigation into this issue. I dont think the /var/log/journal was huge, but I'll check it. – Dushyant Bangal Mar 6 at 13:13
  • What about rsyslog? Are they not rotated/cleared automatically? – Dushyant Bangal Mar 6 at 13:14
2

You can use systemd-tmpfiles to manage the journal. Using it, you should be able to clean and disable the logs with:

rpi ~$ sudo systemd-tmpfiles --clean
rpi ~$ sudo systemd-tmpfiles --remove

You can do more specific things. Have a look at man systemd-tmpfiles.

You can also check /etc/tmpfiles.d/*.conf (see man tmpfiles.d) why your journal is not cleaned up automatically.

If there are only one huge log file daemon.log and its predecessor daemon.log.1, you should look into it what daemon creates so much output and stop it doing that.

Update respected to the comments:
Because you are knowing what applications are generating the huge log messages and if you don't need them you can send them to the black hole /dev/null. Here an example how to do it with a simple ls command:

rpi ~$ ls >/dev/null >&2

This will send all standard messages and all error messages into the black hole.

  • /etc/tmpfiles.d/ is empty. I dont want to clean these manually or disable the logs. I just want the journal to delete the old logs automatically as it does by default, but apparently not in this particular case. – Dushyant Bangal Mar 7 at 6:22
  • @DushyantBangal I have updated the answer with the last paragraph. – Ingo Mar 7 at 9:15
  • Thanks, the post is helpful, but doesn't really solve my issue. I already know what is generating the logs (first line of my question), and I need that. But I dont want it to fill up the entire disk (which it prevents with default settings, but not happening in this case) – Dushyant Bangal Mar 12 at 6:37
  • @DushyantBangal How long do you need the information? One day? A week? Are there repeated log lines about the same issue? May it be a solution to log on a remote server? – Ingo Mar 12 at 10:42
  • I just don't want it to fill up and die. Other than that, it doesn't matter much. – Dushyant Bangal Mar 16 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.