2

I have some scripts that are running via systemctl. They generate lots of logs.

My raspberry pi crashed, I checked and found out the SD card was full, and there were huge files in /var/log.

I have not set any specific settings for systemd or journalctl to clear the logs.

I read that journalctl logs are cleared automatically, to preserve some disk space.

However, thats not happening in my case. I have a 16GB SD Card, of which max 5 GB is filled normally. All of remaining space was consumed by these logs.

Update:

I had the following two huge files in /var/log:

  1. daemon.log
  2. daemon.log.1

All parameters inside /etc/systemd/journald.conf are commented.

2
  • should I move this to unix.stackexchange.com? Commented Mar 4, 2019 at 7:04
  • There should be the logrotate package to compress and delete logs based on controls in /etc/logrotate.d change the parms in there and you can get logrotate to be more aggressive at deleting old stuff.
    – Dougie
    Commented Mar 4, 2019 at 7:55

2 Answers 2

4

At least on Raspberry Pi Stretch from 11/2018, both journalctl and rsyslog are running. This means that log information is repeated twice. You can safely disable rsyslog, which will stop those from growing, and eliminate the need for logrotate.

journalctl log size is controlled by /etc/systemd/journald.conf. Take a look at 'man journald.conf', especially SystemMaxUse, SystemKeepFree, etc. I havent' used these myself, but you may need to adjust these if /var/log/journal is full of logs as well, although I suspect not, since by default on a Pi the journalctl log is kept in non-permanent storage. If you disable rsyslog, you may want to: cd /var/log; mkdir journal; chmod 755 journal; chmod g+s journal and then restart the system. This will make the journalctl log permanent.

4
  • I removed /var/log/daemon.log and /var/log/daemon.log.1. There are still some huge files in the same dir, but I kept them for further investigation into this issue. I dont think the /var/log/journal was huge, but I'll check it. Commented Mar 6, 2019 at 13:13
  • What about rsyslog? Are they not rotated/cleared automatically? Commented Mar 6, 2019 at 13:14
  • @DushyantBangal were you able to keep systemctl from creating huge log files? From reading all these answers, it is still not clear to me if you were successful, and would like to know how you managed to do this in the end so I can apply your solution. I was also under the assumption that journalctl logs were cleared automatically. Commented Jun 7, 2020 at 18:52
  • @diazdeteran it never happened again. I think it just happened 1 or 2 times. Never found a solution. Commented Jun 23, 2020 at 19:17
2

You can use systemd-tmpfiles to manage the journal. Using it, you should be able to clean and disable the logs with:

rpi ~$ sudo systemd-tmpfiles --clean
rpi ~$ sudo systemd-tmpfiles --remove

You can do more specific things. Have a look at man systemd-tmpfiles.

You can also check /etc/tmpfiles.d/*.conf (see man tmpfiles.d) why your journal is not cleaned up automatically.

If there are only one huge log file daemon.log and its predecessor daemon.log.1, you should look into it what daemon creates so much output and stop it doing that.

Update respected to the comments:
Because you are knowing what applications are generating the huge log messages and if you don't need them you can send them to the black hole /dev/null. Here an example how to do it with a simple ls command:

rpi ~$ ls >/dev/null >&2

This will send all standard messages and all error messages into the black hole.

7
  • /etc/tmpfiles.d/ is empty. I dont want to clean these manually or disable the logs. I just want the journal to delete the old logs automatically as it does by default, but apparently not in this particular case. Commented Mar 7, 2019 at 6:22
  • @DushyantBangal I have updated the answer with the last paragraph.
    – Ingo
    Commented Mar 7, 2019 at 9:15
  • Thanks, the post is helpful, but doesn't really solve my issue. I already know what is generating the logs (first line of my question), and I need that. But I dont want it to fill up the entire disk (which it prevents with default settings, but not happening in this case) Commented Mar 12, 2019 at 6:37
  • @DushyantBangal How long do you need the information? One day? A week? Are there repeated log lines about the same issue? May it be a solution to log on a remote server?
    – Ingo
    Commented Mar 12, 2019 at 10:42
  • I just don't want it to fill up and die. Other than that, it doesn't matter much. Commented Mar 16, 2019 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.