0

I have a RPi 4 running 3 web apps which I had running without issues for a few years. About 6 months ago the original SD card (32gb) died so I replaced it with a new 64gb card and reinstalled everything. Raspbian (bullseye) running headless (no desktop etc. installed), apache, and my web apps.

I've just been on vacation and returned home to find the RPi not working, discovered the SD card is full. I have a external USB drive attached which the web apps store their outputs on, so AFAIK no files (at least no large files) should ever be written to the SD card.

I don't know linux at all, so I did a google search and found a couple of commands to run to try to find what was consuming the space, but really I haven't been able to figure it out. The RPi is booting, and my apps are starting, but the apps then just output (in the logs) insufficient space). The external USB has 1.45TB free (of 5TB), so that's not the issue.

If I run sudo df -h I get the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        57G   55G     0 100% /
devtmpfs        1.7G     0  1.7G   0% /dev
tmpfs           1.9G  4.0K  1.9G   1% /dev/shm
tmpfs           759M  2.8M  757M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p1  255M   31M  225M  12% /boot
/dev/sda2       4.6T  3.1T  1.6T  67% /mnt/usbstorage
tmpfs           380M     0  380M   0% /run/user/1000

I am assuming the /dev/root is my SD card? How do I find and remove what is taking up all the space? Yesterday I cleared out /tmp and ran apt clean which gave me a little, but as you can see the little I got yesterday is now gone.

I do realise I probably have a problem with one of my apps writing too much where it shouldn't be (logs maybe), but at this stage I just need to clear out a few GB just so I can get things to run "normally" to figure out what the apps are doing wrong.

This was never an issue on the previous SD card, it was never full, so clearly something I've set up incorrectly in the past few months.

The RPi is on 24/7 (always has been), but we have had a number of short power outages recently (off for a few seconds then on again, sometimes several times a day) so potentially there's just files that should have been removed have been orphaned due to loss of power. But I don't know where to look.

4
  • 1
    You could try cd /, sudo du -skx *|sort -n to check where the biggest part of your space is used, then repeat the command in the respective directory. Or omit the s option to get a fully recursive listing: sudo du -kx / | sort -n. Candidates for growing or many files are /var/log and subdirectories and /var/cache/apt/archives. Maybe some application is configured to produce debug output or verbose output.
    – Bodo
    Commented Apr 4, 2023 at 16:47
  • 1
    @Bodo Could you add this as an answer please? Using sudo du -skx *|sort -n directed me fairly quickly to (I think) the culprit. One of my apps has created a database on the SD card with metadata about the files it's creating (correctly) on the USB drive. It seems to need to move that database also to the USB drive
    – Midavalo
    Commented Apr 4, 2023 at 17:48
  • Have you run journalctl --disk-usage? This will tell you what the archived and active journals are taking up.
    – Seamus
    Commented Apr 5, 2023 at 0:01
  • @Midavalo I made an answer out of the comment.
    – Bodo
    Commented Apr 5, 2023 at 10:27

3 Answers 3

3

With the following commands you get a list of disk usage summaries for files and directories in the root directory /, sorted by size.

cd /
sudo du -skx *|sort -n

If necessary repeat the command in the directories with the biggest size.

You could also omit the s option to get a fully recursive listing

sudo du -kx / | sort -n

Candidates for growing files or many files are /var/log and subdirectories and /var/cache/apt/archives. Maybe some application is configured to produce debug output or verbose output.

1

The most likely culprit is the logs overflowing. Use the following command to see the space used.

sudo du -hd1 /var/log

Personally I would just delete any directory which seems to be a culprit.

sudo rm -rf /var/log/culprit

The directory will reappear at next boot and start to fill again. An examination of the logs will show what is going wrong.

1
  • Thanks - I too thought it might be logs, however my /var/log directory was relatively small and tidy. I think I've figured what it is now though
    – Midavalo
    Commented Apr 4, 2023 at 18:20
1

Run sudo du -h -d 1 / which will show the usage of top level directories.

You can then drill down unto the largest by replacing / with e.g. /var etc to find where the space is used.

See https://raspberrypi.stackexchange.com/a/120083/8697

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.