df shows SD card full but I can't figure out what it's full of

Question

I have a Pi 4 running Raspberry Pi OS that I use as an Emby server. It's just a fresh install of Raspberry Pi OS with Samba and Emby, nothing else. It has been working fine for months, but yesterday Emby wouldn't start due to a "No space left on device" error.

I checked with df and sure enough there's no space on the internal SD card:

user@Server:/mnt $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   29G     0 100% /
devtmpfs        805M     0  805M   0% /dev
tmpfs           934M     0  934M   0% /dev/shm
tmpfs           934M   11M  924M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           934M     0  934M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   50M  203M  20% /boot
/dev/sdd1       3.6T  2.5T 1010G  72% /mnt/storage1
/dev/sdc1       3.6T  2.5T 1004G  72% /mnt/storage2
/dev/sdb1       9.1T  5.0T  3.7T  58% /mnt/media2
/dev/sda1       9.1T  5.0T  3.7T  58% /mnt/media1
tmpfs           187M     0  187M   0% /run/user/1000

So what's taking up all the space? I ran du in / to check:

user@Server:/ $ sudo du -sh *
0   bin
50M boot
0   dev
3.9M    etc
48K home
0   lib
16K lost+found
4.0K    media
15T mnt
240M    opt
du: cannot access 'proc/1036/task/1036/fd/4': No such file or directory
du: cannot access 'proc/1036/task/1036/fdinfo/4': No such file or directory
du: cannot access 'proc/1036/fd/3': No such file or directory
du: cannot access 'proc/1036/fdinfo/3': No such file or directory
0   proc
24K root
11M run
0   sbin
4.0K    srv
0   sys
32K tmp
1.3G    usr
1.5G    var

I don't understand the four lines about proc/1036, but otherwise this all looks correct. There's 15 TB in mnt, which is on external drives. And everything else adds up to about 5 GB. So there should be around 25 GB free on the SD card. But there's not.

This question describes a similar problem that was resolved by expanding the fileystem. I definitely expanded the file system when I first set it up. Just in case, I ran sudo raspi-config and expanded it again. No change. So that doesn't seem to be the issue. Another question shows a very similar issue, but with no resolution.

How can I figure out what is taking up all my space?

Answer 1

I figured it out thanks to this answer: There were large files in the mount directory (on the system SD card) for one of my external drives. With the drive mounted, that directory is completely hidden.

I have a nightly cron job that runs rsync to clone one of my external media drives to another. At some point the backup drive became unmounted while the primary was still mounted. So when rsync tried to clone /mnt/media1 to /mnt/media2, it was unknowingly writing to the /mnt/media2 directory on the system SD card. It wrote 29 GB and filled up the card then quit. Then the next time I rebooted, the media2 drive mounted successfully over /mnt/media2, hiding the 29 GB backup from du and every other way I looked.

Once I unmounted the external drives and took a look it was obvious. I deleted the files from /mnt/media2 and everything is back to normal.

Answer 2

I ALWAYS check that the target drive is mounted in RW mode. Otherwise rsync will still try to copy somewhere, and may copy to your SD Card (until full).

BACKUP_MOUNTED1=$(mount | awk '/PiData/ {print $6}' | grep "rw")

You may find the following useful to check SD usage.
It works on any directory including / and excludes virtual files.

#! /bin/bash
# 2021-02-06
# Print directory usage - sorted by size

if [ $# -eq 0 ] ; then
    UDIR='.'
else
    UDIR=$1
fi

if [ $UDIR == '/' ]; then
    # exclude virtual directories, mount points etc which don't physically exist
    sudo du -hd1 --exclude proc --exclude run --exclude sys  --exclude dev  --exclude boot --exclude mnt  --exclude media $UDIR | sort -h
else
    sudo du -hd1 $UDIR | sort -h
fi