Disk full: find it, fix it, and the du/df mismatch

No space left on device is the most honest error Linux produces, and it still manages to lie to you twice: once when du and df disagree about where the space went, and once when the disk isn't actually full at all - the inodes are. A disk full incident on a Linux box is a solved problem if you check things in the right order. Here is that order.

First move: confirm what "full" means

df -h        # block usage per filesystem
df -i        # inode usage per filesystem

Two distinct failure modes hide behind the same error:

• df -h shows 100% → you're out of blocks. Continue below.
• df -h shows 60% but df -i shows IUse% 100% → you're out of inodes. Writes fail even with gigabytes free. Jump to the inode section.

Also note which filesystem is full. /var/log on its own partition failing is a very different incident from / failing, and overlay filesystems on container hosts point you at the container runtime, not the OS.

Find the space with du - and mind the -x

du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20

-x matters: it stops du from crossing filesystem boundaries, so you don't waste five minutes "finding" 40 GB that's actually on a different, healthy mount. Recurse into the biggest directory and repeat. If the box has ncdu installed, use it - it's the same walk with a UI:

ncdu -x /

Usual suspects, fastest first:

journalctl --disk-usage                          # systemd journal
du -sh /var/log/* 2>/dev/null | sort -rh | head  # classic logs
docker system df                                 # images, stopped containers, build cache
find / -xdev -size +500M -type f 2>/dev/null     # any single large file

When du and df disagree: deleted-but-open files

This is the classic. df says 95% used; du can only account for 60%. The gap is files that were deleted while a process still holds them open. The directory entry is gone (so du can't see it), but the blocks aren't freed until the last file descriptor closes (so df counts them). It's almost always a log file someone rm'd while the daemon kept writing.

lsof +L1

+L1 lists open files with a link count of zero - exactly the deleted-but-open set. Output looks like:

COMMAND   PID USER  FD  TYPE DEVICE  SIZE/OFF NLINK  NODE NAME
java     4172 app    4w REG  259,1  8589934592    0 91234 /var/log/app/debug.log (deleted)

8 GB held hostage by fd 4 of PID 4172. Your options, in order of decreasing politeness:

1. Restart or reload the service. systemctl restart app closes the fd, kernel frees the blocks. Cleanest fix.
2. Truncate through /proc if you can't afford a restart:

: > /proc/4172/fd/4

That truncates the open file in place without touching the process. Note the general lesson: never rm an active log file - truncate it instead (: > /var/log/app/debug.log), or better, let logrotate do it with copytruncate or a proper reopen signal.

Inode exhaustion: full with free space

If df -i shows 100%, something created millions of tiny files. Find the directory with the most entries:

find / -xdev -type d -size +1M 2>/dev/null   # directories whose *entry list* is huge
for d in /var/spool /var/lib /tmp /var/cache; do
  echo -n "$d: "; find "$d" -xdev 2>/dev/null | wc -l
done

Common culprits: PHP/session directories, mail queues (/var/spool), cache directories with per-request files, and CI runners that never clean workspaces. Delete with find ... -delete rather than rm * - the shell glob itself can fail at that scale.

Cleaning up without making it worse

Freeing space is where blast radius lives. Deleting the wrong thing turns a full disk into a data-loss incident. Safe moves, roughly in order:

journalctl --vacuum-size=200M          # cap the journal
logrotate --force /etc/logrotate.conf  # force a rotation cycle
apt-get clean || dnf clean all         # package caches
docker system prune -f                 # dangling images/containers (check first!)

Two things worth knowing before you reach for anything scarier:

• ext4 reserves ~5% of blocks for root by default. On a 1 TB data volume that's 50 GB of emergency headroom: tune2fs -m 1 /dev/nvme0n1p2 reclaims most of it. Legitimate on data disks; leave the reservation alone on /.
• On Kubernetes nodes, a filling disk triggers image garbage collection and then pod eviction well before 100% - if you're here because pods died, read the eviction side too: evicted pods and node pressure. Evicted pods that come back and die again land you in CrashLoopBackOff with a cause that isn't in the container at all.

Verify, then prevent

Don't declare victory on the rm - declare it on the measurement:

df -h /var; df -i /var

Then make the recurrence someone else's non-problem: log rotation with size caps, SystemMaxUse= in journald.conf, disk alerts at 80% not 95%, and separate partitions for /var/log so a chatty app can't take down the root filesystem. The forensic follow-up - what filled it and when - is a journalctl time-window query away.

A full disk is also a canonical interview scenario, because it tests exactly this ordering discipline under time pressure - it shows whether you check df -i, whether you lsof +L1 before rebooting, and whether you delete anything you shouldn't. If you want reps, practice Linux troubleshooting deliberately on systems you're allowed to break.

FAQ

Why does df show 100% but du shows much less? Deleted files still held open by a running process. du walks directory entries, which no longer exist; df asks the filesystem, which still owns the blocks. lsof +L1 finds them; restarting the holding process (or truncating via /proc/<pid>/fd/<n>) frees the space.

Can a disk be "full" with free space available? Yes - inode exhaustion. Millions of small files consume all inodes while blocks remain free. df -i confirms it. The fix is deleting the small-file hoard; the prevention is cleanup jobs and, at mkfs time, a higher inode ratio for small-file workloads.

Is it safe to delete files in /var/log? Truncating is safer than deleting: : > file keeps the inode and open file descriptors valid. If you delete a live log file, the daemon keeps writing to the deleted inode and you're back to the du/df mismatch. Rotate or truncate; don't rm.

Why do writes fail at 95% usage for non-root users? ext4 reserves about 5% of blocks for root so the system stays operable when users fill the disk. tune2fs -m adjusts the percentage; reducing it on data volumes is routine.