Fix: disk full, find what is eating the space

A service stops writing, a database refuses connections, apt fails, or a log fills with:

write error: No space left on device
FATAL:  could not write to file "pg_wal/xlogtemp.1234": No space left on device

The fix has two halves: free enough space to get the system breathing again, then find out what grew and make sure you hear about it next time before it reaches 100 percent.

Step 1: which filesystem, and is it space or inodes?

df -h
df -i

Look at the Use% column in both. A filesystem can be out of inodes (the second command) while df -h shows space free; that happens when millions of tiny files pile up, typically in a mail queue, a session directory or a PHP cache. The rest of this article uses / as the example mount point; substitute yours.

Step 2: find the big directories

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 15
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -n 15
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null | sort -k5 -h | tail -n 20

-x (and -xdev for find) stay on one filesystem, so a mounted NAS or /proc does not pollute the numbers. Walk down one level at a time into the biggest entry until you reach the files. If ncdu is installed, sudo ncdu -x / does the same interactively and lets you delete from inside it. Careful: deleting from ncdu is immediate and permanent.

Step 3: the usual suspects

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

sudo du -sh /var/cache/apt /var/lib/apt/lists
sudo apt clean

sudo du -sh /var/log
sudo ls -lhS /var/log | head

docker system df
docker system prune --volumes

sudo du -sh /var/lib/snapd /var/tmp /tmp ~/.cache

• Journal: --vacuum-size removes the oldest archived journal files until the total is under the limit. Careful: those logs are gone. Set SystemMaxUse= in /etc/systemd/journald.conf so it does not grow back.
• Package caches: apt clean (or dnf clean all) deletes downloaded package files; they are re-downloaded when needed, so this is safe.
• Logs that rotate by size but never got rotated: a single multi-gigabyte error.log usually means a program logging the same error thousands of times a second. Fix the error, then truncate (see Step 4).
• Docker: stopped containers, dangling images, build cache and unused volumes. Careful: docker system prune --volumes deletes every volume not attached to a running container, including databases of stopped stacks. Run docker system prune without --volumes first and look at docker volume ls before going further.
• Old kernels on /boot: sudo apt autoremove on Debian and Ubuntu removes kernels that are no longer needed.

Step 4: the space that du cannot see

If df says the disk is full but du adds up to far less, something holds a deleted file open. The space is only returned when the last process closes it. Find the process:

sudo lsof -nP +L1 | sort -k7 -n | tail -n 10

+L1 lists files with fewer than one link, that is, deleted ones. Restart the process named in the first column (systemctl restart nginx, for example) and the space is freed. If you cannot restart it, truncate the file through the process’s file descriptor: with PID 1234 and FD 7 from the lsof output, run sudo truncate -s 0 /proc/1234/fd/7.

The second invisible case is data hidden under a mount point: files were written to /mnt/data while the disk was not mounted, then the mount covered them. Check with a bind mount of the root filesystem: sudo mkdir /mnt/rootfs && sudo mount --bind / /mnt/rootfs && sudo du -sh /mnt/rootfs/mnt/data, and unmount with sudo umount /mnt/rootfs afterwards.

For live log files, never rm them while the writer is running; that creates exactly the deleted-but-open problem above. Empty them in place instead:

sudo truncate -s 0 /var/log/nginx/error.log

Step 5: alert before it happens again

A disk that hit 100 percent will do it again. Add a Prometheus rule that fires on the trend, not only on the threshold, so there is time to act:

- alert: DiskFillingUp
  expr: |
    predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
    and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.2
  for: 30m
  labels:
    severity: ticket
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} will be full within 24h"

Without Prometheus, the Server health check script in the Linux admin section reports filesystems above a threshold, and a systemd timer can run it hourly. Either way, the goal is to hear about 85 percent on a Tuesday afternoon instead of 100 percent on a Sunday night.

Official documentation

• journald.conf: SystemMaxUse and related settings
• docker system prune
• PromQL: predict_linear