Fix: No space left on device

A program, a package update or a log rotation fails with an error that ends in:

No space left on device

This is the standard text for the kernel error ENOSPC; the program usually prints its own name and the file it was writing in front of it. Services may crash, databases may stop accepting writes and you may not even be able to save a file in an editor. Work through the causes below in order.

Step 1: which filesystem is full?

df -h

Look for Use% at or near 100%. The Mounted on column tells you where: often /, /var or /boot. Note that /tmp may be a separate in-memory tmpfs.

Cause 1: big files or directories

Find what uses the space on that filesystem. -x keeps du from wandering into other mounted filesystems:

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -n 15
sudo find /var -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

Typical culprits and safe ways to shrink them:

• The systemd journal: check with journalctl --disk-usage, trim with sudo journalctl --vacuum-size=500M.
• Old log files in /var/log, especially rotated .gz or .1 files and huge application logs. Rotated copies can be deleted; for a log a program is still writing, see cause 3.
• Package caches: sudo apt clean on Debian/Ubuntu, sudo dnf clean all on RHEL family systems.
• Old kernels filling /boot: sudo apt autoremove removes ones no longer needed on Debian/Ubuntu. On RHEL family systems, the installonly_limit setting in /etc/dnf/dnf.conf controls how many are kept.
• Container images, backups and core dumps left in /var/lib, /home or /root.

Warning: deleting is permanent. Look at a file before removing it, never delete files under /var/lib belonging to a database or package manager by hand, and prefer the tool that owns the data (journalctl, apt, dnf, your container runtime).

Cause 2: out of inodes

Every file uses one inode, and a filesystem has a limited number. Millions of tiny files (session files, mail queues, cache entries) can use them all while df -h still shows free space:

df -i
sudo find /var -xdev -type f | cut -d/ -f2-4 | sort | uniq -c | sort -n | tail

If IUse% is at 100%, find the directory with the most files and clean it up at the application level. ext4 cannot add inodes to an existing filesystem; XFS allocates them dynamically, so this mostly affects ext4.

Cause 3: deleted files that are still open

If you deleted a big log file but df still shows the disk full, a running process still has the file open. The space is only released when that process closes it. List deleted-but-open files:

sudo lsof +L1

Restart the service that holds them (sudo systemctl restart name) and the space comes back. Next time, empty a live log instead of deleting it: sudo truncate -s 0 /var/log/app/big.log.

Cause 4: the volume really is too small

If the data is legitimate, add space. On LVM, check sudo vgs for free space and grow the volume online with lvextend --resizefs (see the LVM entries). Otherwise move data to another disk and mount it there.

Prevent it next time

• Monitor disk and inode usage and alert well before 100%.
• Make sure every application log is covered by logrotate or goes to the journal.
• Put fast-growing data (/var/log, database files, uploads) on its own filesystem so a full disk cannot take down the root filesystem.

Official documentation

• errno(3): ENOSPC
• df(1)
• lsof(8)