rootFind the full filesystem with df -hT, then drill down with du -xh --max-depth=1. If df says full but du adds up to less, a deleted file is still held open (lsof +L1). If you get “No space left on device” while df shows free space, you are out of inodes (df -i).
Symptoms
No space left on devicewhen writing or creating filesdf -hshows 100% on/,/var,/tmpor an application mount- Services crash, logins hang, or package updates fail
Diagnose
$ df -hT # which filesystem, what type
$ df -i # inodes: IUse% at 100% means no new files
$ sudo du -xh --max-depth=1 /var | sort -h | tail
$ sudo du -xh --max-depth=1 /var/log | sort -h | tail
$ sudo find /var -xdev -type f -size +500M -exec ls -lh {} \;
-x keeps du on a single filesystem so the numbers line up with df. Repeat one level deeper until you reach the culprit.
Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 xfs 20G 20G 64K 100% /var $ sudo du -xh --max-depth=1 /var | sort -h | tail -3 1.2G /var/lib 17G /var/log 18G /var
When df and du disagree: deleted files still open
A process keeps writing to a log that someone deleted, so the space is not released.
$ sudo lsof -nP +L1 | grep /var
Preferred fix: restart or reload the owning service so it closes the file. If you cannot restart it, you can truncate the open file through /proc:
$ sudo sh -c ': > /proc/PID/fd/FD' # PID and FD come from the lsof output
Do not truncate database files, journal files or anything you cannot identify. Confirm the file is a log first. Do not rm files under /var/lib to “make room”.
When there is space but you cannot create files: inodes
$ df -i /var $ sudo find /var -xdev -printf '%h\n' | sort | uniq -c | sort -n | tail # directories holding the most files
Millions of tiny files (session files, mail queue, cache) are the usual cause. Delete the junk by age, for example with find DIR -type f -mtime +7 -delete after reviewing the list.
Safe cleanup
# rotate and compress logs now $ sudo logrotate -f /etc/logrotate.conf # systemd journal: keep only 200 MB $ sudo journalctl --vacuum-size=200M # package caches (RHEL 8/9) $ sudo dnf clean all # old kernels beyond installonly_limit $ sudo dnf remove --oldinstallonly
Other places that hide space: reserved blocks on ext4 (tune2fs -l DEVICE | grep -i reserved), LVM snapshots and thin pools (lvs -a), container storage (podman system df) and core dumps in /var/lib/systemd/coredump.
Fix it properly: grow the filesystem
If the data is legitimate, extend it. With LVM and free space in the volume group:
$ sudo vgs $ sudo lvextend -r -L +5G /dev/vg01/lv_var # -r also resizes the filesystem
Without -r, resize the filesystem yourself (xfs_growfs /var or resize2fs DEVICE). XFS can grow but not shrink. Details: Extending LVM.
Common mistakes
- Deleting a big log with
rmwhile the process still has it open: no space is freed. - Running
du /without-xand wandering into/proc, NFS mounts and other filesystems. - Cleaning up and never fixing log rotation or monitoring, so it returns next week.
- Forgetting inodes when
df -hlooks fine.
Production considerations
Free a little space safely first (rotate or vacuum logs) so the application can recover, then investigate. Record what grew and why, add an alert at 80% and 90%, and review the logrotate policy. For root or database volumes, plan the extension in a change window.
Interview questions
df shows 100% but du shows much less. Why?
Deleted files are still held open by a process (lsof +L1), or the space is used by something du cannot see: reserved blocks, snapshots, or data under a mountpoint hidden by another mount. Restart the process to release the space.
“No space left on device” but df shows free space. What is wrong?
Inode exhaustion (df -i) is the classic cause. Also consider quota limits and a full thin pool.
How do you extend /var online?
Add a PV or use free extents in the VG, then lvextend -r -L +SIZE /dev/VG/LV. XFS and ext4 can grow online.
More: Inodes Q&A · Backup and recovery.