rootTroubleshooting by Symptom
Start from the symptom: disk full, read-only filesystem, high load, high memory, I/O wait, service will not start, network down, multipath, LVM and boot failures.
Read-only checks first. Never run a destructive command until you have confirmed the device, filesystem or host. When in doubt, take a snapshot or backup and open a change.
Disk full
“No space left on device”, a filesystem at 100%, or applications failing to write.
df -hT
du -xh --max-depth=1 /var | sort -h | tail
df -i # inodes
lsof +L1 # deleted files still held openWhy it happens: Large or runaway logs, deleted-but-open files, inode exhaustion, snapshots or thin pools filling up.
Filesystem read-only
“Read-only file system” when writing, or a mount that flipped to ro.
findmnt -no TARGET,OPTIONS /path
dmesg -T | grep -i -E 'remount|I/O error|EXT4-fs|XFS'
lsblk -o NAME,RO,MOUNTPOINT
multipath -ll # SAN: all paths down?Why it happens: The kernel remounted ext4 read-only after I/O or metadata errors, the XFS log shut down, a SAN path/LUN failed, or the mount was ro by configuration.
High load
Load average far above the CPU count; the server feels slow.
uptime; nproc
vmstat 1 5
top -b -n1 | head -20
iostat -xz 1 3Why it happens: Load counts runnable and uninterruptible (D-state) tasks, so CPU saturation, I/O waits and memory pressure can all raise it.
High memory / OOM
Free memory near zero, heavy swapping, or processes killed by the OOM killer.
free -h
ps -eo pid,user,%mem,rss,cmd --sort=-%mem | head
vmstat 1 5 # si/so = swap in/out
dmesg -T | grep -i -E 'out of memory|oom-killer'Why it happens: A leaking or oversized process, too little RAM or swap for the workload, or cgroup limits. Check the “available” column, not “free”.
High I/O wait
CPU is mostly idle but the system is slow; %wa is high in top or vmstat.
vmstat 1 5 # wa and b columns
iostat -xz 1 5 # await, %util, aqu-sz
pidstat -d 1 5 # per-process disk I/O
ps -eo state,pid,wchan:20,cmd | awk '$1=="D"'Why it happens: Slow or saturated storage (disk, SAN path, NFS server), a backup or scan hammering the disk, or failing hardware.
Service won’t start
systemctl start fails, or the service is “failed” or flapping after a reboot.
systemctl status SERVICE -l --no-pager
journalctl -xeu SERVICE --no-pager | tail -40
systemctl cat SERVICE
ss -ltnp # port already in use?Why it happens: Bad configuration, a missing ExecStart binary or user, a port conflict, permissions or SELinux, or an unmet dependency.
Network not working
No connectivity, wrong IP, or names not resolving.
ip -br addr; ip route
nmcli device status
ping -c3 GATEWAY; ping -c3 8.8.8.8
getent hosts example.comWhy it happens: Interface down or misconfigured, wrong gateway or VLAN, DNS failure, firewall rules or a duplicate IP.
SSH connection refused
“Connection refused”, a timeout, or a login that is rejected.
systemctl status sshd
ss -ltnp | grep ':22'
firewall-cmd --list-all
sshd -t; journalctl -u sshd -e --no-pager | tailWhy it happens: sshd not running or listening elsewhere, a firewall or security group, a bad sshd_config, SELinux port labels, or key/permission problems. Refused means a reply came back; a timeout usually means a firewall drop.
Multipath / SAN path failure
Paths shown as failed or faulty, a missing LUN, or I/O errors from SAN storage.
multipath -ll
multipathd show paths
lsscsi; ls /sys/class/fc_host/
dmesg -T | grep -i -E 'multipath|scsi|fail|I/O error' | tailWhy it happens: A failed HBA, cable, switch port or controller; zoning or masking changes; or stale paths after a LUN was removed.
LVM won’t extend / missing
“Insufficient free space”, a VG or PV that is missing, or a filesystem that did not grow.
pvs; vgs; lvs
vgs -o +vg_free_count,vg_free
lsblk
lvextend -r -L +5G /dev/VG/LV # after checking free spaceWhy it happens: No free extents in the VG, a new disk not yet added with pvcreate and vgextend, or lvextend run without -r so the filesystem was not resized.
Boot failure / rescue mode
Drops to emergency shell, kernel panic, or stuck at GRUB or dracut.
systemctl --failed
journalctl -xb -p err --no-pager
cat /etc/fstab; findmnt --verify
ls /boot; grubby --default-kernelWhy it happens: A bad /etc/fstab entry, a missing initramfs or kernel, a failed filesystem check, or a storage driver or LVM problem after patching.
SELinux denied
Permission denied although file modes and ownership look right.
getenforce
ausearch -m avc -ts recent
ls -lZ /path
restorecon -Rv /pathWhy it happens: The file has the wrong SELinux context, a non-standard port or a boolean is not enabled. Fix the label or boolean; do not leave SELinux disabled.
Zombie / D-state processes
Defunct (Z) processes pile up, or processes are stuck in uninterruptible sleep (D).
ps -eo pid,ppid,stat,wchan:20,cmd | awk '$3 ~ /^[ZD]/'
ps -o pid,ppid,cmd -p PPID
cat /proc/PID/stack # root, kernel stack of a D process
dmesg -T | tail -30Why it happens: Zombies are finished children whose parent has not called wait(); fix or restart the parent. D-state is usually blocked on storage or NFS and cannot be killed until the I/O returns.