rootOn Linux, load average counts tasks that are runnable plus tasks in uninterruptible sleep (usually waiting on disk or NFS). Compare it to the CPU count (nproc), then run vmstat 1 5: high us/sy means CPU-bound, high wa or b means I/O-bound, and si/so means memory pressure.
Step 1: read the load correctly
$ uptime $ nproc
02:14:09 up 41 days, 3:12, 2 users, load average: 18.40, 16.10, 9.73 8
Load of 18 on 8 CPUs means tasks are queueing. The trend matters: 1, 5 and 15-minute averages tell you whether it is getting worse (18 → 16 → 9.7 here) or recovering.
Step 2: separate CPU, I/O and memory
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 14 0 214532 81220 912344 0 0 9120 4100 2210 3900 4 3 12 81 0
| You see | Meaning | Next |
|---|---|---|
r high, us+sy high, id low | CPU-bound | top, pidstat -u 1 |
b high, wa high | Waiting on disk/NFS (the example above) | iostat -xz 1, pidstat -d 1 |
si/so non-zero, free low | Memory pressure, swapping | free -h, ps --sort=-%mem |
st high (VM) | Hypervisor is stealing CPU | Ask the virtualisation team |
Step 3: find the culprit
# CPU: press 1 for per-core, P to sort by CPU, M to sort by memory $ top $ pidstat -u 1 5 # disk: await = latency in ms, %util near 100 = saturated $ iostat -xz 1 5 $ pidstat -d 1 5 # processes stuck in D state (uninterruptible) $ ps -eo state,pid,wchan:20,cmd | awk '$1=="D"' # history, if sysstat is collecting $ sar -q -s 01:00:00 -e 03:00:00
Common causes
- A runaway process or a tight loop (CPU).
- Backup, antivirus, a database checkpoint or a batch job (I/O).
- A slow or failing disk, a SAN path problem, or a hung NFS server (D-state processes).
- Too little RAM, causing swap storms (see high memory).
- Many short-lived processes (fork storms) from cron or scripts.
Common mistakes
- Reading the load number without dividing by the CPU count.
- Assuming high load always means high CPU. A box can be 90% idle with a load of 30 because everything is blocked on I/O.
- Trying to
kill -9D-state processes. They cannot be killed until the I/O returns; fix the storage. - Restarting the application before capturing evidence.
Production considerations
Capture first, then act: top -b -n1 > /tmp/top.txt, vmstat 1 10, iostat -xz 1 5, ps -efL and the time. Check what changed (deploy, patch, new cron job, backup window). If you must restart something, do it on one node of a pool first.
Advanced: why uninterruptible tasks raise load
Linux includes tasks in TASK_UNINTERRUPTIBLE (state D) in the load average, unlike many Unix systems that count only runnable tasks. That is why NFS and storage stalls show up as load even when the CPUs are idle.
Interview questions
What does a load average of 10 mean on a 4-core server?
On average about 10 tasks were runnable or blocked in uninterruptible sleep, 2.5 per core. It is overloaded or blocked on something. Check vmstat and iostat to see whether it is CPU or I/O.
CPU is 90% idle but load is 25. How?
Many processes are in D state, waiting for storage or NFS. Look at iostat -xz, wa in vmstat and the D-state process list.
More: Performance monitoring and SAR · What iostat -xm tells you · High load on Solaris.