The kernel (or the mount options) put the filesystem in read-only mode. Confirm it with findmnt, read dmesg -T for I/O or filesystem errors, fix the underlying storage problem first, then repair the filesystem while it is unmounted and remount it read-write. Remounting without fixing the cause usually just flips back.
Symptoms
touch: cannot touch 'file': Read-only file systemcp: cannot create regular file '...': Read-only file systembash: cannot create temp file for here-document: Read-only file system- Services fail to start because they cannot write a PID file, socket or log.
Why it happens
- ext4 hit an error and the filesystem is set to
errors=remount-ro(check withtune2fs -l DEVICE | grep -i behavior). The kernel remounts it read-only to avoid more damage. - XFS shut down after a metadata or log error. XFS does not remount read-only; further access usually returns
Input/output errororStructure needs cleaning. - The storage failed underneath: a SAN LUN lost all paths, a disk or controller is failing, or a VM datastore went read-only.
- It was mounted
roon purpose or by mistake: anrooption in/etc/fstab, a read-only export (NFS), a snapshot, an ISO or squashfs image, or a read-only block device.
Diagnose
Run these read-only checks first. Use root or sudo.
$ findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS /data $ grep -E ' ro[ ,]' /proc/mounts $ lsblk -o NAME,SIZE,RO,TYPE,MOUNTPOINT $ dmesg -T | grep -i -E 'remount|I/O error|EXT4-fs|XFS|Buffer I/O|blk_update_request' $ journalctl -k -p err -b --no-pager | tail -50
Expected output when ext4 remounted itself read-only:
/data /dev/mapper/vg01-lv_data ext4 ro,relatime,seclabel [Tue Oct 1 02:14:09 2026] EXT4-fs error (device dm-3): ext4_find_entry:1455: inode #2: comm app: reading directory lblock 0 [Tue Oct 1 02:14:09 2026] EXT4-fs (dm-3): Remounting filesystem read-only
If the block device itself shows RO 1 in lsblk, the problem is below the filesystem. On a SAN host, also check the paths:
$ multipath -ll $ multipathd show paths $ smartctl -H /dev/sdX # local disks (smartmontools)
Fix
Case 1: no errors in dmesg, the mount is simply ro
$ sudo mount -o remount,rw /data
# make it permanent: remove "ro" from the line in /etc/fstab
$ findmnt --verify
Case 2: errors in dmesg (storage or filesystem damage)
- Stop writers. Stop the application and anything else using the mount (
fuser -vm /data). - Fix the storage first. Restore SAN paths, replace the failing disk, or ask the storage team. Take an LVM or array snapshot or a backup if you can.
- Unmount and repair.
# ext4 $ sudo umount /data $ sudo fsck -f /dev/mapper/vg01-lv_data # XFS: dry run first, then repair $ sudo umount /data $ sudo xfs_repair -n /dev/mapper/vg01-lv_data $ sudo xfs_repair /dev/mapper/vg01-lv_data
xfs_repair -L zeroes the XFS log and can lose recent changes. Use it only if a normal repair refuses to run and the data is backed up. Never run fsck or xfs_repair on a mounted filesystem.
For the root filesystem you cannot unmount it while running. Boot into rescue mode (or add fsck.mode=force to the kernel command line for ext4), repair, then reboot.
$ sudo mount /data # or: mount -o remount,rw / $ touch /data/.rw-test && rm /data/.rw-test $ sudo systemctl start APPLICATION
Common mistakes
- Running
mount -o remount,rwrepeatedly while the disk is still failing. - Using
fsck -yblindly on production data without a backup or snapshot. - Checking only the filesystem and forgetting multipath or HBA errors higher up the log.
- Repairing a mounted filesystem.
Production considerations
Treat this as an incident: capture dmesg, journalctl -k and multipath -ll before you change anything, tell the application owner, and plan the repair in a window. In a cluster, fail the service over first. Afterwards find out why it happened (failing disk, path flap, array event) and close the problem record.
Advanced: choose how ext4 reacts to errors
$ tune2fs -l /dev/mapper/vg01-lv_data | grep -i 'errors behavior'
$ sudo tune2fs -e remount-ro /dev/mapper/vg01-lv_data # continue | remount-ro | panic
# or per mount: mount -o errors=remount-ro ...
On databases and clusters some teams prefer panic so the node fails fast and another takes over.
Interview questions
A production filesystem suddenly became read-only. What do you do?
Confirm with findmnt; read dmesg/journalctl -k for the cause; check the storage layer (multipath, disk health); stop writers; back up or snapshot; unmount and run fsck or xfs_repair; remount, test with a write, restart services, then do root-cause analysis.
Why does an XFS filesystem not go read-only like ext4 does?
XFS responds to serious metadata or log errors by shutting the filesystem down. Operations then fail with I/O or “structure needs cleaning” errors until you unmount and run xfs_repair.
How can you tell whether the mount or the block device is read-only?
The mount options show ro in findmnt or /proc/mounts; the device flag is the RO column of lsblk (or blockdev --getro).
More: fsck in the RHEL guide · SAN and multipath · Space free but cannot create files.