🟧 HP-UX, Common Ops & Legacy

HP-UX 11i (PA-RISC/Itanium) · Cross-platform common operations · Legacy Unix reference

HP-UX 11i Common Ops Legacy Reference 29+ Sections

🟧

HP-UX

Dedicated HP-UX track — PA-RISC and Itanium hardware, LVM with VxFS, EFI boot, swinstall + Quality Pack, Ignite-UX, glance, vPars vs HPVM, Serviceguard, plus the cross-OS comparison table.

🟧

1. Platform & Architecture

2 Q&As

💾

2. Disk, LVM & VxFS

2 Q&As

🚀

3. Boot, Patching & Kernel

3 Q&As

📦

4. Ignite-UX & SAM/SMH

2 Q&As

⚡

5. Performance & Monitoring

1 Q&As

🖥

6. vPars, Integrity VMs & Serviceguard

2 Q&As

🔥

7. Advanced Scenarios & Cross-OS

3 Q&As

Sections below are organized topic-wise within this OS. Click any topic in the sidebar or a card above. Each Q&A shows a badge for source document (e.g. RHEL Admin, L3 SME, Solaris SME) and a depth level (Foundational → Advanced).

🟧

1. Platform & Architecture

HP-UX 11i v1/v2/v3 (B.11.31), PA-RISC vs Itanium, hardware/version/patch identification commands.

01 Concept Foundational What is HP-UX and on what hardware does it run? L3 SME ▾

Answer

HP-UX is Hewlett Packard's commercial UNIX, derived from System V. It runs only on HP/HPE proprietary hardware: PA-RISC (HP 9000) and Itanium (HPE Integrity rx/Superdome) servers. HP-UX 11i v3 is 64-bit. End of standard support is 2025–2026, so most estates are being migrated, but L3 admins are still expected to support them through end of life.

Release	Internal Name	Notes
HP-UX 11i v1	B.11.11	PA-RISC era; legacy
HP-UX 11i v2	B.11.23	First Itanium release
HP-UX 11i v3	B.11.31	Current and final; PA-RISC + Itanium; agile addressing

02 Commands Foundational How do you identify HP-UX version, hardware model and patch level? L3 SME ▾

Answer

uname -a # Kernel + arch (ia64 or 9000/800)

uname -r # e.g. B.11.31

swlist -l bundle | head # Installed bundle inventory

swlist -l bundle PHKL* PHCO* PHNE* # List installed patches

swlist -l bundle QPKBASE # Quality Pack base

model # Hardware model

machinfo # CPU, memory, firmware, serial

getconf MACHINE_MODEL # Machine model string

/opt/ignite/bin/print_manifest # Full system manifest

swlist is the HP-UX equivalent of rpm -qa. Patch naming: PHCO_xxxx (commands), PHKL_xxxx (kernel), PHNE_xxxx (network), PHSS_xxxx (subsystem). Knowing this breakdown impresses interviewers — most candidates just say “patches.”

💾

2. Disk, LVM & VxFS

End-to-end LV creation, the mknod group gotcha, VxFS Online JFS resize and Online JFS licensing trap.

01 Concept Foundational Compare HP-UX LVM with Linux LVM. How do you create an LV and filesystem end-to-end? L3 SME ▾

Answer

Concept	Linux LVM	HP-UX LVM
Init disk	pvcreate /dev/sdb	pvcreate /dev/rdsk/c0t1d0
Allocation unit	Physical Extent, default 4 MiB	Physical Extent, default 4 MB
Create VG	vgcreate vg_data /dev/sdb	vgcreate /dev/vg01 /dev/dsk/c0t1d0
VG metadata	/etc/lvm/	/etc/lvmtab
Create LV	lvcreate -L 20G -n lv_app vg_data	lvcreate -L 20480 -n lv_app /dev/vg01
Filesystem	mkfs.xfs / mkfs.ext4	newfs -F vxfs /dev/vg01/rlv_app
Online grow	lvextend -L +10G -r	lvextend then fsadm / extendfs
Default FS	XFS / ext4	VxFS (HFS legacy)

# Full end-to-end on HP-UX 11i v3:

ioscan -funC disk # Discover disks

insf -e # Create device special files

pvcreate -f /dev/rdsk/c2t1d0 # Initialise as PV

mkdir /dev/vg01 # VG directory

mknod /dev/vg01/group c 64 0x010000 # Group device node

vgcreate /dev/vg01 /dev/dsk/c2t1d0

lvcreate -L 20480 -n lv_app /dev/vg01 # Size in MB on HP-UX

newfs -F vxfs /dev/vg01/rlv_app

mkdir /app && mount /dev/vg01/lv_app /app

echo '/dev/vg01/lv_app /app vxfs delaylog 0 2' >> /etc/fstab

HP-UX VG creation needs a unique minor number in the group device file (0x010000 for vg01; vg02 would be 0x020000). Forgetting mknod group is the most common HP-UX LVM mistake on the job.

02 Commands Intermediate How do you extend an HP-UX LV and grow the VxFS filesystem online? L3 SME ▾

Answer

vgdisplay vg01 # Check Free PE

lvextend -L 30720 /dev/vg01/lv_app # Extend to 30 GB

# Online resize for VxFS (requires Online JFS):

fsadm -F vxfs -b 30720000 /app # Size in 1KB blocks

# Old method (offline) — HFS or no Online JFS licence:

umount /app && extendfs /dev/vg01/rlv_app && mount /app

# Add a new disk if VG is full:

pvcreate -f /dev/rdsk/c3t0d0

vgextend /dev/vg01 /dev/dsk/c3t0d0

VxFS online resize requires the Online JFS (HP OnlineJFS) licence — a frequent interview gotcha. If the customer does not have it, you must unmount and use extendfs.

🚀

3. Boot, Patching & Kernel

Integrity EFI boot, swinstall + Quality Pack + swverify, kctune dynamic tunables and Oracle-on-HP-UX tunables.

01 Concept Intermediate Describe the HP-UX boot process on Integrity (Itanium) hardware. L3 SME ▾

Answer

Power on → POST runs from PDC/EFI firmware.
EFI Boot Manager presents options (HPUX, Service Menu, Shell).
EFI loads HPUX.EFI (the HP-UX bootloader) from /EFI/HPUX on the EFI System Partition.
HPUX.EFI loads /stand/vmunix (the kernel) and the AUTO file determines boot mode.
Kernel initialises, mounts / from /etc/fstab, runs /sbin/init.
init reads /etc/inittab and runs /sbin/rc which executes /sbin/rcN.d/Sxx scripts for the target run level.

# Interrupt boot to access EFI:

# At boot prompt press any key, then select 'EFI Shell'

fs0: # Switch to the EFI partition

hpux # Run the HP-UX loader manually

boot -is /stand/vmunix # Boot single-user (interactive)

boot -lm /stand/vmunix # LVM maintenance mode

# Once running:

/usr/sbin/setboot # Show / change primary/alt paths

/usr/sbin/setboot -p <hwpath> # Set new primary boot path

/usr/sbin/lvlnboot -v # Validate boot LV layout

02 Commands Intermediate How do you install patches on HP-UX using swinstall, and what is a Quality Pack? L3 SME ▾

Answer

Patches arrive as depots (.depot or directory format). The Software Distributor (SD-UX) tools (swinstall, swremove, swlist, swverify) manage them. A Quality Pack (QPK) is a tested bundle of patches HP releases periodically — usually you install QPK plus selected critical patches rather than individual fixes.

# Inspect a depot before installing:

swlist -d -s /var/tmp/qpkbase.depot

# Install from a depot (interactive TUI):

swinstall -s /var/tmp/qpkbase.depot

# Install all patches non-interactively:

swinstall -x autoreboot=true -x patch_match_target=true \

-s /var/tmp/qpkbase.depot \*

# After install, verify integrity:

swverify \* # Verify all software

swlist -l patch | wc -l # Count installed patches

swlist -l bundle | grep -i quality # Confirm QPK level

# Remove a bad patch:

swremove PHKL_45678

# Cleanup committed patches to free space:

cleanup -c 1 # Keep 1 superseded version

Mention swverify in interviews — it audits all installed software against its manifests, similar to rpm -V but built into SD-UX. Senior admins run it after every patch session.

03 Commands Intermediate How do you tune the HP-UX kernel? Compare with Linux sysctl. L3 SME ▾

Answer

HP-UX 11i v3 uses dynamic kernel tunables via kctune. Earlier versions required rebuilding the kernel with kmtune + kmadmin and a reboot. Most modern HP-UX tunables can change without a reboot — read the Changeable column in kctune -v.

kctune # List all tunables (like sysctl -a)

kctune -v maxfiles # Detailed view of one tunable

kctune maxfiles=4096 # Set value (immediate if Dynamic)

kctune -B maxfiles=4096 # Boot-time change (next reboot)

kctune -h maxfiles=4096 # Held; applies on next reboot

# Common tunables:

# maxuprc Max processes per user

# nproc Max system processes

# maxfiles Per-process open file limit

# shmmax/shmmni SysV shared memory (Oracle on HP-UX)

# dbc_min_pct / dbc_max_pct Buffer cache size

For Oracle on HP-UX, tunables like shmmax, shmmni, semmns, maxdsiz_64bit and maxssiz_64bit are critical. Naming maxdsiz_64bit shows you have actually run Oracle on HP-UX.

📦

4. Ignite-UX & SAM/SMH

Network OS install, make_net_recovery for DR, golden-image cloning, and web admin via SMH.

01 Concept Intermediate What is Ignite-UX and how do you use it for OS deployments and disaster recovery? L3 SME ▾

Answer

Ignite-UX is HP's equivalent of Kickstart (RHEL) or JumpStart / AI (Solaris) — a network installation framework that also produces system-recovery archives. It is core to every HP-UX shop for disaster recovery.

Network OS install: PXE-like boot from an Ignite server, then automated install using a configuration profile.
System recovery: make_net_recovery creates a bootable network archive of a live system; make_tape_recovery writes one to DDS/LTO tape.
Cloning: capture one golden system with make_net_recovery, then deploy it to dozens of identical servers.
Restore: boot the target from Ignite or the recovery tape and the system is rebuilt automatically.

# Make a bootable network recovery archive (client → Ignite server):

make_net_recovery -s ignite-server -a 192.168.0.50:/var/opt/ignite/recovery

# Tape-based recovery to /dev/rmt/0mn:

make_tape_recovery -a /dev/rmt/0mn -v

# On the Ignite server — clients and archives:

/opt/ignite/bin/manage_index -l

# View manifest of a recovered system:

/opt/ignite/bin/print_manifest

Knowing make_net_recovery by name (and being able to say it ran nightly for 100+ HP-UX servers in your environment) is one of the strongest HP-UX seniority signals you can give.

02 Concept Foundational Explain SAM / SMH and when you use them versus the CLI. L3 SME ▾

Answer

SAM (System Administration Manager) is the legacy text-based menu admin tool. SMH (System Management Homepage) is the web-based replacement on HP-UX 11i v3 at https://server:2381. Both wrap CLI commands; senior admins use the CLI but understand SAM/SMH for occasions when the CLI command name escapes them or a junior screen-shares.

sam # Launch SAM (TUI)

smh # Launch SMH (opens web)

# Useful SAM modules: Disks & FS, Users, Networking, Kernel, Performance

⚡

5. Performance & Monitoring

glance key-bindings (c/d/m/t), and HP-UX↔Linux tool mapping for live performance triage.

01 Commands Intermediate Which performance tools do you use on HP-UX and how do they map to Linux equivalents? L3 SME ▾

Answer

Concern	HP-UX Tool	Linux Equivalent
Real-time top	top, glance, gpm	top / htop
Per-process detail	glance (text), gpm (GUI)	atop / pidstat
CPU statistics	sar -u, vmstat	sar -u, vmstat, mpstat
Memory	vmstat, swapinfo	free, vmstat
Disk I/O	sar -d, iostat	iostat -xz
Per-PID I/O	glance (D drill-down)	iotop -o
Network	netstat -i, lanadmin -g	ss / ip / ethtool
Historical	OVPA / MeasureWare	sar history / sysstat

glance # Live system view

vmstat 1 5 # CPU/memory/I/O snapshot

sar -u 1 10 # Average CPU usage

sar -d 1 5 # Disk activity per device

swapinfo -tm # Memory + swap, in MB

ioscan -funC disk # Disk hardware inventory

ps -ef # Full process listing

glance is the HP-UX killer feature — press 'c' for CPU drill-down, 'd' for disk, 'm' for memory, 't' for threads. Naming these key-bindings tells the interviewer you have actually lived inside glance for hours.

🖥

6. vPars, Integrity VMs & Serviceguard

vPars vs Integrity VMs vs nPars (common confusion), plus Serviceguard cluster diagnostics with cmviewcl.

01 Concept Advanced Compare HP-UX Virtual Partitions (vPars), Integrity VMs (HPVM) and nPars. Which would you choose when? L3 SME ▾

Answer

Technology	Granularity	Isolation	Best For
nPar (hard partition)	By cell board	Electrical (no shared HW)	Maximum isolation, large Superdome
vPar (virtual partition)	By CPU/RAM share	Software-level, hard-partitioned CPUs	Multiple independent OS instances on one box
Integrity VM (HPVM)	Full hypervisor	Hypervisor-managed	Server consolidation, dynamic workloads

# vPar — list and manage:

vparstatus # List vPars + resources

vparcreate -p vpar1 -a cpu::2 -a mem::4096

vparmodify -p vpar1 -a cpu::4 # Live CPU add

vparboot -p vpar1

# Integrity VM — like KVM/VMware:

hpvmstatus # List guest VMs

hpvmcreate -P guest1 -O hpux -c 2 -r 4G -d disk:scsi::file:/var/opt/hpvm/guest1/disk1

hpvmstart -P guest1

hpvmconsole -P guest1 # Attach to guest console

vPars and HPVM are NOT the same — interviewers test this. vPars are hard partitions of physical CPUs sharing one chassis; HPVM is a hypervisor running fully virtualised guests. Mixing them up is an instant red flag.

02 Commands Advanced What is HP Serviceguard and how do you check the health of a Serviceguard cluster? L3 SME ▾

Answer

HP Serviceguard (SG/HP-UX) is the HA clustering product for HP-UX. Concepts mirror Pacemaker/RHEL HA: package groups (resources), heartbeat networks, cluster lock disk (quorum), failover policies. Packages contain a control script and a configuration ASCII file.

cmviewcl # Cluster + package state (the main one)

cmviewcl -v -p mypkg # Verbose view of one package

cmrunnode <node> # Start cluster services

cmhaltnode <node> # Halt cluster on a node (graceful)

cmrunpkg -n <node> <pkg> # Start a package on a node

cmhaltpkg <pkg> # Stop a package

cmmodpkg -e <pkg> # Enable auto-run

cmcheckconf -C /etc/cmcluster/cluster.ascii # Validate config

cmapplyconf -C /etc/cmcluster/cluster.ascii # Apply config

# Common logs:

tail -f /var/adm/syslog/syslog.log

tail -f /etc/cmcluster/<pkg>/<pkg>.log

When Serviceguard fails to start a package, ALWAYS check the package log /etc/cmcluster/<pkg>/<pkg>.log first — cmviewcl only says “down”, never why.

🔥

7. Advanced Scenarios & Cross-OS

Live perf hang triage with kcusage, EFI boot recovery via vmunix.prev, and the 15-task HP-UX↔Solaris↔RHEL comparison table.

01 Scenario Advanced SCENARIO: An HP-UX 11i v3 production server is intermittently hanging for 30–60 seconds every few hours. Application teams report transaction timeouts. No errors in syslog. Walk through your L3 investigation. L3 SME ▾

Answer

Take ownership on the bridge. Confirm symptoms with timestamps from app logs vs. system metrics.
Live capture during the next hang: keep glance, vmstat 1, sar -d 1 and iostat 1 running in screen sessions.
CPU vs. I/O: in glance press 'c' — any CPU pinned in sys%? Press 'd' — any device with await > 50ms?
Memory: swapinfo -tm; check paging via vmstat pi/po.
Kernel pressure: kctune for table limits hitting their max (nfile, nproc, nflocks); use kcusage for live consumption.
Filesystem: VxFS intent-log saturation — vxstat -f and vxfsstat -v /app.
Network: lanadmin -g mibstats <ppa> for errors; netstat -s for retransmits.
HBA / SAN: ioscan -funC disk, EMS hardware events in /var/opt/resmon/log.
If still unresolved: cstm diagnostics and HPE Support with the OS manifest + supportinfo bundle.
Document the RCA, update the runbook, share with L1/L2.

kcusage is the HP-UX-specific tool nobody remembers — it shows which kernel tunables are approaching their limit. Mentioning it during a perf scenario is a strong L3 signal.

02 Critical Advanced SCENARIO: An HP-UX server will not boot — it stops at the EFI prompt after a recent kernel patch. How do you recover? L3 SME ▾

Answer

Connect via the iLO/MP console — the EFI shell will already be visible.
From EFI: fs0: to access the EFI partition; ls to confirm HPUX.EFI is present.
Boot the previous kernel manually: hpux /stand/vmunix.prev (HP-UX keeps the previous kernel as vmunix.prev after a patch).
Once booted, set the system to use the previous kernel: kconfig -a previous to activate the previous configuration.
If kernel files are missing, boot from an Ignite-UX recovery image: boot lan.<ignite-ip> install from EFI.
Restore the system from the most recent make_net_recovery archive.
Investigate which patch caused the issue and engage HPE; raise a problem ticket.

HP-UX maintains the previous kernel automatically — this is why kernel patching on HP-UX feels safer than Linux. Never skip kconfig verification after a kernel patch — the rollback path depends on it.

03 Commands Reference Quick reference — side-by-side comparison of common tasks across HP-UX, Solaris and RHEL. L3 SME ▾

Answer

Task	HP-UX	Solaris	RHEL
OS version	uname -r	uname -r / cat /etc/release	cat /etc/os-release
Patches	swlist -l patch	showrev -p / pkg list	rpm -qa / dnf list installed
Install pkg	swinstall -s	pkgadd / pkg install	rpm -ivh / dnf install
Boot mgr	EFI + HPUX.EFI	GRUB / OBP	GRUB2
Single user	boot -is	boot -s	systemctl rescue
LVM PV init	pvcreate /dev/rdsk/...	format → labelled	pvcreate /dev/sd*
VG create	vgcreate /dev/vg01 ...	zpool create	vgcreate vg ...
FS create	newfs -F vxfs	zfs create	mkfs.xfs / mkfs.ext4
Online grow	fsadm (Online JFS)	ZFS native	xfs_growfs / resize2fs
Disk listing	ioscan -funC disk	format / zpool status	lsblk
Network cfg	lanadmin / nwmgr	ipadm / dladm	nmcli / NetworkManager
Live perf	glance	prstat / dtrace	top / atop
Cluster	Serviceguard (cmviewcl)	Oracle Cluster	PCS / Pacemaker
Auto install	Ignite-UX	JumpStart / AI	Kickstart
Recovery	make_net_recovery	flarcreate / UAR	Relax-and-Recover (ReaR)

If you can rattle off this comparison table fluently, you sound exactly like an L3 SME with hands-on multi-OS experience. Print it, study it, and reproduce it from memory.

🌐

Common / Cross-OS

Topics that span all OSes — clustering (PCS/GFS2/Veritas), remote hardware management, scripting, automation (Ansible/Salt), backup/DR, ITSM, L3 support, RCA, capacity planning and mentoring.

🔩

1. Remote Hardware Mgmt

5 Q&As

🔗

2. Clustering

7 Q&As

☁️

3. vSphere / ESXi & UCS

4 Q&As

📈

4. Performance Methodology & Capacity Planning

3 Q&As

🔐

5. Patching, Security & Compliance (Heterogeneous)

3 Q&As

⚙️

6. Automation

9 Q&As

💿

7. Backup & DR

4 Q&As

🎯

8. L3 Support, RCA & Vendor Escalation

4 Q&As

📋

9. ITIL, Incident Mgmt & Behavioural

7 Q&As

🧑‍🏫

10. Mentoring, Docs & Behavioural Stories

5 Q&As

🔩

1. Remote Hardware Mgmt — HP iLO & Cisco UCS/UCSM

Out-of-band server management with HP iLO 3/4 and Cisco UCS Service Profiles for stateless compute.

01 Concept Foundational What is HP iLO and what can you do with it? RHEL Admin ▾

Answer

iLO (Integrated Lights-Out) is HP's out-of-band management processor — a dedicated chip with its own network port, independent of the host OS and power state.

Integrated Remote Console (IRC) — KVM-over-IP video/keyboard/mouse access.
Virtual Media — mount an ISO remotely for OS install or rescue.
Power control — graceful shutdown, hard reset, power cycle.
Integrated Management Log (IML) — hardware fault diagnostics (drives, DIMMs, NIC, thermal).
hponcfg / RIBCL — CLI and scriptable XML configuration of iLO from the host OS.
iLO4 adds REST API (Redfish) and an HTML5 console over iLO3.

# From the host OS — dump current iLO config

hponcfg -g

# Push a corrected iLO config (RIBCL XML), zero OS downtime

hponcfg -f corrected_ilo.xml

hpasmcli -s 'show temp; show fans'

02 Critical Advanced A remote server is unreachable via SSH and you cannot get to the data centre. Walk me through your iLO troubleshooting workflow. RHEL Admin ▾

Answer

Access the iLO interface on its dedicated management network (OOB) — independent of production.
Open the Integrated Remote Console for a direct video feed — OS crashed, hung on boot, or kernel panic?
Review the IML for hardware alerts: failed drives, memory errors, NIC faults, thermal events.
Check the System Event Log (SEL) for BIOS/hardware-level errors.
If the OS is hung: attempt graceful shutdown via iLO; only hard-reset after documenting the decision.
If the OS cannot boot: mount a rescue ISO via Virtual Media.
Document every action in the incident ticket with timestamps.

Saying “management network / OOB” signals you understand iLO network segmentation. Mentioning hponcfg for config-from-OS shows real depth.

03 Concept Foundational Explain the Cisco UCS architecture and the role of UCS Manager, Fabric Interconnects, and Service Profiles. RHEL Admin ▾

Answer

UCS Manager (UCSM) — centralised management for blade/rack servers; runs on the Fabric Interconnect pair.
Fabric Interconnects (FI) — redundant A/B pair providing unified Ethernet + FC connectivity and failover.
Service Profiles — hardware identity (MAC, WWN, UUID, boot policy, firmware) defined in software, not bound to physical hardware.
vNICs / vHBAs — virtual interfaces presented to the server from FI-level policies.
UCS KVM — Java/HTML5 console (equivalent to iLO for Cisco hardware).

04 Concept Advanced Explain how a UCS Service Profile enables server replacement without reconfiguration. RHEL Admin ▾

Answer

A service profile binds all hardware identity (MAC, WWN, UUID, boot policy, network/storage policies) to a logical object rather than the physical blade. When a blade fails I disassociate the profile from the dead hardware and reassociate it to a spare. The new blade inherits identical MACs and WWNs, boots from SAN, and is operational in minutes. From the network and storage perspective the server is identical — this is what makes UCS compute truly stateless and ideal for HA.

This demonstrates senior-level UCS understanding. Interviewers can tell whether you have actually performed a service-profile move or are reciting theory — speak from experience.

05 Scenario Advanced SCENARIO: A UCS blade shows intermittent network drops; the application times out every few hours. Diagnose. RHEL Admin ▾

Answer

UCSM Faults & Events for the server — vNIC errors, fabric failover events, link state changes.
Fabric path — is traffic consistently on FI-A or FI-B? Unexpected failover between FIs shows here.
VIC adapter statistics — increasing error counters, dropped packets, CRC errors at the adapter.
vNIC pinning policy — misconfigured uplink pinning causes thrashing between uplinks.
Physical layer — DC team inspects blade seating, mid-plane connectors, chassis backplane.
If no software root cause: open a Cisco TAC case with the UCSM tech-support bundle. Convert to a Problem record if recurring.

🔗

2. Clustering — PCS/GFS2 & Veritas

Corosync/Pacemaker/STONITH, GFS2 distributed filesystem, split-brain recovery and Veritas Cluster Server.

01 Concept Foundational Explain the RHEL HA cluster stack: Corosync, Pacemaker, PCS, GFS2 and fencing. RHEL Admin ▾

Answer

Corosync — cluster communication/membership layer; manages quorum (corosync.conf).
Pacemaker — resource manager; controls resource agents (VIPs, services, filesystems).
pcs — command suite (replaces ccs/crm in RHEL 7+) to configure and manage the cluster.
GFS2 — cluster-aware filesystem allowing concurrent read/write from multiple nodes; requires DLM.
STONITH / Fencing — 'Shoot The Other Node In The Head'; mandatory for split-brain prevention.

pcs status # Cluster + resource status

pcs resource show # Configured resources

pcs cluster start --all

pcs node standby <node> # Move resources off a node

pcs stonith show # Fencing config

corosync-quorumtool -l # Quorum + membership

dlm_tool status # DLM lockspace health (GFS2)

02 Critical Advanced A two-node Pacemaker cluster shows one node OFFLINE in pcs status. How do you safely investigate and recover? RHEL Admin ▾

Answer

pcs status — confirm which node is offline; check STONITH status; never assume fencing did not occur.
Check Corosync on the affected node; review /var/log/cluster/corosync.log for ring errors or split-brain.
Verify quorum: corosync-quorumtool -l — lost quorum freezes resource management.
If the node was fenced, find and fix WHY before re-adding it.
Rejoin only after confirming health: pcs node unstandby <node>.
For GFS2: check dlm_tool status and gfs2_tool df before allowing concurrent mounts.

NEVER force-join a node with stale data or an unknown state — on GFS2 this can cause data corruption. The STONITH/fencing caution is exactly what senior candidates raise and juniors miss.

03 Scenario Advanced SCENARIO: Both nodes in a two-node GFS2 cluster show themselves as primary (split-brain). What do you do? RHEL Admin ▾

Answer

This is a P1 — split-brain with GFS2 can cause data corruption. Escalate immediately.

Do NOT touch both nodes simultaneously. Identify the last known-good node from logs and monitoring.
Fence the suspect node via STONITH: pcs stonith fence <node> — ensures only one node has FS access.
With only one node active, run fsck.gfs2 on the volumes in single-node mode.
Review Corosync logs on both nodes for root cause — network partition, deadlocked DLM, misconfigured quorum.
Restore the second node only after root cause is fixed, under a controlled change.
Post-incident: full RCA; add a quorum device (qdevice) for better two-node split-brain resolution.

04 Commands Intermediate What are the core Veritas Cluster (VCS) concepts and diagnostic commands you use? RHEL Admin ▾

Answer

Service Group — a collection of resources that fail over together.
Resource & Agent — managed entities (IP, mount, app) and the agents that control them.
Failover vs Switchover — failover is unplanned (node dies); switchover is a planned move.
VxVM (Volume Manager) — disk groups, volumes, plexes; VxFS — Veritas File System.

hastatus -sum # Cluster + service group summary

hagrp -state # Service group states

hagrp -switch <sg> -to <node> # Controlled switchover

hares -state # Resource states

vxdg list / vxprint -ht # VxVM disk groups & volumes

05 Concept Concept Two-node Pacemaker cluster shows one node OFFLINE. Investigate and recover safely. Solaris SME ▾

Answer

pcs status — check STONITH; never assume fencing did not occur.
Corosync logs for ring errors / split-brain; corosync-quorumtool -l for quorum.
If fenced, find WHY before re-adding the node.
pcs node unstandby <node> only after confirming health.
For GFS2: check dlm_tool status before concurrent mounts.

NEVER force-join a node with stale/unknown state — on GFS2 this can cause data corruption. This caution separates senior from junior.

06 Scenario Scenario SCENARIO: GFS2 split-brain — both nodes show primary. Filesystem errors. What do you do? Solaris SME ▾

Answer

P1 — escalate immediately. Do NOT touch both nodes at once.
Identify last known-good node; fence the suspect via pcs stonith fence <node>.
Run fsck.gfs2 in single-node mode.
Root cause in Corosync logs (partition / DLM deadlock / quorum).
Restore node 2 only after fix; add a qdevice for two-node resolution.

07 Commands Commands Core Veritas Cluster (VCS) concepts and diagnostic commands. Solaris SME ▾

Answer

hastatus -sum # Cluster + service group summary

hagrp -state ; hagrp -switch <sg> -to <node>

hares -state ; vxdg list ; vxprint -ht

☁️

3. vSphere / ESXi & UCS — Production Scenarios

ESXi storage-latency triage, unresponsive-host recovery, UCS Service Profile concept and remote OOB workflows.

01 Critical Critical A VM shows high disk latency. Guest, host, or storage layer? Solaris SME ▾

Answer

Layer	Tool	Metric / Interpretation
Guest	iostat -x	await, %util — high await+low %util = path issue
ESXi Host	esxtop → d	DAVG/KAVG/GAVG — high KAVG = ESXi saturation
Storage path	esxcli storage san fc stats get	Round-robin path balance
Datastore	vCenter	Noisy-neighbour congestion

DAVG vs KAVG vs GAVG separates real production troubleshooters. GAVG = DAVG + KAVG.

02 Scenario Scenario SCENARIO: ESXi host stops responding to vCenter; VMs still serving traffic. Response? Solaris SME ▾

Answer

ssh <esxi-mgmt-ip> ; esxcli vm process list

tail -100 /var/log/hostd.log /var/log/vpxa.log

/etc/init.d/hostd restart ; /etc/init.d/vpxa restart

# Resolves most 'not responding' issues without touching VMs

03 Concept Concept How does a UCS Service Profile enable server replacement without reconfiguration? Solaris SME ▾

Answer

A service profile binds MAC, WWN, UUID, and boot policy to a logical object, not the blade. Reassociate to a spare blade → identical identities, boots from SAN, operational in minutes. Compute becomes stateless.

04 Critical Critical Remote server unreachable via SSH, no DC access. iLO workflow. Solaris SME ▾

Answer

Access iLO on the dedicated management network (OOB).
Open Integrated Remote Console — OS crashed, hung, or kernel panic?
Review IML for hardware alerts (drives, DIMMs, NIC, thermal).
Graceful shutdown via iLO; hard-reset only after documenting.
Mount rescue ISO via Virtual Media if OS won't boot.
hponcfg -g / -f to read/push iLO config from the OS with zero downtime.

📈

4. Performance Methodology & Capacity Planning

Cross-OS layered methodology, Oracle DB tuning side-by-side (RHEL/Solaris/HP-UX), and quarterly capacity planning.

01 Concept Methodology Walk me through your methodology when a production server (any OS) is slow. L3 SME ▾

Answer

Layered approach: CPU → Memory → Disk → Network → Recent changes

Layer	RHEL	Solaris	HP-UX
Real-time	top / htop	prstat -a	glance / top
CPU	mpstat -P ALL 1	mpstat 1 / sar -u	sar -u / glance c
Memory	free -h ; vmstat	vmstat / prstat -s rss	swapinfo -tm ; vmstat
Disk	iostat -xz 1 ; iotop	iostat -xnz 1	sar -d ; iostat
Network	ss -s ; sar -n DEV	netstat -s ; dlstat	lanadmin -g ; netstat
History	sar / atop	sar -A	OVPA / MeasureWare

Always correlate slowness with a recent change — patch, config change, new cron job. 80% of production slowness has a change-related root cause.

02 Commands Intermediate Which kernel parameters do you tune for an Oracle Database server on each OS? L3 SME ▾

Answer

Parameter	RHEL (sysctl)	Solaris (/etc/system)	HP-UX (kctune)
Max SHM segment	kernel.shmmax	shmsys:shminfo_shmmax	shmmax
SHM identifiers	kernel.shmmni	shmsys:shminfo_shmmni	shmmni
Semaphores	kernel.sem	semsys:seminfo_*	semmns / semmni
Max open files	fs.file-max	rlim_fd_max	maxfiles_lim
Swappiness	vm.swappiness=10	lotsfree / desfree	vps_ceiling
HugePages	vm.nr_hugepages	use_brk_lpg = 1	vps_pagesize

# RHEL — apply permanently

vim /etc/sysctl.conf && sysctl -p

# Solaris — via /etc/system (reboot) or projmod (live)

projmod -s -K 'project.max-shm-memory=(priv,64G,deny)' user.oracle

# HP-UX — kctune is dynamic where possible

kctune shmmax=68719476736

kctune -h maxfiles=4096 # Held; next boot

03 Concept Advanced How do you do capacity planning for a Unix/Linux estate? Inputs and outputs? L3 SME ▾

Answer

Capacity planning is what separates senior L3 from operational L2. The quarterly process I run:

Collect baseline: sar/OVPA/sysstat for at least 30 days of CPU, memory, disk I/O, network, swap.
Compute peak-of-peak vs average for each metric to spot under-provisioning hidden by averages.
Trend analysis: regression / moving average to project 6 and 12 months out.
Map to business growth: e.g. “DB transactions/sec growing 8% per quarter → CPU at 80% by Q3.”
Apply the 70% rule: any resource sustained above 70% at month-end peak → recommend upgrade.
Output: written capacity report with charts, recommendations, hardware quotes, and a change roadmap.
Review with infra, DBA, app teams; secure budget approval; track delivery.

Mention OpenLMI/Performance Co-Pilot (RHEL), OVPA/MeasureWare (HP-UX), and Solaris kstat — naming the platform-specific historical tool by name signals you've actually done this exercise.

🔐

5. Patching, Security & Compliance (Heterogeneous)

Quarterly patch cycle across RHEL/Solaris/HP-UX, hardening matrix and compliance evidence pipeline.

01 Concept Process How do you run a quarterly patching cycle across RHEL, Solaris and HP-UX? L3 SME ▾

Answer

Phase	Action	Output
Plan (Week 1)	Identify advisories: Red Hat CVE feed, Oracle Solaris MOS, HPE ITRC. Map to CMDB.	Patch matrix per OS
Test (Week 2)	Deploy to Dev + UAT. Regression + smoke tests with app team.	Test sign-off
Stage (Week 3)	Deploy to DR/Staging. Confirm cluster failover still works post-patch.	DR sign-off
Prod (Week 4)	Wave-based rollout. Snapshot before, validate after.	Patch report

Auditor: “How do you patch HP-UX?” Answer: “We install the latest Quality Pack plus targeted critical patches via swinstall, validated with swverify. Rollback is the previous kernel kept automatically by HP-UX, plus a make_net_recovery archive taken the night before.”

02 Concept Intermediate What is your approach to hardening across RHEL, Solaris and HP-UX? L3 SME ▾

Answer

Area	RHEL	Solaris	HP-UX
SSH	PermitRootLogin no, keys only	Same; ssh in SMF	Same; HP Secure Shell
Mandatory access	SELinux Enforcing	RBAC + Trusted Extensions	HP-UX Bastille
Firewall	firewalld / nftables	ipfilter / pf	IPFilter
Audit	auditd	BSM (auditconfig)	audsys / audevent
File integrity	AIDE	Solaris Audit + BART	Bastille file integrity
Compliance scan	OpenSCAP	Solaris Audit baselines	Bastille / SCM
Password policy	/etc/security/pwquality	/etc/default/passwd	/etc/default/security

03 Concept Advanced How do you prove compliance across a heterogeneous fleet? L3 SME ▾

Answer

Single source of truth: a CMDB row per server with OS, version, patch level, owner, last-scan date.
Scheduled scans: OpenSCAP for RHEL, Bastille for HP-UX, BART/scap for Solaris; results in a central dashboard.
Drift detection: Ansible playbooks run in check-mode quarterly — any node that drifts is auto-ticketed.
Evidence pack: before/after configs, scan reports, change tickets, sign-offs — assembled for audit on demand.
Exceptions: deviations logged in the risk register with a compensating control and expiry date.

“We ran an annual external audit across 200+ servers across RHEL, Solaris and HP-UX with zero critical findings, because I maintained the OpenSCAP/Bastille/BART evidence pipeline end-to-end.”

⚙️

6. Automation — Bash, Python & Ansible

Production-grade multi-OS Bash, Python REST scripting, and Ansible playbooks across heterogeneous Unix.

01 Commands Intermediate Write a production-grade Bash script to check disk usage across servers and alert above 85%. RHEL Admin ▾

Answer

#!/bin/bash

# Production-grade disk alert script

THRESHOLD=85

SERVERS=("server1" "server2" "server3")

ALERT_LOG="/var/log/disk_alerts.log"

for host in "${SERVERS[@]}"; do

ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" \

'df -h --output=pcent,target' 2>/dev/null \

| grep -vE '^Use|tmpfs|devtmpfs' \

| while read pct mount; do

usage="${pct//%/}"

if [ "$usage" -ge "$THRESHOLD" ]; then

echo "$(date +%FT%T) ALERT: $host $mount $pct" | tee -a "$ALERT_LOG"

done \

|| echo "$(date) WARN: cannot reach $host" >> "$ALERT_LOG"

done

Interviewers look for ConnectTimeout (prevents hanging), BatchMode=yes (key-based, no prompt), filtering tmpfs/devtmpfs, and timestamped logging. These prove you've debugged scripts in production at 2 a.m.

02 Commands Intermediate How would you write a Python script to pull server metrics from a REST API and produce a report? RHEL Admin ▾

Answer

import requests, sys

API = "https://monitoring.internal/api/v1"

HDRS = {"Authorization": "Bearer <token>", "Accept": "application/json"}

def get_metrics(host):

try:

r = requests.get(f"{API}/servers/{host}/metrics",

headers=HDRS, timeout=10) # always set timeout

r.raise_for_status() # raises on 4xx/5xx

return r.json()

except requests.exceptions.Timeout:

print(f"WARN: timeout for {host}", file=sys.stderr)

except requests.exceptions.HTTPError as e:

print(f"ERROR: {host}: {e}", file=sys.stderr)

return None

Key signals: explicit timeout, raise_for_status(), specific exception types, errors to stderr. This is production-quality, not 'works on my machine'.

03 Commands Intermediate Write a production-grade Bash script to check filesystem usage across heterogeneous Unix servers (RHEL, Solaris, HP-UX) and alert above 85%. L3 SME ▾

Answer

#!/bin/bash

# Production-grade multi-OS disk alert.

set -uo pipefail

THRESHOLD=85

ALERT_LOG="/var/log/disk_alerts.log"

SERVERS_FILE="/etc/admin/servers.txt"

detect_os() {

ssh -o ConnectTimeout=5 -o BatchMode=yes "$1" 'uname -s' 2>/dev/null

}

while read -r host; do

[[ -z "$host" || "$host" =~ ^# ]] && continue

os=$(detect_os "$host")

case "$os" in

Linux) DFCMD="df -h --output=pcent,target | tail -n +2" ;;

SunOS) DFCMD="df -h | awk 'NR>1 {print \$5, \$6}'" ;;

HP-UX) DFCMD="bdf | awk 'NR>1 {print \$5, \$6}'" ;;

*) echo "$(date) WARN: unknown OS for $host" >>"$ALERT_LOG"; continue ;;

esac

ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" "$DFCMD" 2>/dev/null \

| grep -vE 'tmpfs|devtmpfs|proc|cdrom' \

| while read pct mount; do

usage="${pct//%/}"

[[ ! "$usage" =~ ^[0-9]+$ ]] && continue

[ "$usage" -ge "$THRESHOLD" ] && \

echo "$(date +%FT%T) ALERT $host ($os) $mount $pct" | tee -a "$ALERT_LOG"

done

done < "$SERVERS_FILE"

Per-OS handling (Linux df, HP-UX bdf), ConnectTimeout, BatchMode, comment-filtering, timestamped logging — exactly the signals interviewers look for. Most candidates write a Linux-only df script; call out the multi-OS handling.

04 Concept Foundational Why Ansible for a heterogeneous Unix estate? Design considerations? L3 SME ▾

Answer

Agentless: only Python on the target — RHEL has it natively, Solaris and HP-UX have it via depot/IPS.
SSH-based: works through existing jump-host architecture, no special ports.
Idempotent: safe to re-run; declarative state instead of imperative scripts.
Group/host vars let one playbook handle RHEL/Solaris/HP-UX differences via facts and when-clauses.
AWX/Ansible Tower for RBAC, scheduling, audit trail and approvals — critical for L3 production use.

05 Task Advanced TASK: Show an Ansible playbook that patches a heterogeneous Unix fleet correctly. L3 SME ▾

Answer

---

- name: Heterogeneous Unix patching

hosts: unix_estate

serial: 10% # Wave-based rollout

max_fail_percentage: 5 # Stop if 5% fail

any_errors_fatal: false

gather_facts: true

tasks:

- name: Pre-check — record kernel + uptime

shell: "uname -r; uptime"

changed_when: false

- name: RHEL — apply security patches

ansible.builtin.dnf:

name: "*"

state: latest

security: true

when: ansible_os_family == 'RedHat'

- name: Solaris 11 — pkg update

community.general.pkg5:

name: '*'

accept_licenses: true

when: ansible_distribution == 'Solaris' and ansible_distribution_major_version|int == 11

- name: HP-UX — swinstall from depot

shell: |

swinstall -x autoreboot=false -x patch_match_target=true \

-s {{ hpux_depot_path }} \*

when: ansible_system == 'HP-UX'

- name: Reboot if required (RHEL)

reboot: { reboot_timeout: 600 }

when: ansible_os_family == 'RedHat' and needs_reboot.rc == 1

serial + max_fail_percentage together are what make the playbook safe in production. Without them one playbook run can take down the whole fleet. Always mention these — most candidates forget.

06 Concept Intermediate How do you handle secrets and inventory at scale? L3 SME ▾

Answer

Ansible Vault for encrypted variables (passwords, API tokens) — vault password via CyberArk / HashiCorp Vault, never on disk.
Dynamic inventory from CMDB/ServiceNow so groups always match reality.
Group_vars per OS: group_vars/rhel.yml, solaris.yml, hpux.yml.
Tower/AWX for credentials, scheduled jobs, role-based approvals, audit logs.
Run all playbooks in --check --diff first in production-like environments.

07 Scenario Scenario SCENARIO: Push a config change to 200 servers with Salt safely. Solaris SME ▾

Answer

Canary in staging: salt -G 'role:web and environment:staging' state.apply.
10% of prod: salt -G 'environment:production' --batch-size 10% state.apply.
Monitor with salt-run jobs.lookup_jid; states are idempotent.
Full rollout only after the 10% batch is clean, under change control.

Never say 'I'd just run it on all 200'. Batch-size + staged rollout signals production maturity.

08 Scenario Scenario SCENARIO: After AD password change, users can't SSH; SSSD serves cached creds. Solaris SME ▾

Answer

sssctl user-show <user> # Cache state + last pw change

sssctl cache-expire -u <user> # Invalidate cache

chronyc tracking # >5min skew breaks Kerberos

# debug_level=7 in sssd.conf, restart, read domain log

Clock skew is the most common silent cause of Kerberos/SSSD failures — check time sync first.

09 Commands Commands Production-grade Bash disk-alert script — what makes it senior-level? Solaris SME ▾

Answer

ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" \

'df -h --output=pcent,target' 2>/dev/null \

| grep -vE '^Use|tmpfs|devtmpfs' | while read pct mount; do ...

ConnectTimeout, BatchMode=yes, filtering tmpfs, timestamped logging — proves you've debugged scripts in production at 2 a.m.

💿

7. Backup & DR — NetBackup / Veeam

RPO/RTO design, NetBackup architecture and commands, Veeam vs NetBackup, and DR test procedure.

01 Concept Foundational Explain RPO and RTO, and how they drive your backup design. L3 SME ▾

Answer

RPO (Recovery Point Objective) — how much data you can afford to lose. RPO 1 hour ⇒ backups (or replication) every hour.
RTO (Recovery Time Objective) — how long it can take to restore service. RTO 4 hours ⇒ entire restore + app validation must fit in 4 hours.
Tier-1 typically: RPO 15 min via replication, RTO 1 hour via warm standby.
Tier-3 typically: RPO 24h via nightly backup, RTO 24–48h via restore.
Backup design must serve both — incremental for RPO, full + cataloguing for RTO.

02 Commands Intermediate Describe NetBackup architecture and the key admin tasks you perform. L3 SME ▾

Answer

Master Server — runs the catalogue, schedules and policies.
Media Server(s) — move data between clients and storage; offloads the master.
Client — runs the NetBackup agent on every protected server.
Storage Units — disk pools (MSDP de-dupe), tape libraries, cloud targets.
Policies — what to back up, when, retention, replication, encryption.

bpps -a # All NetBackup processes

bpclntcmd -pn # Client → master connectivity

bppllist # List policies

bpbackup -p <policy> -h <client> # On-demand backup

bprestore -R <renamefile> -L <log> ... # Restore

bperror -problems -hoursago 24 # Errors in last 24h

nbstlutil list # SLP state

bpdbjobs -summary # Job summary

Quote bpdbjobs in interviews — every NetBackup admin uses it daily. Pair with bperror -problems -hoursago 24 for the morning health check routine.

03 Concept Foundational How does Veeam back up Linux servers? Compare with NetBackup. L3 SME ▾

Answer

Veeam targets VM-heavy environments — integrates with vSphere/Hyper-V to take application-consistent snapshots and ships them to a repository. For physical Linux, the Veeam Agent for Linux (VAL) handles file/volume/image-level backups.

Aspect	NetBackup	Veeam
Strength	Enterprise multi-OS, tape, large scale	VMware-native, fast, simple UI
Client install	Agent on every host	Agentless for VMs; VAL for physical
Granularity	File, image, app-aware	Image-based + app-aware
Replication / DR	AIR, SLP, Auto Image Replication	Backup Copy, Cloud Connect, Replication
Best fit	Mixed RHEL/Solaris/HP-UX + tape	Heavily virtualised; vSphere first

04 Scenario Advanced How do you run a DR test for a heterogeneous Unix estate without disturbing production? L3 SME ▾

Answer

Quarterly: schedule a DR rehearsal in the change calendar — CAB, app teams, business stakeholders informed.
Isolated DR network: stand up recovered servers on an isolated VLAN with no production routes.
Restore each tier from the most recent backup: NetBackup catalogue restore, Veeam image restore, Ignite-UX make_net_recovery, Solaris UAR / beadm, RHEL ReaR.
Application validation: app team smoke tests; capture screenshots and timings as evidence.
Document timing per restore step; compare to RTO targets.
After test: tear down, archive evidence pack, raise findings tickets, update DR runbook.
Annual: at least one tier-1 system restored to production switch-over and back.

An untested backup is not a backup. If you have never restored an HP-UX system from make_net_recovery, you do not actually have a DR plan for HP-UX. State this conviction in the interview — it differentiates senior thinkers.

🎯

8. L3 Support, RCA & Vendor Escalation

P1 takeover playbook, RCA methodology, recurring-incident problem management and vendor escalation.

01 Concept Process Walk me through how you take over a Major (P1) incident from L2. L3 SME ▾

Answer

Join the P1 bridge within the 15-minute SLA. Introduce yourself; take ownership of the technical lead role.
Confirm scope and impact in the first 5 minutes — systems, business services, user count.
Read the existing timeline L2 has captured; ask only the questions L2 has not already answered.
Triage layers in parallel: OS, network, storage, app — assign owners on the bridge.
Communicate every 15 minutes — even if “still investigating, no change” — to the incident manager.
Mitigation first, root-cause second: get the service back, then dig.
After resolution: own the RCA, present at the PIR within 48 hours, drive preventive actions.

Priority	Response	Resolution Target	Update Cadence
P1 Critical	15 min	4 hours	Every 15–30 min
P2 High	30 min	8 hours	Hourly
P3 Medium	4 hours	24 hours	Daily
P4 Low	Next business day	72 hours	On request

02 Concept Advanced What is your RCA methodology and what makes a good RCA document? L3 SME ▾

Answer

Use 5-Whys plus Ishikawa (fishbone) — never stop at the first technical cause.
Distinguish trigger, root cause, and contributing factors.
Timeline first: every action with timestamps, who did what.
Preventive vs. corrective: corrective fixes this incident; preventive stops the class recurring.
Track preventive actions in the change calendar with owners and due dates — RCA without follow-through is theatre.

A good L3 RCA is one page maximum: Summary, Timeline, Impact, Trigger, Root Cause, Contributing Factors, Preventive Actions with owners + due dates. Anything longer is not read by stakeholders.

03 Scenario Advanced SCENARIO: Six P2 incidents over a month — same symptom (random app crashes) on different Solaris zones — but no problem record raised. What do you do? L3 SME ▾

Answer

Pull incident history; correlate timestamps, zone brand, host, OS version, recent changes.
Pattern: all six in solaris10 branded zones, hosts patched in the last 30 days, between 02:00 and 04:00 (backup window).
Raise a formal Problem Record — reactive to proactive.
Hypothesis: backup snapshot pressure under solaris10 brand emulation; verify in /var/adm/messages + DTrace on next reproduction.
Raise a Change to reschedule backups or tune zone resource caps; track preventive actions.
Brief leadership: pattern found, RCA in progress, interim workaround applied, permanent fix in Change CHG-xxxxx.

“I noticed the L2 team kept resolving the same incident; nobody had raised a problem ticket. I built the correlation report, opened the problem record, drove the RCA, and reduced incident volume by 75% the following month.”

04 Concept Advanced How do you handle vendor escalations across Red Hat, Oracle and HPE? L3 SME ▾

Answer

Raise a case BEFORE the bridge gets desperate — escalate early, downgrade later if you self-resolve.
Severity 1 cases require 24x7 follow-the-sun engagement: confirm timezone hand-off.
Provide the right diagnostics upfront: sosreport (RHEL), explorer (Solaris), supportinfo / cstm (HP-UX) — saves a full handoff cycle.
Track the case ID in the incident ticket; demand a named TAM/CSE for Sev 1; escalate to vendor management if stalled.
On resolution: get the vendor RCA in writing and attach it to your internal RCA.

📋

9. ITIL, Incident Mgmt & Behavioural

P1 handling, problem management discipline, prioritisation, and STAR-structured behavioural answers.

01 Concept Process How do you handle a Major (P1) incident involving multiple teams? RHEL Admin ▾

Answer

Join the P1 bridge immediately; declare yourself technical lead for the OS layer.
Triage within the first 15 minutes: server health, network, services, recent changes.
Maintain a live timeline — every action with timestamps.
Coordinate App, DBA, Network, Storage teams; prevent siloed working.
Apply 5-Whys to find root cause while restoring service in parallel.
Update the incident manager every 15–30 minutes.
After resolution, draft the RCA and present the PIR within 48 hours.

Priority	Response	Resolution Target	Updates
P1 Critical	15 min	4 hours	Every 15–30 min
P2 High	30 min	8 hours	Hourly
P3 Medium	4 hours	24 hours	Daily
P4 Low	Next business day	72 hours	On request

02 Scenario Behavioural SCENARIO: An engineer wants an urgent production firewall change with no change ticket — client demo in 2 hours. RHEL Admin ▾

Answer

Explain that an Emergency Change Request takes under 30 minutes and the risk of an undocumented firewall change outweighs the inconvenience.
Help them draft the ECR immediately and fast-track it through the CAB on-call approver.
Implement within 40 minutes — still in time for the demo.
Afterwards, send a one-pager on raising emergency changes so they can self-serve next time.

“Process exists to protect the environment and the team. A good ops engineer enforces it even under pressure — but makes compliance as fast as possible.”

03 Scenario Behavioural SCENARIO: Six P2 incidents over a month — same symptom, different RHEL servers — but no problem record raised. What do you do? RHEL Admin ▾

Answer

Proactively pull incident history and correlate: timestamps, server model, OS version, recent changes.
Pattern: all RHEL 7.6, same hardware, incidents 02:00–04:00 during backup windows.
Raise a formal Problem Record — moving from reactive to proactive.
Hypothesis: backup software causing memory pressure / I/O starvation → OOM kills or storage timeouts. Verify in /var/log/messages and dmesg.
Raise a Change to tune backup I/O priority (ionice), reschedule, or patch a known kernel bug.
Communicate a brief to management: pattern identified, RCA in progress, interim workaround applied, permanent fix in Change #.

04 Concept Behavioural How do you prioritise when handling multiple critical issues at once? RHEL Admin ▾

Answer

Own the P1 (business-stopping) immediately and set up the bridge.
Delegate initial investigation of the P2 (degraded but running) to a colleague.
Snooze the P3 (disk alert) for 30 minutes with a plan to address after the P1 stabilises.
Update all tickets every 15 minutes — stakeholders should never have to ask for status.

Always structure behavioural answers with STAR (Situation, Task, Action, Result) and end with a measurable result and a Lessons-Learned tail.

05 Scenario Scenario SCENARIO: System rebooted unexpectedly, no warning. Find root cause (Solaris + RHEL). Solaris SME ▾

Answer

last reboot ; who -b

tail -500 /var/adm/messages | grep -i 'panic|error' # Solaris

journalctl -b -1 | grep -i 'panic|oops|error' # RHEL

mdb -k unix.0 vmcore.0 / crash .../vmcore

fmadm faulty / dmesg | grep -i 'mce|oom'

06 Concept Concept 6 P2 incidents/month, same symptom, no problem record. What do you do? Solaris SME ▾

Answer

Pull incident history; correlate timestamps, hardware, OS version, changes.
Pattern: all RHEL 7.6, same hardware, 02:00–04:00 during backups.
Raise a Problem Record — reactive to proactive.
Hypothesis: backup I/O starvation → OOM kills; verify in messages/dmesg.
Change to tune backup ionice / reschedule / patch kernel bug.

07 Scenario Behavioural Engineer wants an urgent production firewall change with no ticket. 2 hours to demo. Solaris SME ▾

Answer

Explain an ECR takes <30 min and undocumented firewall risk outweighs the inconvenience.
Help draft the ECR, fast-track via CAB on-call approver.
Implement within 40 min — still in time for the demo.
Send a one-pager so they can self-serve next time.

“Process protects the environment and the team. Enforce it under pressure — but make compliance fast.”

🧑‍🏫

10. Mentoring, Docs & Behavioural Stories

Documentation discipline (runbook/SOP/KB), mentoring L1/L2 into L3, and ready-to-deliver STAR stories.

01 Concept Concept What documentation do you produce and maintain as an L3 SME? L3 SME ▾

Answer

Runbooks — step-by-step procedures for known operational tasks (boot from rescue, fail over a Serviceguard package).
Standard Operating Procedures (SOPs) — process documents (patching cycle, change ticket workflow).
Knowledge Base articles — searchable, symptom-led; one article per recurring symptom L2 should resolve.
Architecture documents — diagrams of zone layouts, Serviceguard configs, network topology; refreshed annually.
RCA library — completed incident RCAs, indexed by service and root cause class.

Name the SPECIFIC types and what each is for. “Yes I document things” is a junior answer. “Runbook for ops, SOP for process, KB for L2, architecture for new joiners, RCA library for problem management” is a senior answer.

02 Concept Concept How do you mentor L1/L2 engineers to grow into L3? L3 SME ▾

Answer

Weekly 1:1 — 30 minutes per engineer; review their tickets, identify learning gaps, set one skill goal.
Shadowing — pair them with you on P1 incidents (read-only → assistant → primary with you backing).
Ticket review — two of their tickets per week, walk through what could have been faster or deeper.
Topic of the month — one deep dive (e.g. September: Serviceguard) with a runbook deliverable owned by the L2.
Stretch assignments — give them ownership of small projects so they grow ownership, not just technical skills.
Visible recognition — credit them in incident reports and to their manager.

“I mentored two L2 engineers into L3 over 18 months — one took over my Solaris on-call rotation, the other now leads our HP-UX migration project.” Concrete outcomes beat generic mentoring claims.

03 Scenario Behavioural Tell me about a P1 incident you owned end-to-end. L3 SME ▾

Answer

Situation: name the service, the business impact (revenue / user count / SLA risk).
Task: what was YOUR responsibility on the bridge.
Action: 3–4 concrete technical actions you led — not what the team did.
Result: quantified — “Resolved in 47 minutes, no SLA breach, identified the kernel bug that led to permanent fix in next patch cycle.”

04 Scenario Behavioural Tell me about an automation you built that delivered measurable value. L3 SME ▾

Answer

Have a concrete example with numbers: “I replaced a manual patching SOP with an Ansible playbook that handles RHEL, Solaris and HP-UX. Patch night went from 8 hours of 3 engineers to 2 hours of 1 engineer — 75% effort reduction and zero post-patch incidents in the next six months.”

05 Scenario Behavioural Tell me about a time you pushed back on a process and made it better. L3 SME ▾

Answer

Banking and enterprise environments love this question. Example: “Our change process required separate tickets for related changes. I proposed a parent-child template; reduced ticket overhead by 40% and improved CAB visibility.” Show you respect process but improve it.

📚

Legacy Reference — Original OS Admin L3 Hub

The full original OS Admin L3 interview-prep content with its original styling preserved (52 Q&As across 12 topics covering Linux/Solaris/AIX/VMware/ITSM, plus a quick command reference). Kept here so nothing is lost from the earlier work.

🔧

1. Patching & Pkg Mgmt

6 Q&As

🚀

2. Boot, GRUB & initramfs

4 Q&As

💾

3. LVM, Storage & FS

6 Q&As

🔐

4. Security & Hardening

5 Q&As

⚡

5. Performance

4 Q&As

🌐

6. Networking & DNS

3 Q&As

📜

7. Shell Scripting

3 Q&As

🚨

8. Common Errors & Fixes

6 Q&As

☁️

9. VMware

6 Q&As

🌞

10. Solaris 10/11 & ZFS

3 Q&As

🔵

11. IBM AIX

2 Q&As

📋

12. ITSM & Change Control

4 Q&As

01 Concept How do you perform Unix/Linux OS patching activity? ▾

Answer

OS patching is a structured, multi-phase activity performed during approved maintenance windows. The approach varies by OS but follows the same discipline.

Phase 1 — Pre-Patching Preparation

Identify applicable patches from vendor advisories, Red Hat Satellite, or internal patch baseline
Download and stage patches in a local repository (avoid patching directly from internet on production)
Raise a Change Request with implementation plan, rollback plan, and risk assessment
Take VM snapshot or system backup before starting
Notify application teams and obtain maintenance window approval from CAB

Phase 2 — RHEL / Oracle Linux

# Check available updates
yum check-update # RHEL 7
dnf check-update # RHEL 8/9
# Apply all updates
yum update -y
# Security patches only
yum update --security -y
# Apply patch for specific CVE
yum update --cve CVE-2024-1234 -y
# Check if reboot required
needs-restarting -r
# View patch history
yum history
yum history info <ID>
yum history rollback <ID> # Rollback if needed

Phase 2 — Solaris

# Solaris 11 — IPS
pkg update # Update all
pkg update pkg://solaris/kernel # Specific package
pkg list -u # List updatable packages
# Solaris 10 — patchadd
patchadd -M /patch_dir 123456-07
showrev -p | grep 123456 # Verify patch applied

Phase 2 — AIX

# AIX — Apply TL/SP via NIM or smitty
smitty update_all # Apply all available updates
oslevel -s # Check current TL/SP level
alt_disk_install -b bosboot -d hdisk1 # Zero-downtime via alt disk

Phase 3 — Post-Patch Validation

Verify kernel version: uname -r
Check all services: systemctl list-units --state=failed
Review logs: dmesg | tail -30, journalctl -p err -n 50
Run application smoke tests with app team sign-off
Delete snapshot after 48-hour stability confirmation
Update change ticket, CMDB, and patch inventory

Phased patching order: Dev → UAT/Staging → DR → Production. Never patch production without lower-env validation. Always have a rollback plan documented in the change ticket.

02 Scenario Server not booting after patching — how do you troubleshoot and recover? ▾

Answer

Step 1 — Access via Console

VM: Access VMware vCenter console (not SSH — network may be unavailable)
Physical server: Access via iLO / DRAC / IPMI remote console

Step 2 — Boot with Previous Kernel (Fastest Fix)

At GRUB menu, press Esc or hold Shift to interrupt automatic boot
Select the previous kernel entry from the list
If boots successfully → the new kernel caused the regression

# After booting with old kernel, remove bad kernel:
rpm -q kernel # List installed kernels
yum remove kernel-<new-version> # Remove bad kernel
grub2-mkconfig -o /boot/grub2/grub.cfg # Regenerate GRUB config
grub2-set-default 0 # Set default kernel

Step 3 — GRUB Broken (Rescue Mode)

# Boot from RHEL rescue ISO → Troubleshooting → Rescue
chroot /mnt/sysimage
# Reinstall GRUB (BIOS system):
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg
# Reinstall GRUB (UEFI system):
grub2-install --target=x86_64-efi --efi-directory=/boot/efi
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Step 4 — Rebuild initramfs

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
# Or for specific version:
dracut -f /boot/initramfs-4.18.0-305.el8.x86_64.img 4.18.0-305.el8.x86_64

Step 5 — Emergency Shell (fstab UUID issue)

journalctl -xb # See exact mount error
blkid # Get correct UUIDs
vi /etc/fstab # Fix UUID mismatch
mount -a # Test before reboot

Boot with previous kernel via GRUB is the fastest recovery path — know this step by heart. Always take a VM snapshot before patching.

03 Concept How do you manage quarterly patching on 100+ servers with minimal downtime? ▾

Answer

Download patches to local Red Hat Satellite / YUM mirror — servers never hit internet directly
Create a patch schedule coordinated with change management and application owners
Test patches in UAT/Staging first — document any issues or conflicts
Patch in waves: DR first → Staging → Production clusters (rolling, not all at once)
Use Ansible playbooks to execute patching in parallel batches with health check validation between waves
Take VMware snapshots before each wave; remove after 48-hour stability confirmation
Submit post-patching report: patch compliance %, any failures, rollbacks performed

# Ansible patch playbook (simplified logic)
---
- hosts: staging_servers
  tasks:
    - name: Apply security patches
      yum: name=* state=latest security=yes
    - name: Check if reboot needed
      command: needs-restarting -r
      register: reboot_needed
    - name: Reboot if required
      reboot: reboot_timeout=300
      when: reboot_needed.rc == 1

Mention phased approach and Ansible automation — shows you can handle scale efficiently.

04 Critical A critical CVE (CVSS 9.5) is released for OpenSSH. You have 4 hours to patch 20 production servers. What do you do? ▾

Answer

Raise an Emergency Change Request immediately — get CAB or designated approver sign-off
Download the patched OpenSSH package to local repo and verify checksum
Test on one non-production server: update → verify SSH connectivity → check version
Write a remediation script with automatic rollback if SSH connectivity fails post-update
Execute in parallel batches using Ansible or pssh across all 20 servers
Validate SSH on each server after patching; log results in real time
Submit evidence report (before/after rpm -q openssh, CVE scan results) to security team within 24 hours

yum update openssh openssh-server -y
rpm -q openssh # Verify new version
systemctl restart sshd
ssh -o BatchMode=yes server01 'echo ok' # Test connectivity

Always raise an emergency change ticket FIRST — even if it takes 15 minutes. Making production changes without authorization exposes you and the bank to audit risk.

05 Concept How do you perform vulnerability remediation? ▾

Answer

CVSS Score	Priority	Timeline	Change Type
≥ 9.0 Critical	P1	72 hours	Emergency Change
7.0–8.9 High	P2	7–14 days	Normal Change
4.0–6.9 Medium	P3	Next quarterly cycle	Standard Change
< 4.0 Low	P4	Annual review	Deferred

# Check available security updates and CVEs:
yum updateinfo list security
yum update --cve CVE-2024-1234 -y
yum update --advisory RHSA-2024:1234 -y
# Verify remediation:
rpm -q --changelog <package> | grep CVE
openssl s_client -connect server:443 2>&1 | grep Protocol

Submit scan evidence after remediation — the security team runs a rescan and you must show the CVE is closed. Always document in the change ticket.

06 Commands What is the difference between hotfix, patch, and service pack? How do you prioritize them? ▾

Answer

Type	Scope	When Applied	Example
Hotfix	Single critical bug or CVE	Immediately / Emergency window	CVE OpenSSL patch
Patch	Multiple bug fixes, security updates	Scheduled monthly/quarterly	RHEL kernel update
Service Pack	All patches bundled for a version	Major maintenance window	RHEL 8.6 → 8.8 update, AIX TL

Prioritization is based on CVSS score. I never rely solely on vendor classification — I cross-reference with our asset inventory to determine actual exposure and impact.

01Concept Explain the Linux boot process. What is the role of initramfs during boot? ▾

Answer — 6 Stages

Stage	Component	What Happens	Failure Symptom
1	BIOS/UEFI	POST, locates bootloader from MBR/EFI partition	No display, beep codes
2	GRUB2	Loads vmlinuz + initramfs, passes kernel params	GRUB rescue prompt
3	Kernel	Decompresses, hardware init, mounts initramfs	Kernel panic at early boot
4	initramfs	Loads drivers, activates LVM, mounts real root	Emergency shell "cannot find root"
5	systemd	PID 1 — starts services per default target	Services fail, partial boot
6	Login	getty / sshd ready for user login	Login prompt not appearing

Role of initramfs

initramfs = Initial RAM Filesystem — a compressed cpio archive in /boot/
Provides minimal userspace environment BEFORE real root filesystem is accessible
Contains: storage drivers (SCSI, NVMe, FC HBA), LVM tools (to activate VGs), encryption modules (LUKS), udev for device naming
Without initramfs, kernel cannot access root on LVM, SAN, or encrypted disks
After mounting real root → hands control to systemd on disk

# Inspect initramfs contents:
lsinitrd /boot/initramfs-$(uname -r).img
# Rebuild initramfs (RHEL — uses dracut):
dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
dracut -f --regenerate-all # Rebuild for ALL kernels

02Scenario Server boots to emergency shell: "Failed to mount /data". It's a production DB server. How do you recover? ▾

Answer

Emergency shell appears = systemd failed to mount a filesystem in /etc/fstab
Run journalctl -xb to see the exact mount failure reason
Run blkid to list all block devices and their UUIDs

# Common cause — UUID mismatch in /etc/fstab:
blkid /dev/sdb1 # Get actual UUID
vi /etc/fstab # Update UUID to match blkid output
mount -a # Test all mounts
exit # Continue boot
# If SAN device missing — allow boot without /data:
# Add 'nofail' option to /etc/fstab entry temporarily:
/dev/mapper/data  /data  xfs  defaults,nofail  0 0
# Filesystem corruption — run fsck (filesystem unmounted):
fsck -y /dev/sdb1

Know the 'nofail' mount option — it lets the system boot even if a non-critical disk is missing. Critical for banking systems where you need to restore service quickly.

03Commands How do you reinstall GRUB and rebuild initramfs? ▾

Answer

Boot from Rescue ISO → Chroot

# At ISO boot: Troubleshooting → Rescue → Option 1 (Continue)
chroot /mnt/sysimage # Enter the installed system

Reinstall GRUB2

# BIOS system (check: ls /sys/firmware/efi — if dir missing, it's BIOS):
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg
# UEFI system:
grub2-install --target=x86_64-efi --efi-directory=/boot/efi
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Rebuild initramfs

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
ls -lh /boot/ # Verify files exist
exit && reboot

04Critical Explain the steps to troubleshoot kernel panic errors. ▾

Answer

Step 1 — Capture with kdump

systemctl status kdump # Verify kdump is enabled
ls /var/crash/ # vmcore dump files after crash

Step 2 — Check panic message from previous boot

journalctl -b -1 | grep -i 'kernel panic\|oops\|Call Trace'
grep -i 'kernel panic\|BUG\|oops' /var/log/messages

Step 3 — Analyze vmcore with crash utility

yum install crash kernel-debuginfo-$(uname -r) -y
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore
# Inside crash utility:
bt # Backtrace — shows kernel stack at panic
log # Kernel message buffer
ps # Processes at crash time
vm # Virtual memory info

Common Causes & Fixes

Panic Type	Visible In	Fix
NULL pointer dereference	bt output — driver module	Update or blacklist driver
Hardware memory error	mcelog, EDAC dmesg	Replace faulty DIMM
OOM (Out of Memory)	oom-kill in /var/log/messages	Tune vm.overcommit, add RAM
SAN I/O error	HBA errors in dmesg	Fix multipath, check SAN

# Collect full diagnostic bundle for vendor support:
sosreport

01CommandsHow do you extend LV and filesystem size?▾

Answer

# Step 1 — Check VG free space:
vgs # Check VFree column
vgdisplay vg_data # Detailed VG info
# Step 2 — Extend LV + Resize filesystem (one command):
lvextend -L +20G -r /dev/vg_data/lv_app
# -r flag: auto-resizes ext4 OR xfs filesystem after extend
# Manual 2-step method:
lvextend -L +20G /dev/vg_data/lv_app
xfs_growfs /mountpoint # XFS — uses mountpoint
resize2fs /dev/vg_data/lv_app # ext4 — uses device
# If VG has no space — add new disk first:
pvcreate /dev/sdc
vgextend vg_data /dev/sdc
lvextend -l +100%FREE -r /dev/vg_data/lv_app
# VMware VM — expand disk without downtime:
# 1. Expand virtual disk in vCenter (VM Settings → Hard disk)
echo 1 > /sys/class/scsi_disk/0:0:0:0/device/rescan
partprobe /dev/sdb
pvcreate /dev/sdb
vgextend vg_data /dev/sdb
lvextend -l +100%FREE -r /dev/vg_data/lv_app
# Verify:
df -h /mountpoint
lvdisplay /dev/vg_data/lv_app

The -r flag (lvextend -r) is the shortcut that extends LV and resizes filesystem in one step — mention this in the interview.

02CriticalLVM Volumes not showing when we run vgs — what could be the reason and how do you fix it?▾

Answer

VG is inactive — most common reason after disk replacement, SAN path loss, or system migration
PV device is not visible — underlying disk/LUN not accessible to the OS
VG metadata corrupted — LVM metadata on the PV is missing or corrupt
LVM filter in /etc/lvm/lvm.conf excludes the device
Disk not scanned yet — new disk added but LVM hasn't discovered it

# Check PVs including inactive:
pvs -a
pvdisplay
# Check block devices visible to OS:
lsblk
ls /dev/sd*
# Rescan for PVs:
pvscan
# List all VGs including inactive:
vgs -a
vgdisplay --partial
# FIX — Activate VG:
vgchange -ay vg_data # Activate specific VG
vgchange -ay # Activate ALL VGs
# If VG moved from another system:
vgimport vg_data
vgchange -ay vg_data
# Restore from LVM metadata backup:
vgcfgrestore -f /etc/lvm/backup/vg_data vg_data

vgchange -ay is the first command to try whenever LVs are missing. Never use --partial flag in production without understanding the implications — it can activate a VG with missing PVs and potentially cause data corruption.

03CriticalFilesystem shows space free but no new files can be created. Why?▾

Answer — Inode Exhaustion

When df -h shows available space but you get "No space left on device", the cause is inode exhaustion — the inode table is full.

# Diagnose:
df -i # Show INODE usage — look for 100% in IUse%
df -ih # Human-readable
# Find directory consuming most inodes (most files):
find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20
# Common culprits:
# /var/spool/mail  /tmp  /var/spool/exim/input  app temp dirs
# Fix — delete unnecessary small files:
rm -rf /var/spool/exim/input/*
find /tmp -mtime +7 -delete
# Check inode settings:
tune2fs -l /dev/sda1 | grep -i inode # ext4
xfs_info /mountpoint # XFS

Can filesystem space increase also increase inode count?

Filesystem	Can Extend Add Inodes?	Reason
ext4	NO	Inode count fixed at mkfs time — cannot change after creation
XFS	YES (effectively)	XFS allocates inodes dynamically from free space

# Create ext4 with higher inode density (at mkfs time only):
mkfs.ext4 -i 8192 /dev/sdb1 # 1 inode per 8KB (default is 16KB)
mkfs.ext4 -N 10000000 /dev/sdb1 # Set explicit inode count

This is a classic trick question — df -h looks fine but you cannot create files. Always check df -i when you see "No space left on device".

04CommandsExplain about multipathing (DM-Multipath) on Linux.▾

Answer

DM-Multipath provides multiple I/O paths between a server and SAN storage for redundancy and load balancing. If one path fails, I/O switches to alternate path — transparent to the application.

# Check multipath status:
multipath -ll # Show all devices with path status
multipathd -k # Interactive console
# Sample output interpretation:
mpatha (3600508b...) dm-0 HP,LOGICAL VOLUME
|-+- policy='round-robin 0' prio=1 status=active
| |- 2:0:0:1 sdb 8:16 active ready running   ← active path
| |- 3:0:0:1 sdc 8:32 active ready running   ← active path
`-+- policy='round-robin 0' prio=0 status=enabled
  |- 2:0:1:1 sdd 8:48 active ready running   ← standby path
# Troubleshoot failed/ghost paths:
multipath -ll | grep -i 'fail\|ghost'
dmesg | grep -i 'multipathd\|scsi\|hba'
# Flush and rediscover:
multipath -F && multipath

Policy	Behavior	Use Case
failover	One active path, rest standby	Simple redundancy
round-robin	Distribute I/O across all paths	Load balancing
multibus	All paths in one group, all active	Max throughput

05CriticalHow do you repair a filesystem?▾

Answer

CRITICAL RULE: NEVER run fsck or xfs_repair on a MOUNTED filesystem. Always unmount first or do it from rescue mode for root filesystem.

ext4 — fsck

umount /dev/sdb1 # Unmount first!
fsck -n /dev/sdb1 # Dry run — check only
fsck -y /dev/sdb1 # Auto-repair all errors
e2fsck -fy /dev/sdb1 # Force check + auto-repair

XFS — xfs_repair

umount /dev/sdb1
xfs_repair /dev/sdb1 # Repair XFS
xfs_repair -n /dev/sdb1 # Dry run
# If 'dirty log' error:
xfs_repair -L /dev/sdb1 # Force clear log (LAST RESORT)

Root filesystem (cannot unmount)

# Boot from rescue ISO → chroot → run fsck
chroot /mnt/sysimage
fsck -y /dev/sda2

Check disk health before repair

smartctl -a /dev/sda # Check SMART data for reallocated sectors
dmesg | grep -i 'I/O error\|EXT4-fs error\|XFS'

06Scenario/var filesystem hits 100% at 3 AM causing application failures. How do you recover?▾

Answer

# Immediate — identify top space consumers:
df -h # Confirm /var is full
du -sh /var/* | sort -rh | head -20 # Find biggest directories
du -sh /var/log/* | sort -rh | head -10
# Common culprits in banking environments:
# /var/crash — core dumps (delete if not needed)
rm -rf /var/crash/*
# /var/log — oversized application logs
find /var/log -name "*.log" -size +100M -mtime +3
gzip /var/log/app/app.log.2024*
# Quick space via audit logs (only after security approval):
service auditd stop
truncate -s 0 /var/log/audit/audit.log
service auditd start
# Extend LV if VG has space:
lvextend -L +10G -r /dev/vg_sys/lv_var

After recovery: implement 80% disk alert in monitoring tool
Configure proper logrotate for offending application
Document RCA and submit post-incident report

01ConceptWhat is your approach to server hardening as per bank security policy?▾

Answer

Area	Action	Commands/Files
SSH	Disable root login, key-auth only, SSHv2, idle timeout	/etc/ssh/sshd_config
SELinux	Set to Enforcing mode	setenforce 1, /etc/selinux/config
Firewall	Default deny, allow only required ports	firewall-cmd --add-service
PAM	Password complexity, account lockout, aging	/etc/security/limits.conf
Audit	auditd for privileged commands, file integrity	/etc/audit/rules.d/
Kernel	sysctl hardening, disable IP forwarding, SYN cookies	/etc/sysctl.conf
Packages	Remove unused packages and services	yum remove, systemctl disable
USB	Disable USB storage module	blacklist usb-storage in modprobe

# Key sshd_config settings:
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
Protocol 2
ClientAliveInterval 300
ClientAliveCountMax 2
AllowUsers admin1 admin2
# Kernel hardening (/etc/sysctl.conf):
net.ipv4.ip_forward = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
kernel.dmesg_restrict = 1
# Validate compliance after hardening:
oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_cis_server_l1 /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml

Mention OpenSCAP/CIS benchmarks and running a compliance scan post-hardening — shows process maturity beyond just making changes.

02ConceptHow can you provide permission to a user to run a command as root?▾

Answer

# ALWAYS use visudo — never edit /etc/sudoers directly
visudo
# Allow user 'john' to run systemctl (with password):
john    ALL=(ALL)    /bin/systemctl
# Allow without password prompt:
john    ALL=(ALL)    NOPASSWD: /bin/systemctl restart httpd
# Allow group 'dba':
%dba    ALL=(ALL)    ALL
# Preferred — drop-in file in /etc/sudoers.d/:
visudo -f /etc/sudoers.d/john
chmod 440 /etc/sudoers.d/john
# Test the rule:
sudo -l -U john # What can john run?
# View sudo audit trail:
grep sudo /var/log/secure
ausearch -m USER_CMD -sv yes

Use specific command paths — never give ALL without strong justification
Avoid NOPASSWD for destructive commands (rm, dd, chmod, shutdown)
All sudo usage is logged in /var/log/secure and auditd automatically
Review sudo rules quarterly as part of access review process

03ConceptExplain PAM (Pluggable Authentication Modules) and how you use it.▾

Answer

PAM is a framework that decouples authentication from applications. Configuration files are in /etc/pam.d/ — one per service (sshd, login, sudo).

# Key PAM modules used in banking:
# Password complexity — /etc/security/pwquality.conf:
minlen = 12
dcredit = -1     # At least 1 digit
ucredit = -1     # At least 1 uppercase
ocredit = -1     # At least 1 special char
lcredit = -1     # At least 1 lowercase
# Account lockout (/etc/pam.d/sshd — pam_faillock):
auth required pam_faillock.so preauth deny=5 unlock_time=900
auth [default=die] pam_faillock.so authfail deny=5 unlock_time=900
# Password history — prevent reuse of last 12:
password sufficient pam_unix.so remember=12
# Unlock a locked account:
faillock --user john --reset
# Check failed attempts:
faillock --user john

Always keep a root session open while testing PAM changes — a misconfiguration can lock ALL users out of the system.

04ConceptExplain about ulimit — what is it and how is it configured?▾

Answer

ulimit controls resource limits for processes — preventing a single user/process from consuming all system resources.

ulimit -a # View all current limits for session
ulimit -n 65536 # Open file descriptors (nofile)
ulimit -u 16384 # Max processes (nproc)
ulimit -c unlimited # Core dump size
# Permanent — /etc/security/limits.conf:
oracle   soft   nofile   65536
oracle   hard   nofile   65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
# For systemd services — in unit file [Service] section:
LimitNOFILE=65536
LimitNPROC=16384
# Check limits for a running process:
cat /proc/<PID>/limits

Limit	Flag	Common Value	What It Controls
nofile	-n	65536	Open file descriptors (critical for Oracle, Tomcat)
nproc	-u	16384	Max processes (low value causes fork: retry error)
core	-c	unlimited	Core dump size for debugging
stack	-s	10240	Stack size in KB

05ScenarioA service account is logging into production via SSH password — violating key-auth policy. What do you do?▾

Answer

First — gather evidence: grep the logs, document scope before taking any action
Report to Information Security team IMMEDIATELY — this is a policy violation security incident
After IS team authorization — enforce key-auth only for this account
Coordinate with app team to implement proper key-based auth
Scan all other servers for the same issue

# Gather evidence:
grep <service_account> /var/log/secure | grep 'Accepted password'
last <service_account> # Login history
ausearch -m USER_AUTH -sv yes | grep <account>
# Enforce key-auth only for specific account:
# Add to /etc/ssh/sshd_config:
Match User svc_account
    PasswordAuthentication no
systemctl reload sshd
# Scan all servers for password-auth logins:
for srv in $(cat /etc/server_list.txt); do
  ssh $srv "grep 'Accepted password' /var/log/secure | grep svc_account"
done

Report to IS team FIRST — never take remediation action before authorization in a banking environment. Evidence preservation is as important as the fix.

01ConceptHow do you troubleshoot server slowness issues?▾

Answer — Layered Approach: CPU → Memory → Disk → Network

Step 1 — CPU

uptime # Load avg vs CPU count
top # or htop # Real-time CPU per process
mpstat -P ALL 1 3 # Per-CPU stats
ps -eo pid,ppid,%cpu,%mem,cmd --sort=-%cpu | head -15

Step 2 — Memory

free -h # Available RAM and swap
vmstat 1 5 # si/so = swap in/out (bad if > 0)
grep -i 'oom\|kill' /var/log/messages # OOM kills
ps -eo pid,%mem,cmd --sort=-%mem | head -10

Step 3 — Disk I/O

iostat -xz 1 3 # %iowait >20% = bottleneck, await >20ms = high
iotop -o # Per-process I/O (top for disk)

Step 4 — Network

ss -s # Socket summary
netstat -s | grep -E 'retransmit|error'
sar -n DEV 1 3 # Network throughput
ethtool eth0 # NIC speed/duplex

Step 5 — Recent Changes & Logs

rpm -qa --last | head -20 # Recent package changes
journalctl -p err -n 50
dmesg | tail -30
last reboot

Always check if the slowness correlates with a recent change — patch, config change, or new cron job. 80% of slowness issues have a change-related root cause.

02CriticalHow do you fix Disk I/O bottleneck issues?▾

Answer

# Confirm I/O bottleneck:
iostat -xz 1 5
# Key metrics:
# %iowait > 20%  = CPU waiting for I/O
# await (ms) > 20ms on SSD = high latency
# %util > 80%    = disk near saturation
# Identify process causing high I/O:
iotop -o
pidstat -d 1 5
lsof -p <PID> # Files the process has open
# OS-level tuning:
# Set I/O scheduler:
cat /sys/block/sda/queue/scheduler
echo 'mq-deadline' > /sys/block/sda/queue/scheduler # Good for DB workloads
echo 'none' > /sys/block/nvme0n1/queue/scheduler # NVMe SSDs
# Increase read-ahead for sequential I/O:
blockdev --setra 4096 /dev/sda
# Tune dirty page writeback:
sysctl vm.dirty_ratio=15
sysctl vm.dirty_background_ratio=5

Cause	Symptom	Fix
Runaway log writes	High w/s on app partition	Log rotation, reduce verbosity
Backup running	Sequential read spikes	Reschedule to off-peak hours
Memory too low / swapping	si/so in vmstat > 0	Increase RAM or swap
VMware noisy neighbor	Latency, not throughput	vMotion VM to less-loaded host
SAN path degraded	Intermittent high await	Check multipath -ll, escalate

03ConceptHow do you improve server performance?▾

Answer

# Apply tuned performance profile:
tuned-adm list
tuned-adm profile throughput-performance # Best for servers
tuned-adm profile latency-performance # Low latency workloads
tuned-adm active # Check current profile
# CPU — reduce swapping tendency:
sysctl vm.swappiness=10 # Default 60, lower = less swapping
sysctl vm.vfs_cache_pressure=50 # Retain VFS cache longer
# HugePages for Oracle/Java (reduces TLB misses):
echo 1000 > /proc/sys/vm/nr_hugepages
# Disable services not needed:
systemctl disable bluetooth cups avahi-daemon --now
# Network tuning:
sysctl net.core.rmem_max=16777216
sysctl net.core.wmem_max=16777216
sysctl net.ipv4.tcp_fin_timeout=15
# Filesystem mount performance options:
# Add noatime,nodiratime to /etc/fstab for non-OS partitions

Always measure before and after tuning — use sar to collect baseline data for comparison. Performance changes without measurement data are just guesses.

04ConceptWhat kernel parameters do you tune for an Oracle Database server?▾

Answer

# /etc/sysctl.conf — Oracle Database tuning:
kernel.shmmax = 68719476736     # Max SHM segment (≥ SGA size)
kernel.shmall = 16777216        # Total SHM pages
kernel.sem = 250 32000 100 128  # Semaphores for Oracle processes
fs.file-max = 6815744           # Max open file descriptors
net.core.rmem_max = 4194304     # Network buffer
net.core.wmem_max = 1048576
vm.swappiness = 10              # Minimize swapping for DB
vm.nr_hugepages = 1000          # Pre-allocated HugePages for SGA
kernel.panic_on_oops = 1        # Reboot on kernel oops
# Apply permanently:
sysctl -p
# Oracle pre-install RPM automates most of this:
yum install oracle-database-preinstall-19c

01CriticalA server is unable to resolve domain names. How would you fix it?▾

Answer — Layered DNS Troubleshooting

# Step 1 — Confirm the problem:
nslookup google.com
dig google.com
host google.com
# Step 2 — Check DNS config:
cat /etc/resolv.conf
# Should contain:
nameserver 192.168.1.1
nameserver 8.8.8.8
search company.local
# Step 3 — Test DNS server reachability:
ping 192.168.1.1
dig @192.168.1.1 google.com # Query specific DNS server
nc -zv 192.168.1.1 53 # Test port 53 connectivity
# Step 4 — Check resolution order:
cat /etc/nsswitch.conf | grep hosts
# Should be: hosts: files dns
# Step 5 — Check /etc/hosts for conflicts:
cat /etc/hosts
# Step 6 — Check NetworkManager / systemd-resolved:
systemctl status NetworkManager
resolvectl status
# Fix — Set DNS permanently via nmcli:
nmcli con mod 'eth0' ipv4.dns '192.168.1.1 8.8.8.8'
nmcli con up 'eth0'

02ConceptExplain NIC Bonding/Teaming modes and which mode do you use in banking production?▾

Answer

Mode	Name	Redundancy	Load Balance	Switch Config
Mode 0	Round Robin	No	Yes	Required
Mode 1	Active-Backup	Yes	No	None needed
Mode 2	XOR	Yes	Yes	Required
Mode 4	802.3ad / LACP	Yes	Yes	LACP required
Mode 5	Adaptive TLB	Yes	Tx only	None needed
Mode 6	Adaptive LB	Yes	Yes	None needed

Banking production: Mode 4 (LACP/802.3ad) for app servers needing throughput + redundancy. Mode 1 (Active-Backup) for management interfaces or where switch LACP config is not possible.

# Check bond status:
cat /proc/net/bonding/bond0 # Active interface, mode, link status
ip addr show bond0

03ConceptIs it possible to add multiple IP addresses to one single NIC card?▾

Answer

Yes — called IP aliasing or secondary IPs. A single NIC can have multiple IP addresses, all sharing the same MAC address and physical link.

# Method 1 — Temporary (lost on reboot):
ip addr add 192.168.1.101/24 dev eth0
ip addr add 192.168.1.102/24 dev eth0
ip addr show eth0 # Verify all IPs
# Method 2 — Permanent via NetworkManager:
nmcli con mod 'eth0' +ipv4.addresses 192.168.1.101/24
nmcli con mod 'eth0' +ipv4.addresses 192.168.1.102/24
nmcli con up 'eth0'
# Method 3 — Legacy ifcfg alias (RHEL 6/7):
# Create /etc/sysconfig/network-scripts/ifcfg-eth0:1
DEVICE=eth0:1
IPADDR=192.168.1.101
NETMASK=255.255.255.0
ONBOOT=yes

01ConceptDo you have shell scripting knowledge? What is a for loop?▾

Answer

# Basic for loop syntax:
for variable in list; do
    commands
done
# Example 1 — Check disk on multiple servers:
#!/bin/bash
SERVERS="server01 server02 server03"
for SERVER in $SERVERS; do
    echo "=== $SERVER ==="
    ssh -o ConnectTimeout=5 $SERVER 'df -h | grep -vE "tmpfs|devtmpfs"'
    [ $? -ne 0 ] && echo "WARNING: Cannot reach $SERVER"
done
# Example 2 — C-style counter loop:
for ((i=1; i<=10; i++)); do
    echo "Iteration $i"
done
# Example 3 — Loop over files (compress old logs):
for FILE in /var/log/app/*.log; do
    [ -f "$FILE" ] && gzip "$FILE" && echo "Compressed: $FILE"
done
# Example 4 — Loop from file (server list):
while IFS= read -r SERVER; do
    ssh $SERVER 'uptime'
done < /etc/server_list.txt

02CommandsHow do you use awk, sed, and grep for log analysis?▾

Answer

# grep — pattern matching:
grep -i error /var/log/messages | grep -v kernel
grep -c 'Failed password' /var/log/secure # Count occurrences
grep -A 5 'FATAL' /app/logs/app.log # 5 lines after match
grep -n 'ORA-' /var/log/oracle.log # With line numbers
# awk — field-based parsing:
# Extract source IPs from SSH brute force — sort by frequency:
awk '/Failed password/ {print $11}' /var/log/secure | sort | uniq -c | sort -rn
# Sum a numeric column:
awk '{sum += $5} END {print "Total:", sum}' /var/log/transfer.log
# Print specific time range:
awk '$3 >= "08:00:00" && $3 <= "09:00:00"' /var/log/messages
# sed — in-place text manipulation:
# Extract time window from log:
sed -n '/Oct 1 08:00/,/Oct 1 09:00/p' /var/log/messages
# Replace config value:
sed -i 's/PermitRootLogin yes/PermitRootLogin no/g' /etc/ssh/sshd_config
# Combined — unique Oracle errors with timestamps:
grep 'ORA-' /app/logs/app.log | awk '{print $1, $2, $NF}' | sort -u

03ConceptShell scripting vs Ansible — when do you use each?▾

Answer

Criteria	Shell Script	Ansible
Style	Procedural	Declarative (desired state)
Idempotency	Must code manually	Built-in — safe to run twice
Scale	1–few servers	Hundreds of servers in parallel
Agent	None needed	None (agentless via SSH)
Use case	Monitoring, log mgmt, ad-hoc tasks	Config deployment, patch orchestration
Availability	Always available	Requires Ansible installed

My practice: Shell scripts for monitoring, log management, and single-server tasks. Ansible playbooks for configuration deployment (NTP, SSH hardening, sudoers), rolling patch cycles, and compliance enforcement across the entire server fleet.

01Error"Too many open files" error▾

Cause & Fix

Cause: Process exceeded its 'nofile' ulimit — max open file descriptors limit reached.

lsof -p <PID> | wc -l # Count open files for PID
cat /proc/<PID>/limits # Current limits for the process
ulimit -n # Current session limit
# Fix — /etc/security/limits.conf:
appuser  soft  nofile  65536
appuser  hard  nofile  65536
# For systemd service:
LimitNOFILE=65536 # In [Service] section of unit file
systemctl daemon-reload && systemctl restart <service>

02Error"bash: fork: retry: Resource temporarily unavailable"▾

Cause & Fix

Cause: nproc ulimit reached — max number of processes for the user is exhausted. Often caused by a process creating too many threads or a fork bomb.

ulimit -u # Current nproc limit
ps -u <username> | wc -l # Count user's processes
ps -eo pid,ppid,user,cmd | grep <user> | head -20
# Fix — /etc/security/limits.conf:
username  soft  nproc  16384
username  hard  nproc  16384
# Kill runaway processes consuming slots:
pkill -u <username> <processname>

03Error"No space left on device" — related to inodes▾

Cause & Fix

Cause: Inode table is full — millions of small files (sessions, temp files, mail spools) have exhausted the inode count even though disk space is available.

df -i # Check IUse% — if 100%, inode exhaustion
find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20
# Fix:
rm -rf /var/spool/exim/input/*
find /tmp -mtime +1 -type f -delete
find /var/log -name "*.log" -mtime +30 -delete

04Error"Cannot allocate memory"▾

Cause & Fix

Cause: System is out of physical RAM and swap, OOM killer is active, or virtual address space is exhausted.

free -h # Available RAM and swap
vmstat 1 3 # Check si/so columns
dmesg | grep -i 'oom\|kill' # OOM killer activity
ps -eo pid,%mem,cmd --sort=-%mem | head -10
# Quick fix — add temporary swap:
dd if=/dev/zero of=/swapfile bs=1G count=4
mkswap /swapfile && chmod 600 /swapfile
swapon /swapfile
# Kill top memory consumer (after app team approval):
kill -9 <PID>

05Error"device is busy" when unmounting filesystem▾

Cause & Fix

Cause: Files or processes are still using the filesystem — open file handles, processes with working directory inside the mount.

fuser -mv /mountpoint # Show all processes using the mount
lsof /mountpoint # Open files on the filesystem
# Fix — close application cleanly, then unmount:
fuser -km /mountpoint # Kill processes (use with care!)
umount /mountpoint
# Lazy unmount — detaches when all handles close:
umount -l /mountpoint

06Error"LVM Volume not found / inactive"▾

Cause & Fix

pvs -a # Check PVs including inactive
lsblk # Check block device availability
pvscan # Rescan for PVs
# Activate VG:
vgchange -ay vg_data
vgchange -ay # All VGs
# If moved from another system:
vgimport vg_data && vgchange -ay vg_data

01ConceptHow do you take a VM snapshot and what are the rules?▾

Answer

# PowerCLI — Create snapshot:
New-Snapshot -VM 'server01' -Name 'Pre-Patch-2025-10-15' -Quiesce -Memory:$false
# Revert (rollback):
Set-VM -VM 'server01' -Snapshot (Get-Snapshot -VM 'server01' -Name 'Pre-Patch-2025-10-15')
# Delete snapshot (after validating stability):
Remove-Snapshot -Snapshot (Get-Snapshot -VM 'server01' -Name 'Pre-Patch') -Confirm:$false
# List all snapshots:
Get-Snapshot -VM 'server01'
# Consolidate snapshots (if needed):
# vCenter: Right-click VM → Snapshots → Consolidate

Maximum snapshot age in production: 48 hours. Stale snapshots grow indefinitely (redirect-on-write) and can fill datastores. Never run production VMs in snapshot state long-term.

Quiesce option: Freezes guest filesystem I/O before snapshot for consistency — requires VMware Tools to be installed and running.

02ConceptWhat if VM snapshot fails?▾

Answer

Cause	Fix
Datastore out of space	Free datastore space, or extend; check Storage → Datastores → Monitor
VMware Tools not installed/old	Install/update open-vm-tools inside guest OS
Snapshot consolidation needed	Right-click VM → Snapshots → Consolidate
Too many existing snapshots	Delete old snapshots first (max recommended: 3-4)
Backup running (disk locked)	Wait for NetBackup/Veeam backup to complete
Guest OS quiesce failure	Check /var/log/vmware-vmsvc.log inside guest

# Check VMware Tools inside guest:
systemctl status vmtoolsd
vmware-toolbox-cmd -v
# Update VMware Tools:
yum update open-vm-tools -y

03ConceptWhat is vMotion? Explain the types.▾

Answer

vMotion is VMware's live migration technology — moves a running VM from one ESXi host/datastore to another with zero downtime (typically < 1 second interruption).

Type	Moves	Requirement
vMotion	CPU + Memory (compute)	Shared storage between hosts
Storage vMotion	VM disk files	VM stays on same host
Enhanced vMotion	Compute + Storage	No shared storage needed

# Requirements:
# - Shared storage (SAN/NFS) accessible by both hosts
# - Dedicated VMkernel port with vMotion enabled (10GbE recommended)
# - Compatible CPU families or EVC mode on cluster
# - vCenter required (cannot do vMotion from ESXi directly)
# PowerCLI — Migrate VM:
Move-VM -VM 'server01' -Destination (Get-VMHost 'esxi02')

Banking use case: use vMotion to evacuate all VMs from an ESXi host before patching it. DRS automates vMotion for load balancing across the cluster.

04ConceptCan you explain about VM templates?▾

Answer

A VM template is a master golden image for deploying new VMs consistently. It is a non-runnable copy of a configured VM.

Item	Template	Snapshot	Clone
Purpose	New VM deployments	Rollback point	Copy for testing
Runnable?	No	Yes (parent VM)	Yes
Uses	Gold image for fleet	Pre-change safety net	Dev/test copies

# Convert VM to template:
# vCenter: Right-click VM → Template → Convert to Template
# PowerCLI — Deploy from template:
New-VM -Name 'newserver01' -Template 'RHEL8-Gold-Template' -VMHost 'esxi01' -Datastore 'DS01'
# Update template (quarterly):
# 1. Convert template back to VM
# 2. Power on → apply patches → harden → power off
# 3. Convert back to template
# 4. Document template version in naming convention

05ConceptExplain about VMware Tools — why is it important?▾

Answer

Function	Why It Matters
Time synchronization	Critical for Kerberos, Oracle RAC, banking timestamps
Quiesced snapshots	Consistent backup — requires VMware Tools
Graceful shutdown/restart	vCenter can cleanly shut down VM (not force power off)
Memory balloon driver	Hypervisor reclaims unused guest memory under pressure
VMXNET3 NIC driver	Paravirtual — much faster than emulated e1000
PVSCSI driver	High I/O performance for Oracle/SQL Server
Guest metrics	vCenter shows OS-level CPU/memory/disk inside guest

# Install (preferred — open-vm-tools, in standard repos):
yum install open-vm-tools -y
systemctl enable vmtoolsd --now
# Verify:
systemctl status vmtoolsd
vmware-toolbox-cmd -v

06ScenarioVMware vCenter shows 90% CPU Ready on a VM. The application team says it's slow. Diagnose and resolve.▾

Answer

CPU Ready means the vCPU is ready to run but the physical CPU is not available — the hypervisor host is overcommitted.

# In esxtop (on ESXi host) — press 'v' for VM view:
# %RDY > 5% = concerning, > 20% = severe
# %CSTP = co-stop for multi-vCPU VMs
# At OS level — CPU Ready shows as steal time:
top # st% column (steal time) in CPU stats
# PowerCLI — check CPU Ready stats:
Get-Stat -Entity $vm -Stat cpu.ready.summation -Start (Get-Date).AddHours(-4)

Check if the ESXi host cluster is overcommitted (too many vCPUs vs physical cores)
Coordinate with VMware team to vMotion the VM to a less-loaded host immediately
Consider reducing vCPU count — fewer vCPUs = less co-scheduling pressure (counterintuitive but effective)
Escalate for capacity addition if entire cluster is consistently overloaded
Document correlation between CPU Ready % and application response time for capacity report

01ConceptWhat are the key administrative tasks you perform on Solaris 10 and 11?▾

Answer

Task	Solaris 10	Solaris 11
Patching	patchadd, patchrm	pkg update
Package install	pkgadd	pkg install
Service management	svcadm, svcs	svcadm, svcs (same)
Zones	zoneadm, zonecfg	zoneadm, zonecfg
Network	ifconfig, ndd	ipadm, dladm
ZFS	zfs, zpool	zfs, zpool (same)
Deployment	JumpStart	AI (Automated Installer)

# SMF — Service Management Facility:
svcs -a # All services status
svcs -xv svc:/network/ssh:default # Detailed service info
svcadm enable svc:/network/ssh:default
svcadm restart svc:/network/ssh:default
svcadm disable svc:/network/ssh:default
# Zones management:
zoneadm list -cv # List all zones with status
zoneadm -z zone01 boot # Boot a zone
zlogin zone01 # Login to zone console
zoneadm -z zone01 halt # Halt a zone
# Network (Solaris 11):
ipadm show-addr # Show IP addresses
dladm show-link # Show network links

02ConceptExplain ZFS on Solaris — key features and how you use them.▾

Answer

Feature	Benefit	Command
Copy-on-Write	Crash consistency, instant recovery	Built-in
Snapshots	Instant, space-efficient backups	zfs snapshot pool/ds@snap
Clones	Writable copy of snapshot	zfs clone pool/ds@snap pool/new
Compression	2-4x storage savings for logs	zfs set compression=lz4
Quotas	Capacity management per dataset	zfs set quota=100G
RAIDZ	Native disk redundancy	zpool create -o ashift=12 raidz2

# ZFS pool and dataset management:
zpool status # Pool health
zpool iostat -v 1 # I/O per disk
zfs list # List all datasets with space
zfs list -t snapshot # List snapshots
# Create and manage snapshots:
zfs snapshot data/app@pre-deploy-2024-10-15
zfs rollback data/app@pre-deploy-2024-10-15 # Instant rollback!
zfs destroy data/app@pre-deploy-2024-10-15 # Remove snapshot
# Set compression and quotas:
zfs set compression=lz4 data/logs
zfs set quota=200G data/app
zfs set reservation=50G data/app
# Scrub (verify data integrity):
zpool scrub data
zpool status data # Check scrub results

ZFS snapshot + rollback is one of the most powerful features for OS admins — instant deployment rollback without backup restore time. Mention real-world use: rolling back a failed application deployment in seconds.

03ScenarioA Solaris zone is consuming excessive CPU/memory, impacting other zones. How do you resolve?▾

Answer

# Identify which zone is consuming resources:
prstat -Z # Zone-level resource summary
zonestat 1 # Per-zone resource statistics
# Identify which process inside the zone:
prstat -z zone01 1
# Apply immediate CPU cap (from global zone):
prctl -n zone.cpu-cap -v 200 -i zone zone01 # Limit to 200% (2 CPUs)
# Permanent — configure resource controls via zonecfg:
zonecfg -z zone01
  set cpu-shares=10
  add capped-cpu
  set ncpus=2
  end
  add capped-memory
  set physical=4g
  set swap=8g
  end
  commit
zoneadm -z zone01 apply # Apply without zone reboot

01ConceptExplain AIX's LVM architecture and how it differs from Linux LVM.▾

Answer

Concept	Linux LVM	AIX LVM
Allocation unit	Physical Extents (PE) — 4MB default, fixed	Physical Partitions (PP) — 4–128MB, configurable per VG
VG types	Standard only	Original (32 PVs), Big (128), Scalable (1024)
Mirroring	Via RAID or dm-mirror	Native — mklvcopy
Metadata	/etc/lvm/	ODM (Object Data Manager)
Filesystem	ext4, XFS	JFS2 (preferred)

# AIX LVM commands:
lsvg # List Volume Groups
lsvg -l rootvg # List LVs in VG
lspv # List Physical Volumes (hdisks)
extendvg rootvg hdisk1 # Add disk to VG
mklv -t jfs2 datavg 100 # Create LV (100 PPs)
chfs -a size=+1G /data # Extend filesystem
# AIX LVM mirroring:
mklvcopy lv_data 2 # Create mirror copy
syncvg datavg # Sync mirror after disk replacement
# Check ODM consistency:
lsvgdb rootvg
# AIX performance monitoring:
topas # AIX equivalent of top
nmon # Comprehensive performance tool
errpt -a | head -50 # Hardware/software error report

02ConceptWhat is NIM (Network Installation Management) and how have you used it?▾

Answer

NIM is AIX's network-based infrastructure management system for OS installation, cloning, and updates across LPAR environments.

# NIM resources:
# LPP_SOURCE — installation media
# SPOT       — Shared Product Object Tree (network boot image)
# MKSYSB     — system backup images for cloning
# Check NIM master status:
lsnim -l # List all NIM objects
lsnim -t lpp_source # List available installation sources
# Update client via NIM:
nim -o update -a lpp_source=lpp_7200_03 -a fixes=all lpar01
# Zero-downtime patching with alt_disk_install:
alt_disk_install -b bosboot -d hdisk1 # Install new AIX on alternate disk
# System runs on original disk while patching alternate disk
# Rollback: simply boot from original disk if new disk fails
# Create system backup (MKSYSB):
mksysb -i /dev/rmt0 # Backup to tape
mksysb /tmp/server01.mksysb # Backup to file

alt_disk_install is AIX's unique zero-downtime upgrade feature — the system continues running on the original disk while the new TL/SP is installed on an alternate disk. Instant rollback by simply booting from the original.

01ConceptHow do you handle a Major Incident (P1) involving multiple teams?▾

Answer

Join the P1 bridge call immediately and declare technical lead for the OS layer
Perform initial triage within first 15 minutes: server health, network, service status, recent changes
Maintain a live incident timeline — document every action with timestamps
Coordinate with App, DBA, Network, and Storage teams — prevent siloed working
Apply 5-Why methodology to drill to root cause while simultaneously restoring service
Update the incident manager every 15–30 minutes with status
After resolution: draft RCA document and present PIR within 48 hours

Priority	Response Time	Resolution Target	Update Frequency
P1 — Critical	15 minutes	4 hours	Every 15–30 min
P2 — High	30 minutes	8 hours	Every 1 hour
P3 — Medium	4 hours	24 hours	Daily
P4 — Low	Next business day	72 hours	On request

02ScenarioA developer asks you to open a firewall port on production urgently with no change ticket. What do you do?▾

Answer

Empathize with the urgency but firmly explain that production changes require a change ticket — this is compliance, not bureaucracy
Guide the developer/manager to raise an Emergency Change Request (ECR) immediately — can be approved in 30–60 minutes
While ticket is being raised, collect technical details: source IP, destination IP, port, protocol, business justification — ready to implement the moment approval arrives
Do NOT make the change without an approved ticket — this exposes the bank to audit risk and potential security breach
Once approved: implement, document in the ticket, update firewall rule inventory, and schedule post-implementation review

This is a test of your process discipline under pressure. The answer is ALWAYS: raise the ECR first, then implement. Your job security and the bank's compliance depend on it.

03ConceptAre you using any monitoring tools? What alerts do you usually get?▾

Answer

Tools Used:

IBM Tivoli Monitoring (ITM) / Netcool — enterprise monitoring in banking
Nagios / Icinga — threshold-based service and resource alerting
Zabbix — agent-based monitoring with dashboards
HP Operations Manager (HPOM) — event correlation
vCenter performance alerts — VMware-specific metrics
Custom shell scripts + cron + mailx — lightweight targeted monitoring

Common Alerts & Actions:

Alert	Threshold	My Action
CPU High	>85% sustained 5min	Identify top process, escalate to app team
Memory High	>90%	Check for leak, add swap temporarily
Disk Warning	>80%	du -sh, clean or extend LV
Disk Critical	>90%	Immediate cleanup, alert app team
Filesystem Full	100%	Emergency cleanup, P1 ticket
Inode Exhaustion	>90% IUse	Find small file consumer, clean up
Service Down	Immediate	systemctl restart, RCA if recurring
Backup Failed	Next morning	Retry, fix root cause, document
SSH Brute Force	>10 fails/min	Check secure log, block source IP

04ConceptHow has your ITIL certification influenced the way you work?▾

Answer

Incident Management: I prioritize tickets by impact × urgency matrix, not by who shouts loudest. SLA adherence is tracked and reported weekly.
Change Management: All production changes go through RFC → CAB → Implementation → PIR. No exceptions — this protects both the bank and me personally.
Problem Management: When I see the same incident recurring, I raise a Problem ticket to find and eliminate the root cause. Example: I noticed 80% of disk-full tickets on a cluster came from unmanaged log growth — implemented systematic log rotation that eliminated the recurring incidents entirely.
Continual Service Improvement (CSI): I track SLA trends monthly and identify improvement opportunities. Automation initiatives I've implemented have reduced manual ticket handling by 40%.
Knowledge Management: Every resolved incident gets documented in the knowledge base with RCA and resolution steps — reduces time-to-resolve for repeat issues.

Connect ITIL principles to real examples — "I applied ITIL problem management when..." shows genuine application, not just passing the exam.

Performance

top / htop # CPU/memory
vmstat 1 5 # System overview
iostat -xz 1 3 # Disk I/O
iotop -o # Per-process I/O
sar -A # Historical stats
mpstat -P ALL 1 # Per-CPU
pidstat -d 1 # Per-process I/O
free -h # Memory
uptime # Load average

Storage / LVM

pvs / pvcreate # Physical volumes
vgs / vgextend # Volume groups
lvs / lvextend -r # Logical volumes
vgchange -ay # Activate VGs
xfs_growfs /mnt # Grow XFS
resize2fs /dev/lv # Grow ext4
df -h / df -i # Space / inodes
multipath -ll # Multipath status
fsck -y / xfs_repair # Repair FS

Security

sestatus / getenforce # SELinux
ausearch -m avc # SELinux denials
audit2why / audit2allow
grep sudo /var/log/secure
faillock --user john # Failed logins
faillock --user j --reset
chage -l username # Password aging
visudo # Edit sudoers
sudo -l -U john # John's sudo rights

Networking

ip addr / ip route # IP configuration
ss -tuln # Listening ports
netstat -s # Network stats
dig / nslookup / host # DNS
tcpdump -i eth0 port 80
firewall-cmd --list-all
nmcli con mod eth0 ...
cat /proc/net/bonding/bond0
ethtool eth0 # NIC status

Boot / Kernel

uname -r # Kernel version
grub2-install /dev/sda
grub2-mkconfig -o ... # Regenerate GRUB
dracut -f $(uname -r) # Rebuild initramfs
journalctl -b -1 # Previous boot log
journalctl -p err -n 50
sysctl -a / sysctl -p
dmesg | tail -50
systemctl status kdump

User Admin

useradd / usermod / userdel
groupadd / gpasswd
passwd / chage # Password mgmt
who / w / last # Logged-in users
id username # User info
cat /proc/<PID>/limits # Process limits
lsof -u username # User's open files
ulimit -a # Session limits

Solaris

svcs -a / svcadm # SMF services
zoneadm list -cv # Zone status
zlogin / zonecfg
zfs list / zpool status
zfs snapshot ds@snap
zfs rollback ds@snap
prstat -Z # Per-zone stats
pkg update # S11 patching
patchadd # S10 patching

AIX

lsvg / lspv / lslv # LVM info
extendvg / mklv # Extend storage
chfs -a size=+1G /data
topas / nmon # Performance
errpt -a # Error report
oslevel -s # TL/SP level
smitty update_all # Apply updates
alt_disk_install # Zero-downtime patch
lsnim -l # NIM objects