🌞 Oracle Solaris

All Solaris content — SPARC hardware, ZFS, LDOMs, Zones, Veritas Cluster, Kernel Debugging

Solaris 10 & 11 15 Sections 137+ Q&As SPARC · ZFS · Zones · LDOMs

🌞

Oracle Solaris

All Solaris content — SPARC hardware, ZFS, LDOMs, Zones (including Branded Zones), kernel debugging with mdb/DTrace, Oracle ASM, IPMP and SMF.

📋

1. Solaris 10 vs 11 & Patching

2 Q&As

🔆

2. SPARC Hardware & ILOM

4 Q&As

🔢

3. Disk Major/Minor Numbers & SAN

4 Q&As

💽

4. ZFS Administration

5 Q&As

🖥

5. Zones, Branded Zones & LDOMs

9 Q&As

🔬

6. Kernel Debugging

4 Q&As

🗄

7. Oracle ASM Disk Addition

3 Q&As

🔌

8. Networking, Boot, SMF & Provisioning

4 Q&As

🛠

9. Solaris OS & Kernel

10 Q&As

💿

10. Filesystem & Storage

8 Q&As

⚡

11. Performance

6 Q&As

🌐

12. Networking

5 Q&As

🖥

13. Solaris Zones

18 Q&As

🔆

14. LDOMs

20 Q&As

🔗

15. Veritas Cluster Server

35 Q&As

Sections below are organized topic-wise within this OS. Click any topic in the sidebar or a card above. Each Q&A shows a badge for source document (e.g. RHEL Admin, L3 SME, Solaris SME) and a depth level (Foundational → Advanced).

📋

1. Solaris 10 vs 11 & Patching

Key admin differences between Solaris 10 and 11, IPS, beadm Boot Environments and instant rollback patching.

01 Concept Foundational Key admin differences between Solaris 10 and Solaris 11? L3 SME ▾

Answer

Area	Solaris 10	Solaris 11
Patching	patchadd / patchrm	pkg update (IPS)
Package install	pkgadd	pkg install
Service mgmt	svcadm / svcs (SMF)	svcadm / svcs (same)
Zones	zoneadm / zonecfg	zoneadm / zonecfg (same)
Network	ifconfig / ndd	ipadm / dladm
Default FS	UFS (root) + ZFS	ZFS only (incl. root)
Deployment	JumpStart	Automated Installer (AI)
Patch rollback	Live Upgrade	Boot Environments (beadm)

02 Commands Intermediate How do you patch Solaris 11 and roll back if it goes wrong? L3 SME ▾

Answer

Solaris 11 uses IPS. Combined with ZFS Boot Environments this gives essentially-free, instant rollback — one of the most powerful patching stories on any Unix.

# Create a fallback Boot Environment FIRST

beadm create sol11-pre-patch-$(date +%Y%m%d)

beadm list # Confirm

# Apply updates

pkg update # All updates

pkg update pkg://solaris/kernel # Specific package

pkg list -u # What can be updated

# Reboot into the new BE

reboot

# Roll back if the new BE is broken:

beadm activate sol11-pre-patch-20240315 ; reboot

# Solaris 10 equivalent:

patchadd -M /patch_dir 123456-07

showrev -p | grep 123456

Boot Environments give Solaris 11 a 35-minute patch with instant rollback vs. the old 2-hour patch + 4-hour rollback. Quantify this in interviews — “we cut patching MTTR by ~50%” — and you stand out.

🔆

2. SPARC Hardware & ILOM

SPARC T-Series vs M-Series, ILOM/ALOM out-of-band management, OpenBoot PROM and Fault Management Architecture (FMA).

01 Concept Concept Key differences between SPARC T-Series (T4, T8) and M-Series (M7/M8/M10/M12)? Solaris SME ▾

Answer

Attribute	T-Series	M-Series
Optimised for	Highly-threaded (CMT) workloads	Mission-critical RAS
Threads	T4: 8 cores × 8 threads = 64/socket	High core count with ECC
Key features	Java/web/app servers	Silicon Secured Memory; M12: 32 sockets, 32TB RAM
Best for	Throughput / concurrency	SAP HANA, Oracle DB, core banking

M-Series is your banking answer. Always ask if the environment is T-Series or M-Series before answering architecture questions.

02 Commands Commands Explain ALOM/ILOM and common troubleshooting commands. Solaris SME ▾

Answer

ILOM (Integrated Lights Out Manager) provides out-of-band management independent of the host OS — accessible even when Solaris is crashed or powered off.

-> show /System # Hardware health summary

-> show /System/fans

-> start /SP/console # Attach to host console

-> reset /System # Hard reset

-> show /HOST/fault_log # Fault history

-> set /SP/network/ ipaddress=X

03 Commands Commands What is OpenBoot PROM (OBP)? Key diagnostic commands. Solaris SME ▾

Answer

ok> boot disk # Boot from first disk

ok> boot -s # Single-user mode

ok> boot -r # Reconfigure boot (rebuild /dev)

ok> probe-scsi-all # Scan SCSI/SAS buses

ok> show-devs # List device tree nodes

ok> printenv # Show NVRAM variables

ok> .version # OBP firmware version

04 Concept Concept How do you check hardware errors and faults using Solaris FMA? Solaris SME ▾

Answer

fmadm faulty # Active faults with FRU info

fmstat # FMA module statistics

fmadm repair <UUID> # Clear a fault

fmdump -eV # Verbose error event dump

🔢

3. Disk Major/Minor Numbers & SAN

Solaris device numbering, slice minor encoding, WWN correlation and new SAN LUN discovery workflow.

01 Concept Concept What are major and minor device numbers in Solaris? How do you check them? Solaris SME ▾

Answer

Major number identifies the device DRIVER. Minor number identifies the specific INSTANCE/PARTITION of that device.

ls -l /dev/dsk/c0t0d0s0 # brw-r-- 32,0 → major=32 minor=0

stat /dev/dsk/c0t0d0s0

prtconf -v | grep -A5 'disk'

format # Interactive — all disk instances

02 Concept Concept Explain the minor number encoding for Solaris disk slices. Solaris SME ▾

Answer

# Legacy sd driver formula:

minor = (controller<<16)|(target<<8)|(lun<<3)|slice

# c0t2d0s3 → controller=0, target=2, disk=0, slice=3

# s2 is always the whole-disk partition

prtvtoc /dev/rdsk/c0t0d0s2 # View slice table

03 Scenario Scenario New SAN LUN presented — identify it, check major/minor, add to ZFS. Walk through the workflow. Solaris SME ▾

Answer

# Step 1: Discover

cfgadm -al ; devfsadm -C ; echo | format

# Step 2-3: Identify + check major/minor

ls -lrt /dev/dsk/ | tail -10

ls -l /dev/dsk/c1t5d0s2 # brw-r-- 32,144

# Step 4: Health

iostat -En c1t5d0

# Step 5: Add to ZFS

zpool add datapool c1t5d0 ; zpool status datapool

04 Commands Commands How do you identify disks by WWN/physical path and correlate with device names? Solaris SME ▾

Answer

luxadm probe # Discover FC-attached storage

cfgadm -al -o show_SCSI_LUN # LUN details

prtconf -v | grep 'target-port' # WWN from device tree

readlink -f /dev/rdsk/c0t0d0s2 # Full physical path

💽

4. ZFS Administration

Pool hierarchy, vdev types, disk add/replace/resilver, Boot Environments, compression/dedup, and health/scrub.

01 Concept Concept Explain ZFS pool hierarchy: vdev types and when to use each. Solaris SME ▾

Answer

vdev Type	Redundancy	Use Case
stripe	None	NEVER for production data
mirror	N-1 disk	Boot pools, high IOPS
raidz1	1 disk	General storage, min 3 disks
raidz2	2 disks	Production data pools, min 4 disks
raidz3	3 disks	Archival, min 5 disks
cache (L2ARC)	N/A	SSD random-read acceleration
log (ZIL)	Mirror	SSD sync-write acceleration

zpool create -o ashift=12 datapool raidz2 c1t0d0 c1t1d0 c1t2d0 c1t3d0

02 Commands Commands How do you add and replace disks in a ZFS pool? Solaris SME ▾

Answer

# Add capacity / cache / spare

zpool add datapool raidz2 c2t0d0 c2t1d0 c2t2d0 c2t3d0

zpool add datapool cache c3t0d0

zpool add datapool spare c3t3d0

# Replace a failed disk

zpool offline datapool c1t2d0

zpool replace datapool c1t2d0 c2t5d0

zpool status datapool # Monitor resilver %

03 Concept Concept Explain ZFS Boot Environments (BE) and how they reduce patching downtime. Solaris SME ▾

Answer

A Boot Environment is a complete, independent snapshot of the OS root dataset. Switch between bootable OS states instantly; rollback in seconds.

beadm create sol11-pre-patch-$(date +%Y%m%d)

pkg update ; reboot # Patch active BE

# Rollback — seconds, not hours:

beadm activate sol11-pre-patch-20240315 ; reboot

Boot Environments delivered ~50% patching downtime reduction. Frame it: old = 2hr patch + potential 4hr rollback; BEs = 35min patch + instant rollback.

04 Commands Commands How do you check ZFS pool health and diagnose degradation? Solaris SME ▾

Answer

zpool status -v # Error counts, faulted devices

zpool iostat -v 5 # Live I/O per vdev

zpool scrub datapool # Data integrity scrub

zfs set compression=lz4 datapool

zdb -S datapool # Simulate dedup savings

05 Scenario Scenario SCENARIO: ZFS pool shows DEGRADED, one disk FAULTED. Production impact and recovery? Solaris SME ▾

Answer

zpool status -v datapool # Identify faulted disk

# DEGRADED + raidz2 = still operational, 1 fault tolerance left

zpool scrub datapool # Scrub while sourcing replacement

iostat -En c1t2d0 ; fmadm faulty

zpool offline datapool c1t2d0

zpool replace datapool c1t2d0 c2t5d0

Immediately send a P1 bridge update. ETA for full redundancy = resilver completion time. Do NOT wait until the disk is replaced.

🖥

5. Zones, Branded Zones & LDOMs

Global/NGZ/Kernel/Branded zones, full solaris10-brand walkthrough from flar, resource capping, and LDOM architecture + live migration.

01 Concept Foundational What are Solaris Zones? Compare the different brands. L3 SME ▾

Answer

A Zone is OS-level virtualization in Solaris — multiple isolated user environments sharing one kernel. A brand defines which OS environment the zone emulates, allowing modern hardware to run legacy Solaris OS versions or even Linux.

Zone Type	Brand	Kernel	Best For
Global Zone	global	Host OS	Physical host management
Native NGZ	solaris (S11) / native (S10)	Shared	App isolation on same OS
Solaris 10 brand	solaris10	Emulated S10 on S11	Run legacy Sol 10 apps on Sol 11
Solaris 9 brand	solaris9	Emulated S9 on S10	Legacy retention on Solaris 10
Solaris 8 brand	solaris8	Emulated S8 on S10	Long-tail legacy migration
Kernel Zone	solaris-kz	Own kernel (S11.2+)	Independent patching/upgrade
lx brand	lx	Linux syscalls translated	Run RHEL/CentOS userland on Solaris

02 Task Advanced TASK: Walk through creating a Solaris 10 Branded Zone on a Solaris 11 host — end-to-end. L3 SME ▾

Answer

This is what the JD asks about — running legacy Solaris 10 workloads on modern Solaris 11 hardware via the solaris10 brand.

# Step 1 — Verify the solaris10 brand is installed on the global zone

pkg list | grep brand-solaris10

pkg install pkg://solaris/system/zones/brand/brand-solaris10

# Step 2 — On the source Solaris 10 system, create a flash archive (flar)

flarcreate -S -n srv10-archive -x /export/home /var/tmp/srv10.flar

# Step 3 — Copy the flar to the Solaris 11 host

scp /var/tmp/srv10.flar sol11host:/var/tmp/

# Step 4 — Configure the branded zone

zonecfg -z s10zone

create -t SYSsolaris10 # solaris10 brand template

set zonepath=/zones/s10zone

set autoboot=true

add net

set physical=vnic1

set address=10.0.0.50/24

end

commit ; exit

# Step 5 — Install (clone) the zone from the flar

zoneadm -z s10zone install -a /var/tmp/srv10.flar -u

# Step 6 — Boot and log in

zoneadm -z s10zone boot

zlogin -C s10zone # First-boot config

# Step 7 — Inside the zone, confirm it really is Solaris 10

uname -r # 5.10

zoneadm list -cv # Shows brand=solaris10

The -u flag on zoneadm install performs an unconfigure so the zone boots into the system identification dialog. Without it, the cloned host identity persists — a classic interview gotcha.

03 Concept Advanced How do you design zones architecture for resource isolation? Show CPU + memory + I/O capping. L3 SME ▾

Answer

One zone per application tier — never mix prod and non-prod in the same global zone.
Always cap CPU AND memory — uncapped zones are how a single runaway process takes the whole host down.
Use Resource Pools + Fair Share Scheduler (FSS) so zones get guaranteed minimums under pressure.
Separate ZFS datasets per zone — gives per-zone snapshot/rollback and I/O isolation.
Dedicated VNICs per zone — clean network isolation and easier troubleshooting.

zonecfg -z appzone

add capped-cpu ; set ncpus=4.0 ; end

add capped-memory

set physical=8G ; set swap=16G ; set locked=4G

end

add dedicated-cpu ; set ncpus=2-4 ; end

set cpu-shares=20 # FSS shares relative to siblings

commit ; end

# Apply without rebooting the zone:

zoneadm -z appzone apply

# Monitor in real time:

zonestat 1 ; prstat -Z ; rcapstat 1

Always pair capped-cpu with cpu-shares. The cap is a hard ceiling; the shares decide how un-capped slack is divided when zones compete. Set one without the other and you get starvation or noisy-neighbour problems.

04 Scenario Advanced SCENARIO: A Solaris 10 Branded Zone is consuming 90% of the host CPU and impacting other zones. How do you investigate and fix without taking it down? L3 SME ▾

Answer

prstat -Z 1 5 # Per-zone CPU summary

prstat -Lmz s10zone 1 # Per-LWP inside that zone

# Immediate hard cap (no reboot)

prctl -n zone.cpu-cap -v 200 -i zone s10zone # 2 vCPUs

# Drill into the offending process from the global zone

ps -ef -z s10zone | head

pflags <PID> ; pstack <PID>

# DTrace — highest-value tool here

dtrace -n 'profile-997 /pid==<PID>/ { @[ustack()]=count(); }'

# Permanent cap via zonecfg + live apply

zonecfg -z s10zone

select capped-cpu ; set ncpus=2.0 ; end

commit ; exit

zoneadm -z s10zone apply # Live

“Capped the rogue zone live in under 5 minutes; other zones returned to normal SLA. RCA identified an application memory-leak loop; permanent fix tracked under Change CHG-12345. Updated runbook so L1/L2 can apply the same cap in future.”

05 Concept Concept Explain the LDOM architecture: Control, Service, Root, and Guest domains. Solaris SME ▾

Answer

Domain	Role	Responsibility
Control	Primary / Manager	Manages all domains via ldm CLI; Hypervisor interface
Service	Virtual I/O Provider	Runs vds, vsw, vcc for guests
Root	PCIe root complex	Direct hardware access, SR-IOV
Guest	Virtual Machine	Separate OS/kernel — isolated crash domain

06 Commands Commands Show the complete LDOM creation workflow for a new guest domain. Solaris SME ▾

Answer

ldm add-domain guestdom1

ldm set-core 4 guestdom1 ; ldm set-mem 8G guestdom1

ldm add-vdisk vdisk0 vol0@primary-vds0 guestdom1

ldm add-vnet vnet0 primary-vsw0 guestdom1

ldm bind guestdom1 ; ldm start guestdom1 ; ldm list

07 Concept Concept How do you perform live migration of an LDOM? Prerequisites? Solaris SME ▾

Answer

Same architecture/firmware compatibility on source and target.
Shared SAN storage accessible from both hosts.
Virtual switch on target; SSH key auth between control domains.
Enough free CPU/RAM on target.

ldm migrate-domain -n guestdom1 targethost # Dry run first

ldm migrate-domain guestdom1 targethost # Live migration

08 Concept Concept Compare Solaris Zones: Global, Non-Global, Kernel, Branded. Solaris SME ▾

Answer

Zone Type	Kernel	Best For
Global Zone	Host OS kernel	Physical host management
Non-Global (NGZ)	Shared with GZ	App isolation, dev/test, multi-tenant
Kernel Zone (11.2+)	Own kernel	Run older Solaris on newer GZ
solaris10 brand	Emulated S10	Legacy app support

zoneadm list -cv ; zlogin webzone

zoneadm -z webzone boot / halt / clone srczone

09 Scenario Scenario SCENARIO: A Solaris zone consumes excessive CPU/memory, impacting other zones. Resolve. Solaris SME ▾

Answer

prstat -Z ; zonestat 1 # Per-zone resource usage

prctl -n zone.cpu-cap -v 200 -i zone zone01 # Immediate cap

# Permanent via zonecfg: capped-cpu / capped-memory

zoneadm -z zone01 apply # Apply without reboot

🔬

6. Kernel Debugging — mdb & DTrace

Kernel panic analysis with mdb dcmds, key DTrace one-liners, and latency-spike scenarios.

01 Critical Critical Walk through a complete kernel panic analysis using mdb. Solaris SME ▾

Answer

ls -lh /var/crash/$(uname -n)/ ; dumpadm

mdb -k /var/crash/host/unix.0 /var/crash/host/vmcore.0

> ::status # System status at crash

> ::panicinfo # Panic string + address

> ::stack # Kernel stack of crash thread

> ::msgbuf # Kernel message buffer

> ::kmastat # Kernel memory allocator stats

> ::modinfo # Loaded modules at crash

02 Commands Commands Most important mdb dcmds you use daily. Solaris SME ▾

Answer

dcmd	Purpose
::status	System info + panic message
::panicinfo	Panic details
::msgbuf	Kernel message buffer
::stack	Stack trace of crash thread
::kmastat	Kernel memory allocator statistics
::modinfo	Loaded kernel modules
0xADDR::print struct	Print C structure at address

03 Commands Commands Key DTrace one-liners for performance and debugging. Solaris SME ▾

Answer

# Syscalls per process

dtrace -n 'syscall:::entry /pid==$1/ { @[probefunc]=count(); }' <PID>

# I/O latency by disk

dtrace -n 'io:::done { @[args[1]->dev_statname]=quantize(args[0]->b_elaptime); }'

# Lock contention

dtrace -n 'lockstat:::adaptive-block { @[caller]=sum(arg1); }'

# Syscall errors

dtrace -n 'syscall:::return /arg1!=0/ { printf("%s: %d", probefunc, errno); }'

04 Scenario Scenario SCENARIO: Application latency spikes every 5-10 min, no obvious OS symptoms. How did you use DTrace? Solaris SME ▾

Answer

prstat -mLa 1 # Watch LWP states during spike

dtrace -n 'lockstat:::adaptive-block { @[stack()]=sum(arg1); } tick-10s { printa(@); }'

# FOUND: single mutex held by GC thread blocking 40 app threads

echo 'set ism_off = 0' >> /etc/system # Enable ISM large pages

Result: 35% latency reduction, GC pause from 800ms → 120ms. Always quantify the outcome.

🗄

7. Oracle ASM Disk Addition

End-to-end ASM disk addition on Solaris, persistent permissions across reboots and CANDIDATE-state troubleshooting.

01 Commands Commands End-to-end process to add a new disk to Oracle ASM on Solaris. Solaris SME ▾

Answer

# Phase 1: OS disk prep

cfgadm -al ; devfsadm -C ; echo | format

# Phase 2: Permissions for oracle

chown oracle:dba /dev/rdsk/c2t10d0s2 ; chmod 660 /dev/rdsk/c2t10d0s2

# Phase 3: Verify from oracle user

dd if=/dev/rdsk/c2t10d0s2 of=/dev/null bs=1M count=10

# Phase 4: Add to diskgroup (DBA coordinates)

SQL> ALTER DISKGROUP DATA ADD DISK '/dev/rdsk/c2t10d0s2' NAME DATA_0004;

asmcmd lsdsk --discovery ; asmcmd lsdg

02 Concept Concept How do you make ASM disk permissions persistent across reboots in Solaris? Solaris SME ▾

Answer

Method	Version	Mechanism
/etc/minor_perm	Solaris 10	sd:block oracle:dba 0660
SMF + devfsadm script	Solaris 11	Manifest sets perms after discovery
/etc/devlink.tab + init.d	Both	Post-mount chown on Oracle devices

03 Critical Critical ASM disk shows CANDIDATE state but isn't being used. Diagnose. Solaris SME ▾

Answer

SQL> SELECT path, mode_status, state, failgroup FROM v$asm_disk;

# CANDIDATE = discovered but not in a diskgroup

SQL> ALTER DISKGROUP DATA ADD DISK '/dev/rdsk/c2t10d0s2' NAME DATA_0004;

# Check ASM alert log + verify disk size vs AU

🔌

8. Networking, Boot, SMF & Provisioning

IPMP, LACP/VLAN/VNIC, the Solaris 11 boot process, SMF service management, and UAR vs JumpStart vs AI provisioning.

01 Concept Concept Explain IPMP. How did you reduce downtime by 40%? Solaris SME ▾

Answer

IPMP (IP Network Multipathing) provides NIC failover and load balancing — Active-Active or Active-Standby.

ipadm create-ipmp ipmp0

ipadm add-ipmp -i net0 ipmp0 ; ipadm add-ipmp -i net1 ipmp0

if_mpadm -d net0 # Test failover

Replaced single-NIC configs with IPMP pairs — previous 2-3hr NIC outages now fail over sub-second. 40% downtime reduction.

02 Commands Commands Configure LACP and VLANs in Solaris 11. Solaris SME ▾

Answer

dladm create-aggr -P L4 -l net0 -l net1 aggr0

dladm create-vlan -l aggr0 -v 100 vlan100

dladm create-vnic -l aggr0 vnic0 # For zones/LDOMs

dlstat -i 1 aggr0

03 Commands Commands Critical SMF commands. How do you troubleshoot a failed service? Solaris SME ▾

Answer

svcs -a ; svcs -xv # All states / explain failures

svcs -xv svc:/network/ssh:default

cat /var/svc/log/network-ssh:default.log

svcadm clear svc:/network/ssh:default # Clear maintenance

svccfg import /var/svc/manifest/site/myapp.xml

04 Concept Concept Explain UAR (Unified Archive) and AI vs JumpStart. How did you cut provisioning 60%? Solaris SME ▾

Answer

archiveadm create -z none /var/tmp/goldenimage.uar

archiveadm create --recovery /var/share/recovery/srv01.uar

# AI (Sol 11): XML manifests + IPS; PXE/OBP boot. Replaces JumpStart (Sol 10).

installadm create-service -n sol11-sparc -s sol11.iso -d /ai/images/sparc

Old: 2hr install + 3hr config = 5hr. UAR golden image = 35min. DR: 4hr → 90min. That is the 60% reduction.

🛠

9. Solaris OS & Kernel — L3 Deep Dive

Boot process (OBP→SMF), Solaris 10 vs 11 differences, OBP recovery, SMF internals, post-patch boot failure, run levels/milestones, /etc/system, crash dump analysis with mdb, coreadm, and kernel memory leak identification.

01 Concept Foundational Explain the Solaris boot process from power-on to login prompt. Solaris L3 ▾

Answer

The Solaris boot process has six well-defined phases. Knowing each phase by name and what runs there is fundamental L3 knowledge.

POST — power-on self-test runs from firmware; tests CPU, memory, devices.
OBP (OpenBoot PROM) on SPARC / BIOS-UEFI on x86 — initialises hardware, executes the FCode boot device and loads the bootblk.
Bootblk — primary bootloader on the disk's first sector; loads the secondary bootloader (ufsboot or zfsboot).
Kernel load — secondary bootloader loads /platform/`uname -i`/kernel/unix and /etc/system; kernel mounts root filesystem.
init / SMF — Solaris 10+ uses SMF; svc.startd takes over and brings the system to the milestone target (multi-user-server by default).
Login prompt — getty/sshd start, the system is ready.

# SPARC — interrupt OBP and examine boot path

ok> printenv boot-device

ok> boot -s # Single-user

ok> boot -m milestone=none # Stop after kernel, no SMF services

ok> boot -F failsafe # Sol 10 failsafe miniroot

# Verify after boot

prtdiag -v ; prtconf -vp | head

svcs -xv # Failed services

tail /var/adm/messages

On Solaris 11 (ZFS root) the bootloader is GRUB (x86) or the SPARC OBP loading bootlst → loader.efi → kernel. Boot Environments via beadm make rollback trivial.

02 Concept Foundational What are the differences between Solaris 10 and Solaris 11? Solaris L3 ▾

Answer

Area	Solaris 10	Solaris 11
Package mgmt	SVR4 (pkgadd, pkgrm, patchadd)	IPS (pkg install/update)
Root FS	UFS or ZFS	ZFS only (mandatory)
Boot loader	SPARC: OBP; x86: GRUB Legacy	SPARC: bootlst/loader; x86: GRUB 2
Network config	ifconfig / /etc/hostname.*	ipadm / dladm (profile-based)
Patch rollback	Live Upgrade (LU)	Boot Environments (beadm)
Zones default brand	native	solaris (S11) — solaris10 brand emulates S10
Service mgmt	SMF	SMF (enhanced)
Provisioning	JumpStart	Automated Installer (AI)
Crypto framework	KCF	KCF + native ZFS encryption
Repository	Patchadd from MOS	IPS repo (publisher)

When the interviewer asks “which is harder to administer” — the honest answer is Solaris 11 is easier day-to-day because of IPS + Boot Environments, but harder to migrate to from S10 because the toolchain has completely changed (ifconfig → ipadm, ndd → dladm, patchadd → pkg).

03 Scenario Advanced How do you troubleshoot a Solaris server stuck at the OBP prompt? Solaris L3 ▾

Answer

Stuck at OBP means the system has not progressed past firmware. Either the boot device is missing/corrupt, the boot path is wrong, or hardware has failed.

First confirm OBP responsiveness — type banner; it should print system info. If not, hardware fault or watchdog.
Check the current boot device: printenv boot-device — is the alias pointing at a valid disk?
Probe attached devices: probe-scsi-all and show-disks — does the OS disk respond?
Try booting from an alternate disk: boot disk1 or by full device path.
Boot from network for rescue: boot net — requires a JumpStart/AI server.
Boot from CD/DVD/USB: boot cdrom -s for a single-user rescue shell, then mount the root pool/UFS and inspect.
If you suspect a corrupt bootblk on SPARC, reinstall it: installboot /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t0d0s0.
Check OBP NVRAM variables: printenv; reset them with set-defaults if corrupted.
If hardware suspect, capture and engage Oracle support with the obdiag output.

# Typical recovery sequence

ok> banner

ok> printenv boot-device

ok> probe-scsi-all

ok> show-disks

ok> setenv boot-device disk0 disk1

ok> boot disk1 -s

Never blindly run set-defaults on a production server — it resets all NVRAM including the boot device list. Record printenv first.

04 Concept Foundational What is SMF (Service Management Facility)? How does it differ from init scripts? Solaris L3 ▾

Answer

SMF is Solaris's service management framework, introduced in Solaris 10. It replaces traditional /etc/rc*.d init scripts with an XML manifest-based system that provides dependency management, automatic restarts, and a unified administration interface.

Aspect	Init scripts	SMF
Dependencies	Order = K/S number	Declarative dependencies in manifest
Failure handling	No restart	Auto-restart with backoff
Status visibility	ps + scripts	svcs unified view
Logs	Mixed locations	/var/svc/log/<svc>.log per service
Configuration	Edit script	svccfg / svcprop
State persistence	None	Maintained in repository

svcs -a # List all services + state

svcs -xv # Explain failed services in detail

svcs -d <fmri> # Dependencies of a service

svcs -D <fmri> # Services that depend on this one

svcadm enable / disable / restart <fmri>

svccfg import /var/svc/manifest/site/myapp.xml

svccfg export <fmri> > backup.xml

tail /var/svc/log/network-ssh:default.log

Service states to memorise: online, offline, disabled, maintenance, degraded, uninitialized. Move out of maintenance with svcadm clear <fmri>.

05 Scenario Advanced How do you recover a Solaris system that fails to boot after patching? Solaris L3 ▾

Answer

This is the textbook reason Boot Environments exist on Solaris 11. On Solaris 10 you rely on Live Upgrade or failsafe boot.

Solaris 11 (ZFS root + beadm)

# From GRUB menu (x86) / OBP bootlst (SPARC), select the previous BE

# OR boot into a working BE manually:

ok> boot -L # Lists available BEs

ok> boot -Z rpool/ROOT/sol11-pre-patch

# Once booted, make the old BE the active one

beadm list

beadm activate sol11-pre-patch-20240315

reboot

Solaris 10 (UFS or ZFS)

# Boot failsafe

ok> boot -F failsafe

# When failsafe finds the root FS, mount it on /a

# Roll back patches:

# patchrm -R /a <patchid>

# Or for Live Upgrade: activate previous BE

luactivate Solaris10_pre-patch

init 6

Always create a backup BE before patching on Solaris 11 (beadm create) or a LU BE on Solaris 10 (lucreate). Mentioning this in the interview demonstrates you actually practice safe patching.

06 Concept Foundational Explain Solaris run levels and milestones. Solaris L3 ▾

Answer

Run Level	SMF Milestone	Purpose
0	milestone/none	Powered down, OBP
S / 1	milestone/single-user	Single user, root only
2	milestone/multi-user	Multi-user, no network FS
3	milestone/multi-user-server	Multi-user + network (default)
5	—	Power off (halt)
6	—	Reboot

who -r # Current run level

svcs milestone/\* # Milestone state

init 3 # Change run level

svcadm milestone milestone/multi-user # Same via SMF

svcadm milestone -d milestone/multi-user-server # Make persistent

Boot to milestone=none when SMF itself is broken — gives you a shell with only PID 1 running. From there you can disable broken services and bring the system up.

07 Concept Foundational What is the purpose of /etc/system? Solaris L3 ▾

Answer

/etc/system is read by the kernel at boot and used to set kernel tunables, force-load modules, and exclude modules. It is the legacy way to tune the Solaris kernel; Solaris 11 increasingly uses project resource controls instead, but /etc/system still applies.

# Common entries

set noexec_user_stack = 1 # Stack non-executable (security)

set rlim_fd_max = 65536 # Hard limit, file descriptors

set rlim_fd_cur = 4096 # Soft limit

set semsys:seminfo_semmni = 4096

set shmsys:shminfo_shmmax = 0xFFFFFFFF

exclude: drv/abc # Don't load this driver

forceload: drv/sd # Force-load a driver early

/etc/system mistakes can prevent boot. Always keep a backup line in /etc/system.bak and use boot -as to load with /dev/null as /etc/system if you've corrupted it: ok> boot -a then specify /dev/null when prompted for system file.

08 Commands Advanced How do you analyze a Solaris crash dump? Solaris L3 ▾

Answer

Solaris saves kernel crash dumps to /var/crash/<hostname>/ via savecore. The dump consists of unix.N (namelist) and vmcore.N (image). You analyze with mdb -k.

# Check savecore configuration

dumpadm # Dump device + savecore dir

ls -lh /var/crash/$(uname -n)/

# Open the crash dump in mdb

mdb -k /var/crash/host/unix.0 /var/crash/host/vmcore.0

> ::status # System info + panic message

> ::panicinfo # Panic details

> ::msgbuf # Kernel message buffer

> ::stack # Stack trace of crash thread

> ::cpuinfo -v # Per-CPU state

> ::kmastat # Kernel memory allocator

> ::modinfo # Loaded modules at crash

> ::findstack -v <thread_addr>

> $q # Quit mdb

The single most useful dcmd is ::msgbuf — it shows the last kernel messages before the panic, which often name the offending driver or subsystem. Always start there.

09 Commands Intermediate What are core files and how do you enable them? Solaris L3 ▾

Answer

Core files are process memory snapshots created when a process receives a fatal signal (SIGSEGV, SIGABRT, etc). They are essential for application debugging. Solaris controls them via coreadm.

coreadm # Show current config

# Enable global core file collection in /var/cores

coreadm -g /var/cores/core.%f.%p.%t \

-e global -e process -e log -e global-setid

coreadm -u # Apply now

# Per-user/session

ulimit -c unlimited

# Inspect a core with mdb or pstack

file /var/cores/core.app.12345.1700000000

pstack /var/cores/core.app.12345.1700000000

mdb /opt/app/bin/app /var/cores/core.app.12345.1700000000

Format placeholders to memorise: %f=executable, %p=PID, %t=epoch, %n=hostname, %u=uid. Naming cores well is half the troubleshooting battle.

10 Scenario Advanced How do you identify kernel memory leaks on Solaris? Solaris L3 ▾

Answer

Start with a long-term trend: kstat -m unix -n system_pages -s freemem over time. A steadily falling freemem with no userland growth = kernel leak suspect.
Per-cache view: mdb -k → ::kmastat. Look for caches whose buf inuse and buf total grow without bound.
Profile allocations: ::kmem_log requires kmem_flags = 0x1f (audit). Enable with mdb -kw → kmem_flags/W 1f (only on test systems — high overhead).
Use DTrace: dtrace -n 'fbt::kmem_alloc:entry { @[stack()] = count(); }' to find hot allocation stacks.
Check known offenders: ZFS ARC (kstat -n arcstats), networking buffers, third-party drivers.
If a specific module is suspected, ::findleaks in mdb runs garbage-collection-style leak detection on the live kernel.

mdb -k

> ::kmastat ! head -40

> ::findleaks -d # Run leak detection

> ::memstat # Memory by usage class

> ::arc # ZFS ARC stats

# ARC can legitimately consume large memory — verify before blaming

echo 'set zfs:zfs_arc_max = 8589934592' >> /etc/system # Cap ARC at 8 GB

Do not assume ZFS ARC growth is a leak — ARC reclaims under memory pressure. The actual leak signal is when ARC stays high AND freemem keeps dropping AND application working set has not changed. Cap ARC only after confirming it is actually the cause.

💿

10. Filesystem & Storage — L3 Deep Dive

UFS vs ZFS, pools and datasets, corrupted pool recovery, snapshots and clones, mirror vs RAID-Z comparison, online filesystem expansion, df-vs-ENOSPC troubleshooting, and stale NFS mount resolution.

01 Concept Foundational Explain UFS vs ZFS. Solaris L3 ▾

Answer

Aspect	UFS	ZFS
Volume mgmt	Separate (SVM, VxVM)	Built-in (zpool)
Integrity	Last-mount fsck	Always-on checksums + scrub
Snapshots	fssnap (limited)	Native, instant, unlimited
Resize	growfs (grow only)	Live grow + dataset quotas
Max FS size	~16 TB	16 EB
Block	Fixed 8 KB	Variable up to 1 MB
RAID	SVM external	RAID-Z1/Z2/Z3 built-in
Compression	None	lz4 / gzip native
Boot	Solaris 10 default	Solaris 11 only root option

UFS is rare in greenfield environments today — you encounter it on legacy Solaris 10 root filesystems. ZFS is the default and expected answer for storage architecture questions.

02 Concept Foundational What are ZFS pools and datasets? Solaris L3 ▾

Answer

Pool (zpool) — a storage pool built from one or more vdevs (mirror, raidz, etc). The pool provides aggregated capacity and redundancy. Example: tank.
Dataset — a manageable unit carved from the pool. Types: filesystem (mounted), volume (zvol, block device), snapshot (read-only point-in-time), clone (writable copy of a snapshot).
Datasets inherit properties from their parents (compression, quota, mountpoint).
Naming is hierarchical: tank/data, tank/data/db.

zpool create tank mirror c0t1d0 c0t2d0

zfs create tank/data # Filesystem dataset

zfs create -V 20G tank/oradata # Volume (block device)

zfs create -o compression=lz4 -o quota=100G tank/db

zfs list -t all # List FS, vols, snapshots

zfs get all tank/data # All inherited properties

03 Scenario Advanced How do you recover a corrupted ZFS pool? Solaris L3 ▾

Answer

Run zpool status -v to identify the type of corruption — DEGRADED (redundancy lost but readable), FAULTED (unusable), or specific damaged files.
If checksum errors exist on a healthy pool, run zpool scrub <pool> to verify and self-heal from redundant copies.
If a vdev is FAULTED but the pool is importable: zpool clear <pool> to acknowledge transient errors after fixing the underlying issue.
If a disk replacement is needed: zpool replace <pool> <old-disk> <new-disk>; monitor resilver progress.
If the pool will not import: zpool import -fF <pool> attempts forced import with rewind to the last consistent transaction group.
Last resort for catastrophic damage: zpool import -fFX <pool> (extreme rewind) — discards recent writes; use only after taking advice/backup.
If specific files are corrupted, restore them from snapshots or backup; the file paths are listed in zpool status -v.

zpool status -v tank

zpool scrub tank ; zpool status tank # Watch progress

zpool clear tank # Clear errors after fix

zpool import # List importable pools

zpool import -fF tank # Force + rewind

zpool history tank # Full audit trail of pool

-X (extreme rewind) can roll back hours of committed writes. Always engage Oracle Support before using it in production, and take a snapshot of the disks (if possible) for forensic recovery.

04 Commands Intermediate How do snapshots and clones work in ZFS? Solaris L3 ▾

Answer

A snapshot is a read-only, point-in-time view of a dataset. It is nearly instant and initially consumes no extra space — it grows only as the original dataset diverges. A clone is a writable filesystem created from a snapshot; it shares blocks with the parent until written to (copy-on-write).

# Snapshot — for backup, rollback, replication

zfs snapshot tank/db@before-upgrade

zfs list -t snapshot tank/db

# Rollback to a snapshot (destroys later snapshots)

zfs rollback -r tank/db@before-upgrade

# Clone — writable copy from a snapshot, for dev/test

zfs clone tank/db@before-upgrade tank/db-test

# Promote a clone (swaps the dependency, so the clone can outlive the source)

zfs promote tank/db-test

# Send/recv for replication

zfs send tank/db@snap1 | ssh dr 'zfs recv tankdr/db'

zfs send -i tank/db@snap1 tank/db@snap2 | ssh dr 'zfs recv tankdr/db'

Snapshots used as rolling backups: keep e.g. 24 hourly + 7 daily + 4 weekly + 12 monthly using zfs-auto-snapshot SMF service. Combined with zfs send -i incremental replication, you get an extremely robust DR setup at near-zero cost.

05 Concept Foundational What is the difference between ZFS mirror and RAID-Z? Solaris L3 ▾

Answer

Aspect	Mirror	RAID-Z1	RAID-Z2	RAID-Z3
Redundancy	N-1 disks	1 disk	2 disks	3 disks
Min disks	2	3	4	5
Write perf	Best (parallel)	Lower (parity calc)	Lower	Lowest
Read perf	Best (split reads)	Sequential good	Sequential good	Sequential good
Usable %	50% (2-way)	(N-1)/N	(N-2)/N	(N-3)/N
Best for	DB, low-latency	General storage	Production data	Archive, very large arrays
Resilver speed	Fast (1 disk copy)	Slow (full parity)	Slowest	Slowest

Common rule of thumb: databases get mirrors, file storage gets RAID-Z2. RAID-Z1 is acceptable for small arrays but risky on modern large disks because a second failure during rebuild becomes likely.

06 Commands Intermediate How do you expand a filesystem online? Solaris L3 ▾

Answer

ZFS — automatic

zpool add tank mirror c2t0d0 c2t1d0 # Add a mirror vdev — pool grows

zpool set autoexpand=on tank # New disk capacity used automatically

zpool online -e tank c0t1d0 # Expand single disk after array grow

zfs set quota=500G tank/data # Adjust per-dataset quota

UFS — growfs (works only on growing direction)

# 1. Grow the underlying volume (SVM example)

metattach d10 c0t5d0s7

# 2. Grow the UFS in-place

growfs -M /export/data /dev/md/rdsk/d10

UFS can ONLY grow, never shrink. ZFS pools can only add vdevs, never remove (with some exceptions for mirrors via zpool remove). Plan capacity carefully — a wrong pool layout is permanent.

07 Scenario Advanced How do you troubleshoot filesystem full issues when df shows free space? Solaris L3 ▾

Answer

A classic L3 scenario: df shows 30% free but the application fails with ENOSPC. Several causes — diagnose systematically.

Inode exhaustion (UFS): df -F ufs -o i /mountpoint. If %iused is 100%, you have inode starvation — too many small files. Solution: delete files or reformat with more inodes (newfs -i).
Reserved space (UFS): UFS reserves 10% by default for root. Non-root writes fail before df shows full. Check with tunefs -p, reduce with tunefs -m 5 if needed.
Open deleted files: a process holds a file open after rm. Space is not reclaimed until the process closes the file or restarts. Find with lsof | grep deleted or pfiles <pid>.
ZFS quotas/reservations: zfs get used,quota,reservation tank/data. A dataset may be at quota even if the pool has free space.
ZFS snapshot space: snapshots hold blocks. zfs list -t snapshot -o name,used,refer — old snapshots consume real space.
Pool fragmentation / metaslab issues (ZFS): zpool get fragmentation tank. Above ~80% fragmentation, writes slow drastically even with free space.

# Find deleted-but-open files

lsof 2>/dev/null | grep deleted

pfiles $(pgrep app) | grep -i deleted

# Inode usage (UFS)

df -F ufs -o i /export/data

# ZFS dataset-level

zfs list -o name,used,available,quota,reservation

zfs list -t snapshot -o name,used | sort -hk2

On Solaris, deleted-but-open files are the #1 source of “disk full but df is fine” tickets — particularly with application servers that hold rotated logs open. lsof | grep deleted is your first reflex.

08 Scenario Advanced What are stale NFS mounts and how do you resolve them? Solaris L3 ▾

Answer

A stale NFS file handle occurs when a file or directory on the NFS server is removed/replaced while a client still holds a handle to it. The client gets ESTALE errors. A hung mount happens when the NFS server is unreachable and the client is using a hard mount.

Identify the stuck mount: df hangs, mount shows it. Use nfsstat -m on the client for mount stats.
Check NFS server reachability: ping nfs-srv, rpcinfo -p nfs-srv, showmount -e nfs-srv.
On the client side, processes blocked on the mount appear in 'D' state and can't be killed easily.
Try a forced lazy unmount: umount -f /mnt. On Solaris, umount -f exists but is less reliable; sometimes you need to kill all processes touching the mount first.
Find stuck processes: fuser -c /mnt lists PIDs; fuser -ck /mnt kills them (use carefully).
If the mount is completely wedged and umount fails: identify and restart the NFS server, OR reboot the client as a last resort.
Preventive: use soft,intr,timeo=300,retrans=2 for non-critical mounts; hard,intr for critical data; always use NFS server's HA virtual IP, never physical.

# Diagnose

nfsstat -m

showmount -e nfs-srv

timeout 5 ls /mnt # Doesn't hang the shell

fuser -c /mnt # Processes using the mount

# Forced unmount (with caution)

fuser -ck /mnt && umount -f /mnt

hard mounts WILL hang processes indefinitely if the server is down — but they guarantee data integrity. soft mounts return errors after timeout — fast but can silently corrupt writes. Critical data ALWAYS uses hard,intr.

⚡

11. Performance — L3 Deep Dive

Slow server methodology, vmstat/iostat/mpstat/prstat column-by-column, CPU bottleneck identification, memory pressure signals, swap reservation model, and high load average (CPU vs I/O distinction).

01 Scenario Advanced A Solaris server is slow. How will you troubleshoot? Solaris L3 ▾

Answer

Methodical layered investigation — CPU → memory → I/O → network → application — and always correlate with recent changes.

Baseline first: uptime for load, prstat -a 1 5 for top processes and zones.
CPU saturation? mpstat 1 5 — per-CPU %sys, %usr, %wt. Sustained %sys high = kernel/driver; %usr high = app loop.
Run queue? vmstat 1 5 — r column > CPU count means contention. Also watch w (waiting on resource).
Memory pressure? vmstat 1 — sr (scan rate) > 0 sustained = memory shortage; pages being scanned for eviction.
Swap activity? vmstat 1 si/so > 0 = active paging. Bad.
Disk I/O? iostat -xnz 1 5 — %b (busy) > 80, asvc_t (service time) > 30ms = storage bottleneck.
Network? netstat -i 1, dlstat, nicstat — drops, errors, saturation.
Application? truss -p <pid> for syscall traces; pstack <pid> for stack; DTrace for production-safe deep dive.
Recent changes? showrev -p | tail, pkg history, last reboot, app deploy history. 80% of “sudden slow” incidents tie to a change.

Use prstat -mLa for microstate accounting — it shows what each LWP is actually doing (USR/SYS/TRP/TFL/DFL/LCK/SLP/LAT/STP). A high LAT means waiting for CPU (run queue); high LCK means lock contention. This is the L3 differentiator.

02 Commands Intermediate Explain outputs of vmstat, iostat, mpstat, prstat. Solaris L3 ▾

Answer

vmstat — system-wide virtual memory + CPU

vmstat 1 5

# Key columns:

# r — run queue length (>= CPU count = contention)

# b — blocked on I/O (high = storage wait)

# w — swapped-out procs (should be 0)

# swap — free swap space

# free — free memory (KB)

# sr — page scan rate (>0 sustained = memory pressure)

# si/so — swap in/out (>0 = paging, BAD)

# us/sy/id — user/system/idle CPU

iostat — per-disk I/O

iostat -xnz 1 5

# -x extended, -n names, -z skip idle, -e errors

# Key columns:

# r/s w/s — reads/writes per second

# kr/s kw/s — KB/sec

# wait — # of I/Os waiting in queue

# actv — # of I/Os being serviced

# wsvc_t — wait service time (ms)

# asvc_t — active service time (ms) — >30ms = SLOW

# %w — % time queue not empty

# %b — % time busy (>80 = saturated)

mpstat — per-CPU breakdown

mpstat 1 5

# CPU — CPU id

# minf/mjf — minor/major faults

# xcal — cross-calls (high = inter-CPU communication, suspicious)

# intr/ithr — interrupts (high = NIC/HBA activity)

# csw/icsw — context switches / involuntary

# smtx — spinlock acquisitions on mutex

# srw — spinlock on r/w lock

# usr/sys/idl — per-CPU CPU usage

prstat — Solaris's top, but better

prstat -a # By user + zone summary

prstat -Z # By zone

prstat -L # Per LWP

prstat -mLa # MICROSTATE — the L3 view

# microstate columns:

# USR SYS TRP TFL DFL LCK SLP LAT — must sum to 100

# LAT high = waiting for CPU (run queue contention)

# LCK high = lock contention

# DFL high = data fault, paging

03 Commands Intermediate How do you identify CPU bottlenecks? Solaris L3 ▾

Answer

System-wide: vmstat 1 — r column sustained above CPU count = run-queue saturation.
Per-CPU: mpstat 1 — find imbalanced CPUs (one pegged, others idle = single-threaded bottleneck).
Per-process: prstat -mLa 1 — find threads with high USR or SYS or low LAT.
Drill-down USR-heavy: profile with DTrace — dtrace -n 'profile-997 /pid==<PID>/ { @[ustack()]=count(); }'
Drill-down SYS-heavy: which syscalls? dtrace -n 'syscall:::entry /pid==<PID>/ { @[probefunc]=count(); }'
Lock contention: lockstat -A sleep 5 for kernel locks; plockstat -A <PID> for userland.

# Quick triage one-liners

mpstat 1 5 | awk '$NF<20 {print}' # Show CPUs <20% idle

prstat -mLa 1 1 | head -25

psrinfo -v # CPU physical/virtual layout

cpustat -c PAPI_tot_cyc 1 # Hardware counters

04 Commands Intermediate How do you identify memory pressure? Solaris L3 ▾

Answer

Free memory low: vmstat 1 — free below lotsfree threshold triggers scanning.
Scan rate sustained: vmstat 1 sr column > 0 over time = kernel scanning pages to reclaim. Hard threshold.
Paging: vmstat 1 pi/po column > 0 means active paging — system is short on RAM.
Per-process growth: prstat -s rss sorts by resident set size; check for runaway leaks.
Microstate DFL: prstat -mLa — high DFL (data fault) = thread waiting on page-ins.
ZFS ARC: kstat -n arcstats or ::arc in mdb; ARC can dominate but is reclaimable.
OOM-style events: check /var/adm/messages for “not enough memory” or process kills.

vmstat 1 10 # Watch sr, pi, po, free

swap -s ; swap -l # Swap usage + devices

prstat -s rss # Top by RSS

prstat -s size # Top by VSZ

echo '::memstat' | mdb -k # Memory by usage class

echo '::arc' | mdb -k # ZFS ARC stats

05 Concept Intermediate What is swap reservation in Solaris? Solaris L3 ▾

Answer

Solaris uses an anonymous memory reservation model. When a process allocates memory (e.g. malloc), Solaris reserves enough swap space to back it — even if not yet paged out. This is different from Linux's overcommit model.

Total virtual memory = physical RAM + swap space.
Each anonymous allocation reserves backing store from this pool.
If total reservation exceeds available swap+RAM, allocation fails (no OOM killer like Linux).
Solaris swap is usually small (1×–2× RAM); reservation primarily uses RAM.

swap -s # Reservation summary

# Output: total: used + available = capacity

swap -l # Configured swap devices

swap -a /dev/zvol/dsk/rpool/swap2 # Add a ZFS swap

swap -d /dev/zvol/dsk/rpool/swap2 # Remove (only if unused)

When applications fail with “not enough space” under low load, check swap -s — you may have reserved all swap even though RAM is free. Solaris will refuse new allocations rather than OOM-kill an existing process.

06 Scenario Advanced How would you troubleshoot high load average? Solaris L3 ▾

Answer

Load average on Solaris is the sum of the run queue and processes waiting on disk I/O — not just CPU. So a load of 20 on a 4-CPU box could be CPU-bound, I/O-bound, or a mix.

Confirm load: uptime, w — 1/5/15-minute averages.
Distinguish CPU vs I/O: vmstat 1 — r column is run queue (CPU); b is blocked-on-I/O.
If r dominates: CPU saturation — drill with mpstat + prstat -mLa.
If b dominates: storage waits — drill with iostat -xnz 1 looking for %b > 80 and high asvc_t.
Mix of both: usually one is primary; identify by which is highest more often.
Check for fork bombs / process explosion: ps -ef | wc -l, watch with prstat for sudden process count growth.
Check for runaway zones (in global zone): prstat -Z shows per-zone CPU.
Investigate top consumers; if legitimate workload, that's a capacity problem, not a tuning problem.

🌐

12. Networking — L3 Deep Dive

IPMP modes and commands, network latency triage, logical vs physical interfaces, VLAN configuration (including the S10 PPA gotcha), and NFS performance diagnosis.

01 Concept Foundational Explain IPMP (IP Network Multipathing). Solaris L3 ▾

Answer

IPMP provides NIC failover and load spreading at the IP layer. Multiple physical interfaces are grouped; if one fails, traffic continues on others. Solaris 11 uses link-based failure detection (via dladm) by default, with optional probe-based detection.

Mode	Behavior	Use Case
Active-Active	All interfaces carry traffic; load spread by hash	Throughput-critical
Active-Standby	Standby idle until active fails	Strict failover semantics
Link-based detection	Detects link down (cable, NIC)	Default; fast
Probe-based detection	Pings a target on each interface	Detects upstream switch failures

# Solaris 11 IPMP creation

ipadm create-ipmp ipmp0

ipadm add-ipmp -i net0 -i net1 ipmp0

ipadm create-addr -T static -a 10.0.0.10/24 ipmp0/v4

# Monitor

ipmpstat -i # Per-interface state

ipmpstat -g # Group state

ipmpstat -t # Targets (probe mode)

ipmpstat -p # Probe statistics

# Test failover

if_mpadm -d net0 # Take net0 offline

if_mpadm -r net0 # Restore

IPMP failover is sub-second with link-based detection — quote this in interviews: “Replacing a single-NIC config with IPMP cut our NIC-related outages from 2–3 hours of human intervention to sub-second automatic failover.”

02 Scenario Advanced How do you troubleshoot network latency? Solaris L3 ▾

Answer

Confirm symptoms: end-to-end ping (ping -s), then progressive hops (traceroute).
Local NIC health: dladm show-phys, dladm show-link -s for stats, kstat -n <link> for errors/drops.
Driver stats: nicstat 1 — packets/sec, MBPS, utilisation. High %Util = saturation.
TCP state: netstat -s | grep tcp — retransmits, out-of-order, duplicate ACKs.
Active connections: netstat -an; ss -t in S11. Look for many CLOSE_WAIT/FIN_WAIT.
Packet capture if needed: snoop -d net0 -o /tmp/cap host x.x.x.x; analyze with snoop -i or Wireshark.
DNS: nslookup, dig; slow DNS resolves look like 5-second hangs.
Check IPMP probe statistics for upstream switch issues.

ping -s host 1400 5 # 5 pings, 1400-byte payload

traceroute host

nicstat 1 5

netstat -s | egrep 'tcpRetrans|tcpInDup'

dladm show-link -s -i 1

snoop -d net0 -c 100 host x.x.x.x

TCP retransmissions visible in netstat -s are a strong signal of network loss. tcpRetransSegs as a percentage of tcpOutSegs > 0.5% indicates a real problem somewhere on the path.

03 Concept Foundational Difference between logical and physical interfaces. Solaris L3 ▾

Answer

Physical interface — a NIC port (e.g. net0, e1000g0, ixgbe0). Has a MAC, one link state.
Logical interface — an additional IP address bound on top of a physical or virtual interface (e.g. net0:1, net0:2). Same MAC; multiple IPs.
On Solaris 11 with ipadm, logical interfaces are simply additional addresses on the same IP interface — distinguished by name suffix like net0/v4a.
Use cases: virtual IPs for cluster failover, multi-homing, alias for service migration.

# Solaris 10 — ifconfig logical interface

ifconfig e1000g0:1 plumb 10.0.0.20 netmask + up

# Solaris 11 — ipadm

ipadm create-addr -T static -a 10.0.0.20/24 net0/v4secondary

ipadm show-addr

04 Commands Intermediate Explain VLAN configuration in Solaris. Solaris L3 ▾

Answer

VLANs let you put a single physical NIC on multiple IEEE 802.1Q tagged networks. Solaris creates a VLAN interface on top of the physical link.

# Solaris 11 — preferred dladm/ipadm method

dladm create-vlan -l net0 -v 100 vlan100

ipadm create-ip vlan100

ipadm create-addr -T static -a 10.100.0.5/24 vlan100/v4

# Verify

dladm show-vlan

dladm show-link

ipadm show-addr

# Solaris 10 legacy — PPA encoding

ifconfig e1000g100000 plumb 10.100.0.5 netmask 255.255.255.0 up

# Format: <driver><instance × 1000 + vlan_id>

The Solaris 10 PPA encoding (1000×instance + vlan) is the classic interview trap. e1000g100000 means instance 0, VLAN 100. e1000g1100 means instance 1, VLAN 100.

05 Scenario Advanced How do you diagnose NFS performance issues? Solaris L3 ▾

Answer

Server vs client side: nfsstat -s on server, nfsstat -c on client.
Look at badcalls, retrans, timeouts — non-zero values point to network or server overload.
Server load: is the NFS server itself CPU- or I/O-bound? Use the perf methodology there.
Network path: ping latency client→server should be sub-millisecond on a LAN. Use nicstat on both ends.
Mount options: NFSv3 vs v4, rsize/wsize (default 1 MB is good for modern networks; older systems may have 8 KB).
nfsstat -m on client shows actual negotiated read/write sizes and timing buckets.
If NetApp/array-backed: check array stats; the bottleneck may be downstream of the NFS server itself.
dtrace -n 'nfsv4:::op-*-start { @[probename] = count(); }' profiles op mix.

nfsstat -m # Per-mount stats with timing

nfsstat -c -v 3 # Client NFSv3 stats

nfsstat -s -v 4 # Server NFSv4 stats

share -a # Server exports

dfshares -s nfs-srv # See what server exports

If nfsstat -m shows the timing distribution skewed toward longer buckets (>= 80ms), the server is overloaded. Sub-10ms buckets = healthy LAN NFS.

🖥

13. Solaris Zones — L3 Deep Dive

Zone concepts (GZ/NGZ/Sparse/Whole/Kernel), full creation workflow, zone states, clone and migrate, stuck/failing boot triage, live resource adjustment, corrupted zone recovery, networking inside zones, and three scenario-based questions.

01 Concept Foundational What are Solaris Zones? Solaris L3 ▾

Answer

Solaris Zones are OS-level virtualization — multiple isolated user-space environments sharing a single kernel. Each zone has its own filesystem, process namespace, network identity, and root user, but all run on one Solaris kernel instance. Lightweight (no hypervisor overhead) and high-density (hundreds per host).

Process isolation — zone A cannot see or signal zone B's processes.
Filesystem isolation — zonepath is the root; usually a ZFS dataset.
Network isolation — exclusive-IP zones get their own stack; shared-IP share the global zone's.
Resource isolation — capped CPU, memory, swap; FSS shares; dedicated CPUs.
Branding — a brand defines the runtime environment (solaris, solaris10, lx, etc).

02 Concept Foundational Difference between Global Zone and Non-Global Zone. Solaris L3 ▾

Answer

Aspect	Global Zone (GZ)	Non-Global Zone (NGZ)
Role	The host OS itself	A guest user environment
Privileges	Full system access	Restricted; can't see other zones
Kernel	Owns the kernel	Shares GZ's kernel
Hardware access	Direct	Through GZ
zoneadm visibility	Sees all zones	Sees only itself
Patch/upgrade	Updates kernel + all NGZs	Updates only NGZ's apps
Filesystems	All physical	Only the zonepath

Only the global zone can administer zones (zoneadm, zonecfg). A non-global zone's root cannot install a kernel module, change the system clock, or affect another zone.

03 Concept Foundational Difference between Sparse Root and Whole Root Zones. Solaris L3 ▾

Answer

These are Solaris 10 concepts. They define how the zone's filesystem layout relates to the global zone.

Aspect	Sparse Root	Whole Root
/usr, /lib, /sbin, /platform	Inherited read-only from GZ via loopback	Own private copy
Disk space	Minimal (~100 MB)	Larger (~5 GB)
Patching	Patch GZ; sparse zones get updates automatically	Each zone patched independently
Customization	Cannot modify inherited paths	Full control of /usr et al
Boot time	Faster	Normal
Use case	Dense, identical app servers	Custom builds, third-party tools in /usr

Solaris 11 zones are always whole-root by default; sparse zones were deprecated. If you say “sparse vs whole” referring to a Solaris 11 question, clarify you're describing the legacy concept.

04 Concept Foundational What is a Kernel Zone? Solaris L3 ▾

Answer

A Kernel Zone (introduced in Solaris 11.2) runs its own independent kernel — a true hypervisor-like isolation while still using the Solaris zones management tooling. Brand name: solaris-kz.

Own kernel and SMF — patch the kernel zone independently of the global zone.
Live migration between hosts is supported (cold and warm).
Different Solaris versions can run as KZ guests (within compatibility).
Higher resource overhead than NGZ but lower than full hypervisor VMs.
Requires SPARC T-series or Intel/AMD VT-x with EPT.

zonecfg -z kz1 create -t SYSsolaris-kz

zoneadm -z kz1 install

zoneadm -z kz1 boot

zonecfg -z kz1 info # Note brand=solaris-kz

05 Concept Intermediate Advantages of Zones compared to VMware virtualization. Solaris L3 ▾

Answer

Aspect	Solaris Zones	VMware VMs
Overhead	Near-zero (shared kernel)	Significant (own OS per VM)
Density	100s per host	10s per host
Boot time	Seconds	Minutes
OS diversity	Solaris (or solaris10/lx brand)	Any OS
Patching	GZ patch updates kernel for all	Per-VM patching
Live migration	Kernel Zones only	Full vMotion support
Resource granularity	CPU shares, caps	vCPU + RAM allocation
License model	Free with Solaris	Per-host/socket commercial

Zones win on density and efficiency for homogeneous Solaris workloads. VMware wins for OS diversity and mature operational tooling. The correct architectural answer is “it depends on the workload” — explicitly call out workload type in interviews.

06 Task Intermediate How do you create a new zone? Complete workflow. Solaris L3 ▾

Answer

# Step 1 — Plan and create the zone configuration

zonecfg -z appzone

create # Use SYSdefault template

set zonepath=/zones/appzone

set autoboot=true

set ip-type=exclusive # Or shared

add net

set physical=vnic1

end

add capped-memory

set physical=4G

set swap=8G

end

add capped-cpu

set ncpus=2.0

end

verify # Check the config

commit

exit

# Step 2 — Install

zoneadm -z appzone install

# Step 3 — Boot and configure the OS interactively

zoneadm -z appzone boot

zlogin -C appzone # Console — first-boot SCI

# Step 4 — Verify

zoneadm list -cv

For exclusive-IP zones, create a VNIC in the global zone first: dladm create-vnic -l net0 vnic1. Shared-IP zones reuse the GZ's IP stack — simpler but less isolated.

07 Concept Foundational Explain zone states: Configured, Installed, Ready, Running. Solaris L3 ▾

Answer

State	Meaning	How To Get There
configured	Definition exists in /etc/zones, not installed	zonecfg ... commit
incomplete	Install in progress or interrupted	Failed install left this state
installed	Files installed under zonepath, not booted	zoneadm install
ready	Resources allocated (devices/network), not booted	zoneadm ready
running	Booted, services running	zoneadm boot
shutting_down	Stopping	zoneadm shutdown
down	Halted but resources retained	Transition only
mounted	Read/write access from GZ for upgrade	zoneadm -z X mount

zoneadm list -cv # Show all zones + state

zoneadm -z appzone ready # Allocate resources without boot

zoneadm -z appzone halt # Force stop to installed

zoneadm -z appzone uninstall # Back to configured

08 Commands Foundational Commands used: zonecfg, zoneadm, zlogin. Solaris L3 ▾

Answer

Command	Purpose	Typical Usage
zonecfg	Define / edit zone configuration	zonecfg -z <zone>
zoneadm	Lifecycle (install/boot/halt/clone)	zoneadm -z <zone> boot
zlogin	Log into / execute in zone	zlogin -C <zone> for console
zonename	Print current zone name	zonename
zonestat	Live per-zone resource stats	zonestat 1

# zonecfg subcommands inside the interactive prompt:

zonecfg:appzone> info # Show config

zonecfg:appzone> add net ; set physical=net0 ; end

zonecfg:appzone> select capped-cpu ; set ncpus=4 ; end

zonecfg:appzone> verify

zonecfg:appzone> commit

# zlogin modes

zlogin appzone # Login as root, fast

zlogin -C appzone # Console (incl. first-boot SCI)

zlogin -l user appzone # As a specific user

zlogin appzone svcs -xv # One-shot command

09 Task Intermediate How do you clone a zone? Solaris L3 ▾

Answer

Cloning makes an identical copy of a zone — useful for rapid deployment of standard images or test environments. On ZFS-backed zones it uses snapshot+clone semantics — instant and space-efficient.

# Step 1 — Source zone must be installed (and ideally halted)

zoneadm -z srczone halt

# Step 2 — Configure new zone from source

zonecfg -z newzone create -t srczone

zonecfg -z newzone set zonepath=/zones/newzone

zonecfg -z newzone # Edit IP, hostname, etc to differ

# Step 3 — Clone the installed image

zoneadm -z newzone clone srczone

# Step 4 — Boot — first boot will re-run SCI to take new identity

zoneadm -z newzone boot

zlogin -C newzone

On ZFS, cloning is a snapshot-clone operation — instantaneous and copy-on-write. You can clone a 50 GB zone in seconds with negligible space used until divergence.

10 Task Advanced How do you migrate a zone to another server? Solaris L3 ▾

Answer

Solaris zones support cold migration via detach/attach. Live migration is supported only for Kernel Zones (solaris-kz brand).

# === COLD MIGRATION (detach/attach) ===

# 1. On source — shut down the zone

zoneadm -z appzone shutdown

# 2. Detach the zone (marks it as transferable)

zoneadm -z appzone detach

# 3. Archive zonepath and zonecfg export

zonecfg -z appzone export -f /tmp/appzone.cfg

cd /zones && tar cf - appzone | gzip > /tmp/appzone.tar.gz

# 4. Transfer to target

scp /tmp/appzone.tar.gz /tmp/appzone.cfg targethost:/tmp/

# 5. On target — restore

cd /zones && gzcat /tmp/appzone.tar.gz | tar xf -

zonecfg -z appzone -f /tmp/appzone.cfg

# 6. Validate compatibility, then attach

zoneadm -z appzone attach -u # -u runs update on attach

# 7. Boot

zoneadm -z appzone boot

# === LIVE MIGRATION (Kernel Zones only) ===

zoneadm -z kz1 migrate ssh://target-host

For cold migration, ZFS replication (zfs send | zfs recv) is far faster than tar+scp for large zones. Use zfs snapshot + zfs send to transfer the zonepath dataset directly.

11 Scenario Advanced A zone is stuck in 'installed' state. How will you troubleshoot? Solaris L3 ▾

Answer

installed means files exist on disk but the zone won't transition to ready/running. Common causes: missing devices, network config errors, filesystem permission issues, brand incompatibility.

Try a controlled boot and capture errors: zoneadm -z appzone -v boot (verbose).
Check the zone install log: /var/log/zones/zoneadm.<date>.<zone>.log.
Verify the configuration is valid: zonecfg -z appzone verify.
Check network resources exist: dladm show-link — does the VNIC/datalink referenced in the zonecfg exist?
Check zonepath: ls -ld /zones/appzone — must exist, owned root, mode 700.
If ZFS-backed: zfs list | grep appzone — is the dataset mounted?
Test ready state explicitly: zoneadm -z appzone ready — what error?
Brand mismatch: if the source was a different OS version, you may need zoneadm -z appzone attach -u to update.
Last resort: zoneadm -z appzone uninstall + reinstall after fixing root cause. Backup any data first.

zoneadm -z appzone -v boot 2>&1 | tee /tmp/boot.log

tail -50 /var/log/zones/zoneadm.*appzone*

zonecfg -z appzone info

zonecfg -z appzone verify

dladm show-vnic

12 Scenario Advanced A zone fails to boot. Where do you check logs? Solaris L3 ▾

Answer

Log	Path	Use For
zoneadm operations	/var/log/zones/zoneadm.<date>.<zone>.log	install/attach/boot operations
zone console	zlogin -C <zone>	Boot console (services starting)
Per-zone messages	<zonepath>/root/var/adm/messages	System log inside the zone
GZ messages	/var/adm/messages	Zone-related kernel events visible from GZ
SMF per-service	<zonepath>/root/var/svc/log/...	Failed services inside zone
FMA	fmadm faulty	Hardware/resource faults

ls -lrt /var/log/zones/ | tail

tail -100 /var/log/zones/zoneadm.*appzone*

zlogin -C appzone # Watch console output during boot

tail /zones/appzone/root/var/adm/messages

# If SMF is the problem:

zlogin appzone svcs -xv

Connect to the zone console with zlogin -C BEFORE the boot command in a separate window — that way you see boot output live, including kernel messages and SCI prompts on first boot.

13 Task Intermediate How do you increase memory or CPU for a running zone? Solaris L3 ▾

Answer

Most resource caps can be adjusted live with zoneadm apply; a few require a zone reboot. The pattern is: edit with zonecfg → apply or reboot.

# Increase memory cap (live)

zonecfg -z appzone

select capped-memory

set physical=16G

set swap=32G

end

commit ; exit

# Apply live without reboot (Solaris 11)

zoneadm -z appzone apply

# Increase capped-CPU (live)

zonecfg -z appzone

select capped-cpu ; set ncpus=8 ; end

commit ; exit

zoneadm -z appzone apply

# OR — immediate runtime cap via prctl (no zonecfg edit)

prctl -n zone.cpu-cap -v 800 -i zone appzone # 800% = 8 CPUs

rcapadm -E -z appzone -m 16g # Memory cap

prctl/rcapadm changes are runtime-only — they revert on zone reboot. Always edit zonecfg for persistence, then zoneadm apply to push the change live.

14 Scenario Advanced How do you recover a corrupted zone? Solaris L3 ▾

Answer

Try to attach for repair from the global zone: zoneadm -z appzone mount — gives you read/write access to <zonepath>/root from the GZ.
Inspect and repair: check /etc/passwd, /etc/shadow, /etc/system, broken symlinks. Edit from the GZ.
If SMF is corrupt inside the zone: svccfg -s - repair manifests, or restore from a snapshot.
If filesystem corruption: for UFS-backed zones fsck the zonepath; for ZFS run zpool scrub.
ZFS rollback if you have a snapshot: zfs rollback rpool/zones/appzone/rpool/ROOT/solaris@before-bad-change.
Worst case: clone from a known-good zone, restore data into the new one, retire the corrupted one.
Always take a snapshot BEFORE attempting repair — your first failed repair attempt is data you may need.

zfs snapshot -r rpool/zones/appzone@before-repair

zoneadm -z appzone mount

ls /zones/appzone/root/etc/

# ... do repair work ...

zoneadm -z appzone unmount

zoneadm -z appzone boot

15 Scenario Advanced How do you troubleshoot networking issues inside a zone? Solaris L3 ▾

Answer

First determine IP-type: zonecfg -z <zone> info ip-type. Troubleshooting paths differ for shared vs exclusive.
Shared-IP: the zone shares the GZ's stack — most troubleshooting happens in the GZ.
Exclusive-IP: the zone has its own stack. Run ipadm, netstat INSIDE the zone via zlogin.
Datalink visible: in GZ, dladm show-vnic — is the zone's VNIC up and on the correct underlying NIC?
Inside zone: ipadm show-addr, ipadm show-if, route -n, netstat -rn.
Ping in stages: zone → GZ (anycast same VLAN), zone → default gateway, zone → external. Identify where it stops.
DNS: cat /etc/resolv.conf inside zone; nslookup somehost.
Firewall: svcs -a | grep ipfilter in both GZ and zone. ipf rules can block traffic invisibly.
Packet capture: snoop on the GZ's underlying interface, filter by VLAN/zone IP — useful when you suspect traffic isn't leaving the host.

# In GZ

zonecfg -z appzone info net

dladm show-vnic

dladm show-link -s # Link errors/drops

# In zone

zlogin appzone

appzone# ipadm show-addr

appzone# route -n

appzone# netstat -in

appzone# ping default-gateway

# Snoop from GZ for an exclusive-IP zone

snoop -d net0 -o /tmp/cap host 10.0.0.50

16 Scenario Advanced SCENARIO: Users report application downtime inside a zone while Global Zone is healthy. What will you check? Solaris L3 ▾

Answer

Confirm the zone is running: zoneadm list -cv from GZ.
Log in: zlogin <zone>. Check uptime, load, recent boots: uptime ; who -b.
Application service state: svcs -xv inside zone — failed services?
Application logs: /var/log or app-specific log path; look for errors aligned with the report time.
Resource caps: from GZ prstat -Z 1 5 — is the zone hitting its cpu-cap or capped-memory? rcapstat 1 shows memory cap enforcement.
Inside zone: vmstat 1, iostat -xnz 1, prstat -mLa 1 for thread microstates (LCK, LAT, DFL).
Filesystem: df -h inside zone — full FS often kills apps silently.
Network: ping internal services from inside zone; confirm DNS, NTP.
Recent changes inside the zone: pkg history (S11), application deploy logs.
Take coredumps/snoop if reproducible; engage app team with concrete evidence.

The classic mistake here is assuming the GZ being healthy means the zone is. Zones have their own resource caps, services, filesystems, and networking — investigate inside the zone, not outside it.

17 Scenario Advanced SCENARIO: Zone filesystem is full. How do you identify the cause? Solaris L3 ▾

Answer

Confirm scope: is the zonepath full, or a filesystem mounted inside the zone? df -h both in GZ and zone.
If zonepath: from GZ du -sh /zones/<zone>/root/* sorts by directory.
ZFS dataset usage: zfs list -r rpool/zones/<zone> — see per-dataset used.
Largest files inside zone: find / -xdev -type f -size +1G 2>/dev/null | xargs ls -lh.
Look for runaway logs: /var/log, /var/adm, application logs.
Open-deleted files: lsof | grep deleted in zone — file deleted but still held open.
ZFS snapshots eating space: zfs list -t snapshot -r rpool/zones/<zone>.
Core dumps: coreadm + check /var/cores.
Quotas: if dataset has a quota, the zone may hit dataset quota before pool exhaustion.

zlogin appzone df -h

du -h --max-depth=1 /zones/appzone/root | sort -h | tail

zfs list -r rpool/zones/appzone

zfs list -t snapshot -r rpool/zones/appzone

18 Scenario Advanced SCENARIO: Zone startup takes unusually long. How will you investigate? Solaris L3 ▾

Answer

Quantify: measure with time zoneadm -z appzone boot; compare to a known-good zone.
Watch the boot live: zlogin -C appzone from another window before boot — see where it pauses.
Zone install log: /var/log/zones/zoneadm.*<zone>* for the boot transaction.
SMF inside zone: svcs -xv immediately after boot — failed services delay milestones.
Long-running services: svcs -p shows process trees; check /var/svc/log/<svc>.log for slow startup.
FS check: large UFS filesystems run fsck at boot if not clean — moves to ZFS to eliminate this.
Network timeouts: nfs/client service can hang on missing NFS server; check /etc/vfstab.
If the zone has many services with deep dependency chains, examine the SMF dependency graph for cycles or fan-out.
Resource starvation: if the GZ is under heavy load, zoneadm boot competes for I/O. Check with iostat during boot.

First-boot is always slower because of SCI (System Configuration Interactive). After that, slow boot is almost always a hanging SMF service waiting on a dependency — svcs -xv immediately after boot will name it.

🔆

14. LDOMs — L3 Deep Dive

LDOM architecture (Control/Service/I/O/Guest), CPU strand allocation, LDOMs vs Zones, LDOM creation, vdisk/vnet/vswitch internals, live migration prerequisites and process, migration failure causes, guest boot troubleshooting, delayed reconfiguration, and three production scenarios.

01 Concept Foundational What are LDOMs? Solaris L3 ▾

Answer

LDOMs (Logical Domains, officially Oracle VM Server for SPARC) are full hardware partitions on SPARC servers — each LDOM has its own kernel, OBP, and OS instance. Unlike zones (OS-level virtualization), LDOMs are firmware-level virtualization. They run on the hyper-privileged Solaris kernel on SPARC T-series and M-series systems.

Each LDOM is a complete logical machine — own kernel, own OS, own console.
Resources (CPU strands, memory, I/O) are allocated by the Control Domain.
Multiple OS versions can run side-by-side (Sol 10, Sol 11, even Linux on some platforms).
Live migration supported between compatible hosts.
Lower overhead than VMware — direct hardware access via I/O Domains.

02 Concept Foundational Explain Control Domain, Service Domain, I/O Domain, Guest Domain. Solaris L3 ▾

Answer

Domain Type	Role	Notes
Control Domain (primary)	Manages all other LDOMs via ldm CLI	Required; usually also the I/O Domain
Service Domain	Provides virtual I/O (vds, vsw, vcc) to guests	Disk + network proxy
I/O Domain	Has direct PCIe / PCI-root assignment	Native HW performance for guests
Guest Domain	Runs production workloads	Uses virtual I/O from Service Domain

In small deployments, one domain plays all roles (Control + Service + I/O). In large deployments, separating roles improves availability — a Service Domain reboot can be tolerated by HA-configured guests.

Best practice: configure two Service Domains (primary + alternate) and provide redundant virtual disks / virtual switches from both. This way you can reboot one Service Domain for patching without taking guests down.

03 Concept Intermediate How does CPU allocation work in LDOMs? Solaris L3 ▾

Answer

SPARC processors use strands (hardware threads). T-series chips have many strands per core (e.g. T4 = 8 strands/core, 64 strands per socket). LDOMs allocate strands or whole cores.

set-vcpu assigns logical strands (fine grained).
set-core assigns whole cores — preferred for production to avoid cross-core interference (“whole-core allocation”).
set-crypto assigns crypto units (MAU) on T-series.
Dynamic reconfiguration: add/remove CPU on a running LDOM with ldm add-vcpu/remove-vcpu (no reboot).
Power management: elastic mode lets unused CPUs power down; performance keeps them ready.

ldm set-core 4 guestdom1 # 4 whole cores (32 strands on T4)

ldm set-vcpu 16 guestdom1 # 16 strands (could share cores)

ldm add-vcpu 8 guestdom1 # Live add

ldm remove-vcpu 8 guestdom1 # Live remove

ldm list -o cpu guestdom1 # Show current allocation

Always prefer set-core over set-vcpu for production LDOMs. Sharing cores between LDOMs (“strand stealing”) hurts performance under contention.

04 Concept Foundational Difference between LDOMs and Zones. Solaris L3 ▾

Answer

Aspect	LDOMs	Zones
Virtualization level	Hardware (firmware)	OS-level
Own kernel	Yes	No — shared with GZ
OS diversity	Different Sol versions / Linux	Solaris only (or solaris10 brand)
Overhead	Low (but >0)	Near zero
Density	10s per host	100s per host
Live migration	Yes	Kernel Zones only
Hardware access	Direct via I/O Domain	Always virtualised
Boot time	Minutes (full OS boot)	Seconds
Best for	OS isolation, mixed versions	High density app isolation

The architectural pattern that interviewers love: LDOMs for OS-level isolation; Zones inside each LDOM for app-level density. Combines both technologies for maximum flexibility.

05 Concept Intermediate Can Zones run inside LDOMs? Solaris L3 ▾

Answer

Yes, absolutely — and this is the standard architecture for large SPARC deployments. Each LDOM runs its own Solaris OS, and inside each LDOM you can configure dozens of zones.

Provides two layers of isolation: LDOM for OS-level separation, Zones for app-level density.
Different LDOMs can run different Solaris versions and have different patch levels.
Each zone inside an LDOM still uses that LDOM's kernel — no double-virtualization overhead.
Resource caps apply at both layers: LDOM gets allocated cores; zones get capped within those.
Common in banks and telcos: “2 LDOMs per host (prod + non-prod) × 20 zones each”.

06 Commands Foundational LDOM Commands: ldm list, ldm list-bindings, ldm add-domain, ldm bind-domain. Solaris L3 ▾

Answer

Command	Purpose
ldm list	Concise list of all LDOMs + state + CPU/mem
ldm list -l	Long listing with all resources
ldm list-bindings	Detailed binding map (CPU, memory, vdisk, vnet)
ldm list-services	Virtual services (vds, vsw, vcc)
ldm list-spconfig	Service Processor saved configs
ldm add-domain	Create empty domain definition
ldm bind-domain	Bind resources to a domain (commit allocation)
ldm start-domain / stop-domain	Boot / halt the OS in the domain
ldm migrate-domain	Live migration to another host

ldm list

ldm list -l guestdom1

ldm list-bindings guestdom1

ldm list-services # Vds, vsw, vcc on Control Domain

ldm add-spconfig pre-change # Save current SP config

ldm list-spconfig

Before any significant LDOM change, always save the SP configuration with ldm add-spconfig <name>. This snapshots the entire LDOM layout in the Service Processor — your safety net if something goes wrong.

07 Task Intermediate How do you create a new LDOM? Complete workflow. Solaris L3 ▾

Answer

# Step 1 — Create empty LDOM

ldm add-domain guestdom1

# Step 2 — Allocate resources

ldm set-core 4 guestdom1

ldm set-mem 16G guestdom1

# Step 3 — Add virtual disk

# (Backing store on Control/Service Domain, exported via vds)

ldm add-vdsdev /dev/zvol/dsk/rpool/ldoms/guestdom1-disk0 \

guestdom1-disk0@primary-vds0

ldm add-vdisk vdisk0 guestdom1-disk0@primary-vds0 guestdom1

# Step 4 — Add virtual network interface

ldm add-vnet vnet0 primary-vsw0 guestdom1

# Step 5 — Set console + auto-boot

ldm set-var auto-boot\?=true guestdom1

ldm set-var boot-device=vdisk0 guestdom1

# Step 6 — Bind (commit allocation) + start

ldm bind-domain guestdom1

ldm start-domain guestdom1

# Step 7 — Connect to console for OS install

ldm list -o console guestdom1 # Get console port

telnet localhost <port>

# Step 8 — Save SP config when stable

ldm add-spconfig new-guest-deployed

08 Task Intermediate How do you assign CPU, memory and virtual disks? Solaris L3 ▾

Answer

# CPU — whole cores (preferred for prod)

ldm set-core 4 guestdom1

ldm add-core 2 guestdom1 # Add 2 more (live)

ldm remove-core 1 guestdom1 # Remove 1 (live)

# Memory

ldm set-mem 16G guestdom1

ldm add-mem 4G guestdom1 # Live add

ldm remove-mem 4G guestdom1 # Live remove (may need DR support)

# Virtual disk — three layers:

# 1. Backing store on a Service Domain

zfs create -V 50G rpool/ldoms/guest1-disk0

# 2. Export via vds (virtual disk service)

ldm add-vdsdev /dev/zvol/dsk/rpool/ldoms/guest1-disk0 \

guest1-disk0@primary-vds0

# 3. Attach to guest

ldm add-vdisk vdisk0 guest1-disk0@primary-vds0 guestdom1

# Verify

ldm list -o disk guestdom1

ldm list-services primary # All exported devices

09 Concept Intermediate How do virtual switches work? Solaris L3 ▾

Answer

A virtual switch (vsw) is a software bridge in a Service Domain that connects guest VNETs to a physical network. Multiple VNETs share the vsw; the vsw forwards frames between them and the underlying physical NIC.

# Create a vsw bridged to a physical link

ldm add-vsw net-dev=net0 primary-vsw0 primary

# Attach a guest VNET to that vsw

ldm add-vnet vnet0 primary-vsw0 guestdom1

# Inspect

ldm list-services primary # Shows vsw + connected vnets

ldm list -o network guestdom1

# VLAN-aware vsw

ldm add-vsw net-dev=net0 vid=100 primary-vsw100 primary

# Redundancy via IPMP on multiple VNETs in the guest

# Guest gets vnet0 (from primary-vsw0) and vnet1 (from secondary-vsw0)

# inside guest: ipadm create-ipmp ipmp0 ; add vnet0+vnet1

For network HA, create vsws on TWO Service Domains and give each guest TWO vnets (one from each). Inside the guest use IPMP to combine them. Surviving a Service Domain reboot then becomes a tested capability, not a hope.

10 Concept Intermediate Explain virtual disk services (VDS). Solaris L3 ▾

Answer

VDS is the kernel module in a Service Domain that exports block devices to guest LDOMs. Each exported device is a vdsdev, identified as <name>@<vds>. The guest sees it via the vdc (virtual disk client) driver as a regular disk device.

Backing store can be: physical LUN, ZFS volume, file, or whole disk.
Best perf: physical LUN passed via SR-IOV (rare) or ZFS zvol with logbias=throughput.
Whole disk vs slice: whole disk gives flexibility (guest can repartition); slice locks layout.
Multipath: configure two vdsdevs from two Service Domains, bind both as the same vdisk in the guest — guest uses mpxio across both paths.

ldm add-vds primary-vds0 primary # Create the VDS (one-time)

ldm add-vdsdev /dev/dsk/c0t1d0s2 disk1@primary-vds0

ldm add-vdisk vdisk0 disk1@primary-vds0 guestdom1

# Multipath setup — two vdsdevs from two Service Domains

ldm add-vdisk mpgroup=mygroup vdisk0 disk1@primary-vds0 guestdom1

ldm add-vdisk mpgroup=mygroup vdisk0b disk1@secondary-vds0 guestdom1

11 Concept Intermediate What are prerequisites for LDOM migration? Solaris L3 ▾

Answer

SPARC architecture compatibility — source and target must run compatible CPU generations. ldm list -o domain target shows the supported migration targets.
OS and firmware — same or compatible Solaris release in the Control Domain; matching hypervisor versions.
Shared storage — guest's vdisks must be accessible from BOTH source and target. Usually via SAN.
Network — target must have an equivalent vsw on the same VLAN; source and target on the same VLAN for guest connectivity.
Resources on target — equal or larger free cores/memory available before initiating migration.
SSH key-based authentication between Control Domains.
Cryptographic compatibility — same crypto resources (e.g. MAU) on both ends.
No DR delay in flight — pending delayed-reconfig changes must be applied or cancelled first.

Always run a dry-run first: ldm migrate-domain -n guestdom1 target-host. This validates all preconditions without actually moving anything. The -n flag is your safety net.

12 Task Advanced Explain live migration process. Solaris L3 ▾

Answer

Pre-checks: validate SSH connectivity, target resources, shared storage visibility.
Dry run: ldm migrate-domain -n guestdom1 target-host — verifies all preconditions.
Initiate: ldm migrate-domain guestdom1 target-host.
Phase 1 — Setup: target creates an empty domain definition matching source.
Phase 2 — Memory copy (pre-copy): source streams memory pages to target while guest continues running.
Phase 3 — Dirty page iteration: repeatedly copies pages modified during the previous copy round; converges as dirty rate drops.
Phase 4 — Suspend + final copy: guest briefly suspended; remaining dirty pages + CPU state transferred — typically sub-second.
Phase 5 — Resume on target: target starts the guest; source releases resources.
Verification: ldm list on both hosts confirms guest is now on target with same uptime.

ldm migrate-domain -n guestdom1 target-host # Dry run

ldm migrate-domain guestdom1 target-host # Live migrate

# Watch progress on source

ldm list-domain -o status guestdom1

# Migration with credentials

ldm migrate-domain guestdom1 user@target-host

During the suspend-and-final-copy phase the guest is paused for typically 100ms–2s — enough for TCP retransmits but real applications usually don't notice. For ultra-low-latency apps, test migration impact in pre-prod before declaring “zero downtime”.

13 Scenario Advanced What causes migration failures? Solaris L3 ▾

Answer

Cause	Symptom	Fix
Storage not shared	Migration fails at validation	Confirm both hosts can see the vdisk backing store
VLAN mismatch	Guest loses network on target	Add matching vsw + vlan on target
CPU type mismatch	“Not compatible” error	Check ldm list -o domain target; use newer source
Insufficient resources on target	Validation fails	Free cores/memory on target before migrate
Pending delayed reconfig	Validation fails	ldm cancel-reconf or apply first
Dirty rate too high	Migration never converges	Migrate during low-load window
MAU / crypto mismatch	Validation fails	Match crypto on target
Firmware/hypervisor skew	Compatibility error	Patch one side to match

# Detailed dry-run shows the failing precondition

ldm migrate-domain -n -v guestdom1 target-host

# Check pending delayed reconfig

ldm list -o resmgmt

ldm cancel-reconf primary # If safe to discard

14 Scenario Advanced Guest domain is not booting. How do you troubleshoot? Solaris L3 ▾

Answer

Check domain state: ldm list guestdom1. Is it active, bound, or inactive?
If inactive: bind first ldm bind-domain guestdom1.
If bound but not starting: ldm start-domain guestdom1; check return code.
Connect to console: ldm list -o console guestdom1 gets the port; telnet localhost <port> to see boot output.
OBP visible? If you see ok>, OS is not booting — check boot-device variable.
ok> printenv boot-device; ok> devalias; ok> probe-scsi-all if disks not seen.
Try ok> boot manually; capture errors.
If kernel panic visible: capture ::msgbuf via mdb (after reboot).
Resource issue? ldm list -o resmgmt on Control Domain — pending delayed-reconfig blocks operations.
vDisk backing store accessible? On Service Domain check ldm list-services; verify zvol/file exists.
vNet/vSwitch up? dladm show-link in Service Domain.

ldm list guestdom1

ldm list -l guestdom1 | head

ldm list -o console guestdom1

telnet localhost 5000 # Whatever port

# At ok prompt:

ok> printenv ; devalias ; probe-scsi-all ; boot

15 Scenario Advanced Virtual disk unavailable in guest domain. What checks would you perform? Solaris L3 ▾

Answer

On Control/Service Domain: ldm list-services — is the vdsdev still exported and bound?
Backing store: ls -l the file or zfs list the zvol — does it still exist? Permissions correct?
Inside guest: format or echo | format — does the disk appear? If not, the path is broken.
Inside guest: cfgadm -al for SAN-attached, or devfsadm -C to refresh.
Check guest's /var/adm/messages for I/O errors or device timeout.
If multipath: mpathadm show LU — both paths up?
If migrated recently: did the target Service Domain export the same vdsdev name?
If the vdsdev was accidentally removed: ldm add-vdsdev with the same backing store, then in the guest cfgadm to rescan.

ldm list-services primary # On service domain

ldm list -o disk guestdom1 # Disks attached to guest

# In guest:

echo | format

cfgadm -al

tail /var/adm/messages

16 Concept Advanced Explain delayed reconfiguration. Solaris L3 ▾

Answer

Delayed reconfiguration is a mode the Control Domain enters when you make changes that require an LDOM reboot to take effect. Multiple changes can be queued, then applied together at the next stop/start cycle.

Triggered when you change resources that the OS cannot accept dynamically.
While in delayed-reconfig, most other operations are blocked until you commit or cancel.
Common triggers: changing certain crypto, changing certain memory configs, modifying primary domain heavily.
Visible via ldm list -o resmgmt showing pending changes.

ldm list -o resmgmt # Show pending changes

ldm cancel-reconf primary # Discard pending changes

# Apply: usually requires stop/start of the affected domain

ldm stop-domain primary && ldm start-domain primary

# (Control Domain stop is rare — applies on next reboot)

If you're stuck with a delayed-reconfig blocking other LDOM operations, your two options are: apply (reboot the affected domain) or cancel. There is no third path — many engineers waste hours trying to do other LDM ops while pending changes exist.

17 Scenario Advanced How do you recover an accidentally removed virtual disk service? Solaris L3 ▾

Answer

If ldm rm-vdsdev was run on a live vdsdev, guests immediately lose access. Recovery: re-export with the same name and same backing store; the guest's vdisk reconnects.

# Re-export with identical name + backing store

ldm add-vdsdev /dev/zvol/dsk/rpool/ldoms/guest1-disk0 \

guest1-disk0@primary-vds0

# In the guest, force a rescan or reattach

# Sometimes a quick stop/start of the guest is required:

ldm stop-domain -f guestdom1 && ldm start-domain guestdom1

# If SP config was saved before the mistake, restore it

ldm list-spconfig

ldm set-spconfig pre-incident-config

# Power cycle the host to apply SP config (last resort)

This is why ldm add-spconfig pre-change before any significant operation is best practice. It costs nothing and gives you a single-command rollback. Doubly important on production.

18 Scenario Advanced SCENARIO: Production LDOM suddenly lost network connectivity. Explain troubleshooting steps. Solaris L3 ▾

Answer

Confirm scope: only this LDOM, or others on the same host too? If multiple LDOMs lost network, suspect the host/vsw/physical NIC.
On Control/Service Domain: dladm show-link — physical NIC up? Any errors?
ldm list-services primary — is the vsw still up and bound to the right NIC?
ldm list -o network guestdom1 — is vnet0 still attached?
Inside guest: ipadm show-if, ipadm show-addr — interface up and addressed?
Guest's /var/adm/messages for link state changes.
Ping in stages: guest → vsw IP (on Service Domain), guest → default gateway, guest → external.
If only this guest is affected and link is up: check ARP cache (arp -a), routes, and firewall.
Snoop from the Service Domain on the underlying NIC, filtered by the guest's IP — confirm packets reach/leave the host.
If vsw issue: stop/start the vsw or migrate the guest to a different Service Domain's vsw.

19 Scenario Advanced SCENARIO: CPU utilization is high in one guest domain. How do you determine whether the issue is inside the guest or at hypervisor level? Solaris L3 ▾

Answer

This is the classic L3 question. The key insight: CPU utilization seen inside the guest is relative to what the guest has. The Control Domain sees the actual hardware utilization.

On Control Domain: ldm list — note CPU columns: UTIL% is real hardware utilization for that LDOM. If UTIL% matches what the guest reports, the load is real.
If guest reports 100% but Control sees lower: guest is over-reporting because it doesn't see siblings sharing strands — “whole-core” allocation prevents this.
If Control sees 100% but guest sees lower: guest is being throttled / waiting for CPU at the hypervisor level.
Inside guest: prstat -mLa — high LAT (latency) means threads waiting on CPU; high USR/SYS means real work.
Per-LWP profiling: dtrace -n 'profile-997 /pid==<pid>/ { @[ustack()] = count(); }'.
On Control Domain: cpustat -c <ldom_cpus> 1 for hardware counter activity.
Compare with other LDOMs on the same host — if they're idle but yours is pegged, it's inside your guest.
If hypervisor-level contention: temporarily add cores to the guest (ldm add-core 2 guestdom1) and see if utilization drops — confirms throttling.

20 Scenario Advanced SCENARIO: Live migration is failing between two SPARC servers. What will you verify? Solaris L3 ▾

Answer

Read the dry-run output first: ldm migrate-domain -n -v guestdom1 target names the failing precondition.
CPU compatibility: ldm list -o domain target — same chip family? (T4→T4, M7→M7 generally compatible; mixing is restricted.)
Memory: target has free memory >= guest's allocation?
Cores: target has enough free cores in the right configuration?
Shared storage: all guest vdisks must point to backing stores visible from both sides. Check vds + LUN paths.
Network: equivalent vsw on the same VLAN exists on target?
Firmware/hypervisor: ldm -V shows hypervisor version — should match.
Crypto: if guest has MAU allocations, target must have free MAU.
SSH/auth: passwordless SSH between Control Domains is required.
Pending delayed-reconfig: ldm list -o resmgmt — must be clean.
Time sync: control domains should be in time agreement (NTP).
Logs: /var/svc/log/ldoms-ldmd:default.log on both sides for protocol-level errors.

🔗

15. Veritas Cluster Server — L3 Deep Dive

VCS core concepts (SG/Resource/System/GAB/LLT/HAD), cluster communication and startup, fencing (SCSI-3 PR and coordinator points), admin commands (hastatus/hagrp/hares/hatype), troubleshooting (failover blocked, UNKNOWN resource, HAD down, LLT/GAB issues), five scenario-based questions, and six advanced cross-domain scenarios.

01 Concept Foundational What is VCS? Solaris L3 ▾

Answer

Veritas Cluster Server (now Veritas InfoScale Availability) is an enterprise HA clustering product. It monitors applications and infrastructure, and on failure automatically fails services over to a healthy node — minimizing downtime. Runs on Solaris, AIX, HP-UX, Linux, Windows.

Application-aware: agents probe each resource (IP, mount, app process) for liveness.
Multi-node: 2 to 64 nodes; failover or parallel service groups.
Storage agnostic: works with SAN-backed shared storage; VxVM/VxFS native.
Mature: very common in banking, telco, large Oracle DB environments.
Heavy: requires LLT (heartbeat), GAB (membership), HAD (engine), often I/O fencing.

VCS competes with Sun Cluster, Oracle Clusterware, RHEL Pacemaker, and HP Serviceguard. The principal advantage interviewers want you to know: cross-platform consistency — same tool, same concepts on Solaris/AIX/HP-UX/Linux.

02 Concept Foundational Explain Service Group, Resource, System, Cluster. Solaris L3 ▾

Answer

Concept	Definition	Example
Cluster	Set of nodes running VCS that share configuration	oracle-prod-cluster
System	A single node in the cluster	node1, node2
Service Group (SG)	Collection of resources that fail over together	oracle_sg (= VIP + mount + listener + db)
Resource	Individual managed entity	DiskGroup, Mount, IP, Oracle
Resource Type	Agent that controls a class of resources	Mount, NIC, IPMultiNIC, Oracle
Dependency	Parent depends on Child (start child first)	Oracle depends on Mount depends on DiskGroup

hastatus -sum # Cluster + SG summary

hagrp -list # Service groups

hares -list # Resources

hares -dep # Dependency tree

hatype -list # Resource types

03 Concept Foundational Difference between Failover and Parallel Service Groups. Solaris L3 ▾

Answer

Aspect	Failover SG	Parallel SG
Active where	One node at a time	All listed nodes simultaneously
Use case	Active/passive apps (most DBs)	Stateless apps, monitoring, NTP
Failover behavior	Moves to another node on fault	Independent restart on local node
Shared storage	Required (single owner)	Optional (or independent per node)
Examples	Oracle single-instance, NFS server	ClusterService, NIC monitor

# Defined in main.cf:

group oracle_sg (

SystemList = { node1 = 0, node2 = 1 }

AutoStartList = { node1 }

Parallel = 0 # 0 = failover, 1 = parallel

)

04 Concept Foundational What is GAB? Solaris L3 ▾

Answer

GAB (Group Membership Atomic Broadcast) is the VCS membership and broadcast layer. It tracks which nodes are alive in the cluster and provides reliable broadcast of cluster-wide messages — config changes, fault events. GAB uses LLT for the underlying transport.

Configured via /etc/gabtab — typically gabconfig -c -n <num_nodes>.
GAB “ports” identify subsystems: a (GAB), b (I/O fencing), h (HAD/VCS engine), f (VxFS), v (CVM), w (vxconfigd).
Each port must show membership of all expected nodes; missing port = that subsystem isn't joined.
GAB seed = number of nodes required before the cluster is considered formed.

gabconfig -a # Port membership snapshot

# Output:

# Port a gen ... membership 01 <- all nodes joined

# Port b gen ... membership 01 <- fencing active

# Port h gen ... membership 01 <- HAD running

cat /etc/gabtab

gabconfig -l # Driver state

When the cluster acts strange, gabconfig -a is the second command after hastatus -sum. If port h shows different membership from port a, HAD has crashed on a node even though the kernel still thinks the node is up.

05 Concept Foundational What is LLT? Solaris L3 ▾

Answer

LLT (Low Latency Transport) is the heartbeat/communication layer underneath GAB. It runs over private network links (or shared disks in some configs) and provides fast, reliable node-to-node messaging. Heartbeats detect node failures within sub-second timeframes.

Configured in /etc/llttab (link definitions) and /etc/llthosts (node IDs).
Recommended: at least TWO heartbeat links on separate switches/paths for redundancy.
Detects link failure typically in 16–32 seconds depending on timeout settings.
LLT runs over Ethernet (preferred) or over UDP (LLT-over-UDP) for routed environments.
Does NOT run over the same subnet as data traffic — must be private.

lltconfig # Show LLT config

lltstat -nvv # Node + link status verbose

lltstat -l # Per-link statistics

cat /etc/llttab

# Typical:

# set-node node1

# set-cluster 100

# link net1 /dev/net:1 - ether - -

# link net2 /dev/net:2 - ether - -

cat /etc/llthosts

# 0 node1

# 1 node2

06 Concept Foundational What is HAD? Solaris L3 ▾

Answer

HAD (High Availability Daemon) is the VCS engine — the userland process that owns the cluster configuration, runs agents, and makes decisions about online/offline/failover. It runs on every node and communicates over GAB port h.

Reads config from /etc/VRTSvcs/conf/config/main.cf and types.cf.
Spawns agents — one process per resource type per node — to monitor resources.
If HAD dies, hashadow respawns it. If both fail, the node exits the cluster.
All ha* commands (hares, hagrp, hastop) talk to HAD via local socket.

ps -ef | grep -E 'had|hashadow'

gabconfig -a | grep 'Port h' # HAD membership

tail /var/VRTSvcs/log/engine_A.log

# Stop/start HAD on a node

hastop -local # Stop on local node only

hastop -local -force # Don't fail over SGs

hastart # Restart HAD

07 Concept Intermediate Explain the cluster startup sequence. Solaris L3 ▾

Answer

Node boots; LLT driver loads (llttab) and starts heartbeats on configured links.
GAB driver loads (gabtab); waits for seed (configured node count) before opening ports.
Once GAB seeded: I/O fencing (vxfen) joins port b — coordinates fencing membership.
HAD starts via SMF / init script; joins GAB port h.
HAD reads main.cf; rebuilds cluster state from peers via GAB.
Agents start for each Resource Type used by Service Groups on this node.
Per Service Group AutoStartList: SG comes online on the listed node.
hastatus -sum reports the cluster + SG state.

If the cluster doesn't form, check the sequence: LLT first (lltstat -nvv), then GAB (gabconfig -a), then HAD (engine_A.log). Always work bottom-up — fixing HAD when LLT is broken is wasted effort.

08 Concept Foundational How many heartbeat links should be configured? Solaris L3 ▾

Answer

Minimum two heartbeat links, on physically independent paths. Veritas best practice is two LLT links + a low-priority heartbeat over the public network as backup.

Two LLT links on separate NICs, separate switches.
Optionally a low-pri (“ltm”) heartbeat over the public network for partition detection only.
Three or more LLT links for very critical or geographically distributed clusters.
Each link should have its own subnet/VLAN — never share a switch port group across links.

# /etc/llttab — typical:

# link net1 /dev/net:1 - ether - -

# link net2 /dev/net:2 - ether - -

# link-lowpri net0 /dev/net:0 - ether - - <- backup over public

09 Concept Intermediate What happens if one heartbeat link fails? Solaris L3 ▾

Answer

VCS detects link failure via lltstat — alert in engine log, but cluster continues.
Heartbeats automatically use the remaining link(s).
If only ONE link remains, the cluster enters jeopardy state — single point of communication failure.
If the last link fails: cluster is partitioned; fencing decides which side survives.
Action: repair the failed link ASAP — engage networking; replace cable/NIC/switch port.

lltstat -nvv # Spot the down link

lltstat -l # Per-link stats — packet rates

tail /var/VRTSvcs/log/engine_A.log # 'link down' messages

10 Concept Advanced What is Jeopardy State? Solaris L3 ▾

Answer

Jeopardy is the VCS state when the cluster has only ONE heartbeat link remaining — the redundancy is gone. If that last link fails, VCS cannot distinguish between a node death and a partition, so it changes failover behavior to avoid split-brain.

Service Groups can still come online and fail over normally.
But: if the last heartbeat link goes down, VCS will NOT automatically fail over an SG to the other side — it would risk running on both sides (split-brain).
Operator intervention is then required to determine which side should host the SG.
Jeopardy is a warning state: “one more failure away from manual intervention.”

hastatus -sum # 'Jeopardy' annotated in output

tail /var/VRTSvcs/log/engine_A.log

Treat any Jeopardy alert as a P2 production incident. Restore the missing heartbeat within hours, not days — a single additional failure escalates to a split-brain risk.

11 Concept Advanced How does split-brain occur? Solaris L3 ▾

Answer

Split-brain occurs when nodes can no longer communicate (all heartbeats failed) but each node is still alive. Each side believes the others are dead and tries to take over shared resources — risking double writes to shared storage and data corruption.

All heartbeat links fail simultaneously (network failure, dual switch outage).
Each node loses GAB membership of the others.
Without fencing: each side promotes itself, mounts shared disks, starts the app — DATA CORRUPTION risk.
With I/O fencing: only the side that wins the SCSI-3 reservation race can write to disks; the other side is fenced off (panics or stops resources).

The single most important reason to enable I/O fencing is to prevent split-brain data corruption. Always answer interview questions about fencing with this framing first.

12 Concept Foundational What is I/O fencing? Solaris L3 ▾

Answer

I/O fencing is a mechanism to prevent a node from accessing shared storage when it has been ejected from the cluster. VCS uses SCSI-3 Persistent Reservations (PR) on shared disks: surviving nodes win the reservation race and registered keys remove the ejected node's access — the kernel itself blocks I/O.

Activated automatically on split-brain or node ejection.
Operates at the SCSI command layer — independent of OS or app.
Coordinator disks (or coordination point server / CP server) used to break ties.
Without fencing, two-node clusters cannot safely survive a heartbeat partition.

13 Concept Foundational Why is fencing required? Solaris L3 ▾

Answer

Prevents data corruption caused by two nodes writing to the same shared storage simultaneously.
Prevents split-brain: deterministic decision about which node survives a partition.
Required for production deployments with shared storage. Non-fenced clusters are only safe with shared-nothing architectures.
Many auditors require fencing as a compliance baseline for HA architectures.

Fencing should be enabled in EVERY production VCS cluster with shared storage. Disabling fencing is a serious risk — and is the #1 root cause of catastrophic VCS outages in the wild.

14 Concept Intermediate Explain SCSI-3 fencing and Coordination Points. Solaris L3 ▾

Answer

SCSI-3 fencing uses Persistent Reservations on shared SAN LUNs. Each cluster node registers a unique key on each LUN; only nodes with current registrations can write. When a node is ejected, the survivor preempts its key and the ejected node's writes are rejected by the array — providing OS-independent isolation.

Coordination Points — small LUNs used to break ties in even-node clusters. Three CPs is the standard count; an odd number is required to vote a majority.
Can be: dedicated SAN LUNs (“coordinator disks”), CP servers (network-based), or a mix.
On partition: each side races to gain majority of CPs. The losing side panics (“preempt-and-abort”), removing it from shared storage cleanly.
vxfentab lists the coordinator disks in use.

vxfenadm -d # Fencing state, coordinator membership

vxfenadm -g all -f /etc/vxfentab # Reservation keys per coordinator

cat /etc/vxfentab # Coordinator disks

cat /etc/vxfenmode # scsi3 or customized

vxfenswap -g coordinator -a # Replace a coordinator disk online

15 Concept Advanced What happens if fencing fails? Solaris L3 ▾

Answer

If fencing initialization fails at boot: HAD will refuse to start; the node won't join the cluster.
If a fencing operation fails during a live event: the node attempting the eject typically panics itself to maintain integrity (“panic on fencing loss”).
If coordinator disks are lost (e.g. SAN outage): fencing degrades. Add new CPs or replace before the next event.
Operator action: never disable fencing under pressure. Engage Veritas/Symantec support; let the cluster degrade safely rather than risk corruption.

# Check fencing state

vxfenadm -d # Should show 'cluster member' on each node

# If a coordinator disk failed, replace it online

vxfenswap -g coordinator -a # Atomic swap-in of replacement disk

16 Commands Foundational Explain outputs of hastatus -sum, hasys -state, hares -state, hagrp -state. Solaris L3 ▾

Answer

hastatus -sum — the daily go-to

hastatus -sum

# Sections:

# SYSTEM STATE — each node: RUNNING / FAULTED / EXITED

# GROUP STATE — each SG per node: ONLINE / OFFLINE / PARTIAL / FAULTED

# RESOURCES DISABLED, FROZEN, FAULTED — quick exception list

hasys -state — node state

hasys -state # Per-node state across cluster

hasys -display node1 # Detailed node attributes

# States: RUNNING, EXITED, FAULTED, FROZEN

hagrp -state — SG state per node

hagrp -state # All SGs × all nodes matrix

hagrp -state oracle_sg # One SG

# States: ONLINE, OFFLINE, PARTIAL, FAULTED, STARTING, STOPPING

hares -state — resource state

hares -state # All resources × all nodes

hares -state lvm_dg # One resource

hares -display lvm_dg # All resource attributes

# States: ONLINE, OFFLINE, FAULTED, UNKNOWN

17 Concept Intermediate Difference between Freeze and Persistent Freeze. Solaris L3 ▾

Answer

Freezing a Service Group tells VCS to stop monitoring/managing it — no auto-failover, no fault response. Used during maintenance so VCS doesn't react to expected disruption. Persistent freeze survives cluster restarts; ordinary freeze does not.

Operation	Survives HAD restart?	Survives node reboot?	Typical Use
hagrp -freeze	No	No	Quick maintenance, single op
hagrp -freeze -persistent	Yes	Yes	Long maintenance, planned outage

hagrp -freeze oracle_sg # Non-persistent

hagrp -freeze oracle_sg -persistent # Persistent

haconf -dump -makero # Save config (required for persistent)

hagrp -unfreeze oracle_sg

hagrp -unfreeze oracle_sg -persistent

Always remember to UNFREEZE after maintenance. A persistently frozen SG that nobody unfreezes is a silent disaster waiting to happen — VCS won't fail it over when something actually breaks.

18 Concept Foundational What is AutoStartList? Solaris L3 ▾

Answer

AutoStartList is an attribute on a Service Group listing the nodes where VCS should automatically bring the SG online when the cluster starts. Without it, SGs stay offline until manually started.

hagrp -display oracle_sg -attribute AutoStartList

hagrp -modify oracle_sg AutoStartList node1

hagrp -modify oracle_sg AutoStartList node1 node2 # Tries in order

VCS attempts to start the SG on the first listed node available.
If that node is unavailable, it tries the next, and so on.
Combine with SystemList priorities to control failover destination after fault.

19 Concept Foundational What is SystemList? Solaris L3 ▾

Answer

SystemList defines which nodes are eligible to host a Service Group and their priority. Lower number = higher priority. Combined with policies like FailOverPolicy it determines failover targets.

hagrp -display oracle_sg -attribute SystemList

# Example: { node1 = 0, node2 = 1, node3 = 2 }

# On fault, VCS tries node1, then node2, then node3

hagrp -modify oracle_sg SystemList node1 0 node2 1

# To remove a node from the list:

hagrp -modify oracle_sg SystemList -delete node3

20 Scenario Advanced Service group is not failing over. How do you troubleshoot? Solaris L3 ▾

Answer

hastatus -sum — what state is the SG in? FAULTED, PARTIAL, OFFLINE?
Check freeze state: hagrp -display oracle_sg | grep -i freeze. A frozen SG won't move.
Auto-failover enabled? hagrp -display oracle_sg | grep AutoFailOver.
SystemList — are alternative nodes listed and eligible? Are they up?
Target node's resource state: maybe a dependent resource is FAULTED on the target side, blocking online.
tail /var/VRTSvcs/log/engine_A.log — VCS narrates every decision; the answer is usually in this log.
Resource-specific logs: agent logs in /var/VRTSvcs/log/<agent>_A.log.
Try manual failover: hagrp -switch oracle_sg -to node2. Failure messages name the cause.
Check for cluster-wide blockers: hasys -display | grep Frozen — a frozen node accepts no SGs.

hastatus -sum

hagrp -display oracle_sg

hares -display lvm_dg

tail -100 /var/VRTSvcs/log/engine_A.log

hagrp -switch oracle_sg -to node2

21 Scenario Advanced Resource is in UNKNOWN state. What checks will you perform? Solaris L3 ▾

Answer

UNKNOWN means VCS cannot determine the resource state — usually because its monitor probe is failing or timing out.

Check the resource type's agent: ps -ef | grep <type>Agent — agent process running?
Agent log: /var/VRTSvcs/log/<type>_A.log — what is the agent reporting?
Probe the resource manually: hares -probe lvm_dg -sys node1 — forces a monitor run.
Engine log: /var/VRTSvcs/log/engine_A.log for related decisions.
Resource attributes: hares -display lvm_dg — check MonitorInterval, MonitorTimeout; an underconfigured timeout returns UNKNOWN.
If the underlying app is genuinely confused: check it directly on the node (e.g. vxdg list for a DiskGroup resource).
If agent is hung: haagent -stop <type> -sys node1 then haagent -start <type> -sys node1 to restart it.

hares -probe lvm_dg -sys node1

hares -display lvm_dg

tail /var/VRTSvcs/log/DiskGroup_A.log

haagent -display DiskGroup

haagent -stop DiskGroup -sys node1 && haagent -start DiskGroup -sys node1

22 Scenario Advanced HAD process is down. What actions are required? Solaris L3 ▾

Answer

Check: ps -ef | grep had — confirm absence.
Check hashadow: it should respawn HAD automatically. If hashadow is also gone, both have failed.
Logs: /var/VRTSvcs/log/engine_A.log for the cause of crash (assertion, config error, panic).
Coredump? Solaris saves to /var/crash; check for HAD core files.
Restart manually: hastart. If immediate crash: examine config files — main.cf, types.cf.
If config is corrupt and won't load: copy a known-good main.cf.bak in place and try again. Always keep backups.
If HAD repeatedly crashes due to bug: engage Veritas support with the engine_A.log + coredump; meanwhile keep SGs running on the remaining nodes.
Make sure HAD restart doesn't disrupt active SGs: VCS is designed so HAD failure leaves SGs running; HAD restart re-attaches without disrupting them.

ps -ef | grep -E 'had|hashadow'

tail -200 /var/VRTSvcs/log/engine_A.log

hastart

# Or restart via SMF (Solaris)

svcadm restart system/vcs

23 Scenario Advanced LLT links are down. What files will you verify? Solaris L3 ▾

Answer

lltstat -nvv — confirms which link(s) are down and on which node.
cat /etc/llttab — verify link declarations match physical interfaces.
cat /etc/llthosts — node-id-to-hostname mapping correct.
dladm show-link or ipadm show-if — physical interface state.
Cable / switch: ask networking to verify port up, no errors, correct VLAN.
lltstat -l — packet rates per link; zero rates = no traffic at all.
If LLT itself is broken: lltconfig shows driver state; reload with lltconfig -U + lltconfig -c only after stopping VCS.

lltstat -nvv # Per-link, per-node state

lltstat -l # Statistics

cat /etc/llttab

cat /etc/llthosts

dladm show-link

24 Scenario Advanced How do you troubleshoot GAB port issues? Solaris L3 ▾

Answer

gabconfig -a — see which ports are open and which nodes are in each port's membership.
Identify the missing port: a (gab base), b (fencing), h (HAD).
If port h is missing on a node: HAD isn't running there. Restart HAD (hastart).
If port b is missing: fencing isn't initialized; check vxfenadm -d and /etc/vxfenmode.
If port a is missing entirely: GAB seed not reached. Check cat /etc/gabtab and confirm all expected nodes are configured.
gabconfig -l shows GAB driver state; check cat /etc/gabtab to confirm seed value.
Engine log + GAB messages in /var/adm/messages name protocol-level errors.

gabconfig -a

# Expected for a healthy 2-node cluster:

# Port a gen <num> membership 01

# Port b gen <num> membership 01

# Port h gen <num> membership 01

cat /etc/gabtab

gabconfig -l # Driver state

25 Scenario Advanced SCENARIO: One node in a two-node cluster suddenly goes down. Explain complete failover sequence. Solaris L3 ▾

Answer

Failed node stops sending LLT heartbeats; surviving node's LLT detects loss within timeout (typically 16 seconds).
GAB on survivor sees member loss; cluster membership updates to 1.
I/O fencing kicks in: surviving node races for coordinator disk reservations to confirm it's the legitimate survivor.
If fencing succeeds: HAD on survivor declares the failed node EXITED.
VCS evaluates Service Groups that were ONLINE on the failed node: it will fail them over if they're eligible on this node (SystemList, no freeze, AutoFailOver=1).
For each affected SG: VCS brings up resources in dependency order (DiskGroup → Mount → IP → Application).
Each resource's agent runs its online action; monitor verifies it; next resource starts.
Once all resources are ONLINE, the SG state becomes ONLINE on the survivor.
Application is back in service; clients reconnect via the VIP (which moved to the survivor).
Failed node, when rebooted, joins cluster as PASSIVE (SGs remain on survivor unless re-balanced).

Total downtime in this sequence is usually 30–90 seconds: 16s heartbeat detection + 10s fencing race + 30–60s for resource online sequence (mostly mount + app startup).

26 Scenario Advanced SCENARIO: Service group remains ONLINE on failed node and does not come online elsewhere. What could be the reasons? Solaris L3 ▾

Answer

SG is frozen: hagrp -display oracle_sg | grep Frozen. Frozen SGs don't move.
AutoFailOver=0: SG configured not to auto-fail-over. Operator must manually hagrp -switch.
Target node frozen: hasys -display node2 | grep Frozen. Frozen nodes accept no SGs.
SystemList exhausted: the SG ran out of eligible target nodes (all faulted or not in list).
Resource fault on target: a resource is already FAULTED on the target node (e.g. previous attempt left a stale state).
Fencing failed: cluster cannot complete the eject, so it refuses to move SGs.
Persistent attribute conflict: TriggersEnabled or AdminWait state — engine log explains.
Concurrency violation: another SG with conflicting resources is already online on the target.

tail -200 /var/VRTSvcs/log/engine_A.log # Always check here first

hagrp -display oracle_sg

hasys -display node2

hares -clear lvm_dg -sys node2 # Clear stale FAULT on target

27 Scenario Advanced SCENARIO: Node cannot join the cluster after reboot. How would you investigate? Solaris L3 ▾

Answer

Work bottom up: LLT → GAB → fencing → HAD.
LLT: lltstat -nvv — are heartbeats flowing? Check /etc/llttab matches reality.
GAB: gabconfig -a. If port a is missing on this node, check cat /etc/gabtab seed value.
Fencing: vxfenadm -d — fencing module loaded and joined? Check coordinator disk visibility.
HAD: ps -ef | grep had; check engine_A.log for startup errors.
Config consistency: this node's main.cf must match the cluster's; if it joined and config diverged, HAD refuses.
Cluster ID mismatch: confirm set-cluster in llttab matches the other nodes.
Patch level / kernel: VCS expects matching kernel versions; major patch skew can prevent join.
Network: are private LLT links physically up to the cluster? Many “won't join” issues are misseated cables on the heartbeat NICs.

28 Scenario Advanced SCENARIO: After patching, VCS services are not starting. What logs and commands will you use? Solaris L3 ▾

Answer

SMF/init: svcs -xv — find which VCS-related service failed.
Engine log: /var/VRTSvcs/log/engine_A.log — most VCS startup issues are explicitly logged here.
Kernel modules loaded? modinfo | grep -i 'llt\|gab\|vxfen' — missing modules = kernel/Veritas package mismatch.
Module version vs package version: pkg list VRTS* | grep vcs and modinfo output — patch may have updated kernel but not VRTS package, or vice versa.
Check /etc/llttab, /etc/llthosts, /etc/gabtab still exist and aren't corrupted.
Try manual driver load: /etc/init.d/llt start ; /etc/init.d/gab start — see specific error.
If patch corrupted modules: reinstall the affected VRTS packages; reapply Veritas SP if available.
Veritas support: provide vxexplorer output bundle.
Last resort if it's truly broken: bring up the other node first, then troubleshoot the failed node offline.

svcs -xv

modinfo | grep -i 'llt\|gab\|vxfen\|vcs'

tail /var/VRTSvcs/log/engine_A.log

/opt/VRTS/bin/vxexplorer # Diagnostic bundle for support

29 Task Advanced SCENARIO: How would you safely remove a node from a production VCS cluster? Solaris L3 ▾

Answer

Removing a node is a routine but high-stakes change. Plan and execute methodically; ensure no SG depends on the departing node for HA.

Raise a Change Request; communicate with app teams about reduced redundancy.
Switch any active SGs OFF the departing node: hagrp -switch <sg> -to <other_node>. Verify each one is healthy on its new home.
Remove the departing node from each SG's SystemList and AutoStartList: hagrp -modify <sg> SystemList -delete <node>.
If any resource has the node in FailOver targets or attributes referencing it, clean those too.
Save the running config to disk: haconf -dump.
Stop VCS on the departing node: hastop -local.
Remove fencing reservation for that node: it should drop out of coordinator disk membership automatically.
Edit /etc/llthosts on REMAINING nodes to remove the node's entry; edit /etc/gabtab to lower the seed count.
Restart LLT/GAB or reboot remaining nodes in a rolling fashion if seed change requires it.
On the departed node: stop and disable VCS SMF services; uninstall VRTS packages if decommissioning; remove from inventory.
Update CMDB, runbook, monitoring, on-call rotation.

Never remove a node by just halting it and editing files. The cluster's SystemList entries on every SG will still reference the missing node — VCS will throw errors on every config change attempt until you clean it up properly.

30 Scenario Advanced SCENARIO 1: A Solaris application server is slow. CPU is 20%, memory is 60%, but users complain about slowness. How will you troubleshoot? Solaris L3 ▾

Answer

Low CPU/memory with user-reported slowness means the bottleneck is somewhere other than capacity. Classic causes: I/O, locks, network, application logic, dependent services.

Quantify the slowness: time a specific user operation; ask app team for transaction logs with timestamps.
Storage I/O: iostat -xnz 1 5 — high asvc_t (>30 ms) or %b (>80%) means storage saturation. Disk-bound apps look CPU-idle.
Thread state: prstat -mLa 1 — high LCK = lock contention; high SLP = waiting for resources; high DFL = paging.
Network: nicstat 1; check latency to upstream services (DB, app server, LDAP).
Dependent services: slow DNS, slow LDAP, slow database — application sits idle waiting. Test each from this host.
NFS: nfsstat -m on any NFS mounts; slow NFS server makes the app look slow.
App logs: many “slow” issues are application-level (GC pauses, connection pool exhaustion, lock waits in code).
DTrace: dtrace -n 'syscall:::entry /pid==<pid>/ { @[probefunc] = count(); }' for syscall mix; identify what the app actually spends time on.
Recent changes: showrev -p, app deploy logs — sudden slowness often follows a change.

The single best one-liner: prstat -mLa 1 5. The microstate columns tell you exactly where threads spend time — USR (CPU), LCK (locks), SLP (waiting), LAT (run queue), DFL (paging). 90% of slow-but-not-CPU cases are diagnosed from this output alone.

31 Scenario Advanced SCENARIO 2: A VCS service group keeps faulting every few hours. What logs and commands will you use to identify the root cause? Solaris L3 ▾

Answer

Confirm pattern: grep FAULTED /var/VRTSvcs/log/engine_A.log | tail -20 — what time, which resource, which node?
Identify the resource that triggered the fault: a SG fault originates from ONE resource's monitor failure.
Read that resource's agent log: /var/VRTSvcs/log/<ResourceType>_A.log for the same timestamps.
Resource-specific health: e.g. for a Mount resource, check dmesg//var/adm/messages for I/O errors at the fault timestamp.
Look for periodicity: every N hours suggests a cron job, a backup, or a scheduled GC/maintenance window.
Resource attributes: hares -display <res> | grep -i 'monitor\|timeout' — too-aggressive MonitorTimeout can produce spurious faults.
Underlying issue: app process restarting on its own? Network blip? Storage latency? — fault means VCS monitor returned OFFLINE.
Mitigation: increase MonitorTimeout only after confirming the underlying transient is benign; raising it blindly hides real problems.
Open a Problem Record (not just resolve the incident); recurring SG faults need a documented RCA.

grep 'FAULTED\|going OFFLINE' /var/VRTSvcs/log/engine_A.log | tail -30

awk '/oracle_sg.*FAULTED/{print $1, $2, $3}' /var/VRTSvcs/log/engine_A.log | uniq -c

hares -display oracle_db | grep -i 'monitor\|interval\|timeout'

32 Scenario Advanced SCENARIO 3: An NFS mount is hanging and all commands accessing that filesystem are stuck. How do you recover without rebooting? Solaris L3 ▾

Answer

Quick triage: timeout 5 ls /mnt — confirms the mount is wedged without hanging your shell.
NFS server reachability: ping nfs-srv, showmount -e nfs-srv from another host.
If NFS server is down/restarting: it may recover automatically once the server returns; hard mounts wait indefinitely.
fuser -c /mnt — lists processes touching the mount. These are stuck in 'D' state and can't be killed normally.
Try a forced lazy unmount: umount -f /mnt. On Solaris this often hangs if processes still hold the FS — proceed to next step.
Kill stuck processes: fuser -ck /mnt — this issues SIGKILL. With hard,intr mounts and intr-aware syscalls, processes can be killed; without intr, they may stay stuck.
Once all processes are gone: umount -f /mnt should succeed.
Repair the underlying NFS server first if it's the issue; remount after server is healthy.
Preventive: change to hard,intr,timeo=300,retrans=2 on non-critical mounts. Critical mounts stay hard,intr.

timeout 5 ls /mnt # Confirm hang without hanging shell

showmount -e nfs-srv

fuser -c /mnt # Who's using the mount

fuser -ck /mnt # Kill them

umount -f /mnt

fuser -ck is destructive — those processes lose any unsaved work. In production always confirm with the app owner before killing; if the NFS server can be restored, that's the safer path.

33 Scenario Advanced SCENARIO 4: A zone becomes unresponsive but the Global Zone is healthy. How will you determine whether the issue is CPU, memory, filesystem, or networking? Solaris L3 ▾

Answer

Layered triage — confirm what the zone is and isn't doing, then narrow.

From GZ: zoneadm list -cv — is it running or down?
From GZ: prstat -Z 1 3 — does the zone show CPU/memory consumption? If it's pegged at its cap, that's your answer.
rcapstat 1 — is the memory cap being enforced (pages out)?
From GZ: zlogin appzone uptime — does the zone respond at all? If timeout, the kernel/zone interaction is degraded.
If zlogin works: inside zone prstat -mLa, vmstat 1, iostat -xnz 1.
Filesystem: df -h inside zone; du -sh /* to find space hogs.
Network: from GZ snoop on the zone's VNIC; inside the zone ipadm show-addr, route -n, ping tests.
If zlogin hangs entirely but GZ is healthy: zone may be in a stuck state. Try zoneadm -z appzone reboot -- -i as a careful restart.
If still stuck and zone is critical: forcibly halt and boot: zoneadm -z appzone halt then boot.

34 Scenario Advanced SCENARIO 5: After a live migration of an LDOM, application performance drops significantly. What checks would you perform? Solaris L3 ▾

Answer

Confirm migration completed cleanly: ldm list on target — UTIL%, allocated cores/memory should match source.
Whole-core allocation preserved? ldm list-bindings target-ldom; mismatched core layout vs source can hurt cache locality.
I/O Domain path: on the target, are vdisks served by the same Service Domain as before? A migration might have re-pathed via a slower service domain.
iostat -xnz 1 inside the guest — has asvc_t changed materially compared to before migration?
Network: VLAN/vsw on target equivalent? netstat -i in guest — retransmits or drops post-migration?
CPU generation difference: even compatible generations have small clock/cache differences; if the target is a slower model, you'll see a measurable hit.
Memory placement: post-migration, all guest memory may end up on one socket (NUMA imbalance). On large guests this is significant.
Crypto offload: did MAU/crypto units transfer? If guest uses SSL heavily, missing crypto means software fallback = slow.
Application reconnection state: connection pools, caches, TCP — sometimes “slow after migration” is just cold caches; measure after warmup.
Mitigation: migrate back to source as a test; if perf restored, you've identified target as the issue.

35 Scenario Advanced SCENARIO 6: A node cannot be deleted from a VCS cluster because it is part of the SystemList of multiple service groups. Describe the complete node decommission procedure. Solaris L3 ▾

Answer

Pre-work: identify every Service Group that includes the node in its SystemList: hagrp -list | awk '...' or simply iterate and grep.
Migrate active SGs off: hagrp -switch <sg> -to <other> for each currently ONLINE on the node. Wait for each to be confirmed online elsewhere.
Remove from SystemList: for each affected SG, hagrp -modify <sg> SystemList -delete <node>.
Remove from AutoStartList: hagrp -modify <sg> AutoStartList -delete <node> for any SG that listed the node.
Clean other attributes: FailOverPolicy, PreOnline triggers, agent-specific attributes that reference the node.
Persist config: haconf -dump -makero to save to main.cf.
Stop VCS on the node: hastop -local. This leaves the SGs running on their current homes.
LLT/GAB membership: edit /etc/llthosts on remaining nodes to remove the entry; lower the GAB seed in /etc/gabtab if needed.
Fencing: confirm coordinator disks no longer reference the node's reservation key.
On the decommissioned node: stop+disable SMF services; uninstall VRTS packages if retiring hardware; unzone/unrack as appropriate.
Verify cluster health: hastatus -sum on remaining nodes — confirm clean state, no warnings.
Documentation: update CMDB, runbooks, monitoring inventory, on-call rotation, DR plan.

The whole sequence usually takes 30–60 minutes for a clean two-node-becomes-one operation. Always perform it during a Change window — even though no SG should be impacted, configuration drift is best done with the safety of a controlled change record.

🐧

Solaris — Linux Admin Guide: LDOM, SVM & Health Checks

LDOM live migration, SVM disk mirroring, and Solaris health check commands from the Linux Admin reference guide

LDOM Live Migration Steps

# Pre-migration checks cat /opt/IHSldmcfg/etc/pod_hosts # Verify source/dest frames in hosts /opt/IHSldmcfg/bin/check_pod_disk all # Disk status check # Take XML snapshot for backup ldm ls-constraints -x LDOM > LDOM.xml # Switch to migration user su - ldommig # Perform migration /opt/IHSldmcfg/bin/migrate_domain entcamwp01 mwdt4p01f3-cd # Save configuration post-migration ldm ls-spconfig ldm add-spconfig YYYYMMDD.N-normal # Remove old configs if exhausted ldm rm-spconfig configname # List LDoms and start all ldm ls ldm start -a

Common Error: "Failed to establish connection with ldmd on target" — Solution: On each frame in the pod, run /opt/IHSldmcfg/sbin/PWP_config -c then -s to set up the ldommig account password. Then ensure xmpp_enabled and incoming_migration_enabled are set to true via svccfg.

QWhat is a Solaris LDOM and how does it compare to a zone?

▸ ANSWER

LDOMs (Logical Domains) are hypervisor-level virtualization on SPARC servers — each LDOM has its own OS, CPU cores, memory, and I/O devices. It's full hardware virtualization. Zones (Solaris Containers) are OS-level virtualization — multiple zones share the same kernel. LDOMs provide stronger isolation (separate kernels), Zones provide more lightweight density. LDOMs require SPARC T-series or similar hypervisor-capable hardware. Zones can run on any Solaris system.

QWhy must you save the LDOM spconfig after migration?

▸ ANSWER

The LDOM configuration (spconfig) stored in NVRAM tells the SP (Service Processor) how to configure domains at boot. After a migration, the configuration in NVRAM on both the source and destination frames is outdated. If either frame reboots without an updated spconfig, the LDOM layout won't match reality. ldm add-spconfig YYYYMMDD.N-normal saves the current running config to NVRAM. Always do this on BOTH source and destination frames post-migration.

Solaris SVM Disk Mirroring Procedure

Solaris Volume Manager (SVM) mirroring steps for root + swap + data partitions. Essential for HA on legacy SPARC systems.

1Backup /etc/vfstab: cp /etc/vfstab /etc/vfstab.orig
2Partition mirror disk identical to boot disk: prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2
3Create state DB replicas (at least 3): metadb -a -f -c 3 c0t0d0s7 c0t1d0s7
4Create submirrors for root: metainit -f d10 1 1 c0t0d0s0, metainit d0 -m d10
5Run metaroot: metaroot d0 (updates /etc/vfstab and /etc/system)
6Mirror swap and all other slices similarly
7Reboot: lockfs -fa && init 6
8Attach second submirrors: metattach d0 d20
9Monitor sync: while true; do metastat | grep %; sleep 10; done
10Install boot block: installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0

QWhat are SVM state database replicas and why do you need at least 3?

▸ ANSWER

State database replicas (metadb) store the configuration and state of all meta-devices. SVM uses a voting quorum system — it needs a majority of replicas to be active to function. With 3 replicas, you can lose 1 and still have 2 (majority). With 2 replicas, losing 1 leaves only 1, which isn't a majority, so the system panics (refuses to continue) to prevent data corruption. Best practice: create an odd number (3, 5, 7) spread across multiple disks for maximum fault tolerance.

Solaris Health Check Commands

# System info uname -a; uptime; date; who -r prtdiag -v # hardware diagnostics prtconf -pv | grep -i boot # boot device prtconf -pv | grep -i mem # memory info # Fault management fmdump # fault dump fmadm faulty # list faulty components svcs -xv # failed SMF services # Storage iostat -En # disk errors cfgadm -al # device config (hotplug) echo | format # disk list zpool status; zfs list; zpool list # FC HBA luxadm -e port luxadm probe fcinfo hba-port -l # SVM / VxVM metadb -i; metastat -t vxprint -Aht; vxdg list; vxdisk list; vxtask list # Messages cat /var/adm/messages dmesg