💻 Linux Admin Interview Guide

PV (Physical Volume) — a raw disk or partition initialized for LVM use (e.g., /dev/sdb). It's the building block. VG (Volume Group) — a pool formed by combining one or more PVs. Think of it as a logical disk. LV (Logical Volume) — a slice carved from a VG, equivalent to a partition. It's what you format and mount. The key advantage: LVs can be resized, spanned across disks, and snapshotted without downtime.

First, expand the VG by adding a new PV: pvcreate /dev/sde then vgextend datavg /dev/sde. After that, lvextend -L +200G /dev/mapper/datavg-log -r (the -r flag resizes the filesystem automatically). For virtual disks, rescan first: echo 1 > /sys/class/block/sdc/device/rescan then pvresize /dev/sdc.

LV shrinking is risky — must shrink filesystem BEFORE reducing LV size, never the other way:

1. 1Unmount: umount /home
2. 2Check filesystem: e2fsck -f /dev/mapper/vg-LogVol00
3. 3Shrink FS first: resize2fs /dev/mapper/vg-LogVol00 10G
4. 4Reduce LV: lvreduce -L 10G /dev/mapper/vg-LogVol00
5. 5Re-check FS: e2fsck -f /dev/mapper/vg-LogVol00
6. 6Remount: mount /home

This happens when a process holds the LV open. Fix: dmsetup info -c | grep <lvname> to get the major:minor numbers, then lsof | grep "253,9" to find the PID, then kill -9 <PID>, and retry lvremove. Also check if it's mounted: umount -l /mountpoint before removing.

The -r flag tells lvextend to also resize the filesystem automatically after extending the LV. Without it, you'd have to run resize2fs /dev/mapper/vg-lv manually. It internally calls the appropriate resize tool based on the filesystem type (resize2fs for ext4, xfs_growfs for XFS, etc.).

1. swapoff /dev/mapper/rootvg-lv_swap — disable swap. 2. lvresize -L 16G /dev/mapper/rootvg-lv_swap — resize LV. 3. mkswap /dev/mapper/rootvg-lv_swap — re-initialize swap. 4. swapon -a — re-enable. 5. Verify with free -h and cat /proc/swaps. Note: mkswap wipes old swap signature — data is lost, so swap should be empty before this.

1. Create new LV: lvcreate -n qlikhomelv -L 10G qlikvg. 2. Format: mkfs.ext4 /dev/mapper/qlikvg-qlikhomelv. 3. Mount to temp: mount /dev/mapper/qlikvg-qlikhomelv /mnt. 4. Copy data: cd /home/user; tar -cf /mnt/home.tar *. 5. Backup original dir: mv /home/user /home/user_bkp. 6. Create new dir: mkdir /home/user. 7. Extract: cd /mnt; tar -xf home.tar. 8. Add fstab entry, mount -a, set correct ownership/permissions with chown/chmod.

The 5th field (dump) — 0 means no backup with dump utility, 1 means yes. The 6th field (pass) — controls fsck order at boot: 0 = skip, 1 = check first (root only), 2 = check after root. Never use 1 for non-root filesystems. Best practice: root = 1, data partitions = 2, tmpfs/NFS = 0 0.

Run lvscan to see the status. If an LV shows as inactive, activate with vgchange -ay /dev/datavgc. This is common after disk replacements or importing VGs from other systems. After activation, mount -a to remount filesystems.

Multipath I/O (MPIO) allows a server to access storage through multiple physical paths (HBA ports, switches, storage ports) simultaneously. Benefits: (1) High Availability — if one path fails, I/O continues through other paths. (2) Load Balancing — I/O distributed across paths improves throughput. (3) Non-disruptive maintenance — can replace failed paths without downtime. The OS sees one device even though multiple physical paths exist.

Orphan paths are paths that don't belong to any multipath group. Fix: multipathd show paths | grep -i orphan to identify them, then for each orphan path, get the device name with for i in `multipathd show paths format "%w;%i;%s;%t_%T" | grep undef | awk -F";" '{print $2}'`, find the block device via lsscsi, then delete: echo 1 > /sys/block/$fd/device/delete. Then run multipath -r to reload and service multipathd reload.

1. 1Scan for new LUNs: for i in /sys/class/scsi_host/*; do echo "- - -" > $i/scan; done
2. 2Verify new LUNs visible: inq | grep -i <LUNID>
3. 3Check multipath: multipath -ll | grep -i mpathlt
4. 4Create PV on new mpath device: pvcreate /dev/mapper/mpathlr
5. 5Extend VG: vgextend datavg /dev/mapper/mpathlr
6. 6Extend LV: lvextend -L +200G /dev/mapper/datavg-data -r
7. 7Verify: df -kh

device-mapper-multipath is the open-source Linux kernel solution, configured via /etc/multipath.conf. It's vendor-agnostic. EMC PowerPath is a vendor-specific commercial product from Dell EMC for EMC storage arrays. PowerPath offers advanced features like dynamic load balancing algorithms, better EMC integration, and automated failover. In interviews: know that powermt display dev=all shows PowerPath devices and multipath -ll shows dm-multipath devices.

Error 7641 relates to host credential/certificate issues preventing client-master server communication. Common causes: expired certificates, stale host cache after server migration, or wrong master server configured. Resolution steps: (1) Clear host cache with bpclntcmd -clear_host_cache. (2) Restart all NetBackup and VxPBX services. (3) Run get_nb_certs.sh to regenerate certificates. (4) Use nbcertcmd -getCertificate to pull a fresh cert from the master server. Always verify services are up with bpps -x after resolution.

bpps -x is NetBackup's own process status tool that shows all NB-related daemons in a formatted output with process relationships. It's far more reliable than ps -ef | grep netbackup because it understands the NB process hierarchy and shows the state of each daemon specifically. The -x flag shows extended information including legacy client processes.

-mtime +N — files modified more than N days ago (+31 = older than 31 days). -mtime -N — files modified within the last N days. -type f — only regular files (not dirs/symlinks). -xdev — don't cross filesystem boundaries (stays on same device/mount). Critical for log searches so you don't inadvertently scan NFS/tmpfs mounts. -size +1000000c — files larger than 1MB (c = bytes).

gzip: faster compression/decompression, lower compression ratio, produces .gz files. Best for real-time log compression where speed matters. bzip2: slower but achieves 10-15% better compression, produces .bz2 files. Best for archival where storage savings matter more than speed. In practice: use gzip for active log rotation, bzip2 for long-term archival to tape/cold storage like LTFS.

-i means "in-place" — edits the file directly instead of printing to stdout. On GNU sed (Linux) you can use -i alone; on BSD/macOS sed, you must provide a suffix like -i.bak. Best practice: ALWAYS backup before sed -i: cp -p file file.bak && sed -i '...' file. For critical files like /etc/fstab, /etc/passwd, always backup with a timestamp first.

In sed's replacement string, & represents the entire matched pattern. So s/^pattern/#&/ means: match ^pattern at start of line, and replace it with # + the entire match. This effectively prepends # to comment out matching lines without having to re-type the pattern in the replacement. Example: sed -i 's/^vm.swappiness/#&/' /etc/sysctl.conf comments out the vm.swappiness line.

sed '/pattern/d' deletes the entire line if pattern matches. sed 's/pattern//g' removes only the matched text within a line, leaving an empty line or shortened line. Use /d when you want to completely remove a line (like removing a server entry). Use s// when you want to change a value within a line.

Using a file (cat serverlist.txt) is a best practice because: (1) Easy to modify scope without changing the script. (2) Auditable — the list is a record of what systems were affected. (3) Reusable — same list for pre-check, action, and post-check loops. (4) Reduces typos — hostname is defined once. In large clusters (100+ nodes), hardcoding is impractical. Always verify the list before running destructive operations with a dry-run loop that just echos hostnames.

Use a pipeline combining ps, grep, awk/cut, and kill: kill $( ps -ef | egrep "BD_Setup|rpm" | grep -v grep | tr -s " " | cut -d" " -f2 | tr "\n" " " ). The grep -v grep is critical to exclude the grep process itself. Alternatively: pkill -f "pattern" or killall processname. For user-specific: ps -ef | grep 'username' | awk '{print $2}' | xargs kill.

Standard Unix permissions only support 3 permission sets: owner, group, others. ACLs (Access Control Lists) allow fine-grained permissions for any number of users and groups. Example: standard permissions can't grant read-only access to one specific user who isn't in the file's group — ACLs solve this. The + sign in ls -l output indicates ACLs are set. Use getfacl filename to view them. NFSv4 ACLs use different syntax with nfs4_setfacl and support inheritance flags.

SAR (System Activity Reporter) is part of the sysstat package. It collects system statistics (CPU, memory, I/O, network) on a scheduled basis via cron (usually every 10 minutes). Data is stored in binary format in /var/log/sa/ as files named saNN where NN is the day of month (e.g., sa15 = 15th of month). The sar command reads these files. Compared to top/vmstat, SAR provides historical trending — invaluable for post-mortem analysis of past performance issues.

CPU: ps aux --sort=-%cpu | head -n 15 (descending % CPU). Memory: ps aux --sort -rss | head -n 15 (descending RSS memory). For AIX: topas or svmon -Pt10 for top 10 memory consumers. On Solaris: prstat. For real-time Linux: top then press M for memory sort, P for CPU sort.

iostat -xm 2 shows extended disk statistics in megabytes, refreshing every 2 seconds. Key columns: %util (disk utilization — if near 100%, disk is bottleneck), await (average I/O wait time in ms — should be low), r/s and w/s (reads/writes per second), svctm (service time). Use when investigating slow applications, high I/O wait in top, or storage performance issues. Compare with multipath -ll to correlate to physical paths.

1. last reboot — check when it happened. 2. uptime — verify current uptime. 3. Check /var/log/messages around the reboot time for OOM killer, kernel panic, hardware errors. 4. Check dmesg for hardware errors. 5. journalctl -b -1 — previous boot journal (systemd). 6. Check ILO/iDRAC logs for hardware events (power loss, thermal). 7. Look for scheduled reboots in crontab: crontab -l | grep reboot. 8. Check crash dumps in /var/crash.

Enforcing: SELinux policy is active and violations are blocked and logged. This is the secure default for production systems. Permissive: Violations are logged but NOT blocked. Use during troubleshooting to determine if SELinux is causing an application issue, or when deploying new applications. Disabled: SELinux is completely off. Avoid in production; re-enabling requires a full filesystem relabel. Interview tip: Always prefer Permissive over Disabled for troubleshooting — it's reversible without a full relabel.

targeted: Only specific processes (network daemons like httpd, sshd, etc.) are protected by SELinux policy. Most processes run in the unconfined domain. This is the default and most common. minimum: A subset of targeted — only the most critical processes are protected. mls: Multi-Level Security — enforces Bell-LaPadula model with sensitivity labels. Used in government/military environments with classified data. Extremely strict, complex to manage.

In PAM, the control flag determines how a module's success/failure affects the overall authentication: required: Must succeed for authentication to pass. If it fails, authentication ultimately fails (but all remaining modules still run — this prevents revealing which module failed). sufficient: If this module succeeds and no prior required module has failed, authentication succeeds immediately — no further modules are checked. requisite: Like required, but failure causes immediate termination. Changing pam_access.so from required to sufficient means if access.conf check passes, cron access is granted without further checks.

The RPM database (Berkeley DB files in /var/lib/rpm/) gets corrupted when: (1) System crashes or loses power during an RPM operation. (2) Simultaneous RPM/yum processes creating lock conflicts. (3) NFS mounts used for RPM DB (file locking issues). (4) Out-of-space conditions mid-install. Symptoms: rpm -qa hangs or errors, yum fails with DB errors, package operations fail. The rebuild re-creates the index files from the raw Packages database. If db_verify Packages fails, the corruption is severe and may need restoring from backup.

A keytab (key table) file stores Kerberos principal credentials encrypted with the service's secret key, allowing a service to authenticate to Kerberos without a password prompt. It's essentially a long-lived credential file. Permissions must be 600 (owner read-write only) because: anyone who can read a keytab can authenticate as that Kerberos principal. Wider permissions are a critical security vulnerability. In Cloudera/Hadoop clusters, each service (HDFS, HBase, Solr) has its own keytab owned by its service account.

Tectia SSH (formerly SSH Tectia, originally from SSH Communications Security) is the commercial enterprise SSH product. OpenSSH is the open-source implementation. Key differences: Tectia uses its own key format and stores keys in /etc/opt/SSHtectia/keys/. It supports SSH2 protocol and has different authorization file syntax. Keys need to be converted when migrating between Tectia and OpenSSH servers using ssh-keygen-g3. Tectia also supports FIPS 140-2 mode and has centralized certificate management features.

NSCD (Name Service Cache Daemon) caches NSS lookups (passwd, group, hosts) to reduce latency and AD/LDAP query load. Corruption symptoms: getent passwd hangs or returns wrong data, id username fails, SSH authentication issues, sudo failures. The cache files in /var/db/nscd/ can become corrupted after crashes or LDAP inconsistencies. Fix: stop nscd, delete cache files (they rebuild on start), restart. This is a Level 1 troubleshooting step for "user not found" or AD auth failures.

vm.swappiness controls how aggressively the kernel swaps out anonymous memory pages (application data) to disk. Range: 0-100. Value 0 = avoid swapping unless OOM is imminent. Value 100 = swap aggressively. For most database servers and performance-critical applications: set to 1 (Hadoop/HBase recommendation). For general servers: 10-30. Default is 60 on RHEL/CentOS. High swappiness = excessive swapping = poor performance. Verify with: sysctl vm.swappiness.

vm.max_map_count limits the number of memory map areas a process can have. Default is 65536. Elasticsearch, Hadoop, Spark, and Java applications with large JVMs often require 262144-8000000. Symptom of too-low value: failed to create JVM, OOM errors, memory mapping failures. Elasticsearch specifically requires at least 262144. The GPA/Hadoop teams in the notes required 8000000. Changing this doesn't require reboot — write to /proc/sys/vm/max_map_count for immediate effect.

Dracut is the initramfs (initial RAM filesystem) framework in RHEL. During boot, dracut sets up the early boot environment before mounting root filesystem. If dracut can't complete (missing modules, bad fstab entry, disk issue, wrong kernel specified), it drops to a minimal shell prompt for debugging. Common causes: (1) Wrong default kernel in grubenv. (2) Missing or corrupted initramfs. (3) /etc/fstab entry for a non-existent device. (4) LVM volume group not activating. Fix: set correct kernel with grub2-set-default, or regenerate initramfs with dracut -f.

VCS (now InfoScale Availability) is a high-availability clustering solution. It monitors resources and automatically fails them over to another node when a fault is detected. A Service Group (SG) is a logical collection of resources (IP, disk, application processes, mount points) that represent a service. They all fail over together as a unit. Key concepts: Resources have dependencies (disk must be mounted before app starts). hastatus -sum shows health at a glance. hagrp -switch is used for planned maintenance to move services off a node gracefully.

Freezing a service group (hagrp -freeze) prevents VCS from automatically switching the group to another node even if faults are detected. This is used during planned maintenance — you don't want VCS to failover while you're intentionally stopping services. After maintenance, hagrp -unfreeze re-enables automatic failover. If you forget to unfreeze, manual intervention is required for any subsequent failover. Persistent freezing (-persistent flag) survives VCS restarts.

Tanium is an enterprise endpoint management and security platform that provides real-time visibility and control across all endpoints in an organization. From a sysadmin perspective: the Tanium Client runs on each server and communicates with the Tanium server. Common issues: (1) monitor.db grows too large and causes client to stop working — fix: clear it while service is stopped. (2) /tmp directory fills with trace files — fix: compress old files with find+gzip. (3) Client stops sending data — fix: restart service. It's similar to managing other management agents like IBM ITM or Splunk.