⚙️ ITIL & Unix Operations

KEDB (Known Error Database) is a repository of Known Errors — problems for which a root cause and workaround/fix has been identified. It allows support teams to resolve recurring incidents faster by referencing known solutions instead of starting from scratch. In ITSM tools like ServiceNow or BMC Remedy, KEs are linked to problem and incident records.

A workaround is a temporary solution that restores service without removing the root cause (e.g., restarting a service that crashes every 12 hours). It reduces impact but the problem persists. A permanent fix eliminates the root cause (e.g., patching the memory leak causing the crash). Permanent fixes usually go through the Change Management process.

Key logs: /var/log/messages or /var/log/syslog (system events), /var/log/secure (auth), dmesg (kernel ring buffer), /var/log/cron (scheduled jobs), application-specific logs, last and lastb (logins), ausearch (audit), cluster logs (/var/VRTSvcs/log/engine_A.log for VCS).

CAB (Change Advisory Board) is a group that reviews and approves Normal changes. Attendees typically include: Change Manager (chair), Service Owners, Application Owners, Infrastructure Leads (Unix/Windows/Network/Storage), Security, Business Representatives. ECAB (Emergency CAB) is a subset that convenes for Emergency changes and can approve within hours.

A Rollback Plan is a documented procedure to revert changes and return the system to its pre-change state. You invoke it when: the change fails validation, unexpected errors occur during implementation, SLA breach is imminent, or the business requests abort. For Unix: rollback may include restoring from snapshot, reverting RPM with rpm -e and reinstalling old version, restoring config files from backup, or failing back a cluster service group.

PIR is a review conducted after a significant change to evaluate whether it achieved its objectives, caused any unintended side effects, and to capture lessons learned. For Unix teams: verify services are stable, performance metrics are normal, no new incidents created, and documentation is updated. PIR findings feed back into Change and Problem Management.

1) Raise a Normal Change ticket with full implementation plan. 2) Get CAB approval and maintenance window. 3) Take VM snapshot or backup. 4) Drain active connections / notify users. 5) If clustered, evacuate service groups to peer nodes (hagrp -switch). 6) Apply patch (yum update kernel -y). 7) Reboot. 8) Validate: check services, check new kernel with uname -r. 9) Fail back cluster resources. 10) Monitor for 30-60 mins. 11) If issues: rollback to previous kernel via GRUB.

Least Privilege means users and processes get only the minimum access required to perform their job. In Unix: use sudo to grant specific commands (not full root), use ACLs for fine-grained file permissions, disable root SSH login (PermitRootLogin no in sshd_config), use PAM for authentication policies, regularly audit /etc/sudoers and remove stale entries, enforce password policies with chage.

Example: A banking application. RPO = 15 minutes (means: in a disaster, they can afford to lose at most 15 min of transactions — so replication must run every ≤15 min). RTO = 2 hours (means: the service must be back online within 2 hours of a disaster declaration). In Unix context: RPO drives your backup/replication frequency; RTO drives your DR automation, runbook complexity, and pre-provisioned standby infrastructure.

A DR drill tests the ability to fail over to DR and restore operations within RTO. Preparation: 1) Review and update DR runbooks. 2) Verify replication is healthy. 3) Raise a Change ticket (Normal change). 4) Coordinate with application, DB, and network teams. 5) Define success criteria and rollback plan. 6) Execute: failover, validate services, document any deviations. 7) Post-drill: update runbooks, raise incidents for gaps found, file RCA if RTO/RPO not met.