VERIQTA DevOps Bible — 40-Page Complete Reference

♾️ DevOps Bible

From Engineering Foundations to Production DevOps — The Complete 40-Page Handbook for Engineers

40 Pages Linux · Git · Cloud · Terraform · Ansible Docker · Kubernetes · CI/CD · GitOps SRE · DevSecOps · Observability · Incident Response
📚 Source: VERIQTA Engineering Excellence — DevOps Bible (40-page handbook)
Covering the complete DevOps lifecycle: Develop → Git & Collaboration → CI/CD Pipeline → Container & Kubernetes → Cloud Infrastructure → Monitoring & Observability. Built for: Developers, DevOps Engineers, SREs, Platform Engineers, Cloud Engineers, Tech Leads, Engineering Teams.
♾️

1. DevOps Engineering in the Real World

Build better. Deploy faster. Operate smarter.

CORE

What DevOps Actually Means

DevOps is a mindset, culture, and set of practices that brings development and operations together. It focuses on delivering value to users faster, with higher quality and reliability. DevOps removes silos, improves collaboration, and automates everything from code to production.

Development (DEV)
  • Write and maintain code
  • Build and test applications
  • Commit and manage source code
  • Create automated tests
  • Ensure code quality
  • Package and version artifacts
  • Work closely with operations
Operations (OPS)
  • Provision and manage infrastructure
  • Deploy and run applications
  • Monitor systems and applications
  • Ensure availability and performance
  • Manage security and access
  • Handle incidents and outages
  • Continuously optimize systems
★ Key Point: DevOps = People + Process + Technology. Culture first, tools second.
LIFECYCLE

DevOps Lifecycle — 8 Stages

📋 PLAN
Plan features & requirements
</> CODE
Write & commit changes
🔨 BUILD
Compile & create artifacts
🧪 TEST
Run automated tests & scans
📦 RELEASE
Package to artifact repo
🚀 DEPLOY
Deploy to staging/production
📡 OPERATE
Monitor & maintain
📈 IMPROVE
Collect feedback & iterate
Continuous Feedback Loop: Every stage feeds information back to the previous stages, enabling rapid improvement.
ROLES

DevOps vs SRE vs Platform Engineering

RoleFocusKey Principle
DevOpsCulture and practices uniting Dev+Ops for faster, reliable deliveryCulture drives tooling
SRE (Site Reliability Engineering)Applies engineering to ops. Focuses on reliability, SLIs, SLOs, error budgets, reducing toilReliability is an engineering problem
Platform EngineeringBuilds internal platforms providing self-service tools, scaffolding, golden pathsEnable developer self-service at scale
DevOps is CULTURE. SRE is RELIABILITY. Platform Engineering is ENABLING SCALE.
MINDSET

Production Engineering Mindset

  • Own the full lifecycle from code to production
  • Design for reliability, scalability, and security
  • Automate to reduce manual work and errors
  • Monitor everything that matters
  • Measure, learn, and improve continuously
  • Think in systems, not in isolated components
  • Prepare for failure; build for recovery
  • Focus on customer value and user experience
  • Take responsibility and act with urgency
  • Document, standardize, and share knowledge
THINK LIKE AN OWNER. BUILD LIKE A PROFESSIONAL. OPERATE LIKE A CHAMPION.
🐧

2. Linux for DevOps Engineers

Understand. Operate. Automate. Scale.

LINUX

Linux Architecture & Filesystem

Architecture Layers
  • User Applications — top layer, what users run
  • System Libraries — interfaces to kernel functionality
  • Kernel — manages CPU, RAM, Disk, Network
  • Shell — user interface to the kernel
  • Everything is a file in Linux

Filesystem Hierarchy
PathPurpose
/Root directory
/binEssential binaries
/etcSystem configuration
/homeUser home directories
/varVariable data (logs)
/optOptional applications
/procProcess information
/sysSystem information
/tmpTemporary files
Users, Groups & Permissions
  • Create user: useradd
  • Modify user: usermod
  • Delete user: userdel
  • Switch user: su - <user>
  • View groups: groups
  • Add to group: usermod -aG
  • Principle of least privilege always

Permissionsrwxr-xr--
  • r=Read(4), w=Write(2), x=Execute(1)
  • chmod 755 file — change mode
  • chown user:group file — change owner
  • chgrp group file — change group
SERVICES

Services, Packages & Systemd

Services (systemctl)
systemctl start nginx systemctl stop nginx systemctl restart nginx systemctl status nginx systemctl enable nginx systemctl disable nginx systemctl list-units --type=service systemctl list-units --state=running
Package Management
RHEL/CentOS/Rocky:
yum update (old) yum install yum remove yum search dnf install
Debian/Ubuntu:
apt update apt install apt remove apt search apt upgrade -y
Systemd & Logs
# Logs journalctl -xe journalctl -f journalctl -u -f journalctl -b # boot logs journalctl -b -1 # prev boot tail -f /var/log/syslog tail -f /var/log/messages grep -i "error" /var/log/* # Auth logs /var/log/auth.log
COMMANDS

Essential Production Commands

CategoryCommandDescription
File & Dirls -lahList files with details
File & Dirfind /path -name "file"Find files
File & Dirrm -rf dirRemove directory forcefully
System Infouname -aKernel and system info
System InfouptimeSystem uptime + load
System Infofree -hMemory usage
System Infodf -hDisk usage
Networkingip aIP address
Networkingss -tulnpListening ports
Networkingping hostCheck connectivity
Networkingssh user@hostSSH connection
Logsjournalctl -fFollow all logs
Logstail -f /var/log/syslogLive system log
Logsgrep -i error /var/log/*Search errors
Permissionschmod 755 fileChange permissions
Permissionschown user:group fileChange owner
Processesps auxAll processes
Processestop / htopReal-time monitor
Processeskill -9 <PID>Kill process
MASTER LINUX FUNDAMENTALS. IT IS THE FOUNDATION OF EVERY DEVOPS ENGINEER'S DAY-TO-DAY WORK.
🔧

3. Linux Production Troubleshooting

Find problems fast. Diagnose accurately. Fix confidently.

DIAGNOSE

10 Troubleshooting Areas & Key Commands

AreaKey CommandsWhat to Look For
1. CPU Investigationtop, htop, mpstat -P ALL 1, ps -eo pid,ppid,cmd,%cpu --sort=-%cpu, sar -u 1 5, vmstat 1High %usr (app), high %sys (kernel), high %iowait (disk)
2. Memory Pressurefree -h, top, htop, vmstat 1, swapon --show, ps -eo pid,cmd,%mem --sort=-%mem, cat /proc/meminfoOOM kills in dmesg, swap in/out active, low available memory
3. Disk Utilizationdf -h, du -sh /* 2>/dev/null | sort -h, ncdu /, lsblk, iotop -oPa, sar -d 1 5100% disk usage, inode exhaustion, large log files
4. Filesystem Problemsdmesg | tail, fsck -n /dev/sdX#, mount | column -t, findmnt, umount /mount/point, journalctl -k | tailEXT4/XFS errors, read-only filesystem, corruption
5. Process Failuresps aux, pstree -p, kill -9 <PID>, killall -9 <process>, pkill -f <pattern>Zombie processes, crashed services, runaway CPU/mem
6. Service Failuressystemctl status/start/stop/restart/enable <service>, journalctl -u <service> -fService failed to start, dependency errors, config issues
7. Log Investigationjournalctl -xe, journalctl -f, tail -f /var/log/messages, grep -i "error" /var/log/*Error patterns, timing of failures, service crashes
8. Network Troubleshootingip a, ip r, ss -tulnp, ping <host>, traceroute <host>, dig <domain>, curl -I http://<host>, mtr <host>Timeouts, packet loss, DNS failures, firewall blocks
9. Performance Bottleneckstop, htop, vmstat 1, iostat -xz 1, sar -q 1 5, pidstat 1, iotop -oPa, perf topCPU/memory/disk/network saturation
10. Production WorkflowConfirm scope → Check recent changes → Collect evidence (metrics, logs) → Form hypothesis → Identify root cause → Apply safe fix → Verify → DocumentAlways follow the structured process
RULES

Troubleshooting Golden Rules + Quick Health Check

🔍 OBSERVE
Collect facts, not assumptions
🎯 ISOLATE
Reduce to smallest possible scope
🧪 HYPOTHESIZE
Form theory based on evidence
✅ TEST
Prove or disprove with commands
🔧 FIX SAFELY
Apply least risky effective change
✔️ VERIFY
Confirm the fix and monitor
📝 DOCUMENT
Record what happened and what you learned
CategoryQuick Health Check Commands
System Overviewuptime, w, who -a, hostnamectl
CPUtop, htop, mpstat -P ALL 1, sar -u 1 5
Memoryfree -h, vmstat 1, cat /proc/meminfo
Diskdf -h, du -sh /* 2>/dev/null | sort -h, lsblk
I/Oiostat -xz 1, iotop -oPa, iotop -boPa, sar -d 1 5
Networkip a, ip r, ss -tulnp, ping <host>
Logsjournalctl -xe, journalctl -u <service>, tail -f /var/log/messages
A CALM MIND. A CLEAR PROCESS. THE RIGHT COMMANDS. THAT IS HOW PRODUCTION PROBLEMS ARE SOLVED.
🌐

4. Networking Every DevOps Engineer Must Know

Understand the network. Troubleshoot with confidence. Build reliable systems.

NETWORK

Core Networking Concepts

ConceptDetails
TCP/IP LayersApplication (L7) → Transport (L4) → Internet (L3) → Network Access (L2). Data travels from application to network access.
IPv4 Address32-bit, dotted decimal (192.168.1.10). Network + Host portions. Private: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
IPv6128-bit, hex notation. Replaces IPv4 for scale.
CIDR192.168.1.0/24 = 24 bits network + 8 bits host = 256 addresses. /24 = 255.255.255.0
SubnettingDivide /24 into /26 (4 subnets × 64 hosts). Improves security, performance, organization.
RoutingRouters use routing tables to forward packets. Default route 0.0.0.0/0 = unknown destinations.
DNSTranslates domain names to IPs. Record types: A, AAAA, CNAME, MX, TXT. TTL controls cache duration.
Ports0-1023 well-known (22=SSH, 80=HTTP, 443=HTTPS). 1024-49151 registered. 49152-65535 dynamic.
TCP vs UDPTCP = reliable, ordered, connection-oriented (web, SSH). UDP = fast, connectionless (DNS, VoIP, streaming).
NATAllows private IPs to access public internet. SNAT, DNAT, PAT (NAT Overload).
FirewallsControl inbound/outbound traffic. Stateless (packets) or Stateful (connections). Rules: Allow/Deny by IP, Port, Protocol.
HTTP/HTTPSHTTP (port 80) = unencrypted. HTTPS (port 443) = encrypted with TLS. Always use HTTPS in production.
TLSEncrypts data in transit. Uses certificates, keys, secure ciphers. Ensures confidentiality, integrity, authentication.
Load BalancingDistributes traffic across multiple servers. L4 (transport) or L7 (application). Algorithms: Round Robin, Least Connections, IP Hash.
STRONG NETWORKING KNOWLEDGE IS THE FOUNDATION OF RELIABLE, SECURE, AND SCALABLE SYSTEMS.
🌿

5. Git & Production Version Control

Track changes. Collaborate safely. Ship with confidence.

GIT

Git Repository Workflow & Commands

📁 Working Directory
Make changes
➕ Staging Area
Stage for commit
💾 Local Repository
Commit changes
☁️ Remote Repository
Push commits
OperationCommandsNotes
Basic workflowgit status, git add <file>, git add ., git commit -m "msg", git pushDaily workflow
Branchinggit branch, git branch <name>, git switch <name>, git switch -c <name>Isolate work in branches
Merginggit merge <branch>, git merge --no-ff <branch>Creates merge commit
Rebasinggit rebase <branch>, git rebase -i <branch>, git rebase --abort, git rebase --continueClean linear history
Pull Requestsgit push origin <branch>, git fetch origin, git pull origin <branch>Propose → Review → Merge
Tagsgit tag, git tag -a v1.0.0 -m "message", git push origin v1.0.0Mark releases
Release Branchesgit switch -c release/1.2, git merge release/1.2, git branch -d release/1.2Stabilize before production
Troubleshootinggit restore <file>, git stash, git reset --soft HEAD~1, git revert <commit>, git reflog, git cherry-pick <commit>Fix common mistakes
ProblemCauseSolutionCommand
Accidental changes not committedChanges still in working directoryDiscard or stashgit restore <file> / git stash
Committed to wrong branchWrong commit madeMove to correct branchgit cherry-pick <commit>
Need to undo last commitWrong commit or messageUndo, keep changesgit reset --soft HEAD~1
Pushed wrong commitBad commit already pushedRevert commit safelygit revert <commit>
Merge conflictsChanges between branchesResolve manuallygit status → fix → git add → git commit
Detached HEADChecked out a commitSwitch back to branchgit switch <branch>
Lost commitsCommits not reachableUse refloggit reflog
GOOD VERSION CONTROL IS THE FOUNDATION OF COLLABORATION, QUALITY, AND RELIABLE DELIVERIES.
☁️

6. Cloud Infrastructure Fundamentals

The building blocks of modern cloud platforms.

CLOUD

Cloud Infrastructure — 11 Building Blocks

Building BlockDescriptionExamples
1. ComputeVirtual machines and containers. Scale up/down on demand. Pay for what you use.VM, Containers, Serverless Functions
2. StorageStore and retrieve data durably and securely. Different types for different needs.Object Storage, Block Storage, File Storage
3. NetworkingConnect resources securely. VPC, subnets, IP addressing, routing.VPC, DNS, VPN, Gateways
4. DatabasesManaged database services for reliability and scale. Relational and NoSQL.RDS, Aurora, DynamoDB, Cloud SQL
5. IdentityManage users, roles, permissions. Least privilege principle.IAM, Roles, Policies, MFA
6. Load BalancingDistribute traffic. Health checks ensure only healthy targets receive traffic.ALB, NLB, Gateway LB
7. Auto ScalingAutomatically adjust capacity based on demand. Scale out or in.Auto Scaling Groups, Scale Sets
8. Availability ZonesPhysically separate data centers within a region. Isolate failures.AZ1, AZ2, AZ3
9. RegionsGeographically separate areas. Choose close to users.us-east-1, eu-west-1, ap-southeast-1
10. Shared ResponsibilityProvider secures the cloud. You secure what is in the cloud.Know your responsibilities
11. High AvailabilityDesign to remain available during failures. Multi-AZ, LB, auto scaling.Eliminate single points of failure
💻 COMPUTE
💾 STORAGE
🌐 NETWORKING
🗄️ DATABASES
🔑 IDENTITY
⚖️ LOAD BALANCING
📈 AUTO SCALING
🏢 AZs
✅ HIGH AVAILABILITY
KEY: Build → Connect → Secure → Scale → Deliver. Design for security, reliability, and performance from day one.
🏗️

7. Production Cloud Architecture

Design resilient, scalable, and highly available cloud systems.

ARCH

Production Cloud Architecture — Multi-AZ Design

ComponentRoleKey Detail
Internet GatewayConnects VPC to internetRequired for public subnet access
DNS (Route 53)Domain name resolutionRoutes users to load balancer
Load Balancer (ALB)Distributes traffic to healthy app instancesSpans multiple AZs
Public SubnetHouses load balancer, NAT GatewaysOne per AZ
Application Tier (Private Subnet)Runs stateless app containers/VMsNo direct internet access
Database Tier (Private Subnet)Primary + Replica across AZsReplication for HA
NAT GatewayPrivate subnet → internet (outbound only)One per AZ for HA
Architecture PatternDescriptionUse Case
3-Tier ArchitectureWeb → App → DB tiers separatedStandard web apps
MicroservicesIndependently deployed servicesScale individual components
Multi-AZ HAResources across multiple AZsProduction workloads
Active-Passive (DR)Primary region + standby regionDisaster recovery
Active-Active (Multi-Region)Live traffic in multiple regionsGlobal, low-latency apps
ServerlessNo infrastructure managementEvent-driven, variable workloads
Blue-GreenTwo identical environments, switch trafficZero-downtime deployments
Key PrincipleDescription
Fault IsolationIsolate failures so issues in one part do not affect others
ScalingScale application tiers horizontally based on demand
ResilienceDesign for failure. Systems recover automatically and quickly.
High AvailabilityUse multiple AZs and remove single points of failure
SecurityLeast privilege and defense in depth everywhere
A STRONG ARCHITECTURE TODAY PREVENTS INCIDENTS TOMORROW. DESIGN FOR SCALE, SECURITY, AND RELIABILITY FROM DAY ONE.
📦

8. Infrastructure as Code (Terraform)

Build, manage, and scale infrastructure with code.

IAC

Infrastructure as Code — Why & Terraform Fundamentals

Why IaC?
  • Version controlled and auditable
  • Consistent and repeatable deployments
  • Automated and faster provisioning
  • Easy to scale and modify
  • Real-time collaboration
  • Reduced manual errors
  • Disaster recovery and restore
  • Enables CI/CD for infrastructure
  • Cost-effective and efficient
  • Infrastructure becomes a product
Declarative Infrastructure
Desired State → Terraform → Real Infrastructure

YOU DEFINE THE DESIRED STATE. TERRAFORM ACHIEVES IT.
Terraform continuously compares your desired state (in .tf files) to real infrastructure and reconciles differences.
Terraform ConceptDescriptionExample
ProvidersPlugins that interact with cloud APIsaws, azurerm, google, kubernetes
ResourcesInfrastructure objects you create and manageaws_instance, aws_s3_bucket
VariablesMake code dynamic and reusableregion, instance_type
OutputsReturn important values after applyinstance IP, endpoint, DNS name
ModulesOrganize and reuse code. Build once, use everywhere.modules/vpc/, modules/eks/
StateTerraform's memory of your infrastructure. Maps config to real resources.terraform.tfstate
Remote StateStore state remotely for collaboration, locking, safety.S3 + DynamoDB, Azure Storage
IdempotencyApply multiple times, same result. No changes if matches desired state.terraform plan shows "No changes"
📝 WRITE
Define in .tf files
🔍 PLAN
Review execution plan
✅ APPLY
Provision/update resources
⚙️ MANAGE
Terraform tracks state
🗑️ DESTROY
Remove when no longer needed
INFRASTRUCTURE AS CODE IS THE FOUNDATION OF MODERN DEVOPS. CODE IT. VERSION IT. AUTOMATE IT. SCALE IT.
🚀

9. Production Terraform

Build, manage, and operate infrastructure at scale.

TERRAFORM

Production Terraform — Best Practices

TopicDetails
Module ArchitectureOrganize into reusable modules. Follow DRY principle. Keep modules small and focused. Publish and version modules.
Environment SeparationSeparate environments with folders/workspaces. Separate state per environment. Different variables per env. (dev/stage/prod)
State LockingPrevent concurrent changes to state. Use state locking to avoid corruption. Always enable locking.
Remote BackendsStore state remotely: AWS S3 + DynamoDB locking, Azure Storage, Google Cloud Storage.
Dependency ManagementUse dependency blocks when needed. Use data sources for existing resources. Run terraform init to install.
Drift DetectionDrift = real infra differs from state. Use terraform plan to detect. Integrate detection in pipelines.
Importing ResourcesBring existing resources under Terraform: terraform import aws_s3_bucket.bucket my-bucket
SecurityLeast privilege IAM. Encrypt state at rest. Do not commit secrets. Use variables and secrets managers.
# Core workflow terraform init # download providers and modules terraform validate # validate configuration syntax terraform plan # preview changes — ALWAYS review first terraform apply # apply approved changes terraform destroy # remove resources # Environment structure environments/ ├── dev/ │ └── main.tf ├── stage/ │ └── main.tf └── prod/ └── main.tf # Module structure modules/ ├── vpc/ ├── subnet/ ├── security-group/ ├── eks/ └── rds/ # Remote backend config terraform { backend "s3" { bucket = "my-tfstate" key = "prod/terraform.tfstate" region = "us-east-1" dynamodb_table = "terraform-locks" encrypt = true } }
INFRASTRUCTURE AS CODE IS NOT JUST AUTOMATION. IT IS RELIABILITY, REPEATABILITY, AND CONTROL AT SCALE.
⚙️

10. Configuration Management — Ansible

Automate, standardize, and maintain infrastructure at scale.

ANSIBLE

Configuration Management with Ansible

ConceptDescription
InventoryDefines hosts Ansible manages. Organize into groups. Static (INI/YAML) or dynamic inventory.
PlaybooksYAML files defining automation tasks. Sequence of tasks, easy to read and version. Reusable and shareable.
RolesOrganize playbooks into reusable roles. Standard structure: tasks/, handlers/, templates/, files/, vars/, defaults/, meta/.
VariablesStore values for reuse in playbooks, roles, templates. Sources: vars, defaults, extra-vars, inventory.
TemplatesGenerate dynamic config files using Jinja2 templating. Insert variables. Ensure consistency across systems.
IdempotencyTasks run multiple times with same result. Prevents drift and unnecessary changes.
SecretsNever store plain secrets in playbooks. Use Ansible Vault. Use external secret stores (Vault, AWS Secrets Manager).
Configuration DriftDetect drift using Ansible runs. Revert to desired state. Schedule regular audits.
# Basic playbook structure - hosts: web become: yes tasks: - name: Install Nginx apt: name: nginx state: present - name: Start and enable nginx systemd: name: nginx state: started enabled: yes - name: Deploy config from template template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf notify: Restart nginx handlers: - name: Restart nginx systemd: name: nginx state: restarted # Run playbook ansible-playbook -i inventory playbook.yml ansible-playbook -i inventory playbook.yml --check # dry run ansible-playbook -i inventory playbook.yml --tags install # Ansible Vault ansible-vault encrypt secrets.yml ansible-vault decrypt secrets.yml ansible-vault edit secrets.yml
📋 DEFINE DESIRED STATE
✍️ WRITE PLAYBOOKS
▶️ RUN ANSIBLE
⚙️ APPLY CHANGES TO NODES
✅ VERIFY & MAINTAIN COMPLIANCE
CONFIGURATION MANAGEMENT IS THE FOUNDATION OF RELIABLE INFRASTRUCTURE. AUTOMATE TODAY. OPERATE BETTER TOMORROW.
🐋

11. Docker Fundamentals

Build. Ship. Run. Anywhere.

DOCKER

Docker Fundamentals — Containers vs VMs

AspectVirtual MachinesContainers
IsolationFull OS per VM (Guest OS + Hypervisor)Process isolation (shared host OS kernel)
Startup timeMinutesSeconds or milliseconds
SizeGBs (full OS image)MBs (just app + dependencies)
Resource usageHigh (full OS overhead)Low (shared kernel)
PortabilityLimited (OS dependent)High (OCI image standard)
ConceptDescription
ImagesRead-only template with everything to run an application. Built from Dockerfile. Versioned and immutable.
ContainersRunning instance of an image. Isolated process environment. Has own writable layer.
DockerfilesText file with build instructions: FROM, COPY, RUN, CMD, EXPOSE. Enables automation and consistency.
LayersImages built in layers. Each instruction adds a layer. Layers are cached and reused. Only changed layers rebuild.
RegistriesStore and distribute images. Public (Docker Hub). Private (ECR, GCR, ACR, Harbor).
VolumesPersist data outside containers. Survive restarts. Named Volumes, Bind Mounts, tmpfs.
NetworksEnable communication. Types: bridge (default), host, overlay, none, macvlan.
Environment VariablesKey-value pairs passed to containers. Set at build or runtime (-e). Keep images reusable.
# Example Dockerfile FROM nginx:alpine COPY ./html /usr/share/nginx/html EXPOSE 80 CMD ["nginx", "-g", "daemon off;"] # Container Lifecycle Flow docker create → docker start → docker ps → docker stop → docker pause → docker unpause → docker restart → docker kill → docker rm # Common commands docker build -t myapp:1.0.0 . docker run -d myapp:1.0.0 docker ps # running containers docker logs docker inspect docker stats # resource usage docker rm -f docker system prune -a # clean up
Docker gives you a consistent, portable, and efficient way to package and run applications.
🔒

12. Production Docker

Build secure. Run efficient. Operate at scale.

PRODUCTION

Production Docker — 10 Best Practices

PracticeDetailsExample
1. Multi-stage BuildsKeep final image small. Exclude build tools and cache.Builder stage (golang:1.22) → Final stage (alpine:3.19)
2. Small ImagesUse minimal base images: alpine, distroless, debian-slim. Remove unnecessary packages.alpine:latest, gcr.io/distroless/base
3. Image TaggingUse consistent tagging: SemVer or date tags. Use immutable tags. Avoid "latest" in production.myapp:1.0.0, myapp:2024-05-10, myapp:sha-4f3a1b2
4. Image ScanningScan for vulnerabilities. Tools: Trivy, Grype, Docker Scout. Fail builds on high/critical.trivy image myapp:1.0.0
5. Non-Root ContainersDo not run as root. Create dedicated user. Drop unnecessary capabilities.RUN adduser -D appuser; USER appuser
6. Resource LimitsLimit CPU and memory. Prevent noisy neighbor issues.--cpus="1.0" --memory="512m" --pids-limit=100
7. Health ChecksDetect failures early. Docker can restart unhealthy containers.HEALTHCHECK --interval=30s --timeout=5s --retries=3 CMD curl -f http://localhost:8080/health
8. Persistent StorageUse volumes for persistent data. Backup important data.docker run -d -v mydata:/var/lib/mysql myapp
9. Container NetworkingUser-defined bridge networks. Enable service discovery. Avoid exposing unnecessary ports.docker network create --driver bridge mynet
10. TroubleshootingCheck container status, inspect, logs, resource usage, networks and volumes.docker ps, docker logs, docker inspect, docker stats
Production Docker is about security, efficiency, reliability, and operational excellence. Build containers that are fast, safe, and production-ready.
☸️

13–17. Kubernetes — Architecture, Workloads, Networking, Storage & Operations

Run containers at scale. Build reliable, self-healing systems.

K8S ARCH

Kubernetes Architecture — Control Plane & Worker Nodes

ComponentRole
API ServerSingle entry point for all admin tasks. Front-end for the control plane. Validates and stores data in etcd.
etcdConsistent key-value store for ALL cluster data: nodes, pods, configs, secrets, state. Source of truth.
SchedulerWatches for new pods and selects the best node based on resources, policies, constraints.
Controller ManagerRuns controllers: Deployment, ReplicaSet, Node, Endpoints. Maintains desired state.
KubeletAgent on each worker node. Ensures containers defined in pods are running and healthy.
Kube-ProxyHandles networking and load balancing for Services on each node.
NodesWorker machines running containerized workloads.
PodsSmallest deployable units. One or more containers sharing network and storage.
USER/CLIENT
→ API SERVER
→ etcd (stores)
→ SCHEDULER (places)
→ NODE/KUBELET (runs)
→ POD (running)
Kubernetes Reconciliation Loop: Desired State (etcd) ↔ Actual State (cluster). Kubernetes continuously compares and takes action to make them match. This is the core of Kubernetes self-healing.
WORKLOADS

Kubernetes Workloads — 10 Types

WorkloadDescriptionUse Cases
PodsSmallest deployable unit. Ephemeral by nature. Containers share network/storage.Single container apps, tightly coupled containers
DeploymentsManages ReplicaSets. Declarative updates. Rolling updates and rollbacks.Stateless apps, web apps, APIs, microservices
ReplicaSetsEnsures specified number of pod replicas running. Usually managed by Deployments.Maintain replica count, high availability
StatefulSetsManages stateful applications. Stable network identities and storage. Ordered deployment.Databases, message queues, stateful applications
DaemonSetsRuns a pod on all (or some) nodes. Keeps pod running on node additions.Logging agents, monitoring agents, network plugins
JobsRuns a pod to completion. Ensures task succeeds. Does not restart on success.Batch processing, one-time tasks, data migration
CronJobsRuns Jobs on a schedule. Uses cron syntax. Automates recurring tasks.Backups, data cleanup, scheduled reports
Rolling UpdatesUpdate pods gradually without downtime. Old pods replaced with new ones.Zero-downtime deployments, safe version upgrades
RollbacksRevert to a previous ReplicaSet or version. Quick recovery from bad deployments.Failed deployments, bug fixes, quick recovery
Application LifecycleFull journey: Create → Deploy → Scale → Update → Monitor → Delete.All production applications
K8S NET

Kubernetes Networking & Traffic

Service TypeDescriptionAccess
ClusterIP (default)Stable virtual IP accessible only within the cluster. Ideal for internal communication.Internal only
NodePortExposes service on a static port (30000-32767) on each node's IP. Accessible from outside.External via Node IP:Port
LoadBalancerProvisions an external cloud load balancer. Internet-facing access to the service.External via cloud LB
IngressHTTP/HTTPS routing to services. Host and path-based routing. TLS termination. Single entry point.External HTTP/HTTPS
ConceptDescription
DNSK8s assigns DNS names to Services. Format: <service>.<namespace>.svc.cluster.local
Network PoliciesControl traffic between pods. Allow/deny based on rules. Applied at namespace or pod level. Enforced by CNI plugin.
Service DiscoveryServices get stable DNS. Clients discover services automatically. No hardcoded IPs. Works across namespaces.
Container NetworkingPods get IPs from cluster network. Flat network model — all pods communicate without NAT. CNI plugins: Calico, Cilium, Flannel.
USER
→ LOAD BALANCER
→ INGRESS (routes)
→ SERVICE/ClusterIP
→ PODS (backend)
Traffic Flow Example: https://app.example.com/api → Ingress → Service (ClusterIP) → Pods. Service load balances traffic to healthy pods.
K8S CONFIG

Kubernetes Configuration & Storage

ResourcePurposeUse For
ConfigMapsStore non-sensitive configuration as key-value pairsApp config, feature flags, env vars
SecretsStore sensitive information securely (base64 encoded)Passwords, API keys, certificates
VolumesStorage accessible to containers in a podTemporary (emptyDir), host (hostPath), persistent (PVC)
PersistentVolumes (PV)Cluster resources representing physical storage. Lifecycle independent of pods.Admin-provisioned storage
PersistentVolumeClaims (PVC)User requests for storage. Binds to a PV. Abstracts storage details from pods.App requesting storage
StorageClassesDefine how storage is provisioned. Specify provisioner and parameters.Dynamic provisioning (AWS EBS, GCE PD)
# ConfigMap example apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_NAME: myapp APP_ENV: production # Secret example apiVersion: v1 kind: Secret metadata: name: db-secret type: Opaque data: username: YWRtaW4= # base64 encoded password: cGFzc3dvcmQ= # PVC example apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: fast-storage
K8S OPS

Production Kubernetes Operations

FeaturePurposeExample
Requests & LimitsGuarantee resources (requests) and cap usage (limits). Prevent noisy neighbor.cpu: 250m/1000m, memory: 256Mi/1Gi
Liveness ProbeDetects unhealthy container → K8s restarts it.httpGet /healthz port 8080
Readiness ProbeDetects when container is ready to receive traffic. Pod removed from Service if fails.httpGet /ready port 8080
Startup ProbeHandles slow-starting containers. Disables liveness/readiness until startup succeeds.httpGet /startup, failureThreshold: 30
HPAAutomatically scales pods based on CPU or custom metrics.minReplicas: 2, maxReplicas: 10, targetCPU: 70%
Pod Disruption BudgetsEnsures minimum pods stay available during disruptions (e.g., node maintenance).minAvailable: 2
Node AffinitySchedule pods on preferred or required nodes.requiredDuringSchedulingIgnoredDuringExecution
Taints & TolerationsPrevent pods from scheduling on certain nodes unless tolerated. Protect dedicated nodes.kubectl taint nodes node1 key=value:NoSchedule
RBACControl who can do what in the cluster. Apply least privilege.Role + RoleBinding, ClusterRole + ClusterRoleBinding
Production Operations Checklist: Set resource requests/limits → Configure all 3 probes → Enable HPA → Configure PDB → Use node affinity where needed → Apply taints/tolerations → Secure with RBAC → Monitor and alert → Backup data → Test recovery and failover.

18–22. CI/CD, Production Pipelines, Artifacts, Deployment & GitOps

Automate. Validate. Deliver. Repeat.

CI/CD

CI/CD Fundamentals & Pipeline Stages

ConceptDescription
Continuous Integration (CI)Developers integrate code frequently into shared repository. Automated build on every commit. Early issue detection.
Continuous Delivery (CD)Code is always in a deployable state. Automated testing + quality gates. Manual approval for production.
Continuous Deployment (CD)Every passing change is automatically deployed to production. Fully automated release.
StageWhat HappensExamples
1. BuildCompile source code, create artifactsmvn clean package, docker build
2. TestRun automated tests to ensure qualityUnit tests, integration tests, pytest
3. PackagePackage for distributionCreate JAR, WAR, or Docker image
4. ScanSecurity, vulnerability, compliance scanningSAST, SCA, Container Scan, Secret Scan
5. DeployDeploy to target environmentKubernetes, VM, Serverless
6. ValidateVerify deployment and application healthSmoke tests, health checks, monitoring
👨‍💻 Developer Push Code
→ CI Server Triggers
→ Build & Compile
→ Test
→ Package Artifact
→ Security Scan
→ Deploy to Staging/Prod
→ Validate & Monitor
→ Feedback Loop
GITOPS

GitOps & Continuous Delivery with Argo CD

GitOps PrincipleDescription
Git as Source of TruthAll configurations in Git. Single source of truth. Versioned and auditable. Enables traceability.
Desired StateDefine desired state in Git. Cluster converges to that state. No manual changes. Declarative.
GitOps WorkflowDeveloper pushes → GitOps tool detects change → Syncs to cluster → Cluster becomes consistent → Continuous monitoring
Argo CDDeclarative CD tool. Monitors Git repositories. Syncs to Kubernetes clusters. Web UI and CLI. Built-in observability.
Drift DetectionGitOps continuously detects and highlights when actual ≠ desired state. Triggers sync to fix.
RollbacksRevert Git commit → GitOps tool detects change → Automatically rolls back → Safe and auditable rollbacks
Environment PromotionPromote changes across environments: Dev → Test → Staging → Prod. Controlled and traceable.
# GitOps repository structure apps/ ├── my-app/ │ ├── base/ │ │ ├── deployment.yaml │ │ ├── service.yaml │ │ └── configmap.yaml │ └── overlays/ │ ├── dev/ (kustomization.yaml, values.yaml) │ ├── staging/ │ └── prod/ # Argo CD sync policies Manual: Sync only when manually triggered Automatic: Sync automatically on changes Automatic + Prune: Sync and remove resources no longer in Git Automatic + Self Heal: Sync and revert any drift automatically # Key commands argocd app get my-app argocd app diff my-app argocd app sync my-app argocd app rollback my-app
GitOps Best Practices: Keep Git repo clean → Use meaningful commit messages → Review changes before merging → Automate tests before promotion → Monitor and alert on drift → Document and follow GitOps standards.
👁️

23–26. Observability, Prometheus, Grafana & Logging

See Everything. Understand Deeply. Act Quickly.

OBSERVABILITY

Observability Fundamentals

PillarDescriptionExamples
MetricsNumeric measurements over time. Aggregated, fast, and efficient.CPU usage, request rate, latency, error rate
LogsDiscrete events with timestamps. Detailed records for deep investigation.Error logs, access logs, application logs
TracesRequest flow across services. Shows latency and dependencies.Distributed request trace across microservices
EventsImportant state changes or notifications. Lightweight and meaningful.Deployment succeeded, pod crashed, alert fired
TelemetryThe signals collected from systems. Includes metrics, logs, traces, and events.Everything your systems emit
MethodFocusUse For
Golden SignalsLatency, Traffic, Errors, SaturationUnderstanding health of any service
RED MethodRate, Errors, DurationService reliability from user perspective
USE MethodUtilization, Saturation, ErrorsInfrastructure health and capacity
Monitoring vs Observability: Monitoring tells you WHAT is happening. Observability tells you WHY. Monitoring = reactive (predefined alerts). Observability = exploratory (answers any question).
PROMETHEUS

Prometheus & Metrics

ConceptDescription
ArchitecturePrometheus Server (Retrieval → TSDB → PromQL → Alerting). Visualized in Grafana. Long-term: Thanos/Cortex.
TargetsAnything exposing metrics via HTTP endpoint. Defined in scrape configs or service-discovered.
ExportersExpose metrics in Prometheus format. Node Exporter (Linux), kube-state-metrics (K8s), mysqld_exporter (DB).
ScrapingPrometheus pulls metrics at regular intervals (default 15s). Configurable per job.
PromQLPowerful query language. Filter, aggregate, analyze. Used in Grafana and alerting rules.
LabelsKey-value pairs adding context. Used for filtering and aggregation. job, instance, method, status, namespace.
Recording RulesPre-calculate and store query results. Improve performance. Keep dashboards fast.
Alerting RulesDefine when alerts fire. Integrated with Alertmanager for routing, annotations, severity.
# PromQL examples rate(http_requests_total[5m]) # requests per second sum(rate(http_requests_total{status=~"5.."}[5m])) # error rate (5xx) 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100) # CPU usage % # Alerting rule example groups: - name: api.alerts rules: - alert: HighErrorRate expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate detected"
LOGGING

Logging & Distributed Tracing

ConceptDescription
Centralized LoggingCollect logs from all systems in one place. Eliminate log silos. Search, analyze, and act fast. Tools: ELK, Loki, CloudWatch, Splunk.
Structured LogsJSON format. Easy to parse and query. Consistent fields. Better filtering and analysis. Essential for automation.
Log AggregationAggregate from multiple sources. Index and store efficiently. Enable fast search, retention, archiving.
Correlation IDsUnique ID per request. Track across services and systems. Include in logs, headers, and traces.
Distributed TracingTrace requests across microservices. End-to-end visibility. Identify latency bottlenecks. Tools: Jaeger, Zipkin, AWS X-Ray.
SpansA span = unit of work. Has start time, end time, duration, tags, logs, metadata. Spans form a trace tree.
OpenTelemetryOpen standard for observability. Collects logs, metrics, traces. Vendor-neutral. One pipeline, multiple backends.
Root Cause InvestigationUse logs + metrics + traces. Follow request path → identify failing component → analyze context → fix root cause.
Best Practices: Use structured logs consistently → Always include correlation IDs → Keep timestamps in UTC → Retain logs and traces appropriately → Monitor the monitoring system → Continuously improve observability.
🛡️

27. SRE Fundamentals

Build reliable systems. Deliver value. Reduce toil.

SRE

SRE Fundamentals — Build Reliable Systems

ConceptDefinitionExample
SLI (Service Level Indicator)Measurable metric quantifying what matters to usersAvailability, Latency, Error Rate, Throughput, Success Rate
SLO (Service Level Objective)Target value for an SLI. Defines reliability level you commit to.99.9% availability over 30 days
SLA (Service Level Agreement)Formal contract with customers. Based on SLOs. Legal/business implications if unmet.99.9% uptime SLA with penalties
Error BudgetAcceptable amount of unreliability = 1 - SLO. When exhausted, stop risky changes.99.9% SLO → 0.1% error budget → 43.2 min/month downtime allowed
AvailabilityUptime / (Uptime + Downtime) × 100. Common targets: 99.9%, 99.99%, 99.999%.99.9% = 8.7h downtime/year
LatencyHow fast your system responds. Measure at P50, P95, P99, P99.9. Tail latency impacts UX.P99 < 200ms
ToilManual, repetitive operational work. Does not scale. Reduces engineer satisfaction. Automate and eliminate.Manually restarting services, processing tickets
Define User Experience (SLIs)
→ Measure with SLIs
→ Set SLOs
→ Manage Error Budgets
→ Reduce Toil & Improve
→ Reliable Systems = Happy Users
SRE IS NOT JUST ABOUT TOOLS. IT IS A MINDSET OF RELIABILITY, OWNERSHIP, AND CONTINUOUS IMPROVEMENT.
Start with what matters to users (SLIs). Set realistic SLOs. Use error budgets to drive decisions. Balance speed and reliability.
🔐

28–30. DevSecOps, Identity, Access & Supply Chain Security

Build secure. Deliver fast. Protect continuously.

DEVSECOPS

DevSecOps — Integrate Security into Every Stage

PracticeDescriptionTools
Shift-Left SecurityMove security activities earlier. Find and fix before production.Code review, threat modeling
Secure SDLCSecurity in every phase: Plan, Code, Build, Test, Release, Deploy, Operate.All stages
SASTStatic Application Security Testing. Analyze source code without executing.SonarQube, Semgrep, Checkmarx
DASTDynamic Application Security Testing. Test running application from outside. Find runtime vulnerabilities.OWASP ZAP, Burp Suite, Invicti
SCASoftware Composition Analysis. Analyze open source dependencies for vulnerabilities.Snyk, OWASP Dependency-Check
Secret ScanningDetect hardcoded secrets/credentials. Scan commits and repos.GitGuardian, Gitleaks, TruffleHog
Container ScanningScan container images for vulnerabilities. Check OS packages and configurations.Trivy, Clair, Anchore
Infrastructure ScanningScan IaC, cloud resources, configurations. Detect misconfigurations.Checkov, tfsec, AWS Inspector
Policy EnforcementEnforce security and compliance policies. Use OPA, Sentinel, Kyverno.OPA, Sentinel, Kyverno, Conftest
Security GatesStop bad code/configs from progressing. Gate on quality, security, compliance.Pipeline gates, quality checks
Security is NOT a checklist. It is a CULTURE, a PROCESS, and a CONTINUOUS PRACTICE. Security is everyone's responsibility.
IAM

Identity, Secrets & Access Control

ConceptDescription
AuthenticationVerify who the user or system is. Methods: Passwords, MFA, Certificates, Tokens. Enforce MFA everywhere.
AuthorizationDetermine what an authenticated identity can do. Based on roles, policies, and permissions. Least privilege.
IAMManage users, groups, roles, policies. Centralize identity and access control. Federated identity and SSO.
RBACGrant permissions based on roles. Roles map to permissions. Scalable. Widely used in K8s and cloud.
Least PrivilegeGrant only minimum access required. Reduces blast radius. Review and remove unnecessary access.
Service AccountsNon-human identities for applications and services. Scoped permissions only. Avoid human credentials.
Secrets ManagementStore secrets securely. Encrypt in transit and at rest. Centralized stores: Vault, AWS Secrets Manager, Azure Key Vault.
Secret RotationRotate secrets regularly. Automate rotation. Revoke old secrets immediately. Reduce exposure window.
Short-lived CredentialsTemporary credentials with short TTL. Tokens, STS, OIDC. Avoid long-lived keys.
Workload IdentityBind workloads to identities (OIDC, IRSA). No static credentials inside workloads.
IDENTITIES CAN BE COMPROMISED. PERMISSIONS SHOULD NOT BE. SECURE ACCESS IS CONTINUOUS WORK.
The right identity. The right access. Only when needed. For only as long as needed.
🚨

31–35. Incident Response, Troubleshooting, Failure Patterns, DR & Postmortems

Detect fast. Respond smart. Restore reliably. Learn continuously.

INCIDENT

Incident Response — 11 Stages

📡 DETECTION
→ 🔔 ALERTING
→ 🔍 TRIAGE
→ 📊 SEVERITY
→ 👥 COMMAND
→ 💬 COMMS
→ 🔧 MITIGATION
→ ✅ RECOVERY
→ 🔎 VERIFY
→ 📅 TIMELINE
→ ⬆️ ESCALATE → RESOLVED
StageWhat to Do
DetectionMonitor systems, detect anomalies early, use metrics/logs/alerts, confirm incident existence
AlertingTrigger rules-based alerts, reduce noise, route to right team, include context in alerts
TriageValidate the incident, identify affected systems, determine impact/scope, gather initial data
Severity ClassificationClassify P1-P4 based on impact/urgency. Communicate severity. Reassess as needed.
Incident CommandAssign incident commander, define roles/responsibilities, coordinate response efforts
CommunicationCommunicate early and clearly. Regular updates. Status pages. Keep stakeholders informed.
MitigationContain impact, apply temporary fixes, reduce user impact, avoid further degradation
RecoveryApply permanent fix, restore normal operations, monitor system behavior, validate dependencies
VerificationVerify system healthy, confirm resolution, validate with monitoring, test critical user flows
TimelineRecord key timestamps, track events in order, include actions taken, support post-incident review
EscalationEscalate when needed, follow escalation path, engage experts, remove blockers quickly
METHODOLOGY

Production Troubleshooting Methodology — 11 Steps

StepActionWhy
1. Define SymptomsIdentify and document the problem clearly. What is broken? What are users experiencing?Start with facts, not assumptions
2. Establish ScopeDetermine impact and scope. Who is affected? Which services?Prioritize investigation correctly
3. Check Recent ChangesReview deployments, config changes, releases, infra updatesMost incidents caused by recent changes
4. Inspect MetricsCheck monitoring dashboards. Look for anomalies, spikes, drops.Data-driven diagnosis
5. Inspect LogsSearch logs for errors, warnings, correlated events.Logs contain the answers
6. Inspect TracesFollow requests across services. Identify slow calls and failures.Find exactly where it broke
7. Test DependenciesVerify health of downstream/upstream services, databases, external APIs.Isolate the root cause layer
8. Form HypothesesCreate possible root cause theories based on gathered data.Structured thinking
9. Validate HypothesesTest and confirm (or eliminate) hypotheses with evidence.Evidence-based diagnosis
10. Mitigate SafelyImplement safest fix or workaround. Minimize risk and blast radius.Fix without making it worse
11. Verify RecoveryConfirm issue is resolved. Monitor to ensure stability.Validate the fix works
What To Avoid
Assuming the root causeAlways gather data first
Making random changesHave a hypothesis before changing anything
Ignoring user impactUsers are the priority
Skipping stepsThe methodology exists for a reason
No communicationKeep stakeholders informed throughout
Not documentingDocument everything for learning and future reference
PATTERNS

Common Production Failure Patterns

Failure PatternSymptomsCommon CausesWhat to Check
CPU SaturationHigh CPU usage, slow response, request timeoutsRunaway processes, unoptimized code, too many requeststop, htop, recent deployments, application logs
Memory ExhaustionHigh memory usage, OOM kills, service crashesMemory leaks, large caches, too many instancesfree -m, top, dmesg (OOM logs), container limits
Disk FullWrites failing, services crashing, logs not writingLarge logs, unclean temp files, no log rotationdf -h, disk I/O (iostat), log directories, df -i
DNS FailureCannot resolve names, intermittent failures, slow requestsDNS server down, wrong config, TTL/cache issuesnslookup/dig, /etc/resolv.conf, CoreDNS logs
Network TimeoutRequest timeouts, slow connections, intermittent failuresNetwork congestion, packet loss, firewall blocksping, traceroute, security groups, network latency
Database ExhaustionDB connections failing, slow queries, high DB CPU/IOToo many connections, long-running queries, no connection limitsDB metrics, active connections, slow query logs
Connection Pool ProblemsConnection timeouts, requests hanging, pool exhaustedPool too small, connections not released, idle timeout mismatchPool metrics, active/idle connections, app logs
Certificate ExpirationSSL errors, services failing, users see warningsExpired cert, auto-renew failed, wrong cert, missing chainopenssl s_client, expiration date, monitoring alerts
Dependency FailureExternal calls failing, errors from API, timeoutsService down, rate limits, network issues, bad configStatus pages, error logs, retries, circuit breakers
Bad DeploymentErrors after deploy, health checks fail, performance dropBug in new release, wrong config, DB migration issues, incomplete rolloutDeployment logs, health checks, application logs, recent changes
REMEMBER: MOST OUTAGES ARE PREVENTABLE. OBSERVE → UNDERSTAND → ACT → VERIFY → IMPROVE.
DR

Rollback, Recovery & Disaster Recovery

DR ConceptDefinitionKey Actions
Application RollbackRevert application to previous known good versionRevert deployment, rollback traffic, verify functionality
Infrastructure RollbackRevert infrastructure changes to stable stateRevert IaC changes, restore previous state, validate environment
Database RecoveryRestore database to consistent and healthy stateRestore from backup, point-in-time recovery, validate data integrity
BackupsMaintain reliable backups for all critical systemsAutomated backups, encrypt, store offsite/off-region, retention policies
Restore TestingRegularly test restores to ensure backups actually workTest full/partial restore, validate data, document results
RPORecovery Point Objective — maximum acceptable data loss (measured in time)Define per system, align backup frequency
RTORecovery Time Objective — maximum acceptable downtime to restore serviceDefine per system, optimize recovery time, automate recovery
Disaster RecoveryRecover systems after major disruption or site failureActivate DR plan, failover services, communicate status, stabilize
Multi-Region RecoveryMultiple regions for high availability and recoveryDeploy across regions, replicate data, configure failover, route traffic
Recovery ValidationValidate systems are fully recovered and operating correctlyRun validation tests, verify functionality, monitor stability
BEST DISASTER RECOVERY IS NOT JUST BACKUPS — IT IS TESTED, VALIDATED, AND PROVEN TO WORK WHEN YOU NEED IT MOST.
POSTMORTEMS

Postmortems & Continuous Improvement

ElementDescription
Blameless PostmortemsFocus on the system, not individuals. No blame. Open and honest. Safe environment. Trust and transparency.
Incident TimelineBuild clear timeline: when it started, what changed, when escalated, when mitigated, when resolved.
Root Cause AnalysisIdentify underlying cause. Go beyond symptoms. Find the real trigger. Ask "Why did this happen?"
Five WhysKeep asking "Why?" until you reach the root cause. Stop at root cause. Clear, deep, verified answer.
Contributing FactorsList all factors that made incident worse: technical, process, people, environmental, tooling gaps.
Corrective ActionsActions to fix the issue quickly and restore stability. Immediate fixes, hotfixes, rollbacks, workarounds.
Preventive ActionsActions to prevent this incident from recurring. Process improvements, automation, monitoring, better testing.
OwnershipAssign clear ownership for each action item. One owner per action. Clear responsibilities.
Follow-upTrack progress until all actions completed. Regular updates. Verify completion. Close the loop.
LearningCapture lessons. Share learnings. Update runbooks. Build a learning culture.
EVERY INCIDENT IS AN OPPORTUNITY TO IMPROVE. CAPTURE LESSONS. TAKE ACTION. PREVENT RECURRENCE.
🏢

36–40. Platform Engineering, Enterprise Architecture, Metrics, Senior Principles & Blueprint

From code to production. Secure. Reliable. Observable. Automated.

PLATFORM

Platform Engineering & Internal Developer Platforms

ConceptDescription
Platform EngineeringBuild and operate platforms for internal users. Abstract complexity. Enable developers to ship faster. Treat the platform like a product.
Developer Experience (DX)Frictionless workflows, fast onboarding, clear docs, self-service tooling, consistent environments, quick feedback loops.
Internal Developer Platform (IDP)Central portal for all platform services. Unified tooling. Self-service infrastructure, pipelines, and components.
Golden PathsOpinionated, secure, well-architected paths. Pre-built patterns for common use cases. Reduce decision fatigue.
Self-Service InfrastructureProvision environments on demand. Automated and API-driven. No manual tickets. Policy-driven and secure.
Service TemplatesPre-approved blueprints including code, IaC, pipelines, testing, security, monitoring.
Developer PortalsSingle place to discover and access everything: catalogs, templates, runbooks, guides, approvals.
Platform APIsExpose capabilities via well-documented APIs. Automate and integrate. Enable self-service and extensibility.
GuardrailsEnforce policies and compliance. Security/cost/performance controls. Prevent misconfigurations.
Platform as a ProductHas users (customers). Measure usage and value. Gather feedback. Iterate and improve. Clear SLAs.
A GREAT PLATFORM EMPOWERS DEVELOPERS. A STRONG PLATFORM SCALES THE BUSINESS.
ENTERPRISE

Enterprise DevOps Architecture — End-to-End

LayerComponentsDetails
Source ControlGit Repositories, Branch Protection, Code Reviews, PRs, Audit TrailsCentralized repos, branch protection, collaboration
CI PlatformAutomated builds, tests, quality checks, security scanning, pipeline as codeFail fast feedback, quality gates
Artifact ManagementStore binaries, packages, container images. Version control, retention, immutability.JFrog Artifactory, Sonatype Nexus, ECR
Security SystemsSAST, DAST, Container/Dependency Scanning, Policy Enforcement, Vulnerability MgmtShift-left, continuous scanning
Infrastructure as CodeTerraform/OpenTofu. Versioned IaC, reusable modules, environment parity, drift detection.Policy as code
KubernetesContainer orchestration: workloads, scaling, service discovery, networking, storage, HA.Self-healing, auto scaling
GitOpsDeclarative state (Argo CD): Git as source of truth, automated sync, drift detection, rollback.Argo CD
ObservabilityPrometheus (metrics), Grafana (dashboards), Loki (logs), Jaeger (traces), alerting, SLO monitoring.Full-stack visibility
Secrets ManagementHashiCorp Vault: centralized secrets, dynamic secrets, rotation, fine-grained access, audit logs.Zero-trust secrets
Cloud InfrastructureAWS/Azure/GCP: compute, networking, storage, databases, managed services, cost optimization.Scalable, secure, reliable
CODE COMMIT
→ BUILD & TEST
→ ARTIFACT PUBLISHED
→ SECURITY VALIDATION
→ INFRA PROVISIONED
→ DEPLOYED ON KUBERNETES
→ GITOPS SYNC
→ MONITOR & OBSERVE
→ MANAGE SECRETS
→ RUN IN CLOUD
DORA

DevOps Performance Metrics — DORA & Engineering KPIs

MetricDescriptionGood Target
Deployment FrequencyHow often code is successfully deployed to productionOn-demand or multiple times per day
Lead Time for ChangesTime from code commit to successful deployment in productionLess than 1 day
Change Failure RatePercentage of changes causing degraded service or requiring rollbackLess than 15%
Mean Time to Recovery (MTTR)Average time to restore service after incident or failureLess than 1 hour
AvailabilityPercentage of time system is operational and functional99.9% or higher
Pipeline DurationTotal time for CI/CD pipeline run from start to finishAs fast as possible — optimize continuously
Build Success RatePercentage of builds that complete successfully without failuresGreater than 95%
Infrastructure ReliabilityHow reliable, resilient, and performant your infrastructure isHigh uptime, low errors
Alert QualityPercentage of alerts that are actionable, accurate, and usefulHigh signal, low noise
Engineering EffectivenessHow effectively the team delivers value to customersContinuously improving
DORA Core Four: Deployment Frequency, Lead Time for Changes, Change Failure Rate, Mean Time to Recovery. These four metrics correlate strongly with elite engineering team performance.
Maturity LevelCharacteristics
EliteHigh performance across all metrics. Continuous improvement. Data-driven culture.
HighConsistently better than industry average.
MediumImproving but inconsistent.
LowFrequent issues and manual processes.
SENIOR

Senior DevOps Engineer — 12 Production Principles

PrincipleFocus Areas
1. Automate Repetitive WorkAutomation, Scripting, CI/CD, Self-service. Increases speed, consistency, reliability.
2. Design for FailureResilience, Redundancy, Timeouts, Retries. Failures will happen. Design systems that expect and survive them.
3. Prefer Simple SystemsKISS, reduce complexity, YAGNI. Simplicity improves reliability, maintainability, and understanding.
4. Make Changes ReversibleRollbacks, Feature Flags, Blue/Green, Canary. Always design changes so you can go back quickly and safely.
5. Reduce Blast RadiusIsolation, Segmentation, Least Privilege. Limit the impact of failures to small, isolated areas.
6. Build Observability FirstMetrics, Logs, Traces, Alerts, Dashboards. You can't fix what you can't see. Instrument everything.
7. Protect ProductionAccess Control, Change Management, Safeguards. Production is the customer experience. Treat it with respect.
8. Document Critical DecisionsArchitecture Decisions, Runbooks, Standards. Good documentation preserves knowledge and prevents repeat mistakes.
9. Understand DependenciesDependency Maps, Communication, Contracts. Know how systems, services, and teams depend on each other.
10. Measure Before OptimizingMetrics, Baselines, Data-driven Decisions. Measure first, then optimize with data, not assumptions.
11. Treat Security as EngineeringThreat Modeling, Scanning, Least Privilege, Compliance. Security is not a step — it's a responsibility in every decision.
12. Own ReliabilitySLOs, Error Budgets, Incident Ownership. You own the outcome. Be accountable for reliability and uptime.
THE MINDSET THAT SETS SENIOR ENGINEERS APART: Think Long Term → Automate Relentlessly → Design Resilient Systems → Deliver Value → Improve Continuously → Lead by Example.

PRINCIPLES TODAY. RELIABILITY TOMORROW.
BLUEPRINT

The Complete DevOps Production Blueprint — 13-Stage Pipeline

1️⃣ CODE
Write, commit, change
→ 2️⃣ GIT WORKFLOW
Branch, PR, review, merge
→ 3️⃣ CI PIPELINE
Build, test, quality, scan
→ 4️⃣ SECURITY GATES
SAST, DAST, scan, policy
→ 5️⃣ ARTIFACT LIFECYCLE
Store, version, promote
→ 6️⃣ INFRA PROVISIONING
IaC, plan, apply
→ 7️⃣ CONTAINER DEPLOY
Build image, scan, push
→ 8️⃣ KUBERNETES OPS
Deploy, health, scale, heal
→ 9️⃣ GITOPS DELIVERY
Git source of truth, sync
→ 🔟 MONITORING
Metrics, logs, traces, alerts
→ 1️⃣1️⃣ INCIDENT RESPONSE
Detect, triage, mitigate
→ 1️⃣2️⃣ ROLLBACK & RECOVERY
Rollback, restore, verify
→ ✅ PRODUCTION READY
ChecklistStatus
Code ReviewedMust pass before merge
Tests PassingAll unit + integration tests green
Security ClearedNo critical/high vulnerabilities
IaC ReviewedTerraform plan reviewed and approved
Observability ReadyMetrics, logs, alerts configured
Runbooks ReadyOperational procedures documented
Backup ValidatedData backup and restore tested
Stakeholders ApprovedChange management sign-off
DevOps Maturity LevelCharacteristics
Level 5 — OptimizedContinuous improvement, data-driven culture, full automation
Level 4 — MeasuredStandardized processes, metrics, SLOs, automation
Level 3 — DefinedDefined processes, some automation, basic monitoring
Level 2 — ManagedBasic automation, inconsistent processes
Level 1 — InitialManual, ad-hoc, reactive
GREAT DEVOPS TEAMS DON'T JUST DEPLOY SOFTWARE. THEY DELIVER RELIABILITY, SECURITY, AND VALUE.