♾️ DevOps Bible
From Engineering Foundations to Production DevOps — The Complete 40-Page Handbook for Engineers
Covering the complete DevOps lifecycle: Develop → Git & Collaboration → CI/CD Pipeline → Container & Kubernetes → Cloud Infrastructure → Monitoring & Observability. Built for: Developers, DevOps Engineers, SREs, Platform Engineers, Cloud Engineers, Tech Leads, Engineering Teams.
1. DevOps Engineering in the Real World
Build better. Deploy faster. Operate smarter.
What DevOps Actually Means
DevOps is a mindset, culture, and set of practices that brings development and operations together. It focuses on delivering value to users faster, with higher quality and reliability. DevOps removes silos, improves collaboration, and automates everything from code to production.
- Write and maintain code
- Build and test applications
- Commit and manage source code
- Create automated tests
- Ensure code quality
- Package and version artifacts
- Work closely with operations
- Provision and manage infrastructure
- Deploy and run applications
- Monitor systems and applications
- Ensure availability and performance
- Manage security and access
- Handle incidents and outages
- Continuously optimize systems
DevOps Lifecycle — 8 Stages
Plan features & requirements
Write & commit changes
Compile & create artifacts
Run automated tests & scans
Package to artifact repo
Deploy to staging/production
Monitor & maintain
Collect feedback & iterate
DevOps vs SRE vs Platform Engineering
| Role | Focus | Key Principle |
|---|---|---|
| DevOps | Culture and practices uniting Dev+Ops for faster, reliable delivery | Culture drives tooling |
| SRE (Site Reliability Engineering) | Applies engineering to ops. Focuses on reliability, SLIs, SLOs, error budgets, reducing toil | Reliability is an engineering problem |
| Platform Engineering | Builds internal platforms providing self-service tools, scaffolding, golden paths | Enable developer self-service at scale |
Production Engineering Mindset
- Own the full lifecycle from code to production
- Design for reliability, scalability, and security
- Automate to reduce manual work and errors
- Monitor everything that matters
- Measure, learn, and improve continuously
- Think in systems, not in isolated components
- Prepare for failure; build for recovery
- Focus on customer value and user experience
- Take responsibility and act with urgency
- Document, standardize, and share knowledge
2. Linux for DevOps Engineers
Understand. Operate. Automate. Scale.
Linux Architecture & Filesystem
- User Applications — top layer, what users run
- System Libraries — interfaces to kernel functionality
- Kernel — manages CPU, RAM, Disk, Network
- Shell — user interface to the kernel
- Everything is a file in Linux
Filesystem Hierarchy
| Path | Purpose |
|---|---|
/ | Root directory |
/bin | Essential binaries |
/etc | System configuration |
/home | User home directories |
/var | Variable data (logs) |
/opt | Optional applications |
/proc | Process information |
/sys | System information |
/tmp | Temporary files |
- Create user:
useradd - Modify user:
usermod - Delete user:
userdel - Switch user:
su - <user> - View groups:
groups - Add to group:
usermod -aG - Principle of least privilege always
Permissions —
rwxr-xr--- r=Read(4), w=Write(2), x=Execute(1)
chmod 755 file— change modechown user:group file— change ownerchgrp group file— change group
Services, Packages & Systemd
RHEL/CentOS/Rocky:
Essential Production Commands
| Category | Command | Description |
|---|---|---|
| File & Dir | ls -lah | List files with details |
| File & Dir | find /path -name "file" | Find files |
| File & Dir | rm -rf dir | Remove directory forcefully |
| System Info | uname -a | Kernel and system info |
| System Info | uptime | System uptime + load |
| System Info | free -h | Memory usage |
| System Info | df -h | Disk usage |
| Networking | ip a | IP address |
| Networking | ss -tulnp | Listening ports |
| Networking | ping host | Check connectivity |
| Networking | ssh user@host | SSH connection |
| Logs | journalctl -f | Follow all logs |
| Logs | tail -f /var/log/syslog | Live system log |
| Logs | grep -i error /var/log/* | Search errors |
| Permissions | chmod 755 file | Change permissions |
| Permissions | chown user:group file | Change owner |
| Processes | ps aux | All processes |
| Processes | top / htop | Real-time monitor |
| Processes | kill -9 <PID> | Kill process |
3. Linux Production Troubleshooting
Find problems fast. Diagnose accurately. Fix confidently.
10 Troubleshooting Areas & Key Commands
| Area | Key Commands | What to Look For |
|---|---|---|
| 1. CPU Investigation | top, htop, mpstat -P ALL 1, ps -eo pid,ppid,cmd,%cpu --sort=-%cpu, sar -u 1 5, vmstat 1 | High %usr (app), high %sys (kernel), high %iowait (disk) |
| 2. Memory Pressure | free -h, top, htop, vmstat 1, swapon --show, ps -eo pid,cmd,%mem --sort=-%mem, cat /proc/meminfo | OOM kills in dmesg, swap in/out active, low available memory |
| 3. Disk Utilization | df -h, du -sh /* 2>/dev/null | sort -h, ncdu /, lsblk, iotop -oPa, sar -d 1 5 | 100% disk usage, inode exhaustion, large log files |
| 4. Filesystem Problems | dmesg | tail, fsck -n /dev/sdX#, mount | column -t, findmnt, umount /mount/point, journalctl -k | tail | EXT4/XFS errors, read-only filesystem, corruption |
| 5. Process Failures | ps aux, pstree -p, kill -9 <PID>, killall -9 <process>, pkill -f <pattern> | Zombie processes, crashed services, runaway CPU/mem |
| 6. Service Failures | systemctl status/start/stop/restart/enable <service>, journalctl -u <service> -f | Service failed to start, dependency errors, config issues |
| 7. Log Investigation | journalctl -xe, journalctl -f, tail -f /var/log/messages, grep -i "error" /var/log/* | Error patterns, timing of failures, service crashes |
| 8. Network Troubleshooting | ip a, ip r, ss -tulnp, ping <host>, traceroute <host>, dig <domain>, curl -I http://<host>, mtr <host> | Timeouts, packet loss, DNS failures, firewall blocks |
| 9. Performance Bottlenecks | top, htop, vmstat 1, iostat -xz 1, sar -q 1 5, pidstat 1, iotop -oPa, perf top | CPU/memory/disk/network saturation |
| 10. Production Workflow | Confirm scope → Check recent changes → Collect evidence (metrics, logs) → Form hypothesis → Identify root cause → Apply safe fix → Verify → Document | Always follow the structured process |
Troubleshooting Golden Rules + Quick Health Check
Collect facts, not assumptions
Reduce to smallest possible scope
Form theory based on evidence
Prove or disprove with commands
Apply least risky effective change
Confirm the fix and monitor
Record what happened and what you learned
| Category | Quick Health Check Commands |
|---|---|
| System Overview | uptime, w, who -a, hostnamectl |
| CPU | top, htop, mpstat -P ALL 1, sar -u 1 5 |
| Memory | free -h, vmstat 1, cat /proc/meminfo |
| Disk | df -h, du -sh /* 2>/dev/null | sort -h, lsblk |
| I/O | iostat -xz 1, iotop -oPa, iotop -boPa, sar -d 1 5 |
| Network | ip a, ip r, ss -tulnp, ping <host> |
| Logs | journalctl -xe, journalctl -u <service>, tail -f /var/log/messages |
4. Networking Every DevOps Engineer Must Know
Understand the network. Troubleshoot with confidence. Build reliable systems.
Core Networking Concepts
| Concept | Details |
|---|---|
| TCP/IP Layers | Application (L7) → Transport (L4) → Internet (L3) → Network Access (L2). Data travels from application to network access. |
| IPv4 Address | 32-bit, dotted decimal (192.168.1.10). Network + Host portions. Private: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 |
| IPv6 | 128-bit, hex notation. Replaces IPv4 for scale. |
| CIDR | 192.168.1.0/24 = 24 bits network + 8 bits host = 256 addresses. /24 = 255.255.255.0 |
| Subnetting | Divide /24 into /26 (4 subnets × 64 hosts). Improves security, performance, organization. |
| Routing | Routers use routing tables to forward packets. Default route 0.0.0.0/0 = unknown destinations. |
| DNS | Translates domain names to IPs. Record types: A, AAAA, CNAME, MX, TXT. TTL controls cache duration. |
| Ports | 0-1023 well-known (22=SSH, 80=HTTP, 443=HTTPS). 1024-49151 registered. 49152-65535 dynamic. |
| TCP vs UDP | TCP = reliable, ordered, connection-oriented (web, SSH). UDP = fast, connectionless (DNS, VoIP, streaming). |
| NAT | Allows private IPs to access public internet. SNAT, DNAT, PAT (NAT Overload). |
| Firewalls | Control inbound/outbound traffic. Stateless (packets) or Stateful (connections). Rules: Allow/Deny by IP, Port, Protocol. |
| HTTP/HTTPS | HTTP (port 80) = unencrypted. HTTPS (port 443) = encrypted with TLS. Always use HTTPS in production. |
| TLS | Encrypts data in transit. Uses certificates, keys, secure ciphers. Ensures confidentiality, integrity, authentication. |
| Load Balancing | Distributes traffic across multiple servers. L4 (transport) or L7 (application). Algorithms: Round Robin, Least Connections, IP Hash. |
5. Git & Production Version Control
Track changes. Collaborate safely. Ship with confidence.
Git Repository Workflow & Commands
Make changes
Stage for commit
Commit changes
Push commits
| Operation | Commands | Notes |
|---|---|---|
| Basic workflow | git status, git add <file>, git add ., git commit -m "msg", git push | Daily workflow |
| Branching | git branch, git branch <name>, git switch <name>, git switch -c <name> | Isolate work in branches |
| Merging | git merge <branch>, git merge --no-ff <branch> | Creates merge commit |
| Rebasing | git rebase <branch>, git rebase -i <branch>, git rebase --abort, git rebase --continue | Clean linear history |
| Pull Requests | git push origin <branch>, git fetch origin, git pull origin <branch> | Propose → Review → Merge |
| Tags | git tag, git tag -a v1.0.0 -m "message", git push origin v1.0.0 | Mark releases |
| Release Branches | git switch -c release/1.2, git merge release/1.2, git branch -d release/1.2 | Stabilize before production |
| Troubleshooting | git restore <file>, git stash, git reset --soft HEAD~1, git revert <commit>, git reflog, git cherry-pick <commit> | Fix common mistakes |
| Problem | Cause | Solution | Command |
|---|---|---|---|
| Accidental changes not committed | Changes still in working directory | Discard or stash | git restore <file> / git stash |
| Committed to wrong branch | Wrong commit made | Move to correct branch | git cherry-pick <commit> |
| Need to undo last commit | Wrong commit or message | Undo, keep changes | git reset --soft HEAD~1 |
| Pushed wrong commit | Bad commit already pushed | Revert commit safely | git revert <commit> |
| Merge conflicts | Changes between branches | Resolve manually | git status → fix → git add → git commit |
| Detached HEAD | Checked out a commit | Switch back to branch | git switch <branch> |
| Lost commits | Commits not reachable | Use reflog | git reflog |
6. Cloud Infrastructure Fundamentals
The building blocks of modern cloud platforms.
Cloud Infrastructure — 11 Building Blocks
| Building Block | Description | Examples |
|---|---|---|
| 1. Compute | Virtual machines and containers. Scale up/down on demand. Pay for what you use. | VM, Containers, Serverless Functions |
| 2. Storage | Store and retrieve data durably and securely. Different types for different needs. | Object Storage, Block Storage, File Storage |
| 3. Networking | Connect resources securely. VPC, subnets, IP addressing, routing. | VPC, DNS, VPN, Gateways |
| 4. Databases | Managed database services for reliability and scale. Relational and NoSQL. | RDS, Aurora, DynamoDB, Cloud SQL |
| 5. Identity | Manage users, roles, permissions. Least privilege principle. | IAM, Roles, Policies, MFA |
| 6. Load Balancing | Distribute traffic. Health checks ensure only healthy targets receive traffic. | ALB, NLB, Gateway LB |
| 7. Auto Scaling | Automatically adjust capacity based on demand. Scale out or in. | Auto Scaling Groups, Scale Sets |
| 8. Availability Zones | Physically separate data centers within a region. Isolate failures. | AZ1, AZ2, AZ3 |
| 9. Regions | Geographically separate areas. Choose close to users. | us-east-1, eu-west-1, ap-southeast-1 |
| 10. Shared Responsibility | Provider secures the cloud. You secure what is in the cloud. | Know your responsibilities |
| 11. High Availability | Design to remain available during failures. Multi-AZ, LB, auto scaling. | Eliminate single points of failure |
7. Production Cloud Architecture
Design resilient, scalable, and highly available cloud systems.
Production Cloud Architecture — Multi-AZ Design
| Component | Role | Key Detail |
|---|---|---|
| Internet Gateway | Connects VPC to internet | Required for public subnet access |
| DNS (Route 53) | Domain name resolution | Routes users to load balancer |
| Load Balancer (ALB) | Distributes traffic to healthy app instances | Spans multiple AZs |
| Public Subnet | Houses load balancer, NAT Gateways | One per AZ |
| Application Tier (Private Subnet) | Runs stateless app containers/VMs | No direct internet access |
| Database Tier (Private Subnet) | Primary + Replica across AZs | Replication for HA |
| NAT Gateway | Private subnet → internet (outbound only) | One per AZ for HA |
| Architecture Pattern | Description | Use Case |
|---|---|---|
| 3-Tier Architecture | Web → App → DB tiers separated | Standard web apps |
| Microservices | Independently deployed services | Scale individual components |
| Multi-AZ HA | Resources across multiple AZs | Production workloads |
| Active-Passive (DR) | Primary region + standby region | Disaster recovery |
| Active-Active (Multi-Region) | Live traffic in multiple regions | Global, low-latency apps |
| Serverless | No infrastructure management | Event-driven, variable workloads |
| Blue-Green | Two identical environments, switch traffic | Zero-downtime deployments |
| Key Principle | Description |
|---|---|
| Fault Isolation | Isolate failures so issues in one part do not affect others |
| Scaling | Scale application tiers horizontally based on demand |
| Resilience | Design for failure. Systems recover automatically and quickly. |
| High Availability | Use multiple AZs and remove single points of failure |
| Security | Least privilege and defense in depth everywhere |
8. Infrastructure as Code (Terraform)
Build, manage, and scale infrastructure with code.
Infrastructure as Code — Why & Terraform Fundamentals
- Version controlled and auditable
- Consistent and repeatable deployments
- Automated and faster provisioning
- Easy to scale and modify
- Real-time collaboration
- Reduced manual errors
- Disaster recovery and restore
- Enables CI/CD for infrastructure
- Cost-effective and efficient
- Infrastructure becomes a product
Desired State → Terraform → Real Infrastructure
YOU DEFINE THE DESIRED STATE. TERRAFORM ACHIEVES IT.
| Terraform Concept | Description | Example |
|---|---|---|
| Providers | Plugins that interact with cloud APIs | aws, azurerm, google, kubernetes |
| Resources | Infrastructure objects you create and manage | aws_instance, aws_s3_bucket |
| Variables | Make code dynamic and reusable | region, instance_type |
| Outputs | Return important values after apply | instance IP, endpoint, DNS name |
| Modules | Organize and reuse code. Build once, use everywhere. | modules/vpc/, modules/eks/ |
| State | Terraform's memory of your infrastructure. Maps config to real resources. | terraform.tfstate |
| Remote State | Store state remotely for collaboration, locking, safety. | S3 + DynamoDB, Azure Storage |
| Idempotency | Apply multiple times, same result. No changes if matches desired state. | terraform plan shows "No changes" |
Define in .tf files
Review execution plan
Provision/update resources
Terraform tracks state
Remove when no longer needed
9. Production Terraform
Build, manage, and operate infrastructure at scale.
Production Terraform — Best Practices
| Topic | Details |
|---|---|
| Module Architecture | Organize into reusable modules. Follow DRY principle. Keep modules small and focused. Publish and version modules. |
| Environment Separation | Separate environments with folders/workspaces. Separate state per environment. Different variables per env. (dev/stage/prod) |
| State Locking | Prevent concurrent changes to state. Use state locking to avoid corruption. Always enable locking. |
| Remote Backends | Store state remotely: AWS S3 + DynamoDB locking, Azure Storage, Google Cloud Storage. |
| Dependency Management | Use dependency blocks when needed. Use data sources for existing resources. Run terraform init to install. |
| Drift Detection | Drift = real infra differs from state. Use terraform plan to detect. Integrate detection in pipelines. |
| Importing Resources | Bring existing resources under Terraform: terraform import aws_s3_bucket.bucket my-bucket |
| Security | Least privilege IAM. Encrypt state at rest. Do not commit secrets. Use variables and secrets managers. |
10. Configuration Management — Ansible
Automate, standardize, and maintain infrastructure at scale.
Configuration Management with Ansible
| Concept | Description |
|---|---|
| Inventory | Defines hosts Ansible manages. Organize into groups. Static (INI/YAML) or dynamic inventory. |
| Playbooks | YAML files defining automation tasks. Sequence of tasks, easy to read and version. Reusable and shareable. |
| Roles | Organize playbooks into reusable roles. Standard structure: tasks/, handlers/, templates/, files/, vars/, defaults/, meta/. |
| Variables | Store values for reuse in playbooks, roles, templates. Sources: vars, defaults, extra-vars, inventory. |
| Templates | Generate dynamic config files using Jinja2 templating. Insert variables. Ensure consistency across systems. |
| Idempotency | Tasks run multiple times with same result. Prevents drift and unnecessary changes. |
| Secrets | Never store plain secrets in playbooks. Use Ansible Vault. Use external secret stores (Vault, AWS Secrets Manager). |
| Configuration Drift | Detect drift using Ansible runs. Revert to desired state. Schedule regular audits. |
11. Docker Fundamentals
Build. Ship. Run. Anywhere.
Docker Fundamentals — Containers vs VMs
| Aspect | Virtual Machines | Containers |
|---|---|---|
| Isolation | Full OS per VM (Guest OS + Hypervisor) | Process isolation (shared host OS kernel) |
| Startup time | Minutes | Seconds or milliseconds |
| Size | GBs (full OS image) | MBs (just app + dependencies) |
| Resource usage | High (full OS overhead) | Low (shared kernel) |
| Portability | Limited (OS dependent) | High (OCI image standard) |
| Concept | Description |
|---|---|
| Images | Read-only template with everything to run an application. Built from Dockerfile. Versioned and immutable. |
| Containers | Running instance of an image. Isolated process environment. Has own writable layer. |
| Dockerfiles | Text file with build instructions: FROM, COPY, RUN, CMD, EXPOSE. Enables automation and consistency. |
| Layers | Images built in layers. Each instruction adds a layer. Layers are cached and reused. Only changed layers rebuild. |
| Registries | Store and distribute images. Public (Docker Hub). Private (ECR, GCR, ACR, Harbor). |
| Volumes | Persist data outside containers. Survive restarts. Named Volumes, Bind Mounts, tmpfs. |
| Networks | Enable communication. Types: bridge (default), host, overlay, none, macvlan. |
| Environment Variables | Key-value pairs passed to containers. Set at build or runtime (-e). Keep images reusable. |
12. Production Docker
Build secure. Run efficient. Operate at scale.
Production Docker — 10 Best Practices
| Practice | Details | Example |
|---|---|---|
| 1. Multi-stage Builds | Keep final image small. Exclude build tools and cache. | Builder stage (golang:1.22) → Final stage (alpine:3.19) |
| 2. Small Images | Use minimal base images: alpine, distroless, debian-slim. Remove unnecessary packages. | alpine:latest, gcr.io/distroless/base |
| 3. Image Tagging | Use consistent tagging: SemVer or date tags. Use immutable tags. Avoid "latest" in production. | myapp:1.0.0, myapp:2024-05-10, myapp:sha-4f3a1b2 |
| 4. Image Scanning | Scan for vulnerabilities. Tools: Trivy, Grype, Docker Scout. Fail builds on high/critical. | trivy image myapp:1.0.0 |
| 5. Non-Root Containers | Do not run as root. Create dedicated user. Drop unnecessary capabilities. | RUN adduser -D appuser; USER appuser |
| 6. Resource Limits | Limit CPU and memory. Prevent noisy neighbor issues. | --cpus="1.0" --memory="512m" --pids-limit=100 |
| 7. Health Checks | Detect failures early. Docker can restart unhealthy containers. | HEALTHCHECK --interval=30s --timeout=5s --retries=3 CMD curl -f http://localhost:8080/health |
| 8. Persistent Storage | Use volumes for persistent data. Backup important data. | docker run -d -v mydata:/var/lib/mysql myapp |
| 9. Container Networking | User-defined bridge networks. Enable service discovery. Avoid exposing unnecessary ports. | docker network create --driver bridge mynet |
| 10. Troubleshooting | Check container status, inspect, logs, resource usage, networks and volumes. | docker ps, docker logs, docker inspect, docker stats |
13–17. Kubernetes — Architecture, Workloads, Networking, Storage & Operations
Run containers at scale. Build reliable, self-healing systems.
Kubernetes Architecture — Control Plane & Worker Nodes
| Component | Role |
|---|---|
| API Server | Single entry point for all admin tasks. Front-end for the control plane. Validates and stores data in etcd. |
| etcd | Consistent key-value store for ALL cluster data: nodes, pods, configs, secrets, state. Source of truth. |
| Scheduler | Watches for new pods and selects the best node based on resources, policies, constraints. |
| Controller Manager | Runs controllers: Deployment, ReplicaSet, Node, Endpoints. Maintains desired state. |
| Kubelet | Agent on each worker node. Ensures containers defined in pods are running and healthy. |
| Kube-Proxy | Handles networking and load balancing for Services on each node. |
| Nodes | Worker machines running containerized workloads. |
| Pods | Smallest deployable units. One or more containers sharing network and storage. |
Kubernetes Workloads — 10 Types
| Workload | Description | Use Cases |
|---|---|---|
| Pods | Smallest deployable unit. Ephemeral by nature. Containers share network/storage. | Single container apps, tightly coupled containers |
| Deployments | Manages ReplicaSets. Declarative updates. Rolling updates and rollbacks. | Stateless apps, web apps, APIs, microservices |
| ReplicaSets | Ensures specified number of pod replicas running. Usually managed by Deployments. | Maintain replica count, high availability |
| StatefulSets | Manages stateful applications. Stable network identities and storage. Ordered deployment. | Databases, message queues, stateful applications |
| DaemonSets | Runs a pod on all (or some) nodes. Keeps pod running on node additions. | Logging agents, monitoring agents, network plugins |
| Jobs | Runs a pod to completion. Ensures task succeeds. Does not restart on success. | Batch processing, one-time tasks, data migration |
| CronJobs | Runs Jobs on a schedule. Uses cron syntax. Automates recurring tasks. | Backups, data cleanup, scheduled reports |
| Rolling Updates | Update pods gradually without downtime. Old pods replaced with new ones. | Zero-downtime deployments, safe version upgrades |
| Rollbacks | Revert to a previous ReplicaSet or version. Quick recovery from bad deployments. | Failed deployments, bug fixes, quick recovery |
| Application Lifecycle | Full journey: Create → Deploy → Scale → Update → Monitor → Delete. | All production applications |
Kubernetes Networking & Traffic
| Service Type | Description | Access |
|---|---|---|
| ClusterIP (default) | Stable virtual IP accessible only within the cluster. Ideal for internal communication. | Internal only |
| NodePort | Exposes service on a static port (30000-32767) on each node's IP. Accessible from outside. | External via Node IP:Port |
| LoadBalancer | Provisions an external cloud load balancer. Internet-facing access to the service. | External via cloud LB |
| Ingress | HTTP/HTTPS routing to services. Host and path-based routing. TLS termination. Single entry point. | External HTTP/HTTPS |
| Concept | Description |
|---|---|
| DNS | K8s assigns DNS names to Services. Format: <service>.<namespace>.svc.cluster.local |
| Network Policies | Control traffic between pods. Allow/deny based on rules. Applied at namespace or pod level. Enforced by CNI plugin. |
| Service Discovery | Services get stable DNS. Clients discover services automatically. No hardcoded IPs. Works across namespaces. |
| Container Networking | Pods get IPs from cluster network. Flat network model — all pods communicate without NAT. CNI plugins: Calico, Cilium, Flannel. |
Kubernetes Configuration & Storage
| Resource | Purpose | Use For |
|---|---|---|
| ConfigMaps | Store non-sensitive configuration as key-value pairs | App config, feature flags, env vars |
| Secrets | Store sensitive information securely (base64 encoded) | Passwords, API keys, certificates |
| Volumes | Storage accessible to containers in a pod | Temporary (emptyDir), host (hostPath), persistent (PVC) |
| PersistentVolumes (PV) | Cluster resources representing physical storage. Lifecycle independent of pods. | Admin-provisioned storage |
| PersistentVolumeClaims (PVC) | User requests for storage. Binds to a PV. Abstracts storage details from pods. | App requesting storage |
| StorageClasses | Define how storage is provisioned. Specify provisioner and parameters. | Dynamic provisioning (AWS EBS, GCE PD) |
Production Kubernetes Operations
| Feature | Purpose | Example |
|---|---|---|
| Requests & Limits | Guarantee resources (requests) and cap usage (limits). Prevent noisy neighbor. | cpu: 250m/1000m, memory: 256Mi/1Gi |
| Liveness Probe | Detects unhealthy container → K8s restarts it. | httpGet /healthz port 8080 |
| Readiness Probe | Detects when container is ready to receive traffic. Pod removed from Service if fails. | httpGet /ready port 8080 |
| Startup Probe | Handles slow-starting containers. Disables liveness/readiness until startup succeeds. | httpGet /startup, failureThreshold: 30 |
| HPA | Automatically scales pods based on CPU or custom metrics. | minReplicas: 2, maxReplicas: 10, targetCPU: 70% |
| Pod Disruption Budgets | Ensures minimum pods stay available during disruptions (e.g., node maintenance). | minAvailable: 2 |
| Node Affinity | Schedule pods on preferred or required nodes. | requiredDuringSchedulingIgnoredDuringExecution |
| Taints & Tolerations | Prevent pods from scheduling on certain nodes unless tolerated. Protect dedicated nodes. | kubectl taint nodes node1 key=value:NoSchedule |
| RBAC | Control who can do what in the cluster. Apply least privilege. | Role + RoleBinding, ClusterRole + ClusterRoleBinding |
18–22. CI/CD, Production Pipelines, Artifacts, Deployment & GitOps
Automate. Validate. Deliver. Repeat.
CI/CD Fundamentals & Pipeline Stages
| Concept | Description |
|---|---|
| Continuous Integration (CI) | Developers integrate code frequently into shared repository. Automated build on every commit. Early issue detection. |
| Continuous Delivery (CD) | Code is always in a deployable state. Automated testing + quality gates. Manual approval for production. |
| Continuous Deployment (CD) | Every passing change is automatically deployed to production. Fully automated release. |
| Stage | What Happens | Examples |
|---|---|---|
| 1. Build | Compile source code, create artifacts | mvn clean package, docker build |
| 2. Test | Run automated tests to ensure quality | Unit tests, integration tests, pytest |
| 3. Package | Package for distribution | Create JAR, WAR, or Docker image |
| 4. Scan | Security, vulnerability, compliance scanning | SAST, SCA, Container Scan, Secret Scan |
| 5. Deploy | Deploy to target environment | Kubernetes, VM, Serverless |
| 6. Validate | Verify deployment and application health | Smoke tests, health checks, monitoring |
GitOps & Continuous Delivery with Argo CD
| GitOps Principle | Description |
|---|---|
| Git as Source of Truth | All configurations in Git. Single source of truth. Versioned and auditable. Enables traceability. |
| Desired State | Define desired state in Git. Cluster converges to that state. No manual changes. Declarative. |
| GitOps Workflow | Developer pushes → GitOps tool detects change → Syncs to cluster → Cluster becomes consistent → Continuous monitoring |
| Argo CD | Declarative CD tool. Monitors Git repositories. Syncs to Kubernetes clusters. Web UI and CLI. Built-in observability. |
| Drift Detection | GitOps continuously detects and highlights when actual ≠ desired state. Triggers sync to fix. |
| Rollbacks | Revert Git commit → GitOps tool detects change → Automatically rolls back → Safe and auditable rollbacks |
| Environment Promotion | Promote changes across environments: Dev → Test → Staging → Prod. Controlled and traceable. |
23–26. Observability, Prometheus, Grafana & Logging
See Everything. Understand Deeply. Act Quickly.
Observability Fundamentals
| Pillar | Description | Examples |
|---|---|---|
| Metrics | Numeric measurements over time. Aggregated, fast, and efficient. | CPU usage, request rate, latency, error rate |
| Logs | Discrete events with timestamps. Detailed records for deep investigation. | Error logs, access logs, application logs |
| Traces | Request flow across services. Shows latency and dependencies. | Distributed request trace across microservices |
| Events | Important state changes or notifications. Lightweight and meaningful. | Deployment succeeded, pod crashed, alert fired |
| Telemetry | The signals collected from systems. Includes metrics, logs, traces, and events. | Everything your systems emit |
| Method | Focus | Use For |
|---|---|---|
| Golden Signals | Latency, Traffic, Errors, Saturation | Understanding health of any service |
| RED Method | Rate, Errors, Duration | Service reliability from user perspective |
| USE Method | Utilization, Saturation, Errors | Infrastructure health and capacity |
Prometheus & Metrics
| Concept | Description |
|---|---|
| Architecture | Prometheus Server (Retrieval → TSDB → PromQL → Alerting). Visualized in Grafana. Long-term: Thanos/Cortex. |
| Targets | Anything exposing metrics via HTTP endpoint. Defined in scrape configs or service-discovered. |
| Exporters | Expose metrics in Prometheus format. Node Exporter (Linux), kube-state-metrics (K8s), mysqld_exporter (DB). |
| Scraping | Prometheus pulls metrics at regular intervals (default 15s). Configurable per job. |
| PromQL | Powerful query language. Filter, aggregate, analyze. Used in Grafana and alerting rules. |
| Labels | Key-value pairs adding context. Used for filtering and aggregation. job, instance, method, status, namespace. |
| Recording Rules | Pre-calculate and store query results. Improve performance. Keep dashboards fast. |
| Alerting Rules | Define when alerts fire. Integrated with Alertmanager for routing, annotations, severity. |
Logging & Distributed Tracing
| Concept | Description |
|---|---|
| Centralized Logging | Collect logs from all systems in one place. Eliminate log silos. Search, analyze, and act fast. Tools: ELK, Loki, CloudWatch, Splunk. |
| Structured Logs | JSON format. Easy to parse and query. Consistent fields. Better filtering and analysis. Essential for automation. |
| Log Aggregation | Aggregate from multiple sources. Index and store efficiently. Enable fast search, retention, archiving. |
| Correlation IDs | Unique ID per request. Track across services and systems. Include in logs, headers, and traces. |
| Distributed Tracing | Trace requests across microservices. End-to-end visibility. Identify latency bottlenecks. Tools: Jaeger, Zipkin, AWS X-Ray. |
| Spans | A span = unit of work. Has start time, end time, duration, tags, logs, metadata. Spans form a trace tree. |
| OpenTelemetry | Open standard for observability. Collects logs, metrics, traces. Vendor-neutral. One pipeline, multiple backends. |
| Root Cause Investigation | Use logs + metrics + traces. Follow request path → identify failing component → analyze context → fix root cause. |
27. SRE Fundamentals
Build reliable systems. Deliver value. Reduce toil.
SRE Fundamentals — Build Reliable Systems
| Concept | Definition | Example |
|---|---|---|
| SLI (Service Level Indicator) | Measurable metric quantifying what matters to users | Availability, Latency, Error Rate, Throughput, Success Rate |
| SLO (Service Level Objective) | Target value for an SLI. Defines reliability level you commit to. | 99.9% availability over 30 days |
| SLA (Service Level Agreement) | Formal contract with customers. Based on SLOs. Legal/business implications if unmet. | 99.9% uptime SLA with penalties |
| Error Budget | Acceptable amount of unreliability = 1 - SLO. When exhausted, stop risky changes. | 99.9% SLO → 0.1% error budget → 43.2 min/month downtime allowed |
| Availability | Uptime / (Uptime + Downtime) × 100. Common targets: 99.9%, 99.99%, 99.999%. | 99.9% = 8.7h downtime/year |
| Latency | How fast your system responds. Measure at P50, P95, P99, P99.9. Tail latency impacts UX. | P99 < 200ms |
| Toil | Manual, repetitive operational work. Does not scale. Reduces engineer satisfaction. Automate and eliminate. | Manually restarting services, processing tickets |
Start with what matters to users (SLIs). Set realistic SLOs. Use error budgets to drive decisions. Balance speed and reliability.
28–30. DevSecOps, Identity, Access & Supply Chain Security
Build secure. Deliver fast. Protect continuously.
DevSecOps — Integrate Security into Every Stage
| Practice | Description | Tools |
|---|---|---|
| Shift-Left Security | Move security activities earlier. Find and fix before production. | Code review, threat modeling |
| Secure SDLC | Security in every phase: Plan, Code, Build, Test, Release, Deploy, Operate. | All stages |
| SAST | Static Application Security Testing. Analyze source code without executing. | SonarQube, Semgrep, Checkmarx |
| DAST | Dynamic Application Security Testing. Test running application from outside. Find runtime vulnerabilities. | OWASP ZAP, Burp Suite, Invicti |
| SCA | Software Composition Analysis. Analyze open source dependencies for vulnerabilities. | Snyk, OWASP Dependency-Check |
| Secret Scanning | Detect hardcoded secrets/credentials. Scan commits and repos. | GitGuardian, Gitleaks, TruffleHog |
| Container Scanning | Scan container images for vulnerabilities. Check OS packages and configurations. | Trivy, Clair, Anchore |
| Infrastructure Scanning | Scan IaC, cloud resources, configurations. Detect misconfigurations. | Checkov, tfsec, AWS Inspector |
| Policy Enforcement | Enforce security and compliance policies. Use OPA, Sentinel, Kyverno. | OPA, Sentinel, Kyverno, Conftest |
| Security Gates | Stop bad code/configs from progressing. Gate on quality, security, compliance. | Pipeline gates, quality checks |
Identity, Secrets & Access Control
| Concept | Description |
|---|---|
| Authentication | Verify who the user or system is. Methods: Passwords, MFA, Certificates, Tokens. Enforce MFA everywhere. |
| Authorization | Determine what an authenticated identity can do. Based on roles, policies, and permissions. Least privilege. |
| IAM | Manage users, groups, roles, policies. Centralize identity and access control. Federated identity and SSO. |
| RBAC | Grant permissions based on roles. Roles map to permissions. Scalable. Widely used in K8s and cloud. |
| Least Privilege | Grant only minimum access required. Reduces blast radius. Review and remove unnecessary access. |
| Service Accounts | Non-human identities for applications and services. Scoped permissions only. Avoid human credentials. |
| Secrets Management | Store secrets securely. Encrypt in transit and at rest. Centralized stores: Vault, AWS Secrets Manager, Azure Key Vault. |
| Secret Rotation | Rotate secrets regularly. Automate rotation. Revoke old secrets immediately. Reduce exposure window. |
| Short-lived Credentials | Temporary credentials with short TTL. Tokens, STS, OIDC. Avoid long-lived keys. |
| Workload Identity | Bind workloads to identities (OIDC, IRSA). No static credentials inside workloads. |
The right identity. The right access. Only when needed. For only as long as needed.
31–35. Incident Response, Troubleshooting, Failure Patterns, DR & Postmortems
Detect fast. Respond smart. Restore reliably. Learn continuously.
Incident Response — 11 Stages
| Stage | What to Do |
|---|---|
| Detection | Monitor systems, detect anomalies early, use metrics/logs/alerts, confirm incident existence |
| Alerting | Trigger rules-based alerts, reduce noise, route to right team, include context in alerts |
| Triage | Validate the incident, identify affected systems, determine impact/scope, gather initial data |
| Severity Classification | Classify P1-P4 based on impact/urgency. Communicate severity. Reassess as needed. |
| Incident Command | Assign incident commander, define roles/responsibilities, coordinate response efforts |
| Communication | Communicate early and clearly. Regular updates. Status pages. Keep stakeholders informed. |
| Mitigation | Contain impact, apply temporary fixes, reduce user impact, avoid further degradation |
| Recovery | Apply permanent fix, restore normal operations, monitor system behavior, validate dependencies |
| Verification | Verify system healthy, confirm resolution, validate with monitoring, test critical user flows |
| Timeline | Record key timestamps, track events in order, include actions taken, support post-incident review |
| Escalation | Escalate when needed, follow escalation path, engage experts, remove blockers quickly |
Production Troubleshooting Methodology — 11 Steps
| Step | Action | Why |
|---|---|---|
| 1. Define Symptoms | Identify and document the problem clearly. What is broken? What are users experiencing? | Start with facts, not assumptions |
| 2. Establish Scope | Determine impact and scope. Who is affected? Which services? | Prioritize investigation correctly |
| 3. Check Recent Changes | Review deployments, config changes, releases, infra updates | Most incidents caused by recent changes |
| 4. Inspect Metrics | Check monitoring dashboards. Look for anomalies, spikes, drops. | Data-driven diagnosis |
| 5. Inspect Logs | Search logs for errors, warnings, correlated events. | Logs contain the answers |
| 6. Inspect Traces | Follow requests across services. Identify slow calls and failures. | Find exactly where it broke |
| 7. Test Dependencies | Verify health of downstream/upstream services, databases, external APIs. | Isolate the root cause layer |
| 8. Form Hypotheses | Create possible root cause theories based on gathered data. | Structured thinking |
| 9. Validate Hypotheses | Test and confirm (or eliminate) hypotheses with evidence. | Evidence-based diagnosis |
| 10. Mitigate Safely | Implement safest fix or workaround. Minimize risk and blast radius. | Fix without making it worse |
| 11. Verify Recovery | Confirm issue is resolved. Monitor to ensure stability. | Validate the fix works |
| What To Avoid | |
|---|---|
| Assuming the root cause | Always gather data first |
| Making random changes | Have a hypothesis before changing anything |
| Ignoring user impact | Users are the priority |
| Skipping steps | The methodology exists for a reason |
| No communication | Keep stakeholders informed throughout |
| Not documenting | Document everything for learning and future reference |
Common Production Failure Patterns
| Failure Pattern | Symptoms | Common Causes | What to Check |
|---|---|---|---|
| CPU Saturation | High CPU usage, slow response, request timeouts | Runaway processes, unoptimized code, too many requests | top, htop, recent deployments, application logs |
| Memory Exhaustion | High memory usage, OOM kills, service crashes | Memory leaks, large caches, too many instances | free -m, top, dmesg (OOM logs), container limits |
| Disk Full | Writes failing, services crashing, logs not writing | Large logs, unclean temp files, no log rotation | df -h, disk I/O (iostat), log directories, df -i |
| DNS Failure | Cannot resolve names, intermittent failures, slow requests | DNS server down, wrong config, TTL/cache issues | nslookup/dig, /etc/resolv.conf, CoreDNS logs |
| Network Timeout | Request timeouts, slow connections, intermittent failures | Network congestion, packet loss, firewall blocks | ping, traceroute, security groups, network latency |
| Database Exhaustion | DB connections failing, slow queries, high DB CPU/IO | Too many connections, long-running queries, no connection limits | DB metrics, active connections, slow query logs |
| Connection Pool Problems | Connection timeouts, requests hanging, pool exhausted | Pool too small, connections not released, idle timeout mismatch | Pool metrics, active/idle connections, app logs |
| Certificate Expiration | SSL errors, services failing, users see warnings | Expired cert, auto-renew failed, wrong cert, missing chain | openssl s_client, expiration date, monitoring alerts |
| Dependency Failure | External calls failing, errors from API, timeouts | Service down, rate limits, network issues, bad config | Status pages, error logs, retries, circuit breakers |
| Bad Deployment | Errors after deploy, health checks fail, performance drop | Bug in new release, wrong config, DB migration issues, incomplete rollout | Deployment logs, health checks, application logs, recent changes |
Rollback, Recovery & Disaster Recovery
| DR Concept | Definition | Key Actions |
|---|---|---|
| Application Rollback | Revert application to previous known good version | Revert deployment, rollback traffic, verify functionality |
| Infrastructure Rollback | Revert infrastructure changes to stable state | Revert IaC changes, restore previous state, validate environment |
| Database Recovery | Restore database to consistent and healthy state | Restore from backup, point-in-time recovery, validate data integrity |
| Backups | Maintain reliable backups for all critical systems | Automated backups, encrypt, store offsite/off-region, retention policies |
| Restore Testing | Regularly test restores to ensure backups actually work | Test full/partial restore, validate data, document results |
| RPO | Recovery Point Objective — maximum acceptable data loss (measured in time) | Define per system, align backup frequency |
| RTO | Recovery Time Objective — maximum acceptable downtime to restore service | Define per system, optimize recovery time, automate recovery |
| Disaster Recovery | Recover systems after major disruption or site failure | Activate DR plan, failover services, communicate status, stabilize |
| Multi-Region Recovery | Multiple regions for high availability and recovery | Deploy across regions, replicate data, configure failover, route traffic |
| Recovery Validation | Validate systems are fully recovered and operating correctly | Run validation tests, verify functionality, monitor stability |
Postmortems & Continuous Improvement
| Element | Description |
|---|---|
| Blameless Postmortems | Focus on the system, not individuals. No blame. Open and honest. Safe environment. Trust and transparency. |
| Incident Timeline | Build clear timeline: when it started, what changed, when escalated, when mitigated, when resolved. |
| Root Cause Analysis | Identify underlying cause. Go beyond symptoms. Find the real trigger. Ask "Why did this happen?" |
| Five Whys | Keep asking "Why?" until you reach the root cause. Stop at root cause. Clear, deep, verified answer. |
| Contributing Factors | List all factors that made incident worse: technical, process, people, environmental, tooling gaps. |
| Corrective Actions | Actions to fix the issue quickly and restore stability. Immediate fixes, hotfixes, rollbacks, workarounds. |
| Preventive Actions | Actions to prevent this incident from recurring. Process improvements, automation, monitoring, better testing. |
| Ownership | Assign clear ownership for each action item. One owner per action. Clear responsibilities. |
| Follow-up | Track progress until all actions completed. Regular updates. Verify completion. Close the loop. |
| Learning | Capture lessons. Share learnings. Update runbooks. Build a learning culture. |
36–40. Platform Engineering, Enterprise Architecture, Metrics, Senior Principles & Blueprint
From code to production. Secure. Reliable. Observable. Automated.
Platform Engineering & Internal Developer Platforms
| Concept | Description |
|---|---|
| Platform Engineering | Build and operate platforms for internal users. Abstract complexity. Enable developers to ship faster. Treat the platform like a product. |
| Developer Experience (DX) | Frictionless workflows, fast onboarding, clear docs, self-service tooling, consistent environments, quick feedback loops. |
| Internal Developer Platform (IDP) | Central portal for all platform services. Unified tooling. Self-service infrastructure, pipelines, and components. |
| Golden Paths | Opinionated, secure, well-architected paths. Pre-built patterns for common use cases. Reduce decision fatigue. |
| Self-Service Infrastructure | Provision environments on demand. Automated and API-driven. No manual tickets. Policy-driven and secure. |
| Service Templates | Pre-approved blueprints including code, IaC, pipelines, testing, security, monitoring. |
| Developer Portals | Single place to discover and access everything: catalogs, templates, runbooks, guides, approvals. |
| Platform APIs | Expose capabilities via well-documented APIs. Automate and integrate. Enable self-service and extensibility. |
| Guardrails | Enforce policies and compliance. Security/cost/performance controls. Prevent misconfigurations. |
| Platform as a Product | Has users (customers). Measure usage and value. Gather feedback. Iterate and improve. Clear SLAs. |
Enterprise DevOps Architecture — End-to-End
| Layer | Components | Details |
|---|---|---|
| Source Control | Git Repositories, Branch Protection, Code Reviews, PRs, Audit Trails | Centralized repos, branch protection, collaboration |
| CI Platform | Automated builds, tests, quality checks, security scanning, pipeline as code | Fail fast feedback, quality gates |
| Artifact Management | Store binaries, packages, container images. Version control, retention, immutability. | JFrog Artifactory, Sonatype Nexus, ECR |
| Security Systems | SAST, DAST, Container/Dependency Scanning, Policy Enforcement, Vulnerability Mgmt | Shift-left, continuous scanning |
| Infrastructure as Code | Terraform/OpenTofu. Versioned IaC, reusable modules, environment parity, drift detection. | Policy as code |
| Kubernetes | Container orchestration: workloads, scaling, service discovery, networking, storage, HA. | Self-healing, auto scaling |
| GitOps | Declarative state (Argo CD): Git as source of truth, automated sync, drift detection, rollback. | Argo CD |
| Observability | Prometheus (metrics), Grafana (dashboards), Loki (logs), Jaeger (traces), alerting, SLO monitoring. | Full-stack visibility |
| Secrets Management | HashiCorp Vault: centralized secrets, dynamic secrets, rotation, fine-grained access, audit logs. | Zero-trust secrets |
| Cloud Infrastructure | AWS/Azure/GCP: compute, networking, storage, databases, managed services, cost optimization. | Scalable, secure, reliable |
DevOps Performance Metrics — DORA & Engineering KPIs
| Metric | Description | Good Target |
|---|---|---|
| Deployment Frequency | How often code is successfully deployed to production | On-demand or multiple times per day |
| Lead Time for Changes | Time from code commit to successful deployment in production | Less than 1 day |
| Change Failure Rate | Percentage of changes causing degraded service or requiring rollback | Less than 15% |
| Mean Time to Recovery (MTTR) | Average time to restore service after incident or failure | Less than 1 hour |
| Availability | Percentage of time system is operational and functional | 99.9% or higher |
| Pipeline Duration | Total time for CI/CD pipeline run from start to finish | As fast as possible — optimize continuously |
| Build Success Rate | Percentage of builds that complete successfully without failures | Greater than 95% |
| Infrastructure Reliability | How reliable, resilient, and performant your infrastructure is | High uptime, low errors |
| Alert Quality | Percentage of alerts that are actionable, accurate, and useful | High signal, low noise |
| Engineering Effectiveness | How effectively the team delivers value to customers | Continuously improving |
| Maturity Level | Characteristics |
|---|---|
| Elite | High performance across all metrics. Continuous improvement. Data-driven culture. |
| High | Consistently better than industry average. |
| Medium | Improving but inconsistent. |
| Low | Frequent issues and manual processes. |
Senior DevOps Engineer — 12 Production Principles
| Principle | Focus Areas |
|---|---|
| 1. Automate Repetitive Work | Automation, Scripting, CI/CD, Self-service. Increases speed, consistency, reliability. |
| 2. Design for Failure | Resilience, Redundancy, Timeouts, Retries. Failures will happen. Design systems that expect and survive them. |
| 3. Prefer Simple Systems | KISS, reduce complexity, YAGNI. Simplicity improves reliability, maintainability, and understanding. |
| 4. Make Changes Reversible | Rollbacks, Feature Flags, Blue/Green, Canary. Always design changes so you can go back quickly and safely. |
| 5. Reduce Blast Radius | Isolation, Segmentation, Least Privilege. Limit the impact of failures to small, isolated areas. |
| 6. Build Observability First | Metrics, Logs, Traces, Alerts, Dashboards. You can't fix what you can't see. Instrument everything. |
| 7. Protect Production | Access Control, Change Management, Safeguards. Production is the customer experience. Treat it with respect. |
| 8. Document Critical Decisions | Architecture Decisions, Runbooks, Standards. Good documentation preserves knowledge and prevents repeat mistakes. |
| 9. Understand Dependencies | Dependency Maps, Communication, Contracts. Know how systems, services, and teams depend on each other. |
| 10. Measure Before Optimizing | Metrics, Baselines, Data-driven Decisions. Measure first, then optimize with data, not assumptions. |
| 11. Treat Security as Engineering | Threat Modeling, Scanning, Least Privilege, Compliance. Security is not a step — it's a responsibility in every decision. |
| 12. Own Reliability | SLOs, Error Budgets, Incident Ownership. You own the outcome. Be accountable for reliability and uptime. |
PRINCIPLES TODAY. RELIABILITY TOMORROW.
The Complete DevOps Production Blueprint — 13-Stage Pipeline
Write, commit, change
Branch, PR, review, merge
Build, test, quality, scan
SAST, DAST, scan, policy
Store, version, promote
IaC, plan, apply
Build image, scan, push
Deploy, health, scale, heal
Git source of truth, sync
Metrics, logs, traces, alerts
Detect, triage, mitigate
Rollback, restore, verify
| Checklist | Status |
|---|---|
| Code Reviewed | Must pass before merge |
| Tests Passing | All unit + integration tests green |
| Security Cleared | No critical/high vulnerabilities |
| IaC Reviewed | Terraform plan reviewed and approved |
| Observability Ready | Metrics, logs, alerts configured |
| Runbooks Ready | Operational procedures documented |
| Backup Validated | Data backup and restore tested |
| Stakeholders Approved | Change management sign-off |
| DevOps Maturity Level | Characteristics |
|---|---|
| Level 5 — Optimized | Continuous improvement, data-driven culture, full automation |
| Level 4 — Measured | Standardized processes, metrics, SLOs, automation |
| Level 3 — Defined | Defined processes, some automation, basic monitoring |
| Level 2 — Managed | Basic automation, inconsistent processes |
| Level 1 — Initial | Manual, ad-hoc, reactive |