🤖 Ansible Automation Platform (Tower/AWX) Lab Guide
Hands-on Controller lab — Inventories · Credentials · Projects · Job Templates · Surveys · Workflows · RBAC · Notifications · API, built on a 3-node lab (Controller + 2 managed nodes).
0. Your Lab Topology & Naming
Reference used throughout every task below — keep this open in a second tab
Lab Inventory Reference
| Hostname | Role | OS | Notes |
|---|---|---|---|
app.local.lab | Controller node (Tower/AWX/AAP) | RHEL 10.2 | Runs the web UI, API, execution engine |
node01.local.lab | Managed node | RHEL 10.2 | Same major OS as controller — good "happy path" node |
node02.local.lab | Managed node | RHEL 8.10 | Older Python/OpenSSL — deliberately used to practice mixed-OS automation and version drift |
All tasks below assume: DNS or /etc/hosts resolves the three FQDNs, SSH key-based auth is possible from app.local.lab to both nodes, and you have a local automation user (e.g. ansible) with sudo rights on node01/node02.
1. Organizations, Users & Base Setup
The container objects everything else lives inside
Task 1 — Create an Organization and a non-admin User
Task
Create an Organization called LocalLab. Inside it, create a user opuser (non-superuser) who will later be granted limited permissions. Do this both via the UI and via the CLI/API so you understand both paths.
Automation Execution → Organizations → Add → Name: LocalLab → Save. Then Access Management → Users → Add → Username: opuser, uncheck "System Administrator", set a password, Save. Then open Organizations → LocalLab → Access → Add → select opuser with role Member.
2. Inventories, Groups & Host Variables
Static inventory, groups by OS, per-host variables, ad-hoc commands
Task 2 — Build a Static Inventory with OS-based Groups
Task
Create Inventory local_lab. Add both nodes as Hosts. Create two Groups: rhel10 containing node01.local.lab, and rhel8 containing node02.local.lab. Set host variable ansible_host if your DNS is unreliable.
Inventories → Add → Inventory → Name: local_lab, Organization: LocalLab → Save. Inside it: Groups → Add → rhel10 → Save; Groups → Add → rhel8 → Save. Hosts → Add → node01.local.lab (Variables field optional):
Repeat for node02.local.lab with its own IP. Then open group rhel10 → Hosts → Associate → node01.local.lab. Same for rhel8 → node02.local.lab.
SUCCESS | rc=0 | pong. If node02 (RHEL 8.10) fails with a Python interpreter error, that's your first real troubleshooting task — see Task 3.Task 3 — Fix Python Interpreter Mismatch (RHEL 8 vs RHEL 10)
Task
RHEL 8.10's default python3 may resolve differently than RHEL 10.2's. Force the correct interpreter on node02 using a group variable instead of editing playbooks.
On node02.local.lab, confirm interpreter location:
In Controller: Inventories → local_lab → Groups → rhel8 → Variables:
ansible_python_interpreter is the textbook answer.Task 4 — Smart Inventory Filtered by Facts
Task
Create a Smart Inventory rhel10_only that automatically includes any host where ansible_distribution_major_version equals 10, sourced from local_lab.
Inventories → Add → Smart Inventory → Name: rhel10_only, Organization: LocalLab, Smart Host Filter:
Save, then open the Smart Inventory's Hosts tab — it should auto-populate with node01.local.lab only, after a fact-gathering job has run at least once against local_lab (facts must be cached first — enable "Use Fact Cache" on the Job Template used to gather facts).
3. Credentials
Machine, Become, Vault, and testing failure modes
Task 5 — Machine Credential with SSH Key
Task
Generate an SSH key pair on app.local.lab, distribute the public key to both nodes, then create a Machine Credential in Controller and validate it with an ad-hoc ping.
Credentials → Add → Name: lab-ssh-key, Credential Type: Machine, Username: ansible, paste the private key from ~/.ssh/aap_lab into "SSH Private Key". Save. Then run an ad-hoc ping job against local_lab using this credential.
Task 6 — Become (sudo) Credential + Deliberate Failure Practice
Task
Add a Become password to the credential (or a separate credential) and run a privileged ad-hoc command. Then deliberately break it three ways and record the exact error each time: (a) wrong username, (b) wrong SSH key, (c) become password omitted while target requires one.
| Deliberate break | Typical error |
|---|---|
| Wrong username | UNREACHABLE! ... Permission denied (publickey,password) |
| Wrong/mismatched SSH key | UNREACHABLE! ... Permission denied (publickey) |
| Become required, no password/NOPASSWD | FAILED! ... Missing sudo password or sudo: a password is required |
Task 7 — Vault Credential for Encrypted Variables
Task
Encrypt a secret string with ansible-vault, reference it from a playbook, and run it in Controller using a Vault credential attached to the Job Template.
In Controller: Credentials → Add → Type: Vault, Vault Password: LabVault123! → Save. On the Job Template, add this Vault credential alongside the Machine credential. Run — Controller decrypts transparently at runtime.
4. Projects (SCM Integration)
Git-backed playbook source, auto-sync, branches
Task 8 — Git-backed Project with SCM Update on Launch
Task
Push a folder of playbooks (see Task 9) to a Git repo (GitHub, GitLab, or a local bare repo). Create a Project in Controller pointing at it, with "Update Revision on Launch" enabled.
Projects → Add → Name: Lab Playbooks, SCM Type: Git, SCM URL: /home/ansible/git/lab-playbooks.git (or your GitHub URL), check Update Revision on Launch. Save — watch the sync job succeed and confirm the playbooks list populates on the Job Template creation screen.
5. Job Templates & Core Playbooks
The playbooks referenced throughout this guide — copy these into your project
Task 9 — Basic Job Template: Server Discovery
Task
Write discovery.yml that reports hostname, OS, kernel, CPU, memory and IP for every host. Create a Job Template Server Discovery using it and review the Job Output panes (Details, Output, Recap).
Templates → Add → Job Template → Name: Server Discovery, Inventory: local_lab, Project: Lab Playbooks, Playbook: discovery.yml, Credentials: lab-ssh-key → Save → Launch. In the results, check the Recap tab for ok/changed/failed counts per host and the Output tab for the raw debug messages.
Task 10 — Idempotent Package Install + Service Management
Task
Write a playbook that installs and starts httpd on all hosts. Run it twice and confirm the second run shows 0 "changed" tasks (idempotency). Then manually stop the service on a node and re-run to prove Controller restores desired state.
Task 11 — Survey: Prompt for Package + Choice List
Task
Add a Survey to a "Package Manager" Job Template with: a required text field package_name, a multiple-choice field package_action (install/remove) with default install. Playbook should act on the survey answers via extra_vars.
On the Job Template → Survey tab → Add:
| Field | Type | Required | Default / Choices |
|---|---|---|---|
| Package Name | Text | Yes | e.g. vim-enhanced |
| Action | Multiple Choice (single select) | Yes | install, remove — default install |
Enable the Survey (toggle at top of the Survey tab) — it's inactive by default even after adding questions. Launch — Controller now shows a form before running.
Task 12 — Jinja2 Template Deployment (per-host config)
Task
Deploy a per-host config file /opt/lab/app.conf using a Jinja2 template, driven by group variables (environment differs per group) and gathered facts.
ENVIRONMENT value — this proves group_vars precedence is working as expected.Task 13 — Conditionals for OS-specific Logic (RHEL8 vs RHEL10)
Task
Using your two real OS versions, write one playbook that installs httpd on RHEL 10 hosts and nginx on RHEL 8 hosts, using when: conditions against ansible_distribution_major_version.
Task 14 — Loops, Handlers & Roles Refactor
Task
Create three local users via a loop, then convert the whole "web server" playbook into a proper role: roles/webserver/{tasks,handlers,templates,defaults,vars,meta}.
6. Extra Vars, Tags & Check Mode
Prompt-on-launch behavior, dry runs, selective execution
Task 15 — Prompt on Launch: Extra Variables
Task
Write motd.yml that writes a custom message to /etc/motd from an extra_vars var motd_message. Enable "Prompt on Launch" for Variables on the Job Template and supply a different value at each launch.
Job Template → toggle "Prompt on Launch" next to Variables → Save. At Launch, Controller shows an Extra Variables box — enter:
Task 16 — Tags: Split Install / Configure / Validate
Task
Tag tasks as install, configure, validate. Create three Job Templates from the same playbook, each using --tags to run only one phase, plus a fourth "Full Deployment" template with no tag filter.
On each Job Template's "Job Tags" field enter install, configure, or validate respectively; leave blank on the "Full Deployment" template to run everything.
Task 17 — Check Mode (Dry Run) and Diff Mode
Task
Launch any Job Template with "Show Changes" (diff mode) and check-mode both enabled, and compare output to a normal run.
On the Launch dialog (or Job Template options), enable both Show Changes and Enable Check Mode. Launch — tasks show what would change (e.g. "package would be installed") without applying anything, and file-content diffs are shown inline for template/copy tasks. Use this before every production-style run.
7. Workflow Templates
Chaining jobs, branching on success/failure, approval gates
Task 18 — Linear Workflow: Precheck → Patch → Postcheck
Task
Create three Job Templates: Precheck (facts + disk space check), Patch (dnf update), Postcheck (re-check facts). Chain them in a Workflow Template so each only runs "on success" of the previous.
Templates → Add → Workflow Template → Name: Patch Workflow → Visualizer → drag Precheck as start node → add Patch node with link type On Success → add Postcheck node with link type On Success from Patch. Save, Launch, watch the graph light up green node by node.
Task 19 — Failure Branch + Notification Job
Task
Add a 4th node Failure Alert that only runs "On Failure" of Precheck. Force Precheck to fail (target a nonexistent host or add a failing assert) and confirm the branch fires while Patch/Postcheck are skipped.
In the Workflow Visualizer, from the Precheck node draw a second link to a new node Failure Alert (any simple debug/notify playbook) with link type On Failure. Temporarily lower the assert threshold above your lab's actual RAM to trigger it, run, then confirm in the workflow graph: Precheck = red, Failure Alert = ran, Patch/Postcheck = grey "not executed".
Task 20 — Approval Node Before Patching
Task
Insert an Approval node between Precheck and Patch with a timeout of 10 minutes. Confirm the workflow pauses and requires a human click to continue, and that it fails-safe on timeout.
In the Workflow Visualizer, click the "+" on the link between Precheck and Patch → Add Approval Node → Name: Manager Approval, Timeout: 600 seconds. Save, Launch — the workflow pauses at "Pending Approval". Go to Jobs → find the Workflow → the Approval node shows Approve/Deny buttons. Test both: Approve continues to Patch; Deny (or letting it time out) marks the workflow failed and Patch/Postcheck never run.
8. RBAC: Organizations, Teams, Roles
Least-privilege access — the #1 real-world admin skill
Task 21 — Execute-Only User vs Admin User
Task
Give opuser (from Task 1) the Execute role only on the Server Discovery Job Template — not Admin. Log in as opuser and confirm they can launch it but cannot edit its playbook path, delete it, or see Credentials details.
Templates → Server Discovery → Access → Add → select opuser, Role: Execute → Save. Log out, log in as opuser. Expected: the template appears under Templates, a Launch (rocket) icon is available, but the pencil/edit icon and Delete are absent or return a 403 if hit directly via API.
| Role | View | Launch | Edit | Delete |
|---|---|---|---|---|
| Read | ✅ | ❌ | ❌ | ❌ |
| Execute | ✅ | ✅ | ❌ | ❌ |
| Admin | ✅ | ✅ | ✅ | ✅ |
Task 22 — Teams and Delegated Inventory Access
Task
Create a Team OpsTeam under LocalLab, add opuser to it, and grant the Team Use role on the local_lab Inventory and Credential (but not Admin), so team members can build their own Job Templates against your lab without touching Credential secrets.
Access Management → Teams → Add → Name: OpsTeam, Organization: LocalLab. Open it → Users → Add → opuser. Then Inventories → local_lab → Access → Add → search Team → OpsTeam, Role: Use. Credentials → lab-ssh-key → Access → Add → OpsTeam, Role: Use. Note: "Use" lets them reference the credential in a Job Template they own without ever viewing the private key.
9. Scheduling & Notifications
Recurring jobs and alerting on success/failure
Task 23 — Schedule a Recurring Job
Task
Schedule Server Discovery to run every 15 minutes. Review the Schedules tab and confirm job history builds up automatically without manual launches.
Job Template → Schedules → Add → Name: Every 15 min, Start Date/Time: now, Repeat Frequency: Custom → RRULE:
Save. After ~30 minutes, Jobs → filter by this template — you should see 2+ automatic runs with "Launched By: Scheduler".
Task 24 — Notification on Job Failure
Task
Create a Webhook Notification Template pointing to a local test receiver (e.g. webhook.site, or a small local HTTP listener), attach it to a Job Template for the Failure event, then trigger a real failure (target a nonexistent host) and confirm delivery.
Administration → Notifications → Add → Type: Webhook, Target URL: http://app.local.lab:9000/ → Save. On the Job Template → Notifications tab → toggle this template ON for Failure (and separately for Success if you want both). Temporarily change the inventory to include node99.local.lab (doesn't exist), launch — job fails with UNREACHABLE, and the webhook receiver logs an inbound POST.
10. Execution Environments & Job Slicing
Container-based runtimes and horizontal scale-out
Task 25 — Inspect and Assign an Execution Environment
Task
Identify which Execution Environment (EE) your Job Templates use by default, and explicitly pin the Server Discovery template to a specific EE image.
Administration → Execution Environments — note the default (usually ee-supported-rhel10 or ee-minimal). Each EE is an OCI container bundling ansible-core, Python, and collections — this is what actually executes your playbook, not the Controller host's own Python. On the Job Template → Execution Environment dropdown → explicitly select one and Save; re-run and confirm the Job Details pane shows that EE's image name.
Task 26 — Job Slicing (simulated with your 2 nodes)
Task
Set Job Slicing to 2 on a Job Template running against local_lab (2 hosts) and observe that Controller splits the run into 2 separate jobs under one workflow-style job, one per slice.
Job Template → Job Slicing field → set to 2 → Save → Launch. Jobs list shows a parent "Slice Job" plus two child jobs, each targeting one host. At real scale (say 100 hosts, slice count 10) this parallelizes work across multiple Controller execution nodes/instance groups — with only 2 hosts you're mainly proving the mechanism, not the performance gain.
11. REST API Automation
Driving Controller from curl / Python instead of the UI
Task 27 — Launch a Job Template and Poll Status via API
Task
Authenticate to the API, list Job Templates, launch Server Discovery by ID, and poll until it completes — all with curl.
12. Capstone Project — Self-Service Patch Portal
Combine every feature above into one real workflow
Task 28 — Build the Full Patch Workflow
Task
Build a single Workflow Template a non-admin can self-launch that: (1) shows a Survey for a maintenance window comment, (2) runs Precheck (facts + disk + service status), (3) waits for Approval, (4) runs Patch (dnf update), (5) reboots if the kernel changed, (6) runs Postcheck, (7) sends a success or failure Notification, all restricted via RBAC so opuser can launch but not edit.
Add the Survey directly on the Workflow Template (not just individual Job Templates) so it prompts once at the top: field maintenance_comment, Text, optional. Then grant opuser/OpsTeam Execute role on the Workflow Template only — they can launch the entire pipeline with one click and never see the underlying playbooks or credentials.
13. Interview Q&A
Concept checks tied directly to the tasks above
Core Concepts
ansible-vault files transparently at job launch.ansible_python_interpreter at the group/host variable level rather than hardcoding it in playbooks, and use when: conditionals keyed off gathered facts like ansible_distribution_major_version for OS-specific task logic.14. Quick Reference
Object hierarchy and common CLI/API one-liners
Command & Concept Cheatsheet
| Task | CLI (awx-cli / controller-cli) |
|---|---|
| Login | awx login --conf.host https://app.local.lab |
| List inventories | awx inventory list |
| Launch a template | awx job_templates launch "Server Discovery" --monitor |
| Check job status | awx jobs get <id> |
| Ad-hoc ping | ansible local_lab -m ping |
| Vault-encrypt a file | ansible-vault encrypt group_vars/all/vault.yml |
| Syntax-check a playbook | ansible-playbook site.yml --syntax-check |
| Dry run from CLI | ansible-playbook site.yml --check --diff |