Document and Smoke-Test Scheduled Automations

Tier: Foundation · Workflow area: Scripting & automation · ~25 minutes per automation

This playbook inventories every scheduled task, cron job and RMM automation in your estate, then generates for each a one-page runbook (what it does, when, as whom, what depends on it) and a smoke test that proves it actually ran — turning silent failures into loud ones.

When to use

• Estate inventory momentstakeovers, audits, "how many scheduled tasks do we even have?" (nobody ever knows). Pairs with pb-22 for the scripts nobody can explain.
• Every new automation you deploythe runbook and smoke test ship with the job from day one; retrofitting is this playbook, but the habit is the goal.
• After a silent-failure incidentthe report that ran empty for months, the sync that stopped syncing; the retro action is sweeping the rest before they bite too.

When NOT to use

• On orchestrator-managed pipelinesJenkins, GitHub Actions, Azure Automation runbooks with built-in alerting already have execution visibility; document the gaps (who owns it, what depends on it), don't duplicate the platform.
• As a one-time project with no maintenance ownerundated runbooks rot into misinformation; each page needs an owner and a review trigger or the sweep just creates future archaeology.
• To smoke-test what should be decommissionedthe inventory will surface jobs whose purpose died years ago; the right artifact for those is a retirement change, not a runbook.

The recipe

1. 1.

   Inventory everything that runs on a schedule. Windows tasks, cron, and the RMM's automation list:

   Copy

   Get-ScheduledTask | Where-Object State -ne Disabled |
     Select-Object TaskName, TaskPath,
       @{n="RunAs";e={$_.Principal.UserId}},
       @{n="LastResult";e={($_ | Get-ScheduledTaskInfo).LastTaskResult}},
       @{n="LastRun";e={($_ | Get-ScheduledTaskInfo).LastRunTime}} |
     Export-Csv .\task-inventory.csv -NoTypeInformation
   

   Copy

   for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u" 2>/dev/null \
     | sed "s/^/$u: /"; done; ls /etc/cron.d /etc/cron.daily
   

   Non-zero LastTaskResult values in that first export are your instant hit list — silent failures already in progress.

2. 2.

   Generate the runbook page per automation. Feed each script (or RMM automation config) to the model:

   Copy

   Write a one-page runbook for this scheduled automation.
   
   Sections:
   - Purpose: what it does and WHY it exists (mark WHY as
     [OWNER TO CONFIRM] if not evident from the script)
   - Schedule & identity: when it runs, as what account, from where
   - Inputs/outputs: files, APIs, databases, emails touched
   - Dependencies: what breaks if this stops; what this breaks if wrong
   - Failure signature: what a failed run looks like (exit codes,
     missing output, log lines) — from the script's actual error paths
   - Manual run procedure: exact command to run it by hand safely
   - Owner and review-by date: [FILL]
   
   Script: [paste]   Schedule/context: [paste from inventory]
   

3. 3.

   Fill the human-only fields. The model reads code; it cannot know that the CSV this job drops is what payroll imports on the 25th. The owner confirms the WHY, the downstream dependents and the business impact — ten minutes per job, and the only part that requires a human.

4. 4.

   Generate the smoke test — proof of effect, not proof of schedule. The test checks the job's output, not its exit code:

   Copy

   Write a PowerShell smoke test for this automation. It must verify
   EFFECTS, not just that the task ran:
   - the expected output artifact exists and is fresher than [interval]
   - the artifact is non-trivial (size/row-count above a floor)
   - key content sanity check: [e.g., header row matches, no ERROR lines]
   On any failure: write an event-log entry and send one alert to
   [channel/email]. On success: silent.
   Automation runbook: [paste the page from step 2]
   

   Sample of the core assertion pattern:

   Copy

   $f = Get-Item "\\fs01\reports\daily-export.csv" -ErrorAction SilentlyContinue
   if (-not $f -or $f.LastWriteTime -lt (Get-Date).AddHours(-26)) {
     Write-EventLog -LogName Application -Source "SmokeTest" -EntryType Error `
       -EventId 9001 -Message "daily-export.csv missing or stale"
     Send-Alert "SMOKE FAIL: daily-export stale ($($f.LastWriteTime))"
   } elseif ((Import-Csv $f.FullName).Count -lt 50) {
     Send-Alert "SMOKE FAIL: daily-export suspiciously small ($($f.Length) bytes)"
   }
   

   The staleness window (26h for a daily job) and the row-count floor are the two numbers that catch the "ran but produced nothing" class.

5. 5.

   Schedule the smoke tests offset from their targets. Each test runs after its automation's window (job at 02:00, test at 04:00) and alerts to a channel someone reads. Review the smoke tests with the pb-21 checklist — they are scripts too.

6. 6.

   Break something on purpose. For each smoke test, force one failure in a controlled way (rename the output file, point the job at a dead path in a test window) and confirm the alert fires and reaches a human. An untested smoke test is a second silent failure waiting behind the first.

7. 7.

   Keep the register. One table: automation | host | owner | runbook link | smoke test | last verified. New automations don't ship without a row; the register is what you hand the auditor, the new starter (pb-20), and your successor.

Make it yours

• RMM shopthe RMM's script scheduler is half your inventory; use its native success/failure alerting as tier one, and add effect-checking smoke tests only for the jobs whose "success" can still mean "did nothing."
• Cloud scheduler usersAzure Automation, Lambda on EventBridge: platform logs cover execution, so your smoke tests live as separate monitor functions checking downstream effects, and the runbook lives in the repo with the code.
• One-person ITdo the inventory and the hit list today, runbook the five scariest jobs this week, and add smoke tests to the three whose silent failure would hurt most; breadth later, blast radius first.

Quality check

• Tests check effects, not exit codes. "Task returned 0" co-exists happily with "produced an empty file"; every test asserts on the artifact.
• Every alert path fired once for real. The deliberate-failure drill (step 6) is checklist-mandatory, per test.
• [OWNER TO CONFIRM] fields all resolved. A runbook with unresolved placeholders is a draft, not documentation — the register tracks completion.
• The inventory is re-run quarterly. New mystery jobs appear; a scheduled diff of the inventory export against the register catches automations that bypassed the process.