Agents are a platform tenant that acts — with credentials, on your infrastructure. This module turns the trust map into infrastructure: guardrails enforced outside the model, per-agent identity and audit, typed MCP surfaces, and eval-gated pipelines that make AI behaviour changes deploy like code.
The rollback showed up in Monday's change review with nobody's name on it — or worse, the wrong name. At 03:12 Saturday, a remediation agent someone had wired up "just to try it" had restarted checkout-api and rolled back a deploy. It worked; the incident resolved.
The problem surfaced two days later, when Divya asked the obvious question — who approved this? — and the audit log answered "Rahul." Except Rahul had been on a flight over the Arabian Sea at 03:12. The agent had been running on his kubeconfig, because that was the credential lying around when it got built, and to the Kubernetes API an agent holding Rahul's token is Rahul.
Nobody could answer the three questions every postmortem asks — which agent, on whose behalf, triggered by what — because the agent had no identity of its own. It hadn't done anything wrong. It had done something unattributable, which on a platform is its own kind of wrong.
M7 built the substrate: models served, gated, metered, observed. This closing module adds the tenant that changes everything — the agent. Coding assistants opening pull requests, remediation agents restarting deployments, self-service copilots provisioning databases: software that acts, with credentials, on your infrastructure. The 2026 platform-engineering consensus says it plainly — agents are becoming a first-class platform persona, with the same RBAC, quotas, and golden paths humans get. The difference between an org where that's true and one where agents run on a developer's personal kubeconfig is not the model. It's the platform, and it's you.
This is also where the track's trust map completes its arc. M1 gave you the four placement tiers — generate-only, advise, act-with-approval, act-autonomously. M3 applied them to incident response. This module turns them into infrastructure: guardrail layers that make the tiers mechanical, an identity model that makes agents auditable, an MCP surface that makes "what an agent may touch" a typed API instead of a hope, and eval-gated pipelines that make AI behavior changes deploy like code. The 76% who won't trust AI in production are not wrong to withhold trust from vibes. They are waiting for exactly this: enforcement they can read.
An agent is a workload with a credential and an intent loop. Everything you know about untrusted multi-tenant workloads applies — least privilege, quotas, egress control, audit — plus one new axiom: the model's output is untrusted input to your platform, always, including when it is usually right.
Your first 15 minutes: grep for the agents already running in your cluster and ask one question of each — whose credential is it holding? Every answer that names a human instead of a per-agent ServiceAccount is a Rahul-on-a-flight incident waiting to happen, and this module's onboarding template is the fix.
"Guardrails for agentic workloads" appears verbatim in AI-platform job postings, and the plural matters. No single control survives contact with a capable model plus a hostile input, so you build defense in depth — five layers, each catching what the previous one misses, each independently auditable. Design them like you design network security zones: assume every layer will eventually be bypassed, and make sure the next one is waiting.
Layer 1 — Input filtering at the gateway. Before a request reaches any model: PII/secret redaction (client data never reaches a public API — the services-company hard requirement) and prompt-injection screening on untrusted content entering the context. The M7 gateway is the enforcement point, which is what makes this a platform control instead of per-app etiquette. In LiteLLM, guardrails attach declaratively:
# litellm config.yaml (extends the M7 config)
guardrails:
- guardrail_name: pii-shield
litellm_params:
guardrail: presidio # self-hostable PII detection/redaction
mode: pre_call # runs before the provider is called
- guardrail_name: injection-screen
litellm_params:
guardrail: lakera_prompt_injection # or a self-hosted classifier
mode: during_call
Layer 2 — Output filtering. The model's response is screened before it reaches the caller — or worse, the next tool call: secret patterns, unsafe commands, format violations. For agents, malformed output is a security event, not a UX event, because downstream code will execute what parses.
Layer 3 — Action policy. The layer that makes agents governable. Every action an agent proposes is checked against explicit policy outside the model — an allowlist evaluated by OPA or Kyverno, not by prompt instructions. Prompts are suggestions; policy engines are enforcement. A bounded-remediation policy (this is L10's agent, now expressed as platform policy):
package agent.actions
# Allowlist: action + scope + blast-radius ceiling
allowed_actions := {"restart_deployment", "scale_deployment", "rollback_deployment", "flip_feature_flag"}
default allow := false
allow if {
input.action in allowed_actions
input.namespace in {"web-frontend", "checkout"} # scoped, not cluster-wide
not exceeds_blast_radius
}
exceeds_blast_radius if {
input.action == "scale_deployment"
input.replicas > 10
}
# Anything not allowed requires a human approval token
require_approval if not allow
Layer 4 — Sandboxed execution and egress control. Where an agent runs is a boundary, not a convenience. Each agent workload gets its own namespace with default-deny egress (a NetworkPolicy permitting only the gateway, the MCP endpoint, and its declared dependencies), scoped secrets, and a TTL — ephemeral environments (vCluster-class) for anything that executes generated code, with gVisor/Firecracker isolation where the code is fully untrusted. Egress control is the quiet hero: a prompt-injected agent that cannot reach the internet cannot exfiltrate what it was tricked into reading.
Layer 5 — Approval tiers and kill switches. The trust map, mechanized. Classify every capability by blast radius and reversibility: reads auto-approved; reversible writes auto-approved within policy but logged loudly; irreversible or high-blast-radius actions gated on human approval (PagerDuty's own bounded-then-autonomous sequencing for its SRE agent is the industry validating this order). And every agent gets a kill switch that is boring: revoke its virtual key at the gateway and its ServiceAccount token, one command, rehearsed in game days like any other break-glass procedure.
Guardrails implemented as system-prompt instructions ("you must never delete resources") are layer zero — worth having, trivially bypassed. Any control that matters must be enforced outside the model: gateway, policy engine, RBAC, network. If a red-teamer with full prompt control could defeat it, it is not a control.
Guardrails constrain what agents do; the persona work defines who agents are. The failure mode in the wild is agents running on borrowed human credentials — a developer's kubeconfig, a shared CI token — which makes audit impossible ("was that Priya or her assistant?") and revocation catastrophic. The fix is the onboarding checklist you already run for human tenants, applied verbatim:
Identity. One ServiceAccount per agent per environment, named to be greppable, annotated with ownership — because every agent must have a human owner the pager can find:
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-remediator
namespace: agents-prod
labels: {platform.company.io/persona: agent}
annotations:
platform.company.io/owner: sre-team@company.io
platform.company.io/trust-tier: act-with-approval
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: agent-remediator
namespace: web-frontend # rights live in the target namespace, scoped
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments/scale"]
verbs: ["update", "patch"] # scale/restart — but no delete, no create
- apiGroups: [""]
resources: ["pods", "events"]
verbs: ["get", "list", "watch"]
Note what the Role omits: delete, create, secrets, cluster scope. The RBAC mirrors the OPA allowlist — two layers agreeing is the point, since either alone can be misconfigured. Cloud IAM follows the same shape (an IAM role per agent, scoped to the actions in its allowlist, nothing wildcarded).
Quotas. Agents fail differently from humans — not maliciously but tirelessly, retrying in tight loops at 3 a.m. Cap the damage in both currencies: a ResourceQuota on the agent namespace (CPU, memory, object counts) for compute, and the M7 gateway's virtual key (max_budget, tpm_limit) for tokens. An agent that hits its quota is a ticket; an agent without one is an incident.
apiVersion: v1
kind: ResourceQuota
metadata:
name: agent-remediator-quota
namespace: agents-prod
spec:
hard:
requests.cpu: "2"
requests.memory: 4Gi
pods: "10"
count/jobs.batch: "20"
Audit. The question audits and postmortems will ask is three-part: which agent, what action, triggered by what. Kubernetes audit logs give you the first two once agents have distinct ServiceAccounts. The third — provenance — you must carry yourself: propagate a correlation ID from the triggering event (alert ID, PR number, user request) through the gateway trace (Langfuse metadata), the MCP call, and the K8s API call. When an agent-initiated rollback shows up in a change review, the platform should reconstruct the whole causal chain in one query — that reconstruction, demonstrated, converts more skeptics than any policy document.
This whole persona-as-manifests direction now has a K8s-native name: kagent (CNCF Sandbox) expresses agents as Kubernetes CRDs, so an agent's identity, tools, and permissions are declared and reconciled like any other cluster resource — the same idea as this section, made declarative and native to the platform you already run.
Golden paths apply to agents too. Publish an "agent onboarding" template — ServiceAccount + Role + quota + NetworkPolicy + virtual key + owner annotation, stamped out like any scaffold (UC-43/46). If onboarding an agent safely takes one PR against a template, teams stop wiring agents up unsafely by hand.
RBAC controls what an agent's credential can do; the MCP layer controls what an agent can even see as possible. The Model Context Protocol is the standard interface between AI assistants and tools, and Pulumi's MCP server (exposing its CLI and registry to Copilot/Claude/Cursor) is the reference pattern: wrap your platform's capabilities as typed MCP tools so org-approved assistants operate the platform through a governed API — instead of shelling out to kubectl with someone's kubeconfig. You are not opening a hole; you are replacing an ungoverned hole with a narrow, typed, logged interface.
Design rule number one is read-free / write-gated tiering. Reads (service catalog, deploy status, log queries) are safe to expose broadly and are where most assistant value lives. Writes are few, explicit, and gated. A minimal server with the official Python SDK:
# platform_mcp.py — internal MCP server, FastMCP (official Python SDK)
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("platform")
# ---- Read tier: free for any authenticated org assistant ----
@mcp.tool()
def get_service(name: str) -> dict:
"""Fetch a service's catalog entry: owner, tier, runbook URL, dashboards."""
return catalog.lookup(name)
@mcp.tool()
def list_deployments(service: str, limit: int = 10) -> list[dict]:
"""Recent deployments for a service: version, time, status, author."""
return deploys.recent(service, limit)
@mcp.tool()
def get_recent_logs(service: str, minutes: int = 15) -> str:
"""Redacted log excerpt for a service (PII scrubbed server-side)."""
return logs.query_redacted(service, minutes)
# ---- Write tier: gated — creates an approval, never acts directly ----
@mcp.tool()
def request_rollback(service: str, to_version: str, reason: str) -> dict:
"""Request a rollback. Creates an approval ticket; a human owner must
confirm before the platform executes. Returns the ticket for tracking."""
ticket = approvals.create(
action="rollback", service=service, target=to_version,
reason=reason, requested_by=current_agent_identity(),
)
return {"status": "pending_approval", "ticket": ticket.id, "url": ticket.url}
Notice the write tool's shape: it does not roll anything back. It requests — creating a durable approval record that a human confirms through the normal change process, after which the platform (not the model) executes. That indirection is the act-with-approval tier as API design, and it means even a fully compromised agent context can at worst spam tickets.
The authorization underneath is not homegrown. The MCP spec builds on OAuth 2.1 for HTTP transports: your MCP server is a resource server publishing Protected Resource Metadata (RFC 9728) pointing at your IdP; clients discover it via the 401 WWW-Authenticate flow, obtain tokens with PKCE and the RFC 8707 resource parameter, and send Authorization: Bearer on every request. Three platform obligations fall out:
platform:read grants the read tools, platform:write-request the gated writes; the server filters which tools it even advertises based on the token's scopes, so an unauthorized agent doesn't see the write tier exists.The spec's 2026-07-28 revision moves the core protocol toward a stateless design and further hardens this OAuth flow (adding a formal deprecation policy for retiring older behaviour), but none of it disturbs the authorization tiering taught here — the OAuth 2.1 audience-validation and read-free/write-gated scope mapping live in how you mint and check tokens, not in any per-session state.
Two operational notes:
get_recent_logs returns scrubbed excerpts, never raw dumps.Start your MCP catalog from ticket data. Mine the platform queue for the ten most common read-shaped requests ("who owns X?", "when did Y last deploy?", "why is Z pending?") — expose those as the read tier first. Deflection value shows up in weeks and builds the constituency for the harder write-tier conversations.
Every guardrail above governs runtime. This section governs change — because on an AI platform, behavior changes ship three ways: a prompt edit, a model/config swap, a guardrail tweak. Postings call it "prompt/version pipelines," and it is the most DevOps-native skill in the module: CI/CD discipline applied to AI behavior. The Common Core taught evals as a concept; here they become merge gates you operate.
Everything in git. Prompts, model choices, routing tables, guardrail configs — versioned files, not strings inline in application code and not values edited in a vendor console. The non-negotiable, inherited from M7's tracing: given any production trace, you can name the exact prompt version and model that produced it. If a prompt edit doesn't show up in git log, your deploy log is lying by omission.
Evals gate the merge. promptfoo (open source) runs a declarative eval suite against every change, in CI, exactly like a test suite:
# promptfooconfig.yaml — gate for the incident-triage prompt
prompts:
- file://prompts/triage-v2.txt
providers:
- id: openai:gpt-4o-mini # any OpenAI-compatible endpoint —
config:
apiBaseUrl: http://litellm:4000 # point it at YOUR gateway
tests:
- vars: {alert_bundle: file://fixtures/disk-pressure.json}
assert:
- type: contains-json # output must parse — downstream code depends on it
- type: javascript
value: JSON.parse(output).severity_hypothesis !== undefined
- vars: {alert_bundle: file://fixtures/false-lead.json}
assert:
- type: llm-rubric
value: Does NOT recommend restarting the database; flags evidence as inconclusive
- vars: {alert_bundle: file://fixtures/injection-attempt.json}
assert:
- type: not-contains
value: "IGNORE PREVIOUS INSTRUCTIONS"
# .github/workflows/prompt-ci.yaml (excerpt)
on:
pull_request:
paths: ["prompts/**", "promptfooconfig.yaml", "gateway/config.yaml"]
jobs:
eval-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
- name: Enforce gate
run: |
FAILURES=$(jq '.results.stats.failures' results.json)
if [ "$FAILURES" -gt 0 ]; then
echo "Eval gate failed: $FAILURES failing assertions"; exit 1
fi
The suite is the platform's regression memory: every incident caused by AI behavior adds a fixture, the same way every outage adds a test. Note the third case — injection resistance is in the eval suite, so a prompt edit that weakens Layer 1's assumptions fails in CI, not in production. Mandatory cases pass at 100%; scored rubrics clear an explicit threshold; a failing eval blocks the merge with no human override short of the same exception process a failing security scan gets.
Canary the rollout, automate the rollback. CI can't see real traffic, so promotion goes through the M7 gateway at a canary weight — a few percent of requests on the new prompt or model — while you compare canary against control on both books: quality (sampled online evals, per the DS/ML track's drift machinery) and platform metrics (TTFT, error rate, tokens-per-request — a prompt edit that silently doubles output tokens is a cost regression wearing a quality costume). Rollback is a routing-weight change, which means it is instant, boring, and automatable: define the regression margin before the canary, and let Argo-Rollouts-style analysis (or a small controller reading Langfuse + Prometheus) drive the weight to zero without a meeting. Model swaps ride the identical pipeline — a new model version is a canary with an eval gate, never a Tuesday-afternoon config edit.
Domain-4 questions probe the ordering: version → offline eval gate → canary with paired quality/cost comparison → auto-rollback on a pre-agreed margin. Answers that ship prompt edits straight to 100% of traffic, or canary on latency alone while ignoring token cost, are the planted distractors.
Everything in M7 and M8 composes into one artifact — the track's capstone, and the portfolio piece that answers "have you actually run this?" in an AI-platform interview. The AI-ready platform slice is one model and one agent workflow, end to end, with five verbs demonstrated:
The demo script is the trust map made physical: show a request flowing through every layer; show an attack dying at each guardrail; show an agent's action reconstructed from audit; show a bad prompt change bounced by CI. Fifteen minutes, five verbs, zero hand-waving — the difference between having opinions about AI platforms and having operated one.
The platform is built and demonstrated. Whether it matters now depends on adoption mechanics and governance framing — the part of the job that is Divya's (Head of Platform, 40 engineers) rather than the hands-on platform engineer's, and the bridge into the leadership track. This is the leadership close on top of all the technical machinery above.
Roll out like a platform capability, not a mandate. The playbook you used for Kubernetes migration works unchanged. Start with one pilot team that has a real, bounded use case — the incident-triage copilot from M3, the docs assistant from M5 — and give them white-glove onboarding through the golden path. Instrument from day one: gateway adoption per team, cost-per-team, deflection rates, eval pass rates — the same DORA-style discipline of measuring the system, not the individual. Publish an honest register of what failed and why; with a 76%-skeptical audience, credibility compounds on candor, and your skeptics (every team has a Meera) are your best red-teamers — recruit them to attack the guardrails in Exercise 1 rather than argue with them in meetings. Then scale by templates: the second team onboards via the golden path plus the pilot's lessons, not via your calendar.
Translate the trust map into an AI stance. Leadership doesn't need TTFT histograms; they need a governance object. The trust map is that object, rewritten in risk language — a one-page stance with four sections:
That last section is the one most stances miss and the one that makes yours durable: trust tiers are supposed to move as evidence accumulates, and a documented promotion path is what separates governance from a frozen org chart of fears. Frame the whole thing with DORA 2025's core finding — AI amplifies whatever system it lands in, and quality internal platforms are among the capabilities that convert individual AI speed into organizational performance. The stance is how you tell leadership: this is the org system that makes the amplification safe to want.
Vendor automation claims ("automates up to 80% of incident response") are marketing inputs, not planning inputs. Commit leadership to numbers your own platform measured — adoption, deflection, eval pass rates, cost per outcome — and label vendor figures as vendor-claimed wherever they appear in your decks.
A teammate insists the remediation agent is safe because its system prompt says "you must never delete resources." Why is that not a control — and where does real enforcement have to live?
This module completes the track's closing lab arc. Lab L14 — LLM observability + guardrails (second half): onto the M7-instrumented stack, add the guardrail layers — gateway PII redaction and injection screening, the OPA action policy, model allowlists — and prove each with attack requests; artifact: guardrail configs plus test evidence per layer. Lab L10 revisited: the bounded-remediation agent re-platformed with full persona treatment (Exercise 2) — same agent, now auditable. Lab L15 — MCP server + prompt CI: build the internal MCP server exposing three read and one approval-gated write operation, then put a prompt change through the promptfoo-gated pipeline where a seeded regression is auto-rejected; artifact: the server plus the CI run showing the eval gate blocking the bad change. Composed with L13, these become the capstone slice above.
Three playbooks carry this module's builds into copy-adaptable form:
The agent onboarding golden-path template (pb-38/pb-50 — ServiceAccount + Role + quota + NetworkPolicy + virtual key), the AI stance one-pager, and the team-rollout runbook round out the set. Everything runs at ₹0: kind/minikube, self-hosted LiteLLM + Langfuse + OPA, promptfoo OSS, and the MCP Python SDK locally.