Policy-as-Code Authoring with Adversarial Test Generation

Tier: Core · Workflow area: CI/CD & release engineering

Rego in particular is a language most engineers touch twice a year; AI assistance plus a strong test discipline turns it from a specialist skill into a routine one.

When to use

• •
  Encoding a review rule you keep repeating by hand ("no 0.0.0.0/0 ingress", "all buckets encrypted", "prod needs deletion protection") into a CI gate.
• •
  Extending Checkov/OPA coverage to org-specific conventions no off-the-shelf rule knows.
• •
  Building the policy layer that backstops AI-generated IaC (the gate chain in pb-08 needs policies to run).

Skip custom policies when a built-in Checkov/tfsec rule already covers it — search first, write second.

Prerequisites

• •
  OPA + Conftest (or Checkov with custom policy support, or Kyverno for K8s admission).
• •
  Sample inputs: real terraform plan JSON or K8s manifests from your repos, both compliant and violating.
• •
  An AI assistant; Rego benefits from a strong model — it is a niche language with limited training data, so verify harder.

The workflow

1. 1.

   State the policy in English first, precisely. "S3 buckets must be encrypted" is underspecified: SSE-S3 or KMS-only? Do exceptions exist? What about buckets created via modules? Ambiguity in, unenforceable policy out.

2. 2.

   Generate policy plus tests in one prompt — tests are non-negotiable.

   Copy

   Write an OPA/Rego policy for Conftest evaluating terraform plan JSON:
   
   Rule: every aws_s3_bucket must have server-side encryption with a customer
   managed KMS key (not AES256/SSE-S3). Exception: buckets tagged
   "public-assets" may use SSE-S3.
   
   Also produce:
   1. Six test inputs as plan-JSON fragments: KMS-encrypted (pass),
      SSE-S3 (fail), no encryption (fail), tagged exception with SSE-S3
      (pass), tagged exception unencrypted (fail), bucket created inside a
      module (must still be caught).
   2. A conftest exec command to run them.
   Deny messages must name the offending resource address and say how to fix.
   

3. 3.

   Run the tests and watch them fail correctly.

   Copy

   opa test policy/ -v
   conftest test tfplan.json --policy policy/
   

   A policy whose violating cases do not fail is worse than no policy — it produces green checkmarks and false confidence.

4. 4.

   Ask for adversarial bypasses — this is the step most teams skip.

   Copy

   Now act as an engineer trying to sneak a non-compliant bucket past this
   policy. List 5 realistic bypasses (module indirection, count/for_each,
   dynamic values unknown at plan time, aliased providers, resource renamed
   via moved block). For each: a plan-JSON fragment, and whether the current
   policy catches it. Fix the policy for the ones it misses.
   

   Common real gaps: policies that only match top-level resources and miss module-nested ones; values that are known after apply; conditionally-created resources.

5. 5.

   Wire into CI as a distinct, blocking gate — policy gates (unlike AI review, pb-04) are deterministic and have earned blocking status:

   Copy

   - run: terraform plan -out=tfplan && terraform show -json tfplan > tfplan.json
   - run: conftest test tfplan.json --policy policy/ --fail-on-warn
   

6. 6.

   Document each policy in-file: the English rule, the exception process, and an owner. Policies without owners become archaeology.

Tool variants

• OPA/Conftest (shown): most portable — same Rego evaluates Terraform plans, K8s manifests, and Dockerfiles.
• Checkov custom policies (Python or YAML): easier authoring if you already run Checkov; AI writes these reliably since they are plain Python.
• Kyverno: K8s-native admission policies in YAML; pair AI drafting with kyverno test fixtures exactly as above.

Verification

The policy is itself a gate in the validate→plan→policy→cost chain, so verify it like infrastructure:

• Validate:opa test green, including all violating cases failing; opa fmt and conftest verify clean.
• Plan: policy run against plan JSON from at least two real repos — one known-compliant (no findings) and one with a seeded violation (caught, with a readable message naming the resource).
• Policy-on-policy: the adversarial bypass list from step 4 is committed alongside the policy with its resolutions.
• Cost: measure added CI time; policy evaluation should be seconds — if Checkov's full run cost hurts, scope custom policies to changed paths.

Pitfalls

• Hallucinated Rego built-ins and input paths. The classic failure: plausible functions that do not exist, or assuming resource_changes[].change.after structure incorrectly. Only opa test against real plan JSON proves the input shape.
• Testing only the happy path. Six passing tests that never exercise the deny branch validate nothing. Violating fixtures are the point.
• Policies that block on known after apply. Values unresolved at plan time need explicit handling (warn, or check at apply) — otherwise you get flaky gates and angry teams.
• Silent policy drift. Provider upgrades change plan-JSON shapes. Re-run the fixture suite in CI on a schedule, not just on policy edits.

Restricted-environment variant

Plan JSON contains account IDs, ARNs, and CIDRs. For client estates: generate the policy from the English rule plus synthetic fixtures (never real plan output) using any approved assistant, then test against real plan JSON only inside the client environment. Rego and Conftest are OSS and run air-gapped; local models handle Rego drafting less reliably than large hosted ones, so budget an extra iteration loop offline.

Success criteria

• •
  Policy merged with a committed test suite where every violating fixture fails and every compliant fixture passes.
• •
  At least one adversarial bypass found in step 4 and fixed before ship.
• •
  Policy gate blocking in CI with readable deny messages that name resource and fix.
• •
  A previously manual review comment ("please encrypt this bucket") has visibly stopped appearing in PR threads.