Lab 02 · K8s Manifests That Survive Validation

The starter set below reproduces four real AI failure modes; you will catch all four with tooling, fix them, then prove your improved prompt stops producing them.

Time & tier

~75 minutes hands-on · Core tier (maps to module M2).

Setup

Install two static binaries (no cluster needed): brew install kubeconform kube-score (or GitHub releases; Linux/Windows binaries exist). $0.

Create manifests/ with these four files — a plausible AI draft for an orders-api service containing four seeded defects:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  labels:
    app: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: ghcr.io/example/orders-api:1.4.2
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: orders-api
spec:
  selector:
    app: orders-api
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-service

Guided steps

1. 1Schema gate. Run kubeconform -strict -summary manifests/. One resource fails: the HPA rejects targetCPUUtilizationPercentage — that field belongs to autoscaling/v1, but the manifest declares autoscaling/v2. This API-version blending is the single most common K8s hallucination: models interpolate between schema generations they saw in training. Fix by asking your assistant for "the autoscaling/v2 metrics block equivalent of 70% CPU", then verify the shape against the kubeconform-fetched schema by re-running until Valid: 4.
2. 2Hygiene gate. Run kube-score score manifests/*.yaml. Read every CRITICAL. Three are seeded:
   • •
     Container Resources — requests exist but limits don't; memory is unbounded.
   • •
     Deployment has PodDisruptionBudget — the PDB exists but its selector says orders-service, so it matches nothing. Silent failure: everything applies cleanly and node drains evict all three replicas at once.
   • •
     Deployment targeted by HPA has replicas configured — replicas: 3 fights the HPA: every kubectl apply snaps the fleet back to 3 mid-scale-event.
3. 3Fix with the assistant, verify with the tool. Paste each finding to your assistant and ask for a minimal patch plus a one-line explanation of the production consequence. Apply, re-run both tools. Target: kubeconform -strict fully valid, and the three seeded CRITICALs gone.
4. 4Triage the rest. kube-score still flags NetworkPolicy, securityContext, imagePullPolicy, ephemeral-storage, anti-affinity. Decide per-item: fix (securityContext is cheap and right) or suppress with an explicit flag such as --ignore-test pod-networkpolicy — the flag is documentation of a decision, so record why in a comment in your Makefile.
5. 5Wire the loop. Create a pre-commit hook (or make verify) that runs kubeconform -strict then kube-score score --exit-one-on-warning with your agreed ignore list. Commit-blocking, not advisory.
6. 6Close the loop at the prompt. Now regenerate the whole set from scratch with an upgraded prompt that encodes what you learned, e.g.: "Generate Deployment/Service/HPA/PDB for orders-api. Constraints: autoscaling/v2 metrics syntax only; no replicas field on HPA-targeted Deployments; every container has requests and limits; PDB selector must equal the Deployment's matchLabels verbatim; must pass kubeconform -strict." Run the gates on the fresh output. Save before/after validation reports.

Verification

• •
  kubeconform -strict -summary manifests/ → Valid: 4, Invalid: 0.
• •
  kube-score score shows none of the three seeded CRITICALs; remaining ignores are listed in your Makefile with reasons.
• •
  git commit with a re-broken PDB selector is rejected by your hook.
• •
  Your step-6 regenerated set passes both gates on first run (or you can show the prompt iteration that got it there).

Completion artifact

The manifests/ repo with fixed YAML, the verify hook/Makefile, and reports/before.txt + reports/after.txt validator output. Self-grading rubric (pass = 7/10):

• Defect capture (0–3): all four seeded defects found via tooling (not by reading this doc) = 3.
• Consequence literacy (0–2): for each defect you wrote the production symptom it would cause = 2.
• Gate automation (0–2): commit-blocking loop works, ignore list documented = 2.
• Prompt improvement (0–2): regenerated output measurably cleaner; both reports kept = 2.
• Minimality (0–1): fixes are surgical diffs, not wholesale regenerated files pasted over history = 1.

Stuck?

• •
  kubeconform needs schema downloads on first run; afterwards add -cache ~/.kubeconform for offline use.
• •
  kube-score exits non-zero by design when criticals exist — that is the gate working, not an install problem.
• •
  If the HPA fix keeps failing strict validation, you are likely nesting target wrong: metrics[].resource.target.type: Utilization with averageUtilization: 70.
• •
  Assistant keeps re-adding replicas? Say "omit the replicas field entirely; the HPA owns scale" — models default to copying the most common Deployment shape.

Client-safe variant

Both validators are offline OSS and never see an LLM — run them on client manifests freely. For generation on restricted engagements, use the client's approved assistant or a local model (Ollama + a coder model handles K8s YAML well), and strip client namespaces/registry hostnames from prompts. Deliverable to the client: the same before/after reports — they double as evidence that AI-assisted work on their estate passed deterministic validation.

Answer key (don't read before attempting)

1. 1hpa.yaml:targetCPUUtilizationPercentage is autoscaling/v1 syntax inside an autoscaling/v2 object — caught by kubeconform -strict (additional property not allowed). Fix: metrics block with type: Resource, target.type: Utilization, averageUtilization: 70.
2. 2pdb.yaml: selector app: orders-service ≠ Deployment label app: orders-api — caught by kube-score "Deployment has PodDisruptionBudget". Fix: match labels verbatim.
3. 3deployment.yaml: requests without limits — kube-score "Container Resources". Fix: add limits (e.g., cpu 500m / memory 256Mi).
4. 4deployment.yaml + hpa.yaml:replicas: 3 on an HPA-targeted Deployment — kube-score CRITICAL; every apply resets live scale. Fix: delete the field.