Kubernetes Manifests and Helm Charts with a Schema-Validation Loop

Tier: Core · Workflow area: Infrastructure as Code

The saving grace: K8s has machine-readable schemas and cheap static analyzers, so a tight generate→validate→score loop catches most of it before anything touches a cluster. This is the K8s edition of the verification-chain discipline.

When to use

• •
  Writing Deployment/Service/HPA/PDB/NetworkPolicy sets for a new service.
• •
  Building or extending Helm charts, values schemas, and templates.
• •
  Upgrading manifests across K8s API versions (the apiVersion graveyard).

Skip AI for copy-tweak jobs where your platform already provides a golden template — use the template; that is what it is for.

Prerequisites

• •
  The loop tools, all OSS: kubeconform (schema validation), kube-score or polaris (best-practice scoring), helm lint + helm template for charts.
• •
  A throwaway cluster (kind/minikube) for the dry-run gate.
• •
  Your cluster's actual K8s version — schema validation against the wrong version is false comfort.

The workflow

1. 1.

   Prompt with the operational envelope, not just the app.

   Copy

   Generate Kubernetes manifests for service "checkout" (container
   registry.internal/checkout:1.8.2, port 8080, /healthz and /readyz
   endpoints).
   
   Requirements:
   - K8s 1.29. Deployment (3 replicas), Service (ClusterIP), HPA
     (CPU 70%, min 3 max 10), PDB (maxUnavailable 1).
   - Resources: requests 200m/256Mi, limits 1000m/512Mi.
   - Liveness AND readiness probes on the endpoints above, sensible
     initialDelay/period for a JVM-free Go service.
   - securityContext: runAsNonRoot, readOnlyRootFilesystem, drop ALL caps.
   - Labels per app.kubernetes.io conventions; selector labels immutable-safe.
   - No default namespace; use "payments". Nothing I did not ask for.
   

2. 2.

   Gate 1 — schema validation against your real version.

   Copy

   kubeconform -kubernetes-version 1.29.0 -strict -summary manifests/
   

   -strict is the point: it fails unknown fields — exactly the silent-typo class (replica:, targetport:) the API server would swallow.

3. 3.

   Gate 2 — best-practice scoring.

   Copy

   kube-score score manifests/*.yaml
   

   Treat CRITICAL findings (missing probes, no resource limits, bad PDB/replica interaction) as build-breaking. Feed findings back to the assistant verbatim; it fixes them well.

4. 4.

   Gate 3 — policy. Your org's Kyverno/OPA policies (pb-07) against the rendered manifests:

   Copy

   conftest test manifests/ --policy k8s-policy/
   

5. 5.

   Gate 4 — server-side dry run and disposable-cluster deploy. Static tools cannot see admission webhooks, quotas, or missing Secrets/ConfigMaps:

   Copy

   kubectl apply --dry-run=server -f manifests/
   kind create cluster && kubectl apply -f manifests/ && kubectl rollout status deploy/checkout -n payments
   

   For Helm: helm lint, then helm template | kubeconform -strict -, then helm install --dry-run — template rendering is where chart bugs live, so validate the rendered output, never just the templates.

6. 6.

   Wire the loop as pre-commit/CI. kubeconform + kube-score + conftest as a pre-commit hook makes the loop free forever; AI-generated or hand-written stops mattering.

Tool variants

• Claude Code / repo-aware agent: reads your existing manifests and Helm conventions, runs the loop itself; keep it away from kubectl apply on real clusters — dry-run and kind only.
• Copilot inline in YAML files: strong for field-level completion; run gates manually — inline suggestions get zero validation.
• Helm-specific: have the AI draft values.schema.json alongside the chart — schema-validated values catch consumer errors and it is tedious authoring AI does well.

Verification

The K8s translation of validate→plan→policy→cost:

• Validate: kubeconform -strict clean against the cluster's actual version.
• Plan:kubectl apply --dry-run=server succeeds; kind-cluster rollout completes and pods pass readiness.
• Policy: kube-score CRITICALs resolved; org Kyverno/Conftest policies green.
• Cost: requests × replicas × HPA-max sanity check — an AI-suggested limits: 4Gi across 10 max replicas is a real node-pool bill; check what the HPA ceiling implies before merge.

Pitfalls

• Silent field typos. The single most common AI-YAML failure; only -strict schema validation catches unknown keys reliably.
• apiVersion time travel. Models suggest deprecated versions (policy/v1beta1 PDBs, autoscaling/v2beta2). Your kubeconform version pin catches this — keep it pinned to reality.
• Probe cargo-culting. Copied initialDelays from JVM examples (60s) on a Go service slows every rollout; probes must match the service's actual startup behavior.
• Helm values drift. Editing templates without updating values.schema.json and docs; validate rendered output for at least two values permutations (default and prod).

Restricted-environment variant

Manifests carry internal registry paths, service names, and namespace topology. For client clusters: the entire validation loop is OSS and runs offline — only the drafting step needs an approved endpoint. Local models (via Ollama) draft standard K8s resources acceptably well since the shapes are heavily represented in training data; keep the same gates and expect one extra fix loop. Redact registry hostnames to placeholders if using public assistants.

Success criteria

• •
  Manifests/chart merged with all four gates green, enforced in pre-commit or CI.
• •
  Rollout on a disposable cluster succeeded before any shared-cluster apply.
• •
  Zero unknown-field or deprecated-apiVersion issues reaching review — the loop caught them.
• •
  Resource envelope (requests/limits × HPA range) reviewed and stated in the PR description.