Diagnose Kubernetes Failures with K8sGPT and HolmesGPT

Tier: Advanced · Hands-on time: ~60 minutes

• K8sGPTruns rule-based analyzers over cluster state (CrashLoopBackOff, pending pods, OOMKills, misconfigured services) and uses an LLM only to explain findings and suggest fixes.
• HolmesGPT (CNCF Sandbox since Oct 2025) — agentic: given an alert or question, it decides which data to fetch (pod logs, events, Prometheus queries, node state) and iteratively refines a root-cause hypothesis inside your environment.

The habit it builds carries over to production: kubectl remains the source of truth; the AI's output is a hypothesis to check, not a diagnosis to apply.

When to use

• •
  A pod/workload failure where kubectl describe output is voluminous and the cause is non-obvious (image pulls, probes, quotas, affinity, PVC binding).
• •
  Triaging an unfamiliar cluster (client handover, acquired system) where you lack mental cache of its quirks.
• •
  An alert fired and you want an agentic first-pass RCA before a human dives in (HolmesGPT's alert-investigation mode).

Skip for failures where the first kubectl describe pod line already tells you the answer, and never let either tool apply anything — they diagnose; you change.

Prerequisites

• •
  A local cluster you may break: kind create cluster or minikube. Do NOT learn these tools pointed at production or a client cluster.
• •
  K8sGPT CLI (brew install k8sgpt or release binary) and HolmesGPT (pipx install holmesgpt or its Helm chart for the in-cluster variant).
• •
  An LLM backend for both: hosted API key, or Ollama locally (both tools support OpenAI-compatible endpoints — this is the zero-cost/restricted path).
• •
  Five prepared breakages (script them): bad image tag, CPU/memory limits causing OOMKill, failing readiness probe, unschedulable pod (impossible nodeSelector), missing ConfigMap reference.

The workflow

1. 1.

   Break the cluster on purpose. Apply your five broken manifests to distinct namespaces so each failure is isolated and you know ground truth for every one.

2. 2.

   Run K8sGPT for the survey pass.

   Copy

   k8sgpt auth add --backend openai --model <model>   # or ollama/localai backend
   k8sgpt analyze --explain --filter Pod,Service,Deployment
   

   Each finding pairs the analyzer's rule-based detection (trustworthy — it read real cluster state) with an LLM explanation and fix suggestion (unverified — audit it). Note which findings K8sGPT missed entirely: analyzers only catch coded patterns.

3. 3.

   Verify each explanation against kubectl before believing it. For every finding: kubectl describe/kubectl get events --field-selector involvedObject.name=<pod> and confirm the quoted reason exists verbatim (e.g. the OOMKilled state, the exact failed-probe message). The detection layer is reliable; the explanation layer can misattribute cause.

4. 4.

   Escalate one failure to HolmesGPT for agentic RCA.

   Copy

   holmes ask "why is the pod <name> in namespace <ns> not becoming ready?"
   

   Watch its tool-call transcript: which kubectl/log/Prometheus queries it chose. This transcript IS the citation trail — every claim in its conclusion should map to a command it actually ran and output it actually saw.

5. 5.

   Audit the transcript, then apply the fix yourself. Re-run 2–3 of the exact commands from the transcript and compare outputs. Then write the corrected manifest with your own hands (or your code assistant), kubectl apply, and confirm recovery with kubectl get pods -w. Score each tool per breakage: correct cause? correct fix? anything invented?

Tool variants

• K8sGPT operator (in-cluster): continuous analysis publishing Results CRs; pair with its integrations (e.g. Prometheus) for standing hygiene reports. Same audit rule applies to explanations.
• HolmesGPT via Robusta / Helm: wires alert sources (Alertmanager, PagerDuty) so investigations trigger automatically and land in Slack — turning this playbook into the automated first-pass for pb-15's verification discipline.
• kubectl-ai and generic assistants (Claude Code with kubectl access): an agent that runs read-only kubectl loops approximates HolmesGPT; constrain it with a read-only kubeconfig/ServiceAccount so "diagnose" cannot drift into "mutate".

Verification

• •
  Every causal claim in an explanation is backed by a kubectl-observable fact you re-checked: event message, container state reason, quota status — found verbatim, not paraphrased.
• •
  HolmesGPT's transcript audit passes: commands it claims to have run reproduce the outputs its conclusion relies on; any conclusion resting on a query it never ran is discarded.
• •
  Scorecard complete for all five breakages: detection hit-rate, explanation accuracy, fix correctness, hallucination count. You should be able to state each tool's weakest failure class.
• •
  The applied fix came from a human-authored (or human-reviewed) manifest, and recovery was confirmed by cluster state, not by the model saying "this should fix it."

Pitfalls

• Explanations that overfit the common case. An ImagePullBackOff explained as "typo in image tag" when the real cause is a private-registry auth failure — both produce the same analyzer finding. The events, not the LLM, disambiguate.
• Giving the agent a cluster-admin kubeconfig. Scope HolmesGPT/kubectl-ai to read-only RBAC. An agentic tool with write access is one bad hypothesis from being an outage amplifier.
• Token-burning on huge clusters.k8sgpt analyze across hundreds of findings with --explain is slow and costly; filter by namespace/resource and triage the raw findings first.
• Treating a passing demo as production readiness. Your five scripted breakages are easy mode. Production adds noisy neighbors, partial data, and multi-cause failures — keep the human verification gate permanently.

Restricted-environment variant

• •
  Client clusters: run both tools with an Ollama/vLLM backend inside the client boundary (k8sgpt auth add --backend localai --baseurl http://ollama:11434/v1); no pod names, logs, or config leave the environment. HolmesGPT similarly accepts OpenAI-compatible local endpoints via LiteLLM.
• •
  K8sGPT's --anonymize flag masks resource names before they reach a hosted backend — use it when a hosted model is approved but naming is sensitive; verify the masking by reading the request payloads in a proxy first.
• •
  Get written client approval before installing any operator in their cluster; the CLI-from-your-workstation mode with read-only RBAC is the lower-friction ask.

Success criteria

• •
  All five scripted failures diagnosed to true root cause, with your scorecard showing where each tool was wrong and how kubectl caught it.
• •
  You can produce HolmesGPT's transcript for one investigation and point to the exact tool-call output supporting each claim in its conclusion.
• •
  Zero mutations performed by any AI tool — every fix human-applied, every recovery cluster-verified.
• •
  You can articulate, in one sentence each, when you'd reach for K8sGPT (broad hygiene survey) versus HolmesGPT (deep single-alert RCA) on a real cluster.