Lab 07 · K8sGPT and HolmesGPT Cluster Debugging

The deliverable is a comparison matrix: per failure class, does each tool find the cause, half-find it, or miss it — and would you let it near your production cluster.

Time & tier

~2 hours hands-on · Advanced tier (maps to module M3). $0 with a local-model or free-tier LLM backend; K8sGPT's rule-based mode needs no LLM at all.

Setup

Requirements: Docker, kind, kubectl, k8sgpt (brew or GitHub releases), HolmesGPT (Homebrew tap or pipx per the repo README; the CLI is holmes). Create the cluster: kind create cluster --name lab07.

Save the five break manifests. Scenarios 1–4 go in namespace lab07; scenario 5 gets its own namespace — a requests.cpu quota would otherwise reject every request-less pod in the namespace and contaminate the other scenarios.

# break-1-crashloop.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: worker, namespace: lab07 }
spec:
  replicas: 1
  selector: { matchLabels: { app: worker } }
  template:
    metadata: { labels: { app: worker } }
    spec:
      containers:
        - name: worker
          image: busybox:1.36
          command: ["sh", "-c", "echo starting worker; sleep 2; exit 1"]

# break-2-oom.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: cache, namespace: lab07 }
spec:
  replicas: 1
  selector: { matchLabels: { app: cache } }
  template:
    metadata: { labels: { app: cache } }
    spec:
      containers:
        - name: cache
          image: busybox:1.36
          command: ["sh", "-c", "tail /dev/zero"]
          resources:
            requests: { memory: 16Mi, cpu: 50m }
            limits: { memory: 32Mi, cpu: 100m }

# break-3-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: web, namespace: lab07 }
spec:
  replicas: 1
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports: [{ containerPort: 80 }]
          readinessProbe:
            httpGet: { path: /, port: 8081 }
            periodSeconds: 5

# break-4-imagepull.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: gateway, namespace: lab07 }
spec:
  replicas: 1
  selector: { matchLabels: { app: gateway } }
  template:
    metadata: { labels: { app: gateway } }
    spec:
      containers:
        - name: gateway
          image: nginx:1.99.9-typo

# break-5-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata: { name: tight, namespace: lab07-q }
spec:
  hard: { requests.cpu: 500m }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: api, namespace: lab07-q }
spec:
  replicas: 2
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: nginx:1.27
          resources:
            requests: { cpu: "1", memory: 64Mi }
            limits: { cpu: "1", memory: 128Mi }

The five break commands:

kubectl create ns lab07 && kubectl create ns lab07-q
kubectl apply -f break-1-crashloop.yaml   # 1: CrashLoopBackOff
kubectl apply -f break-2-oom.yaml         # 2: OOMKilled
kubectl apply -f break-3-probe.yaml       # 3: never Ready
kubectl apply -f break-4-imagepull.yaml   # 4: ImagePullBackOff
kubectl apply -f break-5-quota.yaml       # 5: pods never created

Guided steps

1. 1Establish ground truth first, tools second. For each scenario record in your matrix the kubectl evidence: (1) kubectl -n lab07 get pods + kubectl -n lab07 logs deploy/worker --previous; (2) kubectl -n lab07 describe pod -l app=cache — look for Last State: Terminated, Reason: OOMKilled; (3) describe pod -l app=web — readiness probe failing, connection refused on 8081, pod Running but 0/1 Ready; (4) describe pod -l app=gateway — ErrImagePull, manifest not found; (5) the trap: kubectl -n lab07-q get pods shows nothing at all — the evidence lives in kubectl -n lab07-q describe rs events: exceeded quota: tight, requested: requests.cpu=1, limited: requests.cpu=500m.
2. 2K8sGPT, rule-based ($0, no LLM). Run k8sgpt analyze --namespace lab07 and k8sgpt analyze --namespace lab07-q. Record per scenario what its analyzers surface with no AI attached.
3. 3K8sGPT, explained. Configure a backend — k8sgpt auth add openai (any OpenAI-compatible key), or for the free path a local model via the LocalAI/Ollama backend with your local base URL — then k8sgpt analyze --namespace lab07 --explain. Grade each explanation: cause correct? fix correct? anything invented?
4. 4HolmesGPT. With an LLM key exported, ask targeted questions: holmes ask "why is the worker pod in namespace lab07 crash looping?", then one per scenario, ending with the hard one: holmes ask "why does the api deployment in namespace lab07-q have zero pods?". HolmesGPT is agentic — it chooses which kubectl/log queries to run. Note which evidence it gathered versus what you gathered in step 1.
5. 5Score the matrix. Columns: scenario · kubectl ground truth · K8sGPT (rules) · K8sGPT (explain) · HolmesGPT · verdict per tool (found / partial / missed / hallucinated). Pay attention to scenario 5: tools that only inspect pods miss failures where no pod exists — the absence of an object is the hardest signal for both rule engines and agents.
6. 6Write the trust conclusion: three sentences per tool — what class of problem you would trust it to triage unattended, what always needs a human, and where it slotted into your workflow (before kubectl? after? never?).
7. 7Clean up.kind delete cluster --name lab07.

Verification

• •
  Matrix has all 5 scenarios × 3 tool columns filled, each with a verdict.
• •
  Ground-truth column quotes actual cluster output (a describe line or event), not the manifest's intent.
• •
  Scenario 5's evidence cites the ReplicaSet event, proving you found where quota rejections actually surface.
• •
  At least one recorded disagreement between a tool's explanation and kubectl evidence (if all three columns were perfect everywhere, re-run scenario 5 and look harder).

Completion artifact

comparison-matrix.md (the scored matrix + per-tool trust conclusions + cluster-output excerpts). Self-grading rubric (pass = 7/10):

• Ground truth quality (0–3): correct discriminating evidence per scenario, especially RS events for quota = 3.
• Fair tool grading (0–3): verdicts tied to quoted tool output, hallucinations quoted verbatim = 3.
• Coverage (0–2): all 15 cells filled from actual runs, not assumption = 2.
• Trust conclusions (0–2): per-tool placement decision an on-call peer could act on = 2.

Stuck?

• •
  kind needs Docker running and pulls images on first apply — allow a minute before judging scenarios.
• •
  Scenario 3 shows pod Running and you conclude "no failure": check the READY column (0/1) and Service endpoint absence — that is the failure.
• •
  K8sGPT explain errors about backends: k8sgpt auth list shows what's configured; the rule-based run (step 2) needs nothing.
• •
  HolmesGPT install friction: fall back to running both K8sGPT modes and one manual chat-model session where you paste describe output — the comparison matrix still works with that as column three.

Client-safe variant

kind on your laptop is already client-safe — it holds no client data. The decision point is the LLM backend: on client engagements, point K8sGPT/HolmesGPT at the client-approved tenancy or a local Ollama model, never a personal key, because describe output can leak client naming conventions and internal registries. A services-team pattern worth stealing: rehearse these five failure classes on kind exactly like this lab, so the tooling is proven before anyone asks to run it inside a client cluster.

Answer key (don't read before attempting)

1. 1worker: container exits 1 by design → CrashLoopBackOff; logs show "starting worker" then exit.
2. 2cache:tail /dev/zero allocates unboundedly into a 32Mi limit → OOMKilled (Last State reason), then restart loop. The right fix is "this workload's memory is wrong for its limit," not "raise the limit" reflexively.
3. 3web: readiness probe targets 8081; nginx listens on 80 → probe connection refused, pod never Ready, would receive no Service traffic. Not a crash — availability failure without an error log.
4. 4gateway: tag 1.99.9-typo doesn't exist → ErrImagePull/ImagePullBackOff.
5. 5api: ResourceQuota admits 500m CPU of requests; each replica requests 1 CPU → ReplicaSet's pod creations are rejected at admission. No pod exists, so pod-centric analysis finds nothing; the exceeded quota events hang on the ReplicaSet. This is the scenario that most cleanly separates tools (and engineers) that reason about desired-vs-actual state from those that only inspect existing objects.