Diagnose CI/Build Failures from Logs with AI

Tier: Foundation · Workflow area: CI/CD & release engineering

This playbook gets you a ranked hypothesis list in minutes and a habit of confirming the hypothesis before acting on it.

When to use

• •
  A pipeline failure whose cause is not obvious from the last 20 lines.
• •
  Failures in unfamiliar territory: someone else's pipeline, a toolchain you rarely touch, a flaky infra-level error.
• •
  Recurring intermittent failures where you need pattern analysis across multiple runs.

Skip AI when the error is explicit ("secret FOO not set") — read it and fix it.

Prerequisites

• •
  The full raw log of the failing job (not the truncated UI view — download it).
• •
  The diff or commit range since the last green run.
• •
  Any AI assistant; CI-native assistants (GitLab Duo root cause analysis, Copilot in Actions) work too.

The workflow

1. 1.

   Collect evidence, not just the error. Grab: the failing job log, the last green run's timestamp, and git log --oneline <last-green>..HEAD. Cause is usually in the delta.

2. 2.

   Prompt for ranked hypotheses with evidence, not a single answer.

   Copy

   This CI job started failing. Below: the failing log, and the commits since
   the last green run.
   
   Task:
   1. Identify the first line where the run actually goes wrong (not the last
      error — the first).
   2. Give your top 3 causes ranked by likelihood. For each: the exact log
      lines supporting it, and one cheap check I can run to confirm or kill it.
   3. Say which commits could plausibly have caused it, or "none — likely
      environmental."
   Do not propose fixes yet.
   
   --- LOG ---
   <paste>
   --- COMMITS ---
   <paste>
   

3. 3.

   Run the cheapest confirming check first. Good checks: rerun the failing step locally, docker pull the exact image tag, hit the flaky endpoint with curl, rerun the job with debug logging. Confirm before you fix — this is the step that separates diagnosis from guessing.

4. 4.

   Ask for the fix only after confirmation. "Cause #2 confirmed: the base image tag was repointed upstream. Propose the minimal fix." Minimal matters — assistants over-fix, pinning ten things when one was broken.

5. 5.

   Log the failure class. Keep a running note (repo wiki or team channel) of failure → cause → AI verdict. After a month you will know exactly which failure classes the assistant nails (dependency, cache, syntax) and which it fumbles (runner infra, race conditions) — that is your personal trust map.

Tool variants

• CI-native root-cause analysis (GitLab Duo, Copilot for Actions): one click, no copy-paste, sees the full log. Use it first; escalate to a full-context assistant when its answer is shallow.
• Claude Code / agentic CLI: can run the confirming checks itself (gh run view, local rebuild) — fastest loop, watch what commands it wants to run before approving.
• Chat assistant with pasted logs: works everywhere including restricted setups; pre-trim the log to the failing job and strip secrets first.

Verification

Diagnosis has its own gate chain — treat "the AI said X broke it" as a plan, not an apply:

• Validate: the quoted log lines actually exist in the log (search for them — assistants paraphrase).
• Confirm: at least one independent check (local repro, image pull, rerun) supports the chosen hypothesis before any fix is written.
• Policy: the fix touches only what the confirmed cause requires — reject drive-by pins, version bumps, and || true silencers.
• Prove: the pipeline is green after the fix AND you can articulate why it failed; a green rerun without understanding is a flake deferred, not a fix.

Pitfalls

• The confident wrong answer. On ambiguous logs, assistants pick a plausible story and commit to it. The confirming-check step exists precisely for this.
• Truncated log input. UI-copied logs are often clipped; the smoking gun is frequently in setup output. Download the raw log.
• Fixing the symptom. "Increase the timeout" makes deadline errors disappear without touching the slow dependency underneath. Ask "why did this get slow?" before accepting timeout bumps.
• Secrets in pasted logs. CI logs leak tokens, signed URLs, internal hostnames. Scrub before pasting into hosted tools.

Restricted-environment variant

Client CI logs frequently contain client hostnames, account IDs, and tokens that must not enter public AI tools. Pipe the log through a redaction pass first (sed/awk rules for account IDs, URLs, emails — keep a shared scrub script in your team repo), or run the whole workflow against a local model via Ollama; log triage works well on mid-size local models since the task is extraction, not deep reasoning. On air-gapped client estates, the structured prompt in step 2 still pays off with the client's approved internal assistant.

Success criteria

• •
  Median time from red pipeline to confirmed cause drops measurably (target: under 15 minutes for common classes).
• •
  Zero fixes merged on an unconfirmed hypothesis.
• •
  Your failure-class log shows growing precision: you know when to trust the first AI answer and when to dig in.
• •
  Repeat failures of the same class get diagnosed from your log, not from scratch.