Generate Dashboards and SLOs as Code from Service Metadata

Tier: Core · Hands-on time: ~60 minutes

This playbook uses AI to draft both artifacts as code from the service's actual metadata (its OTel semantic conventions, metric inventory, and service spec): dashboard JSON/Jsonnet reviewed like any PR, and SLO definitions in OpenSLO/Sloth format that a human workshop then validates. Two AI drafting passes, two different gates — dashboards are verified against live data; SLOs are verified against user expectations, which no model can know.

When to use

• •
  New service onboarding: nothing exists, and blank-page cost is why observability lags weeks behind launch.
• •
  Standardization sweeps: 40 services with hand-made dashboards and you want them regenerated consistently from metadata.
• •
  SLO adoption pushes, where teams stall on "what should our targets even be?" and need a credible strawman to argue with.

Skip AI for the one bespoke executive dashboard your org already polishes by hand — generation shines on the long tail of per-service operational dashboards, not the flagship.

Prerequisites

• •
  The service's real telemetry vocabulary: exported metric names/types/labels (from /metrics or your Prometheus), OTel semconv compliance status (pb-26 makes this true), plus service tier, dependencies, and traffic pattern.
• •
  Dashboards-as-code tooling: Grafana dashboard JSON in git at minimum; Jsonnet/Grafonnet, Terraform's Grafana provider, or Perses for the mature path.
• •
  SLO tooling: Sloth (generates Prometheus rules from SLO specs) or OpenSLO format; Nobl9 if you're commercial.
• •
  pb-22 completed — every panel and SLI is a query, and query sanity-checking is assumed here.

The workflow

1. 1.

   Feed real metadata, demand structure. Paste the metric inventory and service context:

   Copy

   Service: checkout-api (tier 1, HTTP+gRPC, deps: payments, inventory, redis).
   Metrics: <pasted inventory with types and labels>.
   Generate a Grafana dashboard as JSON with rows: (1) RED overview (rate,
   errors, duration p50/p95/p99), (2) dependency health, (3) saturation
   (runtime + container), (4) rollout markers via deploy annotations.
   Rules: use ONLY metrics from the inventory — if a standard panel's metric
   is missing, emit a placeholder panel titled "MISSING: <what to instrument>"
   instead of inventing a metric name. Template variables for cluster and
   namespace. Every panel gets a description stating what "bad" looks like.
   

   The MISSING-panel rule turns hallucination pressure into an instrumentation gap report — the most useful side effect of this workflow.

2. 2.

   Review as code, then against live data. Diff the JSON in a PR like any change. Then import to a test folder and check every panel against reality: does it render data, do the numbers match a hand-run query (pb-22's known-answer check), do template variables actually filter? Dead panels and averaged-percentile panels are the classic generated defects.

3. 3.

   Draft candidate SLOs from the same context.

   Copy

   From this service context and metric inventory, propose 2-3 SLIs with:
   - SLI definition as a PromQL ratio (good events / valid events), metrics
     from the inventory only.
   - Three candidate targets (conservative/moderate/aggressive) with the
     rationale and the implied monthly error budget in minutes/requests.
   - What user experience each SLI does and does NOT capture.
   - An error-budget policy draft: thresholds at which we page, freeze
     features, or escalate — as a table.
   Output as an OpenSLO/Sloth YAML spec plus a one-page discussion doc.
   

4. 4.

   Workshop the SLOs — the human gate AI cannot replace. Targets are commitments about user happiness and engineering priorities. In a 45-minute session with the service team: check each SLI ratio against known incidents ("would this SLO have caught the March outage?" — replay historically with the actual PromQL), pick targets against real user expectations and current baseline performance (an SLO you're already violating is a project plan, not an SLO), and edit the budget policy until the team would actually honor the freeze clause.

5. 5.

   Ship both through the pipeline. Dashboard JSON/Jsonnet merged to the dashboards repo with provisioning; Sloth spec compiled to Prometheus recording/alerting rules (sloth generate) and merged. Both artifacts now regenerate reviewably when metadata changes — that is the "as code" payoff.

Tool variants

• Grafana Assistant: generates panels/dashboards in-product against live datasources — lowest friction, but capture the result into the git-provisioned repo; a dashboard that lives only in the UI is regression-in-waiting.
• LLM + Grafonnet/Jsonnet: highest leverage for fleets — generate one library function per service archetype, instantiate per service; the review load drops from 40 dashboards to one template.
• Perses / Dashboards-as-code CRDs: same workflow, stricter schemas — schema validation catches malformed generation before review.
• Sloth (lab default) / Nobl9: Sloth's spec-to-rules compilation is the verification-friendly path: the YAML is small enough to review completely, and the generated Prometheus rules are deterministic.

Verification

• •
  Panel-by-panel data check completed: every panel renders live data or is an intentional MISSING placeholder; at least three panels' numbers cross-checked against hand-run queries per pb-22's known-answer discipline.
• •
  No invented metric names anywhere: grep every metric reference in generated JSON/YAML against the real inventory — this is the citation check for telemetry artifacts, and it must be scripted if you're doing fleets (a 10-line script comparing referenced vs. exported names).
• •
  SLO replay test: each adopted SLI, evaluated over the last 90 days of real data, flags the incidents users actually felt and stays quiet otherwise — attach the replay graph to the SLO PR.
• •
  Error-budget policy signed by the team lead who will enforce it; an unsigned policy is documentation, not policy.

Pitfalls

• Cloned-dashboard syndrome, automated. Generating from a generic prompt without the service's inventory reproduces the same dead panels you were escaping. Metadata-in is the whole trick.
• Vanity SLOs. Models propose 99.9% because it's round and their training data likes it. Baseline first: if current performance is 99.5%, adopting 99.9% assigns an unfunded engineering mandate via YAML.
• Averaged percentiles and cross-instance quantile math sneaking into panels — the same pb-22 silent errors, now persisted and multiplied across a fleet.
• SLO sprawl. Three good SLOs beat nine plausible ones; every SLO implies alerting, review load, and a budget meeting. The workshop prunes; the model never will.

Restricted-environment variant

• •
  Metric inventories reveal architecture; keep them off hosted models under strict MSAs by running generation locally (Ollama/vLLM — dashboard JSON and Sloth YAML are well within local-model capability when the inventory is supplied) or redact label values and service names to placeholders, regenerate mapping on import.
• •
  Client-owned Grafana/Nobl9: artifacts belong in the client's repos and their review process; your deliverable is the generated PR plus the verification evidence, gated by their approvers.
• •
  The SLO workshop is client-facing gold in services companies — the AI strawman turns a stalled "define SLAs" workstream into a 45-minute decision meeting; bill the meeting, not the drafting.

Success criteria

• •
  New-service onboarding produces a data-verified dashboard and 2–3 workshopped SLOs in under half a day, versus the multi-week drift it replaces.
• •
  The MISSING-panel report drove at least one real instrumentation fix (pb-26).
• •
  SLO replay graphs attached to every adopted SLO PR; the team can state each SLO's monthly budget in concrete units without looking it up.
• •
  Six weeks later: dashboards still live in git (zero UI-only drift), and at least one error-budget policy clause has been exercised as written.