Natural-Language to PromQL/LogQL with Sanity Checks

Tier: Foundation · Hands-on time: ~35 minutes

The failure mode is equally mature: a syntactically valid query that measures something subtly different from what you asked — wrong rate window, missing label filter, averaged percentiles — and returns a confident-looking graph. This playbook is the generate-then-interrogate loop that makes NL-to-query safe enough to base decisions on.

When to use

• •
  You need a PromQL/LogQL/vendor query outside your fluent zone: histogram quantiles, rate vs increase, joins on group_left, LogQL aggregations.
• •
  Exploring an unfamiliar system's metrics during onboarding or an incident (paired with pb-14/pb-15).
• •
  Translating a colleague's plain-English question ("are we slower for big tenants?") into a defensible query.

Skip the assistant for queries you write weekly — and never paste an AI query straight into an alert rule or SLO definition without the full check sequence below (for alerts, see pb-25).

Prerequisites

• •
  A Prometheus/Loki stack with data (the lab path: docker compose with Prometheus + node-exporter + Loki + Grafana, or any Grafana Cloud free-tier stack), or your org's existing observability tenant.
• •
  A query assistant: Grafana Assistant, Honeycomb Query Assistant, Datadog Bits — or any LLM chat, which handles PromQL well when given metric metadata.
• •
  Ability to list your actual metric names and labels (/api/v1/label/__name__/values, Grafana Explore autocomplete) — the model must work against real names, not guessed ones.

The workflow

1. 1.

   Give the model your real vocabulary. Paste the relevant metric names, their types (counter/gauge/histogram), and label keys:

   Copy

   Metrics available:
   - http_server_request_duration_seconds (histogram; labels: service, route,
     status_code, le)
   - kafka_consumergroup_lag (gauge; labels: group, topic, partition)
   Write a PromQL query for: p95 request latency for service "checkout",
   5-minute window, excluding health-check routes. Explain each function
   choice in one line, and state what the query does NOT include.
   

   Without this, models hallucinate plausible metric names (http_request_latency_p95) that silently return empty results or, worse, match a different metric that exists.

2. 2.

   Read the query before running it. Check the four classic silent errors: (a) rate() window shorter than scrape interval × 4; (b) histogram_quantile applied without sum by (le) or averaged across instances; (c) label filters using = where you meant =~, or missing entirely (querying ALL services); (d) counters used without rate/increase. The model's own one-line explanations make these fast to spot — a wrong explanation flags a wrong query.

3. 3.

   Run the sanity triplet. Before trusting output, run three cheap checks:

   • •
     Known-answer check: query something you can verify independently (request count for the last hour vs. the load balancer's number; log line count vs. wc -l on a known window).
   • •
     Perturbation check: change one thing with a known effect — widen the window, drop a filter — and confirm the result moves in the expected direction and magnitude.
   • •
     Zero check: point the query at a service/time range where the true answer is zero or empty and confirm it says so (catches wrong-metric matches that always return data).
4. 4.

   Interrogate the semantics. Ask the assistant: What question does this query actually answer, precisely? List 3 ways its answer could differ from "<my original English question>". This surfaces the gap between asked and measured — averaging across pods vs. per-pod p95 is the canonical example — while the query is still in front of you.

5. 5.

   Save the validated pair. Store English question + final query + the sanity-check evidence in your team's query library (a repo file or Grafana snapshot). The library compounds: it becomes few-shot context for future generations and the review baseline for pb-24 dashboards and pb-25 alerts.

Tool variants

• Grafana Assistant: generates and explains against your actual datasources inside Explore — the label-vocabulary problem largely disappears; the semantic checks do not.
• Honeycomb Query Assistant: NL to Honeycomb queries over high-cardinality events; its visual query builder makes the read-before-run step especially easy — use it.
• Datadog Bits / Elastic AI Assistant: same pattern for their DSLs; Elastic's assistant also explains existing queries, useful for auditing dashboards you inherited.
• Plain LLM + metric metadata (lab default): works with any model including local ones; PromQL is extremely well represented in training data, so quality is high when you supply real metric names.

Verification

• •
  Every generated query passed the sanity triplet (known-answer, perturbation, zero check) before its output was shared or acted on — keep the evidence one line each in the query library entry.
• •
  The model's per-function explanation was read and matches what the functions actually do — treat the explanation as a citation and check the one you're least sure of against Prometheus docs; a fabricated function behavior invalidates the query.
• •
  Metric and label names in the query verified to exist via autocomplete/API, not assumed from the model's output.
• •
  For any query informing a decision (capacity, rollback, SLO), a second engineer re-derived the English meaning from the query text alone and it matched your original question.

Pitfalls

• Averaged percentiles.avg(histogram_quantile(...)) over instances is statistically meaningless but renders a beautiful line. The known-answer check against a single instance catches it.
• Empty-result ambiguity. Empty can mean "no such data" (typo'd metric) or "genuinely zero." The zero check plus name verification disambiguates.
• Assistant overconfidence on vendor DSL edge cases. LogQL/vendor DSLs are sparser in training data than PromQL; expect more invented functions there and lean harder on running the query against known data.
• English drift across refinements. After five conversational refinements, the query answers the fifth question, not the first. Re-state the full question in the final prompt and re-run the semantics interrogation.

Restricted-environment variant

• •
  Metric names and label values leak architecture and customer information (tenant="acme-corp"). Under client MSAs, use the client's in-tenant assistant (their Grafana/Datadog seat) or a local model via Ollama — PromQL generation is comfortably within 8B-class capability when metric metadata is supplied.
• •
  Strip label values (keep keys) from the vocabulary you paste to hosted models; values are where customer identifiers live.
• •
  The sanity triplet requires no data egress at all — it runs entirely in the client's stack, so the verification half of this playbook is always client-safe.

Success criteria

• •
  You produce correct queries for 10 test questions (mix of PromQL and LogQL) with every query passing the sanity triplet — and you caught at least one silently-wrong query in the process (if none, your test questions were too easy to teach you anything).
• •
  Time from English question to validated query under 5 minutes for queries that would take you 20+ unaided.
• •
  Your query library has entries with question + query + check evidence, and a teammate has reused one.
• •
  You can name, from memory, the four classic silent PromQL errors and which sanity check catches each.