Every AI-SRE capability in this track — triage, investigation, anomaly detection — is bounded by telemetry quality: an AI investigator cannot cite a span that was never emitted, and anomaly detection over unparsed log soup detects nothing.
Tier: Core · Hands-on time: ~75 minutes
This playbook covers the two AI-assisted workflows that raise that floor, both classic tedious-but-verifiable tasks: (1) generating log parsing rules (grok/VRL/ingest pipelines) from sample logs, proven correct by unit tests; (2) generating OpenTelemetry instrumentation that follows semantic conventions, proven correct in a local collector before it ships. In both, the AI writes the tedious artifact and a deterministic harness — not your judgment — verifies it.
Skip AI for a single trivial pattern you can write faster than you can prompt — the leverage is in volume, gnarly formats (multiline stack traces, mixed formats on one stream), and convention compliance.
vector CLI with its native unit-test support — the lab default), Logstash, or an Elastic/Loki ingest pipeline.Logs — generate rules AND tests together. Paste the samples:
Here are 40 raw log lines from service X (note: lines 12-19 are one
multiline Java stack trace; lines 30+ switch to JSON).
Write a Vector VRL transform that parses every line into: timestamp
(RFC3339), level, service, trace_id (nullable), message, plus
error.class/error.stack for tracebacks. Unparseable lines must route to
a `parse_failure` output with the raw line preserved — never drop.
Also write Vector unit tests covering: one normal line of each format,
the multiline trace, a truncated line, and one line NOT in the samples
that your rule should reject.
Run the harness; iterate on failures, not on vibes. vector test runs the generated tests deterministically. Feed failures back verbatim until green, then — the step that matters — audit the tests before trusting the green: do the expected-output assertions actually check field values (not just "parsing succeeded"), and does the reject case really represent unknown input? AI-generated tests can be sycophantic to AI-generated rules; the test review is your citation check here.
Replay real traffic through the rule. Run a day's worth of raw logs through the transform offline; measure parse rate and inspect the parse_failure stream. Every failure is either a rule gap (iterate) or a genuinely malformed emitter (file a bug upstream — also a win). Ship via PR with the parse-rate number and the test file in the diff.
OTel — instrument against the conventions, not the model's memory. Semconv attribute names are exactly the kind of detail models half-remember (http.status_code vs http.response.status_code changed with semconv stabilization). Paste the current convention tables for your domain into context:
Instrument this Flask/Express/Go handler with OpenTelemetry using the
attached semantic conventions VERBATIM. Spans for: inbound HTTP (server),
the outbound payment call (client), and the DB query — with required
attributes from the tables. Add span events for retry attempts. Do not
invent attribute names: every attribute must appear in the attached
conventions or be namespaced under `app.*` as custom.
Verify in a local collector before merge. Run service + collector + Jaeger; exercise the endpoints; then check in Jaeger/collector debug output: spans present with correct kind and parentage, required attributes populated with real values, trace context propagating across the outbound call (one trace, not three orphans). Grep exported spans for attributes NOT in the conventions or app.* — that grep is your automated hallucinated-attribute detector; wire it into CI if you're doing this across a fleet. Merge with the Jaeger screenshot and the attribute-check output in the PR.
Both halves share one design: the AI produces the artifact, a deterministic harness (unit tests, collector output) produces the evidence, and the human audits the harness — verify the verifier, then let it work for you.
vector test gives the tightest generate-test loop in the logs ecosystem; its remap errors are excellent feedback to paste back to the model.vector test green, replay parse rate reported (target: >99% with all failures explained), zero silent drops — the parse_failure route is present and monitored.app.* namespace — this is the citation check, run as a grep over actual collector output, not over the source code the model wrote.password=... into a queryable field has industrialized a leak. Add redaction transforms in the same PR as parsing, and test them like any rule.vector test, replay, and the local collector need no egress — in restricted shops, lean even harder on harness evidence precisely because model choice is constrained.Telemetry pipelines are production infrastructure — a bad transform deployed fleet-wide can blind you during the next incident. Parse rules ship through the same PR-plus-replay-evidence gate as alert rules (pb-25), no exceptions for "just a parsing tweak."