Executive Communication of AI-System Performance

Tier: Core · Workflow area: Serving & LLMOps

This playbook turns your eval results, cost curves, and monitoring data (pb-49/50/51 outputs) into decision memos leadership can act on — using AI to draft framing and translation, while every number stays human-owned and artifact-grounded.

When to use

• •
  Quarterly/monthly reviews of an AI feature, a go/no-go on expanding one, or a budget ask (GPU purchase, model upgrade, headcount).
• •
  After an incident or rollback, when leadership needs an honest account that preserves trust in the program.
• •
  Standing reporting: turning pb-49's dashboards into a recurring one-pager.

Not for peer-technical audiences — they should get the dashboards and eval tables directly.

Prerequisites

• •
  Real artifacts to report on: eval results with baselines, cost/latency data, monitoring history. This playbook cannot fix missing measurement — it only translates it.
• •
  One sentence, written before you start, stating the decision you want the reader to make (approve, fund, accept risk, change course). No decision, no memo.
• •
  Any capable AI assistant.

The workflow

1. 1.

   Assemble the evidence pack. The 5–10 numbers that matter with baselines and deltas, the cost curve, one concrete failure example and one win example (verbatim, anonymized), and the decision sentence. This curation is your judgment and cannot be delegated — the model can only frame what you select.

2. 2.

   Translate metrics to consequences before drafting. For each metric, write the plain-language consequence yourself: faithfulness 0.87 → "roughly 1 in 8 answers contains an unsupported claim; at 10k answers/month that is ~1,250 needing the citation-check workflow." Ask the AI to stress-test your translations ("what would a skeptical CFO ask about this framing?"), not to invent them.

3. 3.

   Draft with an explicit structure. Give the assistant the evidence pack plus a frame: decision requested up front; three findings max, each as consequence-first-then-number; costs in units leadership uses (₹/$ per resolved ticket, per report generated — not per token); risks stated plainly with mitigations; the ask, sized and dated. Instruct it to use only supplied numbers and to mark any inference it makes as [inference].

4. 4.

   Run the claim audit (pb-01 discipline, executive stakes). Every number traced to the evidence pack; every [inference] either verified and un-marked, or deleted. Then apply the leadership-trust test: is any framing technically true but likely to mislead? ("Accuracy up 40%" when it moved 50%→70% on an easy subset fails this test.) Fix those yourself — models optimize for persuasiveness, and persuasiveness is exactly the failure mode here.

5. 5.

   Prepare the Q&A layer. Have the AI generate the 10 hardest questions the memo invites ("why not just use ChatGPT?", "what happens if the model provider doubles prices?", "when does this break even?"), then write your own answers, grounded in the artifacts. This step converts the memo from a document into a decision meeting you are ready for.

6. 6.

   Close the loop. After the meeting, record which framings landed, which numbers were challenged, and what was decided — this feedback tunes the next cycle's evidence pack, and over quarters becomes your program's decision log.

Tool variants

• Chat assistant with file uploadpaste the evidence pack; the workflow above unchanged.
• Slide/table generation (assistant code-execution or skills for docx/pptx) when the venue is a deck; same claim-audit applies to every chart label.
• Templated pipelinefor standing monthly reports, script the evidence-pack extraction from pb-49 dashboards and reuse a frozen prompt — consistency across months makes trends legible.

Verification

• •
  Every number in the memo traced to a source artifact; audit trail kept alongside the draft.
• •
  The misleading-framing test applied and at least one framing consciously rejected or softened (record it — that judgment is the skill).
• •
  A non-technical colleague reads the memo cold and can state the requested decision and the top risk.
• •
  ROI/cost figures reconcile with pb-49's dashboard numbers exactly.

Pitfalls

• Leading with the metric instead of the meaning. Executives fund consequences and decisions; put "1 in 8 answers needs checking" before "faithfulness 0.87," always.
• Hiding the failure modes. Omitted risks surface later as broken trust; a plainly-stated risk with a mitigation increases credibility and future budget.
• Vanity precision. "87.3% ± unstated noise" invites a challenge you will lose; round to what your eval's sample size actually supports.
• Letting AI-fluent prose oversell. The polish is the danger — audit the draft harder because it reads well, not less.

Restricted-environment variant

Evidence packs contain aggregates and curated examples, so hosted-assistant use is usually fine once examples are anonymized — but revenue figures, client names, and unannounced-product references need the same handling as any confidential memo: placeholder them before drafting, restore after, or draft against a local model. The memo itself is often more sensitive than the metrics; store and share it under your org's document-classification rules, not in a chat history.

Success criteria

• •
  A decision memo, grounded in your real system's artifacts, that secured (or would credibly secure) the stated decision.
• •
  Zero unsupported numbers under audit; Q&A prep answered a real challenge in the meeting.
• •
  A repeatable evidence-pack → memo → decision-log cycle, documented for the next cadence.
• •
  Leadership asks better questions next cycle — the observable sign the translation layer worked.