Translate a CEO's "AI use is now a baseline expectation" mandate into a level rubric that measures observed work quality — not tool telemetry. A well-meaning HRBP draft scores Copilot acceptance rates and seat activation; your job is to reject every telemetry trap in writing and rule fairly on three hard edge cases.
Kavitha Rao is Senior Director of Engineering at Northbeam Software, a fictional 300-person B2B analytics company. This morning the CEO sent the memo every engineering leader now gets eventually: AI fluency is a baseline expectation, and it will be "reflected in performance reviews." By noon, a helpful HRBP produced a draft rubric update — built on the only AI data HR can see: vendor telemetry. Kavitha knows that rubric would punish her best verifier and promote her fastest paste-and-pray merger. Your job is to write the version that survives calibration: measurable, fair, and about the work.
CEO MEMO (excerpt) | To: All staff
"Effective this half, reflexive AI use is a baseline expectation at
Northbeam, like using version control. Before asking for headcount,
teams must show why AI cannot do the work. AI fluency will be
reflected in performance and promotion reviews. We will provide
tooling, training, and time to learn. This is not optional."
CURRENT RUBRIC (Engineer II excerpt, unchanged since 2023)
- Craft: ships correct, maintainable code; tests own work
- Execution: delivers committed scope; unblocks self; sizes well
- Collaboration: reviews others' code constructively; communicates
status honestly; lifts teammates
- Ownership: monitors production; follows through on incidents
HRBP DRAFT UPDATE (v0.9, "AI Fluency" — proposed new rubric row)
Below expectations: Copilot seat inactive or <20% suggestion
acceptance rate; fewer than 5 AI tool sessions/week in telemetry
Meets expectations: seat active; 20-40% acceptance rate; uses
approved AI tools daily per usage dashboard
Exceeds expectations: >40% acceptance rate; top-quartile AI tool
usage minutes; evangelizes AI in team channels
Note: telemetry pulled monthly from vendor admin dashboards; scores
auto-populate the review packet.
MANAGER NOTES | three cases pending your ruling
Case 1 - Riya (Engineer II): screen-reader user; the approved AI
IDE plugin is barely usable with her assistive stack. Uses a chat
model in a browser instead, pastes results into her editor.
Telemetry shows her Copilot seat "inactive for 60 days."
Case 2 - Marcus (Senior): staffed 100% on the Halden account, whose
contract prohibits all AI tools on their codebase. Telemetry:
near-zero. His non-client work shows sharp AI-assisted design docs.
Case 3 - Priyanka (Engineer II): declines AI tools on principle,
citing craft concerns. Output is solid but her throughput now
trails peers who verify well AND use AI. She asks: "Will I be
rated down for not using it?"
Form your own view first — ten minutes, no AI. Read the memo, rubric, and HRBP draft. Write down: what the memo actually requires (and does not), every specific way the HRBP draft could misfire, and your instinct on each of the three cases. The draft looks reasonable at a skim; the point of this pass is to stop skimming.
Interrogate the telemetry metrics. Prompt: "For each metric in this draft rubric row — suggestion acceptance rate, seat activation, sessions per week, usage minutes — explain what it actually measures, how an engineer could score high while doing worse work, and how a strong engineer could score low. Be specific." Verify each claim against your own reasoning; keep the ones that survive as your written rejection rationale.
Write the rejection memo to the HRBP. One page, respectful, specific: why each telemetry metric cannot appear in a review packet — acceptance rate measures how often someone presses tab, not output quality; seat activation measures procurement, not fluency; auto-populated scores convert a vendor dashboard into a performance verdict nobody can appeal. Propose the replacement principle: AI fluency is assessed from observed work, by humans, with evidence.
Draft the replacement rubric row. Anchor each level in observable work behaviors. Below: ships AI output unverified, cannot explain AI-assisted changes in review, or refuses to engage with AI tooling without an articulated reason. Meets: uses AI where it helps, verifies before shipping, can explain every line they submit, flags AI-origin work appropriately. Exceeds: measurably lifts the team — shares prompts and verification patterns, improves review standards for AI code, mentors others. Note what is deliberately absent: any number a vendor dashboard produces.
Write the calibration guidance. One page for managers running calibration: evidence that counts (review threads, design docs, incident behavior, demos of workflow), evidence that does not (telemetry, self-reported enthusiasm, channel activity), and the two failure modes to police in the room — punishing skilled skeptics who verify deeply, and rewarding loud adopters who ship slop.
Rule on Case 1 — accessibility. Write the ruling: Riya's telemetry is irrelevant and her workaround demonstrates more fluency, not less — she adapted tooling to her constraints. Pair the ruling with the systemic action: an accessibility review of the approved toolset, and an approved path for her browser-based workflow with the same data-handling guarantees.
Rule on Case 2 — client-blocked tools. Write the ruling: Marcus is assessed on the work he is permitted to do; contract-driven zero telemetry cannot appear as a fluency deficit. Specify the mechanic: reviews note the constraint explicitly, fluency evidence is drawn from his non-client work, and account-staffing rotations ensure client constraints never silently cap a career.
Rule on Case 3 — the conscientious objector. This one is genuinely hard; write the ruling honestly. Priyanka cannot be rated down for tool telemetry — but the rubric measures outcomes, and if peers deliver comparable quality faster, the throughput gap is real and ratable as an execution outcome, not as a compliance failure. The ruling: her craft concerns get a fair hearing (and feed the review standards), she gets the same enablement time as everyone, and she is told plainly that outcomes are the bar — declining the tool is her right; declining the results it enables has a cost.
Assemble the diff and pressure-test it. Produce the final artifact: rubric diff (old row → new row), calibration guidance, three rulings, rejection memo. Then red-team: "How could a cynical manager or engineer game this rubric?" Patch what you can; document the residual risk.
Any metric that auto-populates a review packet from a vendor dashboard will be optimized within one quarter — and what gets optimized is the metric, not the work.
A complete lab has all of the following:
"Reflected in reviews" is a mandate about outcomes, not telemetry. Rate the work humans can observe — verified quality, honest explanation, lifted teammates — and keep every vendor dashboard number out of the packet, because whatever you put in it is what you will get more of.