Set AI Expectations in Performance Reviews Without Turning Telemetry into a Weapon

Workflow area: Hiring & Performance · Tier: Advanced · Hands-on time: ~40 minutes

The pressure is real: your org has paid for the tools, the CTO has announced the transformation, and somebody in a calibration meeting will eventually say "shouldn't AI usage factor into ratings?" The lazy answer — pull the Copilot dashboard and grade adoption — is measurably wrong (acceptance rates and suggestion counts don't correlate with outcomes), corrosive (engineers will game any usage number within a sprint of learning it's watched, exactly the Goodhart trap from pb-15), and unfair (it punishes the engineer whose domain has thin AI leverage and rewards prompt-spam). The right answer is the one performance systems always give: define the behaviors that constitute skilled work in the AI era, and assess them from the work.

When to use — and when not to

• Use it when an org AI mandate needs translatingbefore the next review cycle, so expectations precede assessment; announcing new criteria in the same cycle they're graded is a fairness failure.
• Use it when calibration debates AI skill inconsistentlyone committee crediting "uses AI heavily" while another discounts it means the rubric is silent; fill the silence deliberately.
• Use it alongside pb-17 and pb-23JD language, onboarding habits, and review expectations should describe the same skills at the same levels, or you hire for one thing and grade another.
• Skip grading individual telemetry, alwaysusage hours, acceptance rates, suggestion counts, and per-person dashboards are banned inputs to any individual's rating in this playbook; that data measures systems (pb-38's territory), never people.
• Skip it where the mandate itself is unsettledif leadership hasn't decided what AI use is actually for (pb-26's stance one-pager), review expectations will encode the confusion; sequence the stance first.

The recipe

1. 1.

   Extract the behaviors from your best evidence. Before prompting, list what your genuinely effective AI-era engineers observably do: they verify model output before building on it, they know when a task is faster by hand, their AI-assisted PRs arrive with real tests, they feed context files and improve them, they catch the model's confident errors in review. This list — not the mandate — is the raw material.

2. 2.

   Draft rubric language per level, with observability enforced.

   Copy

   Draft performance-rubric additions for AI-era engineering
   skills. Inputs:
   - Observed behaviors of effective AI users here: [YOUR LIST]
   - Our existing rubric dimensions and level definitions:
     [PASTE]
   - Our AI policy summary: [APPROVED TOOLS, review gates]
   
   Rules:
   1. Every expectation must be assessable from WORK PRODUCT —
      PRs, review comments, design docs, incident writeups.
      If an expectation would require tool telemetry or
      usage data to assess, rewrite or delete it.
   2. Express skill as JUDGMENT, not volume: knowing when NOT
      to use AI is explicitly part of the skill at every level.
   3. Scale by level: juniors are assessed on verification
      discipline; seniors on judgment and on raising the
      team's AI practice; staff on setting standards others
      follow.
   4. Fold into EXISTING dimensions (craft, execution,
      leadership) rather than creating a standalone "AI"
      dimension that invites usage-counting.
   

3. 3.

   Check the output for observability, then keep the best rows. Good output looks like:

   Copy

   CRAFT (Senior) — addition:
   "AI-assisted work meets the same bar as hand-written work
   and the engineer can explain every line. Evidence: review
   threads where they caught AI-introduced defects; PRs
   showing verification notes; sound choices about which
   tasks used AI and which didn't."
   
   REJECTED by rule 1: "Regularly uses approved AI tools in
   daily work" — assessable only via telemetry. Replaced
   with: "Delivers at the pace and quality the team's AI-
   assisted workflow enables, by whatever mix of methods."
   

   That rejection is the design working: outcome and judgment language survives; usage language dies.

4. 4.

   Write the grandfathering and accommodation rules before the cycle starts. Three groups need explicit protection in the written guidance: engineers hired before the expectations existed get a full cycle of access-to-training before assessment (grandfathering); engineers in domains where approved tools genuinely underperform — safety-critical code, air-gapped environments, a no-AI client contract (pb-33) — are assessed as if the expectation doesn't apply, stated in writing so calibration doesn't ding them; and engineers with accessibility needs interact with these tools differently, which routes through accommodation processes, not ratings. Skipping this step is how a skills rubric becomes a legal exposure.

5. 5.

   Brief calibration, then audit the first cycle. Give every calibrating manager the same one-page guidance: what evidence counts (work product), what's banned (telemetry, usage anecdotes like "I never see her use Copilot"), how the exempted groups are handled. After the cycle, audit a sample of written reviews for banned inputs — one manager writing "low Copilot adoption" in a review re-teaches the whole team that the dashboard is secretly the rubric, and trust, once spent there, doesn't refund.

Make it yours

• Small team or startup: you may not have a formal rubric — write the expectations as a one-page "how we work" note and use it in feedback conversations (pb-03) all year, not just at review time. The telemetry ban still applies; at ten people, dashboard-grading is somehow even more corrosive.
• Large org or GCC: coordinate with HR before touching rubric language — performance criteria changes may trigger works-council consultation, policy review, or parent-org alignment. If the parent org pushes telemetry-based adoption metrics, negotiate the boundary explicitly: adoption metrics report on the program (pb-37, pb-38), rubric behaviors assess the person, and the two datasets never join.
• Teams mid-rollout: stage the expectations with the rollout phases (pb-43) — pilot-cohort engineers get assessed on the new behaviors a cycle before the long tail, matching when each group actually received tools and training; assessing the untrained is the fastest way to convert skeptics into saboteurs (pb-48).

Quality check

• Every expectation passes the work-product testfor each rubric row, name the artifact a manager would cite as evidence; if the honest answer is "the usage dashboard," the row is banned input wearing rubric clothes.
• "Knowing when not to" appears at every levela rubric that only rewards using AI is a volume metric; judgment language is the tell that you're grading skill.
• The exemption rules are written, not oralgrandfathering, domain exceptions, and accommodations exist as documented guidance calibration can cite, because unwritten exceptions evaporate under ranking pressure.
• Expectations shipped a full cycle before assessmentcheck the dates; grading criteria people learned about at review time is the complaint HR will hear first, and they'll be right.
• The first-cycle audit ransample reviews for telemetry language and usage anecdotes; what you catch and correct in cycle one becomes culture by cycle three.