Baseline the Team's AI Skills Without Building a Ranking

Workflow area: Org Design & Change · Tier: Core · Hands-on time: ~35 minutes

Most orgs "know" their AI skill level from two sources: seat-activity dashboards (which measure logins, not capability) and the vibes of whoever talks about AI the loudest. Both are wrong in the same direction — they mistake enthusiasm for skill. The engineers quietly running sophisticated verification workflows are invisible; the loud early adopter shipping unreviewed generated code reads as your best user. A real baseline watches people do realistic tasks against a rubric. And it only works under an explicit non-negotiable: results aggregate to program design, never to individual evaluation — because the first time a baseline touches a performance review, every future assessment measures test-taking.

When to use — and when not to

• Use it before designing any upskilling programpb-46's gap matrix is built from this data; without it you are prescribing before diagnosing.
• Use it before selecting pilot teamspb-43's "representative pilot" needs to know what representative means in your org.
• Use it annually as the program's yardstickthe same scenarios re-run are how you measure skill change instead of attendance.
• Skip it if you cannot guarantee anonymized reportingif leadership pressure will force per-person results into calibration meetings, running the assessment does net harm; fight that battle first.
• Don't bolt it onto performance seasoneven a clean assessment run in review month will be read as an evaluation, and your response rate and honesty collapse together.

The recipe

1. 1.

   Design three scenario tasks, 30–40 minutes total. Practical, on your stack, doable with whatever AI tools the person normally uses:

   Copy

   TASK A — PROMPT & CONTEXT CRAFT (10 min)
   "Here is a real ticket from our backlog [redacted copy]. Show
   how you'd brief an AI assistant to start it: the actual prompt
   or agent instructions you'd write, including what context
   you'd attach and why."
   TASK B — VERIFICATION (15 min)
   "This AI-generated diff claims to implement the ticket. It
   contains 3 seeded flaws [logic bug, subtle security issue,
   spec deviation]. Review it as if approving for merge: what do
   you flag, what do you test, what do you ask for?"
   TASK C — TOOL FLUENCY & JUDGMENT (10 min)
   "For each of these 5 real situations from our work, say
   whether you'd use AI, which tool/mode, and what you'd never
   paste into it. One sentence each."
   Delivery: async in a normal work week, tools allowed, stated
   purpose: 'this designs our training program; results are
   anonymized aggregates; nobody sees individual answers but you.'
   

   Task B's seeded flaws are the heart of the assessment — verification skill is the one that predicts whether AI leverage is safe, and it is invisible in usage dashboards.

2. 2.

   Score with a rubric, assisted but human-owned. Define 0–3 levels per skill before reading any response. Claude drafts the first-pass scoring; a human moderates:

   Copy

   Score these anonymized assessment responses against the rubric.
   RUBRIC:
   - Prompt craft: 0 = vague one-liner … 3 = decomposed task,
     relevant context attached, constraints and output format
     specified, verification plan implied
   - Verification: 0 = approved with style comments … 3 = found
     2-3 seeded flaws, proposed tests, asked for spec
     clarification
   - Judgment: 0 = would paste anything … 3 = correct tool/tier
     choice and data-boundary call in ≥4 of 5 situations
   For each response: level per skill + the quoted evidence line
   that justifies it. Mark borderline calls [MODERATE] for human
   review. Do not attempt to identify authors or writing styles.
   Responses: [PASTE ANONYMIZED BATCH]
   

   A human spot-checks 20% of scores and all [MODERATE] flags — the rubric's credibility is worth more than the hours saved.

3. 3.

   Report as aggregates with the shape, not the names. The output is a distribution per skill per role family: 42% of product engineers at level 2+ on verification; platform team strongest on agentic fluency; judgment scores uniformly high except data-boundary calls in the payments group. Present it to the team before any leadership audience — the people who gave you the data see the picture first, with the program actions it implies (pb-46's matrix) attached.

4. 4.

   Let individuals keep their own results. Each participant privately receives their own scores and the rubric — self-directed development fuel, shared with a mentor only if they choose. This is the honest trade for their honest effort, and it is what makes next year's re-run welcome instead of resented.

5. 5.

   Re-run annually with the same tasks, refreshed seeds. Same scenarios and rubric so deltas are real; new seeded flaws so memory isn't the skill measured. The year-over-year distribution shift is your upskilling program's report card (pb-46, step 6).

Make it yours

• Startup or small team: under ~15 people, anonymized aggregates are a polite fiction — so change the contract instead: run the tasks as an open workshop where everyone scores a partner's Task B against the rubric together. You lose the survey; you gain the calibration conversation, which at this size is worth more.
• Mid-size org: run it via Google Forms or your survey tool with a trusted scorer per 30–40 responses, report by role family, and time it two months before annual planning so the gap matrix lands while budgets are still liquid.
• Large org or GCC: sample rather than census — 25–30% per role family gives you the program-design signal without an org-wide testing event — and put the anonymization pipeline in writing (who holds the key, when raw responses are deleted). In a GCC, aggregate capability data is also a commercial asset: "68% of the account team at level 2+ verification, measured against seeded-flaw tasks" is a client-credible sentence no attendance report can produce.

Quality check

• The seeded flaws were actually findablepilot Task B on two seniors first; if they miss the security seed, the task measures the seed's obscurity, not the team's skill.
• Scores cite quoted evidenceevery level assignment traces to a line in the response; unquoted scores drift toward the scorer's priors.
• Participation was genuinely voluntary and honesta 95% completion rate with suspiciously polished answers means people treated it as a test; check what your comms actually promised.
• Aggregates reached the team before the exec deckorder of disclosure is the trust signal everyone watches.
• The data changed somethinga baseline that didn't alter the program design, pilot selection, or budget ask was an expensive survey; name the decision it fed.