Design a role-based upskilling program from a 12-person skills-baseline dataset. Two rows are planted signals: one hides shadow-AI expertise worth recruiting into a champion role, and one shows perfect quiz scores over failed verification tasks — the trap that breaks every attendance-based training program.
Fatima Al-Rashid is an EM at Cobalt Harbor, a fictional 55-engineer maritime-logistics software company. The exec team funded "AI training for everyone" and procurement wants to buy a video-course bundle with completion tracking. Fatima ran something better first: a scenario-task baseline — every engineer on her 12-person team worked three realistic tasks (prompt a model through a gnarly ticket, verify a plausible-but-flawed AI output, drive their actual toolchain) plus a knowledge quiz. The dataset below is the result. Her thesis, which this lab tests: the 20% who self-teach need recognition, not courses; the 80% need deliberate practice on real work, and the only measurement that matters is skill change — attendance measures chairs.
Scores 1–5 (task-assessed by two reviewers except Quiz, which is auto-scored). DeclTools = tools the engineer declared using; Verify = did they catch the three seeded flaws in the AI-output task?
Name Role Quiz PromptQ Verify ToolFluency DeclTools
Arjun Sr BE 3.2 4.1 4.5 3.8 IDE+chat
Bella Mid FE 2.8 2.4 2.1 2.6 chat
Chen Sr FE 3.5 3.9 4.2 4.0 IDE+chat
Dana Jr BE 2.1 1.9 1.8 2.2 none
Elias Mid BE 4.9 4.7 4.8 4.9 "none"
Farah Sr Plat 3.0 3.3 3.6 3.1 IDE
Gustav Mid BE 4.8 2.2 1.6 2.4 chat
Hana Jr FE 2.5 2.7 2.9 3.3 chat+agent
Igor Sr BE 3.1 2.9 3.4 2.8 IDE
Jia Mid Plat 3.7 3.6 3.9 3.7 IDE+chat
Karan Jr BE 1.8 2.0 2.3 1.9 none
Lena Mid FE 3.9 3.4 3.1 3.5 chat
Notes from assessors:
- Elias declared "none" on the tool survey but completed the
toolchain task faster than anyone, using advanced patterns;
asked assessors "not to make a thing of it."
- Gustav aced every multiple-choice item incl. the hard ones;
in the verify task he approved all three seeded flaws with
confident written justifications for two of them.
- Hana was the only engineer who had configured the agent tool.
Read the dataset before averaging it — ten minutes, no AI. Rank the team by each column separately and note every row where columns disagree sharply. Two rows should stop you cold. Write down what you think each anomaly means and what you would do about it — before the AI smooths them into cohort averages, which is exactly what will happen if you skip this step.
Ask the AI for cohort structure, then audit it. Prompt: "Cluster these 12 engineers into 3-4 skill cohorts for an upskilling program, using the four scores. Show cluster membership and the defining gap of each cluster." Check its treatment of the two anomalous rows specifically: averaging Gustav's 4.8 quiz into a 2.5-ish practical profile, or filing Elias as a beginner because he declared no tools, are the errors to catch. Correct the clusters by hand.
Rule on Elias — the hidden expert. Write the finding: declared "none," performed like the team's strongest practitioner — this is shadow-AI use, almost certainly driven by policy ambiguity or fear of judgment (his "don't make a thing of it" is the tell). The response is graded: amnesty explicitly, a private conversation to learn what he's built, an invitation into a champion/practice-lead role with protected hours — and a fix for whatever made concealment feel safer than candor (usually the policy comms, see Lab 04's rollout note). Punishing or exposing him teaches eleven other people to hide.
Rule on Gustav — the quiz-verified illusion. Write the finding: 4.8 on knowledge, 1.6 on verification, with confident justifications for approving seeded flaws — the most dangerous profile on the team, because certification-style measurement would score him as AI-ready and hand him tools he cannot check. Design his path around supervised verification reps with tight feedback, not more content. Then generalize: write the one-paragraph policy ruling that quiz scores and course completion are inadmissible as skill evidence in this program. This is the row that breaks attendance-based measurement; say so in writing.
Design role-based paths per corrected cohort. For each cohort: the defining gap, the practice focus (prompting fundamentals / verification discipline / toolchain-and-agent fluency), and the format — short workshop, then applied projects on the cohort's actual sprint work with review checkpoints. Name names per cohort. Hana's agent configuration makes her the seed of the agent-fluency track despite junior-level scores elsewhere — spiky profiles get spiky paths.
Protect the time or admit the program is fiction. Write the resourcing clause: 2–3 hours per engineer per week, named in sprint capacity, for eight weeks; champions (Elias, if he accepts; Arjun or Chen as verification leads) get 4. Include the sentence for the exec sponsor: a program funded with licenses but not hours is a bookmark folder.
Build the baseline-to-target matrix. Per engineer: current scores → 8-week targets (realistic: +0.5 to +1.0 on focus dimensions, holds elsewhere), reassessed with the same scenario-task instrument, different seeded content. Program-level success criteria: median Verify ≥ 3.5, no engineer below 2.5 on any dimension, and Gustav's Verify above 3.0 specifically. Explicitly absent from the criteria: course completions, video hours, attendance, quiz deltas.
Write the measurement memo. One page to the exec sponsor: why the video-bundle-with-completion-tracking is being declined or repurposed as optional reference material; what is measured instead (skill change on task performance, baseline vs week 8); the two dataset stories that justify the design — told anonymized ("an engineer with near-perfect quiz scores approved every planted flaw"; "our strongest practitioner was invisible to the tool survey") — and the cost clause from step 6.
Every skills dataset has an Elias — run the anomaly check (declared usage vs demonstrated fluency) before any training purchase, because your best curriculum author is usually hiding in the "none" column.
A complete lab has all of the following:
Upskilling the 80% runs on deliberate practice against real work, measured by skill change on the same instrument that found the gaps. Quizzes certify memory, attendance certifies presence — and both would have promoted the one engineer who approves flaws with confidence.