Lab 10 · Design Red-Team: Attack Your Own Design Before Humans Do

The skill is using an LLM as an adversarial reviewer with a repeatable protocol, instead of the vague "any feedback?" prompt that yields flattery.

Time & tier

Core · ~60 minutes hands-on.

Setup

No code, no API spend — any capable chat assistant on a free tier works (a reasoning-strong model is noticeably better at the attack passes). Create a working folder lab10/ with three empty files: design-v1.md, attack-log.md, design-v2.md.

The scenario you must design for (read carefully — it contains deliberate traps):

Design a rate-limiting service for our API platform.
Requirements from the product team:
- Protect ~40 backend services; limits are per-API-key, e.g. "500 req/min".
- Enforcement should be accurate: marketing has promised customers
  "exact limits, no surprises".
- Latency budget: the limiter may add at most 3ms p99 to any request.
- Platform runs in two AWS regions, active-active; a customer's traffic
  can hit either region.
- Limits are configured by customer-success staff in an internal admin UI
  and should take effect "immediately".
- When the limiter itself has problems, "the API should keep working".
- Expected scale: 60k req/s peak today, tripling within a year.
- Budget note: infra team prefers reusing the existing shared Redis cluster.

Guided steps

1. 1.

   Draft v1 yourself — no AI. 45–90 lines in design-v1.md: chosen algorithm (fixed window, sliding window, token bucket…), data store and data model, enforcement point (gateway middleware? sidecar?), config propagation, failure behavior, and a capacity estimate. Timebox to 20 minutes; an imperfect v1 makes the attack pass more instructive.

2. 2.

   Lens 1 — failure modes. Paste the scenario and your full v1, then:

   Copy

   Act as a principal engineer doing a pre-mortem. Assume this design shipped
   and caused a SEV-1 within 90 days. Give the 5 most plausible incident
   narratives: trigger, propagation, blast radius, and the design decision
   that enabled it. Rank by likelihood × impact. Do not suggest fixes yet.
   

3. 3.

   Lens 2 — scale cliffs. Same conversation:

   Copy

   Now attack scalability only. For each component, identify the first
   quantitative cliff: at what request rate, key cardinality, or config
   count does it degrade, and what is the observable symptom? Show the
   arithmetic (memory per key, Redis ops per request, cross-region traffic).
   

4. 4.

   Lens 3 — security. New message: request a lightweight STRIDE pass focused on tenant isolation, the admin UI config path, spoofed API keys, and what an attacker gains by bypassing or weaponizing the limiter (e.g., exhausting another tenant's quota).

5. 5.

   Lens 4 — operability. Ask: what can on-call not see or not do at 3 a.m. with this design? Demand concrete gaps: missing metrics, no kill switch, no per-tenant override, unclear degraded mode, runbook-shaped holes.

6. 6.

   Triage the attack output. In attack-log.md, table every finding with columns: finding · lens · accept/reject · rationale. Reject at least two findings with a written reason — LLM red-teams overproduce, and rejecting confidently is part of the skill. Expect 15–25 raw findings.

7. 7.

   Revise. Write design-v2.md addressing every accepted finding, ending with a change log: each change mapped to the finding that forced it. Where you consciously accept a risk (e.g., approximate limits for lower latency), say so explicitly — that sentence is what distinguishes a design from a wish.

Verification

Objective checks against the seeded traps (see answer key for the full list):

• •
  Your attack log contains findings from all four lenses, ≥12 total, with ≥2 explicit rejections.
• •
  v2 resolves the contradiction between "exact limits" and "3ms p99 + active-active two regions" in writing (pick a side and say why).
• •
  v2 specifies fail-open vs fail-closed behavior for a Redis/limiter outage and ties it to the "API should keep working" requirement.
• •
  v2 addresses the shared-Redis suggestion explicitly (accept with isolation controls, or reject with cost note).
• •
  Every v2 change traces to a logged finding — no unexplained edits.

Completion artifact

design-v1.md, attack-log.md, design-v2.md. Self-grade (4 points = pass at 3+):

1. 1.
   All four seeded contradictions from the answer key surfaced (by you or the AI) and dispositioned.
2. 2.
   At least one scale cliff is quantified with arithmetic, not adjectives.
3. 3.
   Attack log shows real triage: accepted findings changed v2; rejected findings have defensible rationale.
4. 4.
   v2's change log is complete and traceable.

Stuck?

• The AI is being polite, not adversarial: the persona framing matters — "assume it already failed; write the incident narrative" reliably breaks flattery mode.
• Findings are generic ("add monitoring"): demand specificity — "name the exact metric, its labels, and the alert threshold."
• You can't judge which findings to reject: ask the AI to argue against its own top finding, then decide. Steelman/counter-steelman is cheap.
• v1 paralysis: pick token bucket in Redis with a Lua script and move on; the lab grades the attack-and-revise loop, not the initial choice.

Client-safe variant

This lab needs zero client code — the scenario is fictional, so even strict-NDA engineers can run it verbatim on any public assistant. To apply the protocol to a real client design: strip client names, product names, and any data classifications from the design doc; replace concrete hostnames/regions with placeholders; then run the same four lenses on the sanitized version, or run the whole loop on a client-approved enterprise endpoint (Copilot Chat Business, Bedrock/Azure-hosted model in the client tenancy). The four-lens protocol itself is the deliverable you carry between engagements.

Answer key (don't read before attempting)

The scenario seeds four contradictions and several cliffs a competent red-team must surface. Contradictions: (1) "Exact limits" vs active-active two regions vs 3ms p99 — exact global counting requires cross-region coordination (~tens of ms) or a single home region per key; you must trade to approximate/eventually-consistent limits, region-local quotas (e.g., split 500 into 2×250), or renegotiate the marketing promise. (2) "Keep working when limiter fails" implies fail-open — but fail-open plus a Redis outage means zero protection during exactly the traffic spikes that hurt Redis; v2 needs a bounded degraded mode (e.g., local in-process approximate limiting). (3) "Immediately" effective config vs cached limits — per-request config reads destroy the latency budget; v2 needs push/TTL propagation with a stated staleness bound. (4) Shared Redis cluster — the limiter adds 1–2 Redis ops per request at 60k rps (→180k+ ops/s at 3× growth), can noisy-neighbor the very services it protects, and couples failure domains. Expected cliffs: per-key state × key cardinality (memory), hot-key single-shard saturation for a whale customer, thundering herd at fixed-window reset (argues for token bucket/sliding log), and Lua script CPU on one shard. Security: API keys must be authenticated before counting (else attackers burn a victim's quota by spoofing), admin UI config changes need authz + audit + validation (a fat-fingered 5 req/min is a self-inflicted outage), and limit-status responses shouldn't leak other tenants' quotas. Operability: per-tenant observed-vs-configured rate metric, a global and per-tenant kill switch, shadow/dry-run mode for new limits, and a documented Retry-After contract.