Every tester knows the folklore inputs — the emoji name, the leap-day birthday, the million-character paste. Adversarial datasets turn folklore into inventory: systematically generated malformed payloads, Unicode and RTL text, boundary dates, and oversized inputs, each tagged by the risk it probes, versioned, and wired into tests that assert graceful failure.
Tier: Core · Workflow area: Test data & environments
This playbook combines LLM generation, the big-list-of-naughty-strings, and Hypothesis into a curated adversarial corpus organized by risk category — so "did we test hostile input?" has a diffable answer instead of a shrug.
Define the risk taxonomy first — tags are the product. A flat pile of weird strings is folklore again; the taxonomy makes coverage assessable:
ENC encoding & unicode: combining chars, ZWJ emoji, RTL/bidi, homoglyphs,
normalization pairs (é as NFC vs NFD), null bytes, astral-plane chars
LEN size extremes: empty, 1-char, exactly-at-limit, limit+1, 10MB paste
FMT malformed structure: truncated JSON, wrong types, duplicate keys,
invalid UTF-8 bytes, CSV with embedded delimiters/newlines
TIME boundary dates: 29 Feb, DST transitions, 1970-01-01, 2038-01-19,
year 0/9999, timezone offsets ±14:00, leap seconds
NUM numeric edges: -0, NaN, Infinity, 2^53±1, 2^63±1, 0.1+0.2 sums,
currency with >2 decimals, negative where unexpected
INJ injection-adjacent (handling, not exploitation): quotes, angle brackets,
template syntax, path traversal shapes, format-string tokens
LOC locale traps: Turkish dotless-i casing, German ß expansion, Indic
grapheme clusters, non-Gregorian calendar dates, comma decimals
Seed from BLNS, then expand with the LLM per domain. BLNS covers generic hostility; your domain needs bespoke members:
For a payments form with fields (cardholder_name, UPI VPA, amount in INR,
remarks), generate 10 adversarial values PER category from this taxonomy
(attached). For each: value, category tag, WHY it is adversarial (one line),
and EXPECTED graceful behavior (reject-with-message / accept-and-store-intact
/ truncate-visibly). Output as JSON array.
Include Indic-script and Hindi/Tamil cases for name fields, VPAs with
confusable unicode, amounts exercising paisa rounding.
The expected behavior column is the difference between a fuzz list and a test dataset — it forces the product question "what should happen?" per input, which is where half the findings come from before a single test runs.
Curate like a reviewer, not a collector. Models pad — near-duplicate emoji rows, "adversarial" strings that are merely unusual. Dedupe per category, verify the WHY claims (is that string actually bidi? check), and cap each category at members that earn their runtime. Store as versioned data:
{ "id": "ENC-014", "value": "Zoë👩👩👧👦 نامدار",
"category": "ENC", "why": "ZWJ family emoji + RTL run inside LTR name",
"expected": "accept-and-store-intact; renders without bidi spillover" }
Wire the corpus into parameterized tests asserting graceful failure. The assertion is never "no 500" alone — it is the expected-behavior column:
@pytest.mark.parametrize("case", load_adversarial("ENC", "LOC"))
def test_name_field_handles_hostile_input(case, api):
r = api.create_customer(name=case.value)
if case.expected == "reject-with-message":
assert r.status == 422 and r.json["error"]["field"] == "name"
else: # accept-and-store-intact
assert r.status == 201
assert api.get_customer(r.id).name == case.value # round-trip intact
The round-trip assert catches the classic silent corruptions: NFC/NFD drift, truncated grapheme clusters, emoji butchered by a MySQL utf8 (not utf8mb4) column.
Add the property-based layer for the space between your examples. The corpus is enumerable; Hypothesis explores the gaps and finds new corpus members:
@given(st.text(min_size=0, max_size=200))
def test_name_never_corrupts(api, s):
r = api.create_customer(name=s)
assert r.status in (201, 422) # never 500
if r.status == 201:
assert api.get_customer(r.id).name == s # round-trip law
Every Hypothesis-found failure gets minimized, fixed, and its shrunken example promoted into the versioned corpus with a tag — the two layers feed each other permanently.
Report coverage by category, and route findings by risk. "ENC: 22 cases, 2 failures; TIME: 14 cases, 1 failure (DST double-booking)" is a robustness statement a release decision can use. INJ-tagged findings that look exploitable (reflected markup, executed template syntax) route to the security team immediately — your dataset probes handling, but what it catches may outrank QA.
The taxonomy and the expected-behavior column do the real work. Hostile strings are free; knowing which risk each probes and what graceful looks like is what turns them into regression armor — and into coverage numbers a release gate can actually read.
<script> is stored intact is correct for a storage API and a security hole in a rendering path. Expected behavior is per-surface; when in doubt, ask security.Adversarial corpora are synthetic by nature — no masking needed — but two cautions apply on client work: generated injection-adjacent strings in a shared repo can trip client security scanning (annotate the corpus as test data, exclusion-listed with security's sign-off), and oversized-input tests need environment-owner approval before running against client-hosted staging. Local models generate ENC/TIME/NUM categories perfectly well if external tools are barred; BLNS and Hypothesis are offline already.
Run the corpus against every new input surface in its first PR, not after launch. A parameterized run over an existing, tagged dataset costs minutes in review and turns "we'll harden it later" into a checkbox that is cheaper to tick than to skip — the rare case where rigor is also the path of least resistance.