Synthetic and Edge-Case Data Generation for Testing and Rare-Class Augmentation

Tier: Core · Workflow area: Synthetic data & test fixtures

LLMs excel at schema-constrained generation and adversarial edge-case ideation; the craft is validation — proving synthetic rows are structurally valid, distributionally plausible, and never allowed to contaminate the test set.

When to use

• •
  Building pipeline test fixtures when real data is restricted, scarce, or slow to provision (pairs with pb-09/pb-10).
• •
  Stress-testing pipelines with edge cases production will eventually produce.
• •
  Augmenting a rare class (fraud, defects, rare intents) where collection is expensive.
• •
  Developing against realistic-but-fake data outside the secure environment.

Never use synthetic data in evaluation — evaluation is always on real data.

Prerequisites

• •
  Pydantic for schema enforcement; Faker for identifier-like fields; optionally SDV for statistical tabular synthesis at scale.
• •
  An LLM API for semantic fields and edge-case ideation (cheap-tier models suffice).
• •
  The real data's profile (pb-01) and validation schema (pb-10) — synthetic data is validated against the same contract as real data.

The workflow

1. 1.

   Encode the schema as Pydantic — the generator's hard boundary:

   Copy

   from pydantic import BaseModel, Field
   from typing import Literal
   from datetime import datetime
   
   class SupportTicket(BaseModel):
       ticket_id: str = Field(pattern=r"^TK-\d{8}$")
       created_at: datetime
       channel: Literal["email", "chat", "phone", "social"]
       product_area: Literal["billing", "auth", "api", "mobile", "other"]
       subject: str = Field(min_length=8, max_length=120)
       body: str = Field(min_length=30, max_length=2000)
       priority: Literal["p1", "p2", "p3", "p4"]
       csat: int | None = Field(default=None, ge=1, le=5)
   

2. 2.

   Split fields by generator. Identifiers, timestamps, and numeric draws go to Faker/NumPy (cheap, distribution-controlled); only semantic fields (subject, body) go to the LLM. Sending everything to the LLM wastes tokens and produces subtly non-random "random" numbers.

3. 3.

   Generate semantic fields with distribution and diversity constraints:

   Copy

   Generate 40 support-ticket (subject, body) pairs for product_area=billing,
   priority mix 10% p1 / 30% p2 / 60% p3-p4. Realism constraints: Indian and
   international customer names and locales; mixed tone (angry, confused,
   polite); include typos in ~20%; body must be consistent with subject and
   priority (p1 = outage/blocked language). Vary length: 30% short (<50 words).
   Output as a JSON array matching keys: subject, body, priority.
   Avoid repeating sentence templates across items.
   

   Validate every batch through the Pydantic model with a retry loop; log the rejection rate (rising means prompt and schema disagree).

4. 4.

   Ideate edge cases adversarially:

   Copy

   You are attacking the ticket-ingestion pipeline described by this schema and
   these transforms [paste pb-10 contract]. Generate 30 edge-case rows that are
   SCHEMA-VALID but likely to break processing: emoji-only subjects, RTL text,
   HTML/SQL fragments in body, body in Hindi/Tamil, max-length fields,
   csat null vs missing, midnight/DST timestamps, duplicate ticket_ids across
   batches. For each: the row + one line on what it tests.
   

   Each edge case that breaks something graduates into a pb-10 regression test.

5. 5.

   For rare-class augmentation, generate against the real minority profile. Give the LLM (or SDV, for numeric-heavy tables) the minority class's statistical profile and a few anonymized exemplars, generate candidates, then filter hard: schema-valid, passes the pb-10 validation suite, and — critically — a discriminator check: train a quick real-vs-synthetic classifier; AUC ≈ 1.0 means your rows are stylized caricatures that teach the model to detect synthetic-ness, not the class.

6. 6.

   Enforce the containment rule. Synthetic rows are tagged with an is_synthetic column that survives every join; used in training folds only; never in validation folds, test sets, or reported metrics. Measure value the only honest way — train with and without synthetic rows, compare on untouched real-data test metrics, report both.

7. 7.

   Document provenance. A datasheet per synthetic set: generator versions, prompts, seed, counts, validation pass rates, discriminator AUC, intended use. Undocumented synthetic data becomes indistinguishable from real data in a warehouse within a quarter — a governance incident, not a convenience.

Tool variants

• LLM + Pydantic (this playbook's core): best for text fields and edge-case ideation; batch APIs cut costs at volume.
• SDV / copulas: best for tabular numeric/categorical synthesis preserving joint distributions at scale; no LLM needed; weak on free text — combine: SDV for the table, LLM for text columns.
• Faker-only: sufficient for fixtures where semantic realism is irrelevant; zero cost, perfect control, zero leakage risk.

Verification

• •
  100% of shipped synthetic rows pass the same pandera/Pydantic validation as real data.
• •
  Distribution check: key marginals and 2-3 important joints within tolerance of the target profile — a KS/chi-square table, not eyeballing.
• •
  Discriminator AUC on real-vs-synthetic is meaningfully below 1.0 for augmentation sets (fixtures are exempt — they do not need to fool anyone).
• •
  Augmentation evaluated on real-only test data, with and without synthetic training rows, both numbers reported.
• •
  is_synthetic tag present and verified to survive the full pipeline.

Pitfalls

• LLMs as samplers. Asked for "uniform random amounts," an LLM produces round numbers and suspicious clusters. Numbers come from NumPy; language comes from LLMs.
• Mode collapse in bulk generation. Batch 40 looks diverse; rows 400-4,000 recycle templates. Rotate personas, scenarios, and seeds across batches; monitor n-gram overlap.
• Privacy laundering. Generation prompted with real exemplars can regurgitate them — screen outputs for verbatim overlaps with source rows; "synthetic" is not automatically "safe to share."
• Teaching the model the generator. Over-clean synthetic minority rows create a classifier that detects your prompt style. The discriminator check exists for this.
• Fixture rot. Fixtures frozen while the schema evolves; regenerate from the current Pydantic contract in CI so drift fails loudly.

Restricted-environment variant

• •
  Fixtures-from-schema is the restricted environment's best friend: generation needs only the contract, never real rows — develop against synthetic data outside the secure zone.
• •
  For augmentation where exemplars cannot reach hosted models: use aggregate profiles plus locally-run open models, or SDV entirely locally.
• •
  Any synthetic set derived from real exemplars inherits the source's classification until a privacy review (including the verbatim-overlap screen) clears it — default closed.

Success criteria

• •
  A validated fixture set + adversarial edge-case suite in the pipeline tests, generated in under half a day.
• •
  At least one edge case broke the pipeline before production data did.
• •
  If augmenting: measured real-test-set lift (or an honest null result), containment verified.
• •
  Every stored synthetic set answers "where did this come from?" via its datasheet.