LLMs generate realistic-looking test data effortlessly — and violate your schema, your foreign keys, and your business rules just as effortlessly. Schema-constrained generation flips the trust model: the model proposes rows, Pydantic and the database dispose, and a validation loop feeds every rejection back until the dataset is provably conformant.
Tier: Core · Workflow area: Test data & environments
This playbook builds a repeatable data factory: LLM generation for realism and edge-case intent, hard validation against Pydantic models and relational constraints, deterministic tools (Faker/SDV) for bulk, and a versioned output that seeds environments identically every run.
user1@test.com × 500.Write the schema as executable truth first. Every rule you want enforced must live in the Pydantic model — the LLM prompt is documentation, the validator is law:
class Customer(BaseModel):
id: UUID
email: EmailStr
full_name: str = Field(min_length=1, max_length=120)
locale: Literal["en-IN", "hi-IN", "ta-IN", "en-US", "de-DE"]
created_at: datetime
class Order(BaseModel):
id: UUID
customer_id: UUID # FK enforced at load time
status: Literal["pending", "paid", "shipped", "cancelled", "refunded"]
total: Decimal = Field(ge=0, max_digits=10, decimal_places=2)
placed_at: datetime
@model_validator(mode="after")
def paid_before_shipped(self): ... # cross-field business rule
Prompt for slices, not the world — with the schema in context. Generate in themed batches so intent stays reviewable:
Generate 30 customers as a JSON array conforming EXACTLY to this Pydantic
model (attached). Requirements:
- Locale mix: 40% en-IN, 15% hi-IN, 15% ta-IN, 20% en-US, 10% de-DE,
with names plausible for each locale (scripts included for hi-IN/ta-IN).
- Include deliberate edge members: 1 name at exactly 120 chars, 1 single-char
name, 2 emails with plus-addressing, 1 with an IDN domain.
- created_at spread over the last 3 years, none in the future.
Output raw JSON only — no markdown, no commentary.
The model's actual contribution is realism and edge-case intent — locale-plausible names in the right scripts are exactly what Faker does poorly and tests involving rendering, sorting, and search actually need.
Validate hard; loop rejections back verbatim. The factory's core is a while-loop, not a prayer:
for attempt in range(3):
rows, errors = validate_batch(raw_json, Customer)
if not errors: break
raw_json = llm(f"These {len(errors)} records failed validation. "
f"Errors: {errors_as_json}. Regenerate ONLY the failing "
f"records, corrected. Same rules, raw JSON only.")
Typical loop output — this is the flip working as designed:
batch customers-03: 30 generated, 27 valid
rejected: idx 7 email 'anita@localhost' not valid EmailStr;
idx 12 created_at 2027-01-03 in future; idx 22 locale 'en-GB' not allowed
retry 1: 3 regenerated, 3 valid -> batch complete (30/30)
Enforce referential integrity by loading, not by trusting. Generate parents before children, hand the child-generation prompt the actual parent IDs, then prove the whole graph in the scratch DB:
psql testdata -f load_customers.sql -f load_orders.sql # FKs on, constraints on
# any FK violation = generator bug; fix the pipeline, never hand-edit the data
Split labor: LLM for the interesting 10%, Faker for the bulk. Ten thousand background orders come from Faker templates seeded from the LLM's realistic exemplars; the LLM hand-crafts the personas, edge members, and scenario-critical rows your tests assert against. This keeps cost near zero and the interesting rows curated.
Version the factory and pin the seed. Scripts, prompts, schema, and either the generated dataset or a deterministic regeneration path (fixed Faker seed + cached LLM outputs) live in git. Tests reference dataset versions (testdata-v7), so a failure on Tuesday is reproducible on Thursday. Schema migrations break the factory loudly — a Pydantic model that no longer matches the DB fails at load, which is the drift alarm doing its job.
Never trust, always verify — the LLM is a fluent intern with no memory of your constraints. Everything it emits passes through Pydantic and a real constraint-checking load before any test sees it. The validation loop is not overhead; it is the product.
testdata-v7 from the repo yields identical data (seeds pinned, LLM outputs cached).example.com, .test), phone numbers in reserved ranges.customer_ids "matching" the parents works until it doesn't; pass real IDs in, and let the DB load be the proof.Fully synthetic generation is itself the privacy-safe path — no production extract is ever prompted, so DPDP/GDPR exposure is designed out rather than mitigated. On client engagements: run generation on an approved endpoint or local Ollama, keep the client's schema files inside their repo boundary, and document the factory (schema-as-code, validation loop, reserved identifier ranges) as the evidence pack for their data-protection review. If distributions must mirror client production, mask first (see the PII-masking playbook), train SDV on the masked sample, and keep the LLM away from the extract entirely.
Add one deliberately invalid dataset to the factory — rows that violate each constraint in a known way — and keep it out of the seed path. It is the perfect fixture for testing your own validators, ETL error handling, and API 4xx paths, and the factory that builds good data builds its evil twin for free.