Generate Ingestion Parsers and Connector Scaffolds from Samples

The trick is adversarial testing — the assistant writes the parser, and then you make it write the test cases designed to break its own parser.

When to reach for this

Reach for this when a new feed arrives in a format that needs custom parsing, or when you need connector scaffolding (pagination, rate limits, retries) for an API with published docs. It is the fastest AI win in daily ingestion work because the input is small, self-contained, and easy to verify.

Triggers:

• •
  A vendor file with fixed-width fields, embedded delimiters, or multiline records.
• •
  Regex extraction from semi-structured logs or filenames.
• •
  A REST API to ingest, where docs exist but no off-the-shelf connector does.

The workflow

1. 1.

   Curate representative samples. Take 10–20 real records including the ugly ones — the row with a quote inside a quoted field, the record with the missing optional column, a non-ASCII name. Mask any sensitive values but preserve structure exactly (do not "clean up" whitespace).

2. 2.

   Prompt with samples plus explicit failure behavior.

   Copy

   Write a Python function parse_settlement_line(line: str) -> dict for the
   fixed-width settlement file sampled below.
   
   Layout (from the vendor spec):
   - cols 1-10: account_id (right-padded)
   - cols 11-18: settle_date YYYYMMDD
   - cols 19-30: amount, implied 2 decimals, leading zeros, "-" suffix = negative
   - cols 31-33: currency ISO-4217
   
   Rules:
   - Return Decimal for amount, datetime.date for settle_date.
   - On any malformed field, raise ParseError with the field name and raw
     value — never guess, never silently skip, never coerce.
   - No dependencies outside the standard library + decimal.
   
   Sample lines:
   [paste 10-20 raw lines]
   

3. 3.

   Make it attack its own code. In the same conversation:

   Copy

   Now write pytest cases designed to BREAK this parser: truncated lines,
   amount overflow, "-" suffix handling, non-numeric dates, empty account_id,
   a line with trailing whitespace, and a Unicode character in the account
   field. Include the expected behavior for each per the rules above.
   

   Run the tests. Failures here are the review — fix the parser (in-conversation, scoped) until they pass.

4. 4.

   Verify against volume, not just samples. Run the parser across a full historical file and assert the reconciliation invariants inline:

   Copy

   -- after loading parsed output to a staging table
   SELECT COUNT(*) AS parsed_rows,
          SUM(amount) AS total_amount
   FROM stg_settlements;
   -- compare parsed_rows to `wc -l` of the file and total_amount
   -- to the control-total trailer record the vendor provides
   

5. 5.

   For API connectors, paste the docs — not a memory of them. Give the assistant the actual pagination and rate-limit sections from the API documentation and require: cursor-based pagination handled to exhaustion, Retry-After honored on 429s, exponential backoff with jitter, and idempotent checkpointing so a rerun resumes rather than duplicates. Ask it to enumerate which doc section justifies each behavior — an unjustifiable behavior is a hallucination.

6. 6.

   Dry-run the connector against a sandbox or low-volume window and check the checkpoint file survives a mid-run kill and resume without duplicating records.

Verify before you trust it

• •
  Adversarial test suite (step 3) passes, and you added at least two cases the assistant did not think of.
• •
  Full-file run: parsed row count matches file line count minus header/trailer, exactly.
• •
  Control totals (sum of amounts, record counts from the vendor trailer) match to the cent.
• •
  Malformed-input behavior confirmed by hand: feed one corrupted line, watch it raise, not coerce.
• •
  Connector kill-and-resume test produces zero duplicate records (check by unique key count vs total count).

Pitfalls

• Silent coercion. Unprompted, generated parsers "helpfully" default malformed amounts to 0 or skip bad lines. That is data corruption with extra steps — the explicit raise-on-error rule prevents it.
• Overfitted regex. A pattern that matches all twenty samples can still miss the format's legal variations. Test against the vendor spec's rules, not just your samples.
• Hallucinated API parameters. Assistants invent plausible query params and header names for APIs they half-remember. Everything must trace to the pasted docs (the step-5 justification list enforces this).
• Pagination that stops early. Off-by-one cursor logic passes small tests and drops the last page in production. Verify total fetched count against the API's reported total where available.
• Float money. Generated code loves float for currency. Demand Decimal in the prompt and grep the output for float( before running.

Tools

• Claude / any frontier LLM chatideal for the paste-samples-generate-attack loop; long context handles full API doc sections.
• GitHub Copilotfine for in-editor parser drafting once the pattern is established; weaker at the adversarial-test step, so drive that deliberately.
• Airbyte connector builder (AI assist)turns API docs into a declarative connector for standard REST patterns; use custom code only where it does not fit.

Client-safe variant

Vendor file layouts and API docs are usually not sensitive, but sample records almost always are. On client work, generate the parser from the layout spec plus synthetic sample lines you construct to match the spec (or client samples with every value replaced, structure intact), via approved-tenancy Bedrock/Azure OpenAI or a local Ollama model. Run the adversarial tests and full-file control-total verification inside the client environment on real data — the parser is proven where the data lives, while the prompt never contains a real record.