BXUP Track progress & test yourself — free
Data Engineering · Playbook · Advanced · ~5 min read
DATA ENGINEERING · PLAYBOOK · ADVANCED

Migrate a Legacy ETL Platform with Transpilers plus LLM Refinement

Informatica, SSIS, and DataStage estates are the revenue engine of services-company data work, and the modern migration stack is hybrid: deterministic converters (Lakebridge, SnowConvert AI's SSIS-to-dbt path, BladeBridge-class transpilers) do the bulk mechanical conversion, an LLM pass handles residual and custom logic, and generated parity tests prove the result.

This playbook is the end-to-end operating procedure for one migration wave — assessment through parity sign-off — with the LLM in a bounded, verifiable role.

When to reach for this

Use this when a mapped scope of legacy ETL jobs must land on dbt or a lakehouse, and the estate is too large for hand-rewriting but too custom for pure transpilation. It assumes you already know how to reverse-engineer individual gnarly components (see the stored-procedure archaeology playbook) and generate parity suites (see the reconciliation playbook) — this one sequences the whole wave.

Triggers:

  • A modernization statement of work with hundreds of mappings/packages and a deadline.
  • A transpiler pilot returned 60–85% auto-conversion and someone must own the rest.
  • Legacy platform licenses expiring — a hard cutover date exists.

The workflow

  1. Assess and slice into waves. Run the transpiler's analyzer (Lakebridge analyzer, SnowConvert assessment) across the estate to get per-job complexity and feature usage. Slice waves by dependency cluster, not alphabetically: a wave must be independently testable end-to-end. Park the top 5% most complex jobs for engineer-led rework — they will eat your schedule if hidden inside a wave.

  2. Transpile the wave and disposition the report. Every job lands in one of three buckets: clean conversion (goes straight to parity testing), flagged residue (LLM refinement, next step), or redesign (custom scripts, external calls, platform-specific components with no target equivalent — human design first).

  3. LLM-refine the residue with source semantics pinned. For each flagged component, give the assistant the legacy component's exported definition (XML/JSON) plus the transpiler's partial output:

    This SSIS Derived Column transformation was partially converted.
    Below: the original component XML, the transpiler's dbt output, and
    the transpiler's warning.
    
    Complete the conversion to dbt SQL. Rules:
    - Replicate SSIS semantics exactly, including its NULL propagation
      and type-cast behavior; comment each place target defaults differ.
    - Preserve the transpiler's naming and structure — you are completing
      its output, not replacing it.
    - If the component's behavior cannot be determined from the XML alone
      (e.g., a variable set at runtime), stop and list what you need.
    
    [paste XML + partial output + warning]
    

    Verify each refinement immediately with a component-level fixture: run representative input rows through the new SQL and compare against the legacy component's recorded output for the same rows. Do not defer verification to wave-level testing — a wrong component hides inside aggregate parity for weeks.

  4. Reconstruct orchestration deliberately. Control-flow (precedence constraints, event handlers, retry logic) does not transpile cleanly into DAGs. Have the LLM draft the dependency graph from the package definitions, then review it against the legacy scheduler's actual run history — the documented dependencies and the real ones often differ:

    -- real execution order from legacy run logs
    SELECT job_name, MIN(start_time) AS typical_start
    FROM legacy_run_history
    WHERE run_date = DATE '2026-07-01'
    GROUP BY job_name ORDER BY typical_start;
    
  5. Run wave-level parity. Execute legacy and new pipelines against the same frozen source snapshot; compare every target table with the three-level pattern (row counts, aggregate fingerprints, sampled full rows — see the reconciliation playbook for generation). Mismatches route back to step 3 scoped to the offending component, never to wholesale regeneration.

  6. Parallel-run before cutover. Both systems live on real daily data for an agreed window (typically 5–10 business days), with automated daily parity checks and a divergence log. Every divergence gets a written disposition: translation bug (fix), legacy bug faithfully not reproduced (client decision), or timing artifact (documented). Cutover requires N consecutive clean days — put the number in the plan.

Warning

Vendor automation rates are measured on syntax, not on your estate's custom scripts, external commands, and embedded business logic — which is precisely where migrations slip. Budget the residue at 3–5x the effort per job of the clean bucket, and protect the parallel-run window from schedule compression; it is the only stage that proves the migration rather than asserting it.

Verify before you trust it

  • Transpiler report 100% dispositioned per wave; the redesign bucket has named owners and designs.
  • Every LLM-refined component has a passing component-level fixture comparison against legacy output.
  • Wave parity: all target tables pass three-level comparison on the frozen snapshot; decimals exact.
  • Orchestration graph validated against legacy run history, not just package metadata; SLA-critical paths timed on the new platform.
  • Parallel run: N consecutive clean days with the divergence log at zero undispositioned entries.
  • Rollback rehearsed: legacy remains runnable until sign-off, and someone has actually re-pointed a consumer back once, in test.

Pitfalls

  • Hidden logic in expressions and scripts. Informatica expression transformations and SSIS script tasks carry decades of business rules the transpiler flags or mangles. That residue is the project, not an appendix to it.
  • LLM "completing" ambiguous components by guessing. Runtime variables, environment-dependent connections, and parameterized SQL cannot be inferred from static definitions. The stop-and-list-what-you-need rule exists to prevent confident fabrication.
  • Legacy bug fidelity. The old pipeline truncates instead of rounding; the new one rounds. Parity fails, and the fix is a client decision, not a code change. Log these — reproducing a bug on purpose needs a paper trail.
  • Dependency drift between doc and reality. Packages document one execution order; ops has been running another via manual holds for years. Run-history reconstruction (step 4) is what catches it.
  • Parity against a moving source. Testing while source data changes turns every diff into an argument. Frozen snapshots per wave, refreshed deliberately, keep evidence trustworthy.

Tools

  • Databricks Lakebridgeopen-source analyzer, transpilers (including LLM-based conversion), and a reconcile module; the free-path default for Databricks targets.
  • SnowConvert AIfree assessment and conversion including the SSIS-to-dbt route for Snowflake-bound estates.
  • Claude Code / frontier LLMthe residue-refinement and orchestration-drafting layer, driven with the pinned-semantics prompts above.

Client-safe variant

Legacy ETL definitions encode client business logic end to end, so the default posture is everything-in-tenancy: run Lakebridge/SnowConvert inside the client environment (both work offline), and route LLM refinement through Bedrock or Azure OpenAI in the client's own cloud with logging enabled — this is the pattern most modernization MSAs accept. Where no in-tenancy LLM exists, genericize exported component definitions with a scripted identifier mapping before prompting a local Ollama model, and restore names on output. Snapshots, fixtures, and the parallel-run divergence log all live in the client's platform, forming an audit trail that stands on its own regardless of which AI assisted.

This is the free reading view. Open BXUP to track your progress, run the hands-on labs, and prove it with original, exam-pattern assessments.
© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in