Author an Airflow 3 DAG from a Pipeline Spec

Tier: Foundation · Workflow area: Orchestration, ops & cost

This playbook turns a plain-English pipeline spec into an Airflow 3 DAG that uses the current best-practice patterns — assets, task groups, deferrable operators — with a validation loop that catches the invented operator arguments assistants love to produce.

When to reach for this

• •
  Standing up a new pipeline where you'd otherwise copy an old DAG and mutate it.
• •
  Migrating a cron script or notebook into orchestration for the first time.
• •
  Adding non-trivial structure (dynamic task mapping, task groups, asset-triggered scheduling) to an existing DAG and you don't remember the exact API.

For a one-line change to a working DAG, edit it directly — reviewing a regenerated file costs more than the edit.

The workflow

1. 1.

   Write the spec as bullets before you prompt. Sources, transforms, destination, schedule or triggering assets, retry policy, alerting, backfillability. Vague prompt in, generic DAG out.

2. 2.

   Prompt with Airflow 3 patterns named explicitly. Assistants trained on years of Airflow 2 code regress without this:

   Copy

   Write an Airflow 3 DAG (TaskFlow API) for this pipeline:
   
   - Extract: pull yesterday's orders from Postgres (existing conn id: pg_orders).
   - Transform: run dbt build --select orders_mart via our existing DbtRunOperator pattern.
   - Load-check: row-count sanity query against the mart.
   
   Requirements:
   - Airflow 3 idioms only: @dag/@task decorators, Asset-based scheduling — this DAG
     is triggered when the upstream asset s3://raw/orders/ is updated, not by cron.
   - Use a task group for the extract steps; deferrable operators for any waiting.
   - retries=2 with exponential backoff; on_failure_callback to our Slack utility
     (from include.alerts import notify_slack).
   - Idempotent per logical_date: re-running a past date must be safe (no appends
     without dedup). Say explicitly how each task achieves this.
   - No top-level heavy imports or DB calls (keep the file cheap to parse).
   - Ask before assuming anything about connections or variables I didn't list.
   

   Expected output: one DAG file plus a short list of stated assumptions. The "say how each task is idempotent" line forces the model to design for backfills instead of bolting them on later.

3. 3.

   Static-check before running. Import errors and invented operator kwargs surface for free:

   Copy

   python -c "import dags.orders_pipeline"   # parses & catches bad imports/args at definition time
   ruff check dags/orders_pipeline.py
   

   Then verify every operator and argument against the provider docs for your installed provider versions — the single most common failure is a plausible kwarg that doesn't exist in your version.

4. 4.

   Dry-run tasks locally, then a real run in dev. airflow tasks test orders_pipeline extract_orders (dag id, then task id) exercises individual tasks without scheduler state; then trigger a full run in a dev environment and read the logs for silent misbehavior: skipped-but-shouldn't tasks, sensors polling instead of deferring, an asset trigger that never fires because the producer DAG doesn't declare the outlet.

5. 5.

   Test idempotency explicitly. Run the same logical date twice and diff the target table row counts. If the second run changes anything, the "idempotent" claim was decorative — paste the evidence back to the assistant and have it fix only that task.

6. 6.

   Review as if a junior wrote it. You own this DAG at 3 a.m. If you can't explain a line (a trigger rule, a max_active_tis_per_dag), make the assistant justify or remove it before merge.

Verify before you trust it

• •
  The file imports cleanly and the DAG renders in the UI with the expected graph shape.
• •
  Every uses/operator argument exists in your installed provider version (check docs, not vibes).
• •
  A dev run completes; the asset trigger fires when the upstream updates.
• •
  Same-logical-date re-run is a no-op on the target (idempotency proven, not claimed).
• •
  No secrets in the file; connections referenced by conn id only.

Pitfalls

• Invented operator arguments. The classic failure: a kwarg from a newer/older provider version, or from a different operator entirely. Parsing catches some; docs-checking catches the rest.
• Cron nostalgia. Asked for event-driven, the model still emits schedule="0 3 * * *". If the pipeline should run when data lands, insist on Asset scheduling.
• Non-idempotent inserts. Plain INSERT-append tasks make every retry and backfill a duplication incident. Delete-insert by partition or MERGE, per logical date.
• Heavy top-level code. DB clients or API calls at module level run on every scheduler parse loop. Keep them inside tasks.
• Copy-pasted callbacks that swallow failures. A broken on_failure_callback can itself throw and mask the real error. Test the alert path by forcing one failure in dev.

Tools

• Astronomer Astro IDEAirflow-specialized AI authoring with environment awareness; the fastest path on Astro. OSS Airflow 3 + a generic assistant is the free equivalent.
• Claude Code / Cursorrepo-scale drafting that reads your existing DAGs and matches house conventions (alert utils, operator wrappers) without being told.
• Dagster + Copilotthe same spec-to-asset-graph workflow if your shop orchestrates with Dagster's asset model instead.

Client-safe variant

Client orchestration code usually encodes their system topology — connection names, bucket paths, vendor systems — which is confidential even when the logic is generic. If the client hasn't approved your assistant, describe the pipeline abstractly (placeholder conn ids and paths), generate, then substitute real identifiers inside their environment; or use a locally-hosted model for drafting. Validation is identical and stays entirely client-side (parse, airflow tasks test, dev run, idempotency re-run). Leave behind the spec-bullets format as a template — clients adopt the practice fastest when the next engineer can see spec → DAG → evidence for an existing pipeline.