Generate Bulk Model and Column Documentation from Metadata

Tier: Foundation · Workflow area: Docs, lineage & governance

This playbook turns an undocumented dbt project into one with reviewable draft descriptions for every model and column, plus a CI check that keeps coverage from decaying again.

When to reach for this

• •
  A dbt project (or warehouse schema) where documentation coverage is low and a human-written pass would take weeks.
• •
  Onboarding a new team onto an inherited project where nobody remembers what flg_x_2 means.
• •
  A catalog rollout (Atlan, Collibra, Unity Catalog) that needs seeded descriptions before anyone will use it.

Skip AI drafting for the 10–20 truly business-critical columns (revenue definitions, regulatory fields) — write those by hand first, because they become the examples the AI mimics.

The workflow

1. 1.

   Measure the gap first. Generate a coverage report so you know the before/after. In a dbt project:

   Copy

   dbt docs generate
   python -c "
   import json
   cat = json.load(open('target/manifest.json'))
   models = [n for n in cat['nodes'].values() if n['resource_type']=='model']
   undoc = [m['name'] for m in models if not m.get('description')]
   print(f'{len(undoc)}/{len(models)} models undocumented')
   "
   

2. 2.

   Give the assistant real metadata, not just names. Column names alone produce generic filler. Feed it the model SQL, upstream model descriptions, and (where policy allows) a few sample values per column. With dbt Copilot this context is automatic; with a repo-scale assistant like Claude Code or Cursor, point it at the project. Prompt:

   Copy

   Draft dbt YAML documentation for the model models/marts/fct_orders.sql.
   
   Context you must use:
   - The model SQL (read it — derive column meaning from the transformations, not the names).
   - Upstream descriptions in models/staging/stg_orders.yml — inherit and refine, do not contradict.
   - Our house style: one sentence per column, start with a noun phrase, name the grain
     in the model description, spell out abbreviations on first use.
   
   Rules:
   - If a column's meaning cannot be inferred from the SQL, write "TODO: confirm with owner"
     instead of guessing.
   - Do not invent business rules (thresholds, SLA definitions, currency assumptions).
   - Output only the schema.yml block, no commentary.
   

   Expected output: a models: YAML block with a grain-stating model description and one line per column, with explicit TODOs where inference failed. The TODO rule is the important part — it converts hallucination risk into a visible worklist.

3. 3.

   Batch across the project, one layer at a time. Document staging models first, then marts, so downstream drafts can inherit upstream descriptions. In an agentic tool: "Repeat this for every model under models/staging/ missing a description; commit one file per model."

4. 4.

   Run the human meaning-pass. AI can describe what the SQL does; only a human knows what the business means by it. Review every draft in a PR, and route domain-specific models to their owners. Budget roughly one minute per model — you are editing, not authoring, which is where the 10x comes from.

5. 5.

   Enforce coverage in CI. Add a check so the debt cannot silently return:

   Copy

   # e.g. dbt-checkpoint / dbt_meta_testing style hook
   check-model-has-description --manifest target/manifest.json --fail-under 0.95
   

   Expected outcome: PRs adding undocumented models fail until a description is added — which is now cheap, because the assistant drafts it in the same PR.

Verify before you trust it

• •
  Spot-check 10 random column descriptions against the actual SQL — every one must be derivable from the code or marked TODO.
• •
  Grep the drafts for invented specifics: numbers, currencies, timezones, SLA claims that appear in no source (grep -nE '[0-9]+ (days|hours|%)' models/**/*.yml).
• •
  Confirm inherited descriptions match upstream — a drifted copy is worse than a blank.
• •
  Re-run the coverage report and record before/after numbers in the PR description.

Pitfalls

• Plausible-but-wrong business rules. "Order total in USD including tax" reads perfectly and may be false on both counts. Anything stating a business rule needs a human confirm.
• Name-echo descriptions. "customer_id: The ID of the customer" adds nothing. Reject drafts that only restate the column name; demand the SQL-derived meaning or a TODO.
• Documenting deprecated models. Bulk passes happily polish models nobody should use. Exclude models slated for deletion first, or you legitimize them.
• Sample values leaking PII into YAML. If you paste sample data for context, use masked or synthetic rows — descriptions are committed to git forever.
• One giant PR. A 4,000-line docs PR gets rubber-stamped. Batch by domain so each owner reviews their own 20 models.

Tools

• dbt Copilotmetadata-aware generation inside dbt platform; the lowest-friction path when the team is on a paid dbt plan.
• Claude Code / Cursorrepo-scale agent for dbt Core users; reads SQL and upstream YAML directly and can commit per-model files.
• Databricks Genie Code / AI-generated commentssame pattern for Unity Catalog tables when the estate lives in Databricks.

Client-safe variant

For services-company delivery teams working in a client's project: check the client's AI policy before pointing any assistant at their repo. If public tools are prohibited, run the same workflow with an approved or self-hosted model, and never include sample data rows in prompts — metadata (model SQL and column names) is usually acceptable where data values are not, but confirm in writing. Strip client names and internal system identifiers from any prompt sent to an external API, and keep the generated YAML in the client's repo rather than your own notes. The CI coverage gate is client-friendly evidence of value: "documentation coverage went from 20% to 96%" is a reportable deliverable.