dbt-legacy-migration-skills

dbt legacy migration skills

A set of agent skills that migrate legacy ETL/ELT workloads to dbt - mapping the original workload, translating it into dbt with best practices (tests, docs, contracts), validating the result against real warehouse data, choosing cost-aware materializations for the target cloud, and reporting a legacy-vs-dbt cost comparison. Each targets ≥95% coverage of the migratable workload, with the residual explicitly flagged for human review.

Built for the dbt Wizard CLI; the skills are plain SKILL.md folders and work in any agent that supports the skill format (e.g. Claude Code).

Skills

How they fit together

The six migration skills share the same 8-step workflow and defer the common steps to legacy-to-dbt-migration-foundations:

• Step 0 - Detect environment & cloud (Snowflake / Databricks / BigQuery / Redshift; Fusion vs Core)
• Step 1 - Inventory & map the legacy workload (source-specific)
• Step 2 - Choose the target modeling approach (see below), then classify the workload into it + detect Mesh
• Step 3 - Translate to dbt SQL for the chosen modeling approach, with cost-aware materializations (source-specific answer key)
• Step 4 - Apply best practices: tests (arguments: spec), docs, contracts, snapshots
• Step 5 - Validate: dbt compile gate (free), then legacy-prod vs dbt-dev data parity (explain every difference)
• Step 6 - Cost comparison: measured warehouse consumption (legacy vs dbt), auditable
• Step 7 - Coverage report (confirm ≥95%, flag residual)
• Step 8 - Document changes in migration_changes.md

Each migration skill carries a references/ folder with two source-specific docs: how to parse the source artifact, and the component/transformation → dbt answer key. All generated SQL is dbt Fusion-conformant (cast() not ::, coalesce() not nvl/ifnull, tests via the arguments: nested spec).

Uses the dbt package ecosystem (all Fusion-checked): codegen to scaffold sources/models/YAML, audit_helper to prove legacy-vs-dbt parity, dbt_utils + dbt_expectations for tests, dbt_project_evaluator as the quality gate, datavault4dbt/dbt_date per modeling approach - see skills/legacy-to-dbt-migration-foundations/references/dbt-packages.md.

Target modeling approach (Step 2)

After mapping the legacy workload, the skill asks which target modeling approach to build, then generates the dbt models accordingly:

• Layered (default) - faithful source-aligned staging → intermediate → mart. Lowest risk.
• Data Vault 2.0 - hubs / links / satellites via the datavault4dbt package, with dimensional info marts on top.
• Kimball dimensional - conformed dimensions + fact tables, SCD2 via snapshots, surrogate keys.
• Star schema (pragmatic) - facts + dimensions for a focused subject area.

The generation for each paradigm lives in the shared foundations skill's references - not as separate skills - so the install is just the six migration skills + foundations: skills/legacy-to-dbt-migration-foundations/references/target-modeling.md (legacy→paradigm mapping + performance), building-datavault.md, building-kimball.md, building-starschema.md.

Data Vault path: building-datavault.md is distilled from Scalefree's datavault4dbt agent skills (Apache-2.0) - see ATTRIBUTION.md. The datavault4dbt package itself is installed on demand when Data Vault is chosen (see dbt-packages.md). For deeper Data Vault coverage, install Scalefree's full skill set separately. Kimball and star need nothing extra.

Install

Run these in your terminal, not inside the dbt Wizard prompt. Installing skills is just file copying into ~/.dbt/wizard/skills/; it needs no login and no model call. Pasting the commands into the Wizard chat only makes the Wizard spend a model call to run shell for you (and fails if your dbt Platform auth is not set up). After installing, restart the Wizard so it reloads the skills.

The skills live under skills/<name>/, the standard Agent Skills layout, so any skills tool can install them straight from GitHub, from anywhere:

npx skills add hicham-bab/dbt-legacy-migration-skills   # vercel-labs/skills
# or:  gh skill install hicham-bab/dbt-legacy-migration-skills

Or the bundled one-liner (installs/updates all eight skills):

git clone https://github.com/hicham-bab/dbt-legacy-migration-skills.git
cd dbt-legacy-migration-skills && ./install.sh          # dbt Wizard CLI (~/.dbt/wizard/skills)
#                                  ./install.sh --claude # Claude Code   (~/.claude/skills)
#                                  ./install.sh --codex  # Codex CLI     (~/.codex/skills)

install.sh copies just the eight skill folders (from skills/) into your skills directory and leaves everything else untouched. If you run it outside a clone it fetches the skills from GitHub for you. It's safe to re-run to update; it cleanly replaces only these folders. (Use --dest <path> for a custom location.) After it finishes, restart the agent so it reloads the skill list.

cp -R skills/* ~/.dbt/wizard/skills/

Install all seven together - the five migration skills reference legacy-to-dbt-migration-foundations by relative path, so it must sit alongside them.

Using dbt Wizard on the dbt platform (Studio)

Wizard in the dbt platform does not read ~/.dbt/wizard/skills. It discovers repo-level skills from an .agents/skills/ directory at your dbt project root, scanned at the start of each chat (docs). Studio is a browser IDE with no general shell, so you do not run install.sh (or any of the commands above) inside the platform. Instead, get the seven skill folders into your project repo's .agents/skills/ one of two ways.

Via git (recommended) - in a local checkout of your dbt project (on your machine, outside Studio), add the skills and push:

# run in a local clone of YOUR dbt project, not in Studio
git clone https://github.com/hicham-bab/dbt-legacy-migration-skills.git /tmp/dbt-mig
mkdir -p .agents/skills && cp -R /tmp/dbt-mig/skills/* .agents/skills/
git add .agents/skills && git commit -m "Add dbt legacy-migration skills" && git push

Then in Studio, pull the branch using the built-in git controls (the git button / wayfinder bar handles branch/commit/push/pull; there is no shell). (/tmp/dbt-mig/install.sh --dest "$(pwd)/.agents/skills" does the same copy if you prefer it, but it still runs on your machine, not in the platform.)

Or in the Studio file explorer - create .agents/skills/<skill>/SKILL.md (plus any references/ files) for each skill by hand, pasting the contents from this repo. Fine for one skill, tedious for all seven.

Either way, then:

1. Start a new dbt Wizard chat - skills load at session start, so mid-session edits are not picked up until you open a new chat.
2. Reference one by name, e.g. Use migrating-informatica-to-dbt to migrate models/legacy/wf_orders.XML.
3. Keep .agents/skills/ committed so the whole project team (and every Wizard session on the repo) gets them. A custom skill overrides a built-in one of the same name.

Keep all seven together so legacy-to-dbt-migration-foundations sits alongside the migration skills.

Using with Claude Code or Codex

The same seven skill folders work in other Agent-Skills-compatible CLIs; only the directory differs (each agent scans its own location, not .agents/skills):

• Claude Code - ~/.claude/skills/ (all your projects) or .claude/skills/ in a repo (that project only). Run ./install.sh --claude. Claude Code watches the folder, so a newly added skill is picked up in the current session (restart only if ~/.claude/skills did not exist when the session started). Then just ask, e.g. "use migrating-talend-to-dbt to migrate ...".
• Codex CLI - ~/.codex/skills/ (all your projects) or .codex/skills/ in a repo. Run ./install.sh --codex. Invoke with $migrating-talend-to-dbt, or let it auto-activate from the skill's description.

All seven install together (foundations must sit alongside the five migration skills), and re-running install.sh is safe to update.

Iterating on the skills (frequent edits / beta)

If you are actively changing the skills, symlink instead of copy so your working clone is the live source and you never re-install on each edit:

./install.sh --link            # symlinks into ~/.dbt/wizard/skills; combine with --claude / --codex / --dest

Then edits in skills/ are live: dbt Wizard and Codex pick them up when you start a new session; Claude Code re-reads changed skills within the session (no restart). --link needs a local clone (it points the symlinks at your checkout). Run a plain ./install.sh again to switch back to a real copy. For the dbt platform (Studio) there's no symlink, iterate by committing to .agents/skills/ and starting a new chat.

Usage

Once installed, ask the agent to migrate a project and point it at the source artifacts, e.g.:

• "Migrate this Informatica workflow export to dbt" (point at the .XML)
• "Migrate these Talend jobs to dbt" (point at the .item folder)
• "Migrate this stored procedure to a dbt model and prove parity"
• "Migrate this Matillion pipeline to dbt" (point at the .tran.yaml/.orch.yaml or METL JSON)
• "Migrate this Coalesce project to dbt" (point at the cloned Coalesce Git repo)

The skill runs the 8-step workflow, asks which cloud you're targeting, and produces the dbt models, tests, docs, a parity check, a cost comparison, and a coverage report.

Set the approval posture up front. A migration is a multi-step run (it writes models and runs dbt), so decide once at the start instead of confirming every edit. To run without a prompt at each step, use a workspace-scoped sandbox with approvals off:

dbt-wizard --sandbox workspace-write --ask-for-approval never

workspace-write confines writes to your project; never means the Wizard applies edits and runs dbt without asking (a command that would need to leave the sandbox just fails instead of prompting). The default policy, on-request, is what makes the Wizard ask "Would you like to make the following edits?" at each step; if you stay in that mode, press a ("don't ask again for these files") to stop the prompts for the current files. --dangerously-bypass-approvals-and-sandbox removes the sandbox entirely and is for externally sandboxed / CI runs only.

Scope & caveats

• Only transformation logic migrates to dbt. Ingestion/extract-load (Informatica sessions loading files, Talend tFileInput/tFTP, Matillion S3 Load / connector queries / CDC capture) and control-flow/scheduling have no model equivalent - the skills map those to sources and a scheduler, and exclude them from the coverage denominator rather than force-fitting them.
• Compilation is not correctness. Every skill treats dbt compile as the fast iteration gate and data parity against the warehouse as the proof a migration preserved business logic.
• Dynamic SQL, non-deterministic procedural logic, and proprietary runtime components are routed to the flagged residual, not guessed at.

Eval results

Every push and PR runs a deterministic eval harness over the skills' inventory parsers (the step that reads a legacy export and produces the workload inventory). CI fails on any regression, so the badges above reflect the live state of main.

• Published breakdown: evals/RESULTS.md - per-check results by source, refreshed by CI on every push to main.
• How it's checked: evals/README.md - fixtures, methodology, and the "verified against real vendor exports" discipline behind the numbers.
• Reproduce locally: pip install pyyaml && python3 evals/run_evals.py (add --write to regenerate the report + results.json).

Two layers of evals:

• Parser evals (above) - deterministic unit checks over the inventory parsers, gated in CI.
• Agent eval + scorecard - harbor/ holds Harbor tasks that run a whole agent following the skills against real migrations in a container (targeting DuckDB, no cloud creds) and grade the output on a weighted scorecard: build → data parity (row-for-row vs a held-out expected output) → coverage → structural (idiomatic-dbt signals) → LLM-as-judge (SCD correctness, decomposition, docs/test quality). Five tasks - stored procedure, Talend, Informatica, Matillion, and Coalesce (incl. a Type 2 SCD → dbt snapshot) - each verified solvable: the oracle scores 100% / reward 1, a broken migration drops parity and scores reward 0. Run the aggregate smoke test with python3 harbor/_scorer/run_all.py.

License

Apache-2.0. See LICENSE.