{"id":"63a53b7a-3c05-48f6-b425-de4505e63a10","slug":"data-migration","kind":"skill","name":"data-migration","summary":"Prove a data migration actually landed correctly, on the destination. Row counts and a clean run are compatible with a completely wrong result. Stdlib Python, no dependencies.","owner":"shaheershoaib","repoUrl":"https://github.com/shaheershoaib/data-migration","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"skill","createdAt":"2026-08-26T23:59:33.000Z","addedAt":"2026-10-31T23:53:34.487Z","logoUrl":"https://avatars.githubusercontent.com/u/77290024?u=456ef0fce91858cc5482cde333076d3fcd901bd1&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["data-analytics","developer-tools"],"tags":["claude-code","data-engineering","data-migration","data-quality","etl"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-31T23:54:01.492Z","sample":false,"url":"https://conduits.dev/tools/data-migration","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# data-migration\n\n[![tests](https://github.com/shaheershoaib/data-migration/actions/workflows/test.yml/badge.svg)](https://github.com/shaheershoaib/data-migration/actions/workflows/test.yml)\n\nProve a data migration actually landed correctly, on the destination, where the\nevidence is.\n\nA clean extraction run, matching row counts and a load with no errors are all\ncompatible with a completely wrong result. This is the discipline for the cases\nwhere they are, plus `migration_check.py`, which makes the mechanical half\nexecutable.\n\nPure Python standard library: no dependencies, no network, no database driver.\nCSV, JSONL or JSON in - the format is the transport, not the database - so it runs\nagainst an extract, in CI, or against a fixture.\n\n```bash\npython3 migration_census.py --discover handover/ --out findings.md   # step 0 with no declarations\npython3 migration_census.py --spec census.json                       # step 0, declared\npython3 migration_check.py  --spec spec.json     # the checks that can block\n```\n\n## The problem\n\nEvery failure this skill exists to prevent looked like success at the moment it\nhappened.\n\nA migration is the work type where **the easiest thing to measure proves the\nleast**. Row counts reconcile perfectly when a column was never carried. A load\nreports zero errors when the destination does not enforce its own enums. A\ntransform runs clean over half its domain and looks identical to one that covered\nall of it. A status backfill keyed on an id that had been re-sequenced attaches\nevery row to the wrong entity, completely, with no error anywhere.\n\nThe defining property: **the correctness of the output cannot be observed from the\ninput.** So the whole discipline is about proving it on the destination.\n\n## The check that makes the point\n\nLegacy rows routinely assert two things that cannot both be true. A success flag\nwritten when an operation is *submitted* and never cleared when it later fails,\nsitting beside the status field that records the real outcome. The row says both\n\"settled\" and \"reversed\".\n\nDetecting the contradiction is the easy half. The expensive half is **which one\nwins**, and that is a business rule, not a technical one: a bounced payment is not\nmoney received.\n\n```json\n{\"exclusivity\": [{\n   \"name\": \"a reversal beats a settle-flag - a bounced payment is not money received\",\n   \"key\": {\"source\": \"id\", \"destination\": \"legacy_id\"},\n   \"destination_column\": \"status\",\n   \"states\": [\n     {\"name\": \"reversed\", \"when\": {\"payment_state\": \"REVERSED\"}, \"destination_value\": \"REVERSED\"},\n     {\"name\": \"settled\",  \"when\": {\"is_settled\": \"1\"},           \"destination_value\": \"SETTLED\"}\n   ]}]}\n```\n\n```\nFAIL  exclusivity-precedence\n      case: a reversal beats a settle-flag - a bounced payment is not money received\n        rows_checked: 4\n        source_contradictions: 2      <- the legacy system's mess, reported\n        precedence_violations: 2      <- YOUR transform resolved these the wrong way round\n        missing_in_destination: 0\n        examples: [{\"key\": \"1\", \"matched\": [\"reversed\", \"settled\"],\n                    \"expected\": \"REVERSED\", \"got\": \"SETTLED\"}]\n```\n\nNote the two separate counts. Source contradictions are a property of the legacy\ndata and do not fail the run: you inherited them. Resolving them the wrong way\nround is yours, and that fails.\n\n## What it checks\n\nDeclare only what applies. Each section of the spec is optional.\n\n| check | catches |\n|---|---|\n| `exclusivity-precedence` | two mutually exclusive states on one row, and whether the transform let the right one win - and every affected row the destination never received, counted rather than silently skipped (`allow_missing` declares deliberate skips) |\n| `key-uniqueness` / `key-uniqueness-destination` | a natural key that maps N:1, a blank/sentinel key that cannot join - and, on the destination side, a double-applied load minting duplicate keys while every value stays right |\n| `key-identity` | an id present in both systems that does not mean the same thing, scored against an independent human-readable attribute - the threshold is explicit (`min_match_rate`, default 1.0) and printed, never a silent tolerance |\n| `value-reconciliation` | a mapped column whose landed VALUES silently diverged from the source - compared row by row, column by column, over the full population (step 5, executable); blank or duplicated destination keys are excluded and counted, never resolved last-seen-wins |\n| `column-coverage` | a field that was never carried, invisible to any row count |\n| `coverage-summation` | transformed + skipped + deferred failing to sum to the input - the silent subset that \"ran clean\" |\n| `grain` | a one-to-many flattened to one-to-one, which destroys information while every count still reconciles - and the reverse fan-out, a load applied twice |\n| `destination-contract` | types (int, number, ISO date, enum), required-ness, min/max ranges, all-NULL columns, especially where the destination does not enforce them itself |\n| `counterexamples` | an inferred field meaning, stated as a hypothesis and refuted by a query (`when` values take a scalar or a list) |\n| `provenance` | a hand-supplied mapping artifact older than the data it maps, stale by construction - an entry missing a date, or carrying a non-ISO one, fails rather than silently passing |\n| `row-census` | the baseline, which proves the least and is reported first so it is never mistaken for the answer - it blocks only when a declared input has ZERO rows (the wrong-WHERE extract), unless `allow_empty` says that is deliberate |\n| `merge-ledger` | when several sources feed one destination: every source row appears in the merge ledger exactly once (landed / merged / skipped / deferred), every merged row has one golden row, same-source merges were declared, and an independent attribute turns false merges into a failing count and false splits into a reported one |\n| `evidence-rung` | every semantic decision carries the intake rung it rests on (1 code read / 2 running system observed / 3 schema only): rung 1-2 names its source, rung 3 is blocked with its unblocker named, and a precedence applied through `exclusivity` must rest on rung 1 or 2 - `exclusivity` with no `evidence` section fails |\n\nThe spec itself is validated: an unknown section, contract type, or rule key is a spec\nerror, never a silent skip - a typo'd check is a check that never runs while the run\nlooks green.\n\nExit codes: **0** = every declared check passed, **1** = a check failed (a block,\nbecause the transform is not proven), **2** = the spec is invalid.\n\n## The false-block guard\n\n`tests/` seeds a fixture with one instance of every defect class and asserts each\nis caught. It also runs a **clean** fixture that must produce zero findings:\n\n```\nok    row-census\nok    key-uniqueness\nok    key-uniqueness-destination\nok    key-identity\nok    value-reconciliation\nok    column-coverage\nok    coverage-summation\nok    grain\nok    destination-contract\nok    counterexamples\nok    provenance\n\n0/11 checks failed\n```\n\nThat second half matters as much as the first. A checker that cries wolf gets\nswitched off, and then nothing is checked at all.\n\n```bash\npython3 tests/test_migration_check.py\n```\n\n## The discipline\n\n`SKILL.md` is the loop (about 3,500 words; `references/loop-in-full.md` is the long form with the reasoning behind every rule), and most of it is judgment the tool cannot supply:\nlocate the code that writes and reads each side before you read a schema (a schema is\nshape; the code is meaning) and record where each store is and how it is reached in a\nbrief the project keeps, prove you can reach and write to both systems before designing\nanything, census the source's mess, derive the contract from the destination and enumerate what it\nactually *enforces*, derive semantics from behaviour rather than names, prove the\njoin keys before any bulk operation, decide deliberately what happens to values that\ncannot be mapped, state coverage as in-scope / transformed / skipped, reconcile by\nvalue over the full population rather than a sample, scope the load so it cannot\ndestroy what the destination owns, rehearse at production *size* rather than\nproduction shape, and size the depth of every step to the stakes without ever dropping\nthe key proof, the coverage arithmetic or the by-value reconcile - and, when several\nsources feed one destination, keep a merge ledger that proves every source row landed,\nmerged, skipped or was deferred exactly once, with false merges and false splits counted.\n\nTwo of those exist because a migration across two systems breaks advice that holds\ninside one. There is often no shared surrogate key to fall back on when a natural key\nturns out non-unique, because ids get re-sequenced at migration; and the transfer path\nitself, when source and destination sit on different networks, is routinely the largest\nunbudgeted cost in the job.\n\nIt is written as a Claude Code skill, so an agent loads it automatically when it\nrecognises migration work. It reads perfectly well as a checklist for a human.\n\n```bash\ngit clone https://github.com/shaheershoaib/data-migration ~/.claude/skills/data-migration\n```\n\n## Where this sits relative to existing tooling\n\nPublished migration tooling, and most ORM tooling, covers **schema mechanics**:\nDDL, expand/contract, zero-downtime cutover, rollback, version ordering. It\nassumes the data itself is clean. That is a different problem, and the two\ncompose. Use those for *how* the schema changes; use this for *whether* the data\nthat landed is right.\n\nThe checks here are not novel in data engineering. Assertion frameworks (dbt\ntests, Great Expectations, Soda, data-diff) express exactly these destination-side\nconstraints. Where a project already runs one, prefer **emitting** the contract,\nuniqueness and reconciliation checks into it rather than keeping a bespoke\nvalidator: an assertion in the project's own test runner survives after the\nmigration is over, and a one-off script does not. This tool is for the case where\nthere is no such framework yet, or the migration is happening before one exists.\n\nThe checks matter most *before* the extract exists.\nRun them on the source however you can query it; the script is the backstop for what\nyou pulled, not the first line of defence. Across two systems behind different drivers\nthere is no single connection that spans both, which is why a live-connection mode\nwould not help and would cost the property that lets this run anywhere.\n\n## License\n\nApache License 2.0. Copyright 2026 Shaheer Shoaib. See `LICENSE` and `NOTICE`.\n"}