{"id":"92c851aa-16f3-4672-8403-16b38b8f7d2e","slug":"seanleecoder-rule-trace","kind":"skill","name":"rule-trace","summary":"","owner":"seanleecoder","repoUrl":"https://github.com/seanleecoder/rule-trace","license":"MIT","licenseTier":"permissive","trust":"community","stars":5,"match":null,"fewStars":false,"format":"skill","createdAt":"2026-06-24T22:09:28.000Z","addedAt":"2026-10-14T11:47:49.352Z","logoUrl":"https://avatars.githubusercontent.com/u/35733866?u=178a7e560493a878a88fc600a390627d9da7a894&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["writing"],"tags":[],"runtime":"node","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-17T22:08:52.111Z","sample":false,"url":"https://conduits.dev/tools/seanleecoder-rule-trace","language":"JavaScript","transports":["stdio"],"install":{"npm":"rule-trace"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# rule-trace\n\n[![Skills](https://www.skills.sh/b/seanleecoder/rule-trace)](https://www.skills.sh/seanleecoder/rule-trace)\n\nSee which agent rules actually shaped the work.\n\n**Today:** every substantive agent response ends with a clickable, reviewable trace — which rules were candidates, which shaped the work, and why any rule was waived — instead of trusting \"the agent followed project rules\" as an unverifiable claim.\n\n**Over weeks:** the counters turn those traces into maintenance signal — which rules are dead, which are always loaded but never applied, and which `MUST` rules keep getting silently skipped.\n\n`CLAUDE.md`, `AGENTS.md`, `.cursorrules`, and tool configs are easy to grow and hard to debug. After a few weeks, you usually cannot tell which instructions are still useful, which ones are noise, or whether Claude, OpenCode, Codex, Cursor, and other tools are even loading the same rules.\n\n`rule-trace` turns agent rules into a reviewable loop:\n\n1. Migrate prose rules into stable, citable IDs.\n2. Ask agents to disclose which rules were candidates, applied, or deliberately skipped.\n3. Validate that the catalog, rule files, and importers have not drifted.\n4. Report usage so dead, broad, skipped, or stale rules become visible.\n\nRules loaded into a coding agent are *loaded* context, not *applied* context. A rule that was followed looks identical to one that was ignored unless the agent makes the difference visible. This skill closes that gap with stable rule IDs, trace blocks, a deterministic validator, and cross-session usage reports.\n\nBackground: [Making AI-agent rule application visible - stable IDs and trace blocks](https://seanleecoder.hashnode.dev/making-ai-agent-rule-application-visible-stable-ids-and-trace-blocks).\n\n![rule-trace dashboard flagging dead, low-rate, and un-waived MUST rules from seeded demo data](docs/dashboard.png)\n\n## Why Use It\n\nUse this when your agent rules have become important enough to break things, but still invisible enough that no one can inspect them.\n\n- Give reviewers a concrete trace instead of \"the agent followed project rules\" as an unverifiable claim.\n- Find rules that are never candidates and should be removed, narrowed, or rewritten.\n- Find rules that are always candidates but rarely applied, usually a sign they are too broad, too expensive, or miscoped.\n- Catch `MUST` rules that were in scope but neither applied nor explicitly waived.\n- Fail CI when a catalog ID no longer resolves to a heading, a rule is missing required fields, an importer drifts, or `.opencode/opencode.json` is malformed.\n- Keep supported importer entry points referencing the same canonical rule files when a repo uses more than one agent tool; see the importer support matrix for which tools actually expand references.\n\nThis repository dogfoods rule-trace: `.agents/` contains the live migrated rule set, while `CLAUDE.md` and `AGENTS.md` show the thin-importer end state — though per the importer support matrix in [`importer-wiring.md`](skills/rule-trace/references/importer-wiring.md), this repo's own root `AGENTS.md` uses `@`-imports whose expansion is unconfirmed for Codex CLI. See `examples/demo/` for seeded metrics and a generated dashboard.\n\nThis is not compliance theater. Counts are self-reported by the model, so they are an audit signal, not proof. The value is that previously invisible rule behavior becomes reviewable.\n\n## Core Workflow\n\nMost teams should start with the core loop:\n\n1. **Migrate** existing prose from `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, package READMEs, and docs into ID-based rules.\n2. **Validate** the rule system so every catalog entry resolves, every rule has required fields, and configured importers load the same file set.\n3. **Collect** trace blocks from saved transcripts or a live Claude Code hook.\n4. **Report** candidate/applied/deviation counts into `.agents/metrics/report.json` and an optional `dashboard.html`.\n\n\n| Workflow step | Who runs it | How |\n| --- | --- | --- |\n| Migrate / Init / Audit | agent (skill mode) | ask your agent |\n| Validate | CLI or CI | `rule-trace validate` |\n| Collect | CLI or Stop hook | `rule-trace collect` / hook |\n| Report | CLI | `rule-trace report` |\n\nThe skill also supports `init` for new repos and `audit` for cleanup after you have enough usage data, but `migrate -> validate -> collect -> report` is the main release path. Do not skip collect: `report` only aggregates traces that already exist, so a report run before any traces are collected flags every catalogued rule as dead.\n\n## Before And After\n\nBefore, rules are usually prose in one or more importer files:\n\n```md\n# CLAUDE.md\n\nAlways use pnpm, not npm.\nRun relevant tests before finishing.\nKeep agent importers in sync.\n```\n\nAfter migration, each enforceable rule gets a stable ID and a trigger:\n\n```md\n## ROOT-001\n\n- Scope: repository\n- Applies when: installing dependencies or running package scripts\n- Severity: MUST\n- Rule: Use pnpm, not npm.\n\n## TEST-001\n\n- Scope: tests\n- Applies when: changing runtime behavior\n- Severity: SHOULD\n- Rule: Run the relevant tests before finishing.\n```\n\nAn agent response can then disclose what mattered; the prose is for reviewers, and the fenced `rule-trace` JSON is the machine-stable data layer:\n\n````md\nRule trace\n\n- Candidate rules loaded: [`ROOT-001`](rules/root.md), [`TEST-001`](rules/testing.md)\n- Rules applied: [`ROOT-001`](rules/root.md)\n- Sources: [`.agents/rules/root.md`](rules/root.md), [`.agents/rules/testing.md`](rules/testing.md)\n- Reasoning note: dependency commands were involved, but the change was docs-only.\n- Deviations: [`TEST-001`](rules/testing.md) - docs-only change; no runtime behavior changed.\n\n```rule-trace\n{\"v\":1,\"candidate\":[\"ROOT-001\",\"TEST-001\"],\"applied\":[\"ROOT-001\"],\"deviations\":[\"TEST-001\"]}\n```\n````\n\nAcross sessions, the report turns those traces into maintenance signal:\n\n```text\nTEST-001: candidate 42x, applied 8x, waived 20x\n```\n\nThat does not prove the agent was right. It tells you `TEST-001` is worth reviewing: maybe the trigger is too broad, maybe the rule is too expensive, or maybe the team is repeatedly waiving something it claims to care about.\n\n## What You Get\n\nCore pieces:\n\n- **Stable rule IDs** anchored at markdown `##` headings.\n- **A catalog** (`.agents/rules-catalog.md`) that indexes every rule ID and source file.\n- **Trace blocks** that distinguish candidate rules, applied rules, and deliberate deviations.\n- **A validator** that is deterministic, dependency-free, CI-friendly, and runnable without an agent runtime.\n- **Usage reports** that aggregate traces across sessions into candidate/applied/rate metrics.\n\nOptional adoption support:\n\n- **Thin importers** for tools that support file references, plus documented/generated native formats for the tools that don't — see the importer support matrix for which is which.\n- **A dashboard** that flags dead rules, always-candidate-never-applied rules, low application rate, un-waived `MUST` gaps, and unknown IDs.\n- **CI snippets** for GitHub Actions and GitLab.\n- **A Claude Code Stop hook** for live usage collection.\n- **Scaffolding** for optional CI, hook, and metrics wiring.\n- **Audit mode** for cleaning up rule sets after reports have useful volume.\n\n## Install\n\nPick **one** path for Claude Code — skills.sh and the plugin install the same skill, so they are alternatives, not steps. Combining them wires the live `Stop` hook twice (see the note below).\n\n### Path A — skills.sh (all agents; recommended for teams)\n\n```bash\nnpx skills add seanleecoder/rule-trace\n```\n\nInstalls into every detected agent at once — Claude Code under `.claude/skills/`, OpenCode/Codex/Cursor under `.agents/skills/` — as files in your repo tree that you can **commit**, so teammates get it on clone. Add `-g` for a global install, or `--copy` to copy instead of symlink. Update later with `npx skills update rule-trace`. For live counting on Claude Code, wire the `Stop` hook by hand (one snippet — see [`skills/rule-trace/references/importer-wiring.md`](skills/rule-trace/references/importer-wiring.md)).\n\n### Path B — Claude Code plugin (Claude Code only; hook auto-wired)\n\n```text\n/plugin marketplace add seanleecoder/rule-trace\n/plugin install rule-trace@seanleecoder-skills\n```\n\nInstalls the same skill for Claude Code only, per developer (under `~/.claude/plugins/`, _not_ committed to the repo), and ships a `Stop` hook in [`hooks/hooks.json`](hooks/hooks.json) that is wired automatically — nothing else to do for live counting.\n\n> **Don't combine A and B.** Both wire the recorder. The plugin's command resolves to `${CLAUDE_PLUGIN_ROOT}/…` and the manual one to `$CLAUDE_PROJECT_DIR/…`, so Claude Code can't dedupe them and the recorder runs on every turn from each. It fails _silently_ — `record-trace.mjs` dedupes by message UUID, so the second run just writes nothing — but it spawns a redundant process per turn. Use skills.sh **plus** the manual hook, _or_ the plugin alone. `validate` and `scaffold` warn if they detect both.\n\nFor CI-only validation, use the package CLI without an agent runtime:\n\n```bash\nnpx rule-trace@1 validate\n```\n\n## Quickstart\n\nOnce installed, ask your agent to migrate an existing repo:\n\n```text\nuse rule-trace to migrate this repo's rules\n```\n\nUseful starting points:\n\n- **Existing rules:** ask for `migrate`. The agent gathers `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.opencode/opencode.json`, package READMEs, and any docs you point it at, then splits prose into ID-based rules.\n- **No existing system:** ask for `init`. The agent creates `.agents/rule-trace.md`, `.agents/rules-catalog.md`, an example `.agents/rules/root.md`, and optional thin importers.\n- **Just validate:** run `rule-trace validate` or `node <skill>/scripts/validate-rules.mjs` from the target repo root.\n- **Verify it is working:** after migrate, ask a rule-shaped question such as `Which tests would matter if I changed the validator?`; a substantive response should end with `Rule trace`. If the live hook is wired, confirm `.agents/metrics/traces.jsonl` gained a line. Or try the committed demo first: `node <skill>/scripts/report.mjs --root examples/demo`.\n- **Just count usage:** run `collect` to backfill traces from transcripts, then `report` to build the report and dashboard.\n- **Ready to clean up:** after you have enough trace data, ask for `audit` to classify rules as keep, revise, remove, consolidate, or add.\n\n## Commands\n\nThe deterministic scripts are available directly:\n\n```bash\n# Validate the rule system.\nnode <skill>/scripts/validate-rules.mjs --root <repo>\n\n# Backfill trace events from saved transcripts.\nnode <skill>/scripts/parse-traces.mjs --root <repo> --transcripts <dir>\n\n# Build .agents/metrics/report.json and dashboard.html.\nnode <skill>/scripts/report.mjs --root <repo> --low-rate 0.5 --min-candidates 3 --min-coverage 0.2 --stale-days 30 --since 2026-01-01\n\n# Generate the catalog from rule headings, preserving curated summaries.\nnode <skill>/scripts/generate-catalog.mjs --root <repo> --write\n\n# Materialize generated importers for reference-blind tools such as Cursor/Copilot.\nnode <skill>/scripts/sync-importers.mjs --root <repo> --check\n```\n\nThe CLI exposes the same core tools:\n\n```bash\nnpx rule-trace@1 <validate|collect|report|catalog|scaffold|sync>\n# Pre-registry fallback (unpinned — prefer the registry):\nnpx github:seanleecoder/rule-trace validate\n```\n\n## Counters And Dashboard\n\nTrace blocks carry candidate and applied IDs in human-readable prose plus a fenced `rule-trace` JSON block, so the data exists in transcripts even if future model wording drifts. The collectors append events into one UUID-deduped log at `.agents/metrics/traces.jsonl`.\n\n- **Offline collection/backfill:** `parse-traces.mjs` scans saved transcripts and appends trace blocks. It is re-runnable and tool-agnostic when the transcript records expose a UUID and assistant text.\n- **Live Claude Code hook:** `record-trace.mjs` records each finished main-agent response from a Claude Code `Stop` hook. The plugin wires this automatically; skills.sh and standalone installs can add the hook manually from `references/importer-wiring.md`.\n\n`report.mjs` writes `.agents/metrics/report.json` and `.agents/metrics/dashboard.html`. Tune noisy repos with `--low-rate <0..1>`, `--min-candidates <n>`, `--min-coverage <0..1>`, `--stale-days <n>`, and `--since <ISO-8601 date>`. Pass `--now <ISO-8601 date>` to pin report time for reproducible runs (staleness and `generatedAt` both derive from it) — it does not filter events, that's `--since`.\n\nThe live hook also records finished main-agent responses that omit a trace block, letting the report show trace coverage and warn when coverage is too low to trust dead-rule or low-rate conclusions. Trivial or conversational responses may intentionally omit traces, so 100% coverage is not the target; coverage is a sanity and trend signal.\n\nThe dashboard highlights:\n\n- `deadRules` - catalogued rules that were never candidates.\n- `alwaysCandidateNeverApplied` - rules that came up but never constrained the work.\n- `lowRate` - rules below the configured application-rate threshold.\n- `unwaivedMustGaps` - strongest configured severity rules (default `MUST`) that were candidates but neither applied nor waived.\n- `stale` - rules that were candidates before but have not surfaced within the configured staleness window.\n- `unknownIds` - hallucinated or stale IDs cited by traces.\n\n\n## Stability\n\nSemver covers the public contract teams build around: the trace-block convention (including the prose labels and fenced `rule-trace` JSON format), the rule anatomy and rule-ID grammar, event JSONL fields, `.agents/rule-trace.config.json` keys, CLI command names and documented flags, and validator exit-code semantics.\n\nSemver does not cover internal script file paths or repository layout, dashboard HTML/CSS details, exact console-output wording, or `report.json` field ordering.\n\n## What It Costs\n\nA trace block is roughly 50–100 tokens on substantive responses. Rule files usually replace prose the agent entry points were already loading, so migration is roughly context-neutral; `.agents/rule-trace.md` adds a few hundred tokens once per session. The live Claude Code Stop hook is a local subprocess per turn and is designed never to block the agent. Treat these as rough orders of magnitude, not benchmarks.\n\n## Validation And CI\n\n`validate-rules.mjs` fails on rule-system errors and exits `1`:\n\n- Catalog IDs that no longer resolve to `## ID` headings.\n- Rule headings missing from the catalog.\n- Duplicate rule IDs.\n- Missing required fields: `Scope`, `Applies when`, `Severity`, and `Rule`.\n- Invalid severity values outside `MUST`, `SHOULD`, and `MAY`.\n- Importer drift, where configured agent tools load different rule file sets.\n- Malformed OpenCode config when `.opencode/opencode.json` is present.\n- Trace blocks that cite IDs missing from the catalog when using `--lint-file <path>`. Lint real trace output, not files that document the trace format; quoted examples inside four-backtick fences are ignored, but three-backtick `rule-trace` examples are linted as real traces.\n\nIt warns, but does not fail, when configured importers are absent or numbered IDs have gaps. If a gap is intentional because a rule was retired, add that ID to `retiredIds` in `.agents/rule-trace.config.json` so validation treats the gap as documented retirement rather than drift. If a repo intentionally uses only one agent tool, set `importers` in `.agents/rule-trace.config.json` to just that entry so validation stays quiet.\n\nExample package script:\n\n```json\n{\n  \"scripts\": {\n    \"rules:validate\": \"node .agents/skills/rule-trace/scripts/validate-rules.mjs\"\n  }\n}\n```\n\nMore CI snippets live in `skills/rule-trace/references/ci-wiring.md`.\n\n## Optional Scaffolding\n\n`scaffold-wiring.mjs` writes optional operational glue. It is non-destructive: existing files are left untouched, and merge instructions are printed when manual integration is needed.\n\n```bash\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --all\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --hook\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --gitignore\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --ci github\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --ci gitlab\nnode <skill>/scripts/scaffold-wiring.mjs --root <repo> --ci none\n```\n\nWith no selector flags, `--all` is assumed. Selective flags do only what they say: `--hook` does not create CI, and `--gitignore` does not create a hook.\n\n## Tests And Evals\n\nThe repo includes deterministic tests for the scripts, manifests, docs, and regression cases:\n\n```bash\nnpm test\n```\n\nBehavioral evals exercise the agent-driven `migrate` mode on fixture repos:\n\n```bash\nnode evals/run.mjs\nnode evals/run.mjs --exec --fixtures single-claude-md,oss\nnode evals/run.mjs --exec --agent codex --fixtures single-claude-md,oss\nnode evals/run.mjs --exec --agent codex --codex-sandbox danger-full-access --fixtures single-claude-md\nnode evals/run.mjs --grade-only\n```\n\nThe runner prints the source fixture and generated arm directories after each round so you can inspect or diff before/after outputs. See [`evals/README.md`](evals/README.md) for the eval workflow and `fetch-oss.mjs` for adding a real public-repo fixture.\n\n## Limits\n\nRule traces are self-reported. They tell you what the model claimed was in scope and applied; they do not prove the model complied. Treat the report as a review surface.\n\nA rule that never appears as a candidate is invisible to the counters, so early reports with low volume are directional. The first useful result is often not a perfect score; it is a short list of rules worth deleting, narrowing, or rewriting.\n\n## License\n\nMIT (c) Sean Lee ([seanleecoder](https://github.com/seanleecoder))\n"}