skill-quartermaster

Why

Skill marketplaces ship hundreds of skills at once. Past a few dozen, your agent starts picking the wrong one — and you become the janitor, with no safe way to clean up. Quartermaster fixes both, built on four rules:

1. Trust is earned. New skills start as candidates and advance only by winning trials and passing review. No skill jumps the queue.
2. Fair trials, not stars. Candidate vs. your current skill — same task, same seed, pinned versions.
3. Evidence is append-only. Pass/fail, latency, cost, notes — recorded, never rewritten.
4. Humans decide. The tool proposes; you approve. Everything is audited and reversible.

Proof it works

On 851 real open-source skills: loadout trimmed to 30, 96% of context tokens saved (~57.5k), 0 files deleted, demote→restore byte-identical. And in a live A/B (claude-opus-4-8, 60 tasks), the agent picked the right skill more often with the 30-skill loadout than with all 840 — 97% vs 93%. Full reports: BENCHMARK.md · PERFORMANCE.md · benchmark/ab_eval/.

How it works

• Intake & trust — you clone a candidate repo; Quartermaster scans it without running its code, hashes content, flags risky patterns, and imports only what you accept.
• Scouting engine — qm scout pins exact versions, runs randomized paired A/B jobs in your own sandboxed worker (any provider), and writes an honest dossier: quality deltas, confidence intervals, cost, limitations.
• Human control — three reviewers score the dossier blind; two accepts approve, any safety veto blocks. Even then, lifecycle changes go through your normal review, and qm revert undoes them.

Every skill sits on a reversible ladder — each step logged, only the last one gated by your explicit --yes:

How to run

# Claude Code plugin …
/plugin marketplace add ginaecho/skill-quartermaster
/plugin install quartermaster@skill-quartermaster

# … or zero install (pure Python, stdlib only)
export QM_SKILLS_DIR=~/.claude/skills
python3 bin/qm status

# Daily loop
qm status              # every skill: state, last used, token cost
qm compile "<intent>"  # pick the ~30 skills this project needs
qm review              # approve proposed changes
qm restore <skill>     # bring anything back
qm revert              # undo the last automatic change

# Vet a new skill: scan, then trial
qm intake <cloned-repo> --dry-run       # scan safely — never executes code
qm scout plan <candidate> --suite benchmark/scout/suite.example.json \
  --current-skill <current-default> --source <repo-url> --version <commit>
qm scout run scout-plan.json --runner python --runner-arg <isolated_runner.py>
qm scout report scout-plan.json scout-trials.jsonl
qm scout review scout-evidence.json reviewer-votes.json --author <owner>

qm --help lists the rest (demote, hide, archive, conflicts, gaps, authoring, feedback). A worked Copilot case study lives in benchmark/scout/; schemas and trust boundaries in docs/SCOUTING.md. All state stays in ~/.quartermaster/ — nothing leaves your machine. Tests: python3 -m pytest (128 pass).

Roadmap

Shipped: lifecycle core, telemetry, authoring arm, feedback, Skill Scout. In design for v1.0: semantic discovery, prompt-injection & provenance gates, trust tiers, dashboard — the not-yet parts of the figures above.

Contributing & license

Issues before large PRs, please. Adapters for other harnesses (Cursor, Copilot, Zed) especially welcome. MIT — see LICENSE.