gutcheck

gutcheck

Pause your AI agent. Rate its outputs by gut feel for 15 minutes. Hand it a scorer that actually matches your taste.

My metrics said brighter. My ears said bassier.

I was running an agentic pipeline that generated music and scored it with a crowd-engagement recipe — spectral brightness, motion, glitchy energy. The agent kept "improving" the score, and the tracks kept getting worse.

So the agent paused and served me a tiny local page: one clip at a time, no filenames, random order, keys 1–5. Eighty gut reactions, about fifteen minutes. Then it correlated my ratings against the metric's ingredients.

The recipe was partly inverted. It rewarded bright, glitchy motion; I actually prefer darker, bassier, fuller mixes. And the biggest thing my ratings tracked — "does it sound like a real song" — wasn't in the metric at all. No amount of metric tuning would have found this, because the metric itself was the bug. A later pairwise A/B round steered the generation variants precisely.

gutcheck packages that move as a Claude Code skill your agent reaches for on its own.

What it does

1. Agent runs /gutcheck <dir-or-glob> over any artifacts — audio (wav/mp3), images, text snippets, HTML.
2. You rate in a local, keyboard-driven page: absolute (1–5, ~5 s each) or pairwise (A/B duels, arrow keys). Blind by construction: no filenames, randomized order. Progress autosaves after every keypress; quit and resume anytime.
3. It fits a preference model (mean ratings or a Bradley–Terry fit for duels) and writes:
   • calibration/scores.json — a per-item personalized scorer your agent optimizes against instead of the proxy
   • calibration/report.md — ranking, which filename-encoded variant ingredients drive your preference, and feature↔preference correlations if you pass --features
4. It guards against fooling yourself: if ratings saturate at 5, it recommends a pairwise round; if a coarse group axis (say real/ vs generated/) explains most of the variance, it warns you that you may be detecting group membership, not quality.

Everything runs on 127.0.0.1, stdlib only, nothing leaves your machine.

Install

As a Claude Code plugin (recommended):

/plugin marketplace add mhavelka77/gutcheck
/plugin install gutcheck@gutcheck

Or copy the skill directly:

git clone https://github.com/mhavelka77/gutcheck
cp -r gutcheck/skills/gutcheck ~/.claude/skills/

Or use it without Claude Code at all — it's a plain CLI:

uvx --from git+https://github.com/mhavelka77/gutcheck gutcheck renders/ --open
python3 skills/gutcheck/scripts/gutcheck.py renders/ --open   # zero-install

When your agent reaches for it

The skill teaches Claude to invoke this when:

• it's optimizing anything subjective — audio pleasingness, visual style, text tone, UI feel;
• a metric, model judge, or heuristic disagrees with what you say you perceive (the signal a proxy is lying);
• you ask to rate, rank, or A/B a batch of variants.

It also teaches budget manners: one session, under ~100 judgments, never asks about the same item twice.

Modes

The tool recommends a mode and auto-suggests switching: absolute ratings saturating at the ceiling is exactly when pairwise duels start paying off. Duels are scheduled Swiss-style — after each round it refits and pairs near-ties, so your judgments land where they carry the most information.

FAQ

Why not just have the LLM judge? An LLM judge is another proxy. Fine for grammar; useless when the target is your taste — that information exists only in your head, and 15 minutes of gut reactions extracts a surprising amount of it.

Dependencies? None. Python ≥ 3.9 stdlib for everything, including the Bradley–Terry fit. numpy/pandas are optional (parquet feature tables).

Does it work in Safari? Yes — the server implements HTTP Range responses (206), which Safari requires before it will play audio at all. That one's a hard-won lesson; it ships tested.

Can I add my own feature table? --features features.csv (or parquet), first column = item id. The report gets a Spearman feature↔preference table — this is how I caught my inverted metric.

What does the agent actually consume? calibration/scores.json: items sorted by rank with scores. The skill instructs the agent to read the warnings first, obey the suggested next step, then use the ranking as the fitness function for the next generation round.

License

MIT