preregister

preregister

-- Preregistering your hypothesis --

This skill aims to solve two problems:

Problem 1: Tacit Knowledge: Researchers and engineers have a lot of tacit knowledge that is not available to your agent. This makes them make mistakes.

Problem 2: Insufficient thinking: When coding up experiments has become this cheap we need to think harder about what experiments are worth our time.

This skill de-risks the above problems by having the agent interrogate the you and freeze a preregistration before a new experiment, then hold the analysis and the report to what was written.

Flow: preregister → hypothesis → test → analyze → report with conclusion.

The agent asks in rounds (question and stakes, hypothesis, test design, analysis plan, de-risking, pre-agreed conclusions), records the researcher's answers verbatim, writes experiments/<date>/PREREGISTRATION.md from the form in experiments/_template/, and refuses to start a non-pilot run until it is committed. After the run it runs the preregistered analysis first, appends changes under "Deviations" instead of editing frozen sections, and reports prediction and outcome side by side using the pre-agreed conclusion sentence.

Also here: gpu-lock

.claude/skills/gpu-lock/ holds GPUs on a shared multi-user machine with visible locks: /tmp/gpu-locks/ records which GPUs an experiment holds, which process, and a symlink to its log. gpu_lock.py run starts the job with CUDA_VISIBLE_DEVICES set and, when it ends, kills leftovers, verifies with nvidia-smi that the GPU memory is freed, writes a [gpu-lock] DONE line (exit code, memory check) to the log and removes the locks; status shows every GPU, its lock owner and whether that process is alive. Stdlib only — see its SKILL.md.

Repo layout (the illustration)

.claude/skills/preregister/SKILL.md      the skill
.claude/skills/gpu-lock/                 GPU locks + automatic cleanup on shared machines
AGENTS.md  (CLAUDE.md -> AGENTS.md)      agent instructions wiring the skill into the workflow
experiments/_template/                   skeleton: PREREGISTRATION.md form, config.json, run.py, README
experiments/2026-10-12-example/          a filled-in preregistration (toy experiment) + matching config
research-log/log.md, template.md         the log the skill quotes predictions into
primitives/config.py                     ExperimentConfig: config.json in, cfg.save() on run start
pyproject.toml, uv.lock                  the uv project the scripts run under

Read AGENTS.md, then experiments/_template/README.md, then the example folder. research-log/log.md shows how a preregistered experiment is logged: Expected Outcome quotes the prediction, Actual Outcome is the researcher's.

Install the skill in your own repo

With the skills CLI:

npx skills add reinthal/preregister                      # preregister
npx skills add reinthal/preregister --skill gpu-lock      # gpu-lock

Or copy by hand:

git clone https://github.com/reinthal/preregister
mkdir -p .claude/skills experiments/_template research-log
cp -r preregister/.claude/skills/preregister .claude/skills/
cp -r preregister/.claude/skills/gpu-lock .claude/skills/        # optional
cp preregister/experiments/_template/PREREGISTRATION.md experiments/_template/
cp preregister/research-log/template.md research-log/

Then add the "Preregister before running" convention from AGENTS.md to your own agent instructions, and /preregister in Claude Code (it also triggers when you propose an experiment, ask to scaffold a run, or try to launch a non-pilot run with no PREREGISTRATION.md).

Try it here

uv sync
uv run python experiments/_template/run.py --stage pilot
uv run python experiments/2026-10-12-example/run.py --stage run

Adapting

The skill assumes experiments live in experiments/<date>/ with a config.json holding every hyperparameter, and a research log with Expected/Actual Outcome fields owned by the researcher. Edit the paths in SKILL.md if your project differs. The pre-mortem list in section 5 is a prompt, not a checklist: replace the examples with your own project's past failures.

Origin

Extracted from reinthal/downstream-environment (deception-probe research).