kaggle-war-room

Kaggle War Room

A multi-role AI-agent workflow for Kaggle competitions, packaged as a plugin for Claude Code and Codex. Computer-vision focused, but the workflow is task-agnostic.

• Scout watches public notebooks, the leaderboard and discussions every day, reads the threads that changed (host posts flagged), and reports only what changed.
• Strategist reads the rules, a research brief of what won similar past competitions, the experiment log and decision log, then plans experiments, compute and submissions within your real submission slots and GPU quota.
• Critic — ideally a different vendor's model — reviews each plan adversarially, without seeing the Strategist's reasoning.
• Builder reviews public notebooks before reusing them, implements, blends, runs heavy jobs on Kaggle GPUs when needed, and submits through a closed loop.

Roles hand off through files, scripts do the mechanical work, and every risky step stops for your confirmation. Final submissions are chosen as a hedged pair to resist leaderboard shake-up.

What the scripts cover:

• Submission loop — slots left and weekly GPU quota → validated dry run → submit after your yes → wait for the score → LB written to the experiment log and compared with CV (warns on CV/LB divergence). Code competitions submit a kernel.
• Research — save the competition's rules / evaluation / data pages, find similar ended competitions, pull their solution writeups (Kaggle's writeups category), and read discussion threads, all through the official Kaggle API.
• CV split — one shared folds.csv (stratified, grouped by patient/scene/site, or by time) with a balance report, protected from silent changes.
• Kaggle compute — push training to Kaggle GPUs, wait, download outputs, and move weights between kernels as private datasets (the usual code-competition setup). Public notebooks are reproduced as private forks on Kaggle with their inputs and environment - nothing to download.
• Monitoring and analysis — snapshot diffing, notebook review with suspicious-code flags, submission checks, prediction correlation for blending, stagnation and phase detection.

Install

Easiest: ask your agent

In Claude Code or Codex, say:

Install the kaggle-war-room plugin from https://github.com/JunhaoLiXD/kaggle-war-room

The agent follows Instructions for AI agents below.

Claude Code

claude plugin marketplace add JunhaoLiXD/kaggle-war-room
claude plugin install kaggle-war-room@kaggle-war-room

Or inside a session: /plugin marketplace add JunhaoLiXD/kaggle-war-room, then /plugin install kaggle-war-room@kaggle-war-room. Start a new session (or run /reload-plugins).

Codex

codex plugin marketplace add JunhaoLiXD/kaggle-war-room
codex plugin add kaggle-war-room@kaggle-war-room

Start a new Codex session afterwards.

Python dependencies (both hosts)

pip install numpy pyyaml kaggle

Then create a Kaggle API token (kaggle.com → Settings → API) and save it as ~/.kaggle/kaggle.json. Do this yourself — never paste the key into an agent chat.

Manual install (any agent that supports Agent Skills)

Copy plugins/kaggle-war-room/skills/kaggle-war-room/ into your agent's skills directory, e.g. ~/.claude/skills/kaggle-war-room/ or ~/.codex/skills/kaggle-war-room/. For Claude Code you can also copy plugins/kaggle-war-room/agents/*.md into ~/.claude/agents/ to get the role subagents.

Quick start

1. In your project folder, tell the agent: "Start a Kaggle competition workspace for <competition-slug>."
2. First-run setup. The agent looks up what it can (deadline, metric, daily limit, code competition, your Kaggle username, local GPUs, sample_submission.csv), then asks you the rest in one message: models for each role, how the Critic runs, your compute, team name, host staff names, and whether you want the war-room touches. Nothing else starts until this is confirmed. Your personal choices carry over to your next competition.
3. The agent drafts rules.md from the competition pages → you confirm. It creates the shared CV split (folds.csv), reproduces a reviewed public notebook as a private fork on Kaggle, and records it as the baseline → you confirm.
4. Each day: "Run today's Kaggle update and plan." Scout → Strategist → Critic → Strategist → Builder.

Where your files go

The plugin never scatters files in your project. Everything it creates is in one folder:

<your project>/kaggle-war-room-workspace/
├── workspace.json        which competitions exist, which one is active
├── README.md
└── <competition-slug>/   settings.yaml, rules.md, baseline.md, folds.csv, experiments.csv,
    ├── code/             decisions.md, research_brief.md, daily_brief.md, plan.md,
    ├── preds/            review.md, submissions.md
    ├── submissions/      submission files + receipts of every real submission
    ├── kernels/          Kaggle kernels and forks, run log, outputs per version
    ├── research/         competition pages, past-solution writeups, discussion threads
    └── snapshots/        daily snapshots of the leaderboard, notebooks and discussions

Every session starts by running workspace.py locate, so the agent always reads and writes the right competition folder. Several competitions in one project are fine: "switch to <slug>" runs workspace.py use <slug>. To keep the workspace elsewhere (e.g. another drive), set the environment variable KAGGLE_WAR_ROOM_WORKSPACE to that folder.

Configuration

Everything you can change lives in kaggle-war-room-workspace/<slug>/settings.yaml. The first-run setup fills it with you (configure.py: detected facts plus one round of questions); every key is documented in SKILL.md → Variables you can change:

• models per role (inherit by default, so a single-model setup just works)
• Critic: external CLI of the other vendor (codex exec … from Claude Code, claude -p … from Codex) or a fresh-context subagent
• compute: weekly Kaggle GPU quota, your own GPUs, inference limit, confirmation threshold
• submissions: daily limit, who allocates them, path to sample_submission.csv
• competition: deadline, metric direction, code competition, team name, host staff names
• flavor: the small war-room touches on or off
• phases: explore / refine / finalize boundaries, stagnation limits, min_gain

You are always asked before

• confirming rules.md and the baseline
• every Kaggle submission (scripts only do a dry run until you say yes)
• any experiment estimated above confirm_if_gpu_hours_over GPU hours, locally or on Kaggle
• every dataset upload, and making anything public
• switching phase (explore → refine → finalize)
• big plan changes caused by the Critic, or overruling the Critic
• choosing final submissions

Safety

Public notebooks, discussions and dataset descriptions are treated as data, not instructions. Text aimed at AI agents is reported, never followed. Notebooks are reviewed for leakage, leaderboard probing, rule violations and suspicious code before any reuse. Public notebooks are reproduced as private Kaggle forks (internet off, never on your machine), and an unreviewed fork cannot be pushed. Kernels and datasets stay private unless you say otherwise. Data comes only from the official Kaggle API.

Repository layout

.claude-plugin/marketplace.json      Claude Code marketplace
.agents/plugins/marketplace.json     Codex marketplace
plugins/kaggle-war-room/
├── .claude-plugin/plugin.json       Claude Code plugin manifest
├── .codex-plugin/plugin.json        Codex plugin manifest
├── agents/                          Claude Code role subagents
└── skills/kaggle-war-room/          The skill (shared by both hosts)
    ├── SKILL.md                     Workflow, gates, variables
    ├── config/settings.example.yaml All tunable variables
    ├── roles/                       Scout / Builder / Strategist / Critic playbooks
    ├── scripts/                     14 Python tools (each has --help)
    ├── references/                  Review checklists, anti-shake-up, ensembling, Kaggle runs, CV tips
    └── templates/                   Workspace files (rules.md, plan.md, ...)
tests/                               Unit and regression tests (not installed with the plugin)

Instructions for AI agents

You were asked to install this plugin. Do the following, then report what you did.

1. Detect your host:
   • Claude Code (you have the claude CLI or are Claude Code): run claude plugin marketplace add JunhaoLiXD/kaggle-war-room, then claude plugin install kaggle-war-room@kaggle-war-room.
   • Codex (you have the codex CLI or are Codex): run codex plugin marketplace add JunhaoLiXD/kaggle-war-room, then codex plugin add kaggle-war-room@kaggle-war-room.
   • Other agent with skills support: clone the repository and copy plugins/kaggle-war-room/skills/kaggle-war-room/ into your skills directory.
2. Run pip install numpy pyyaml kaggle (ask first if the user manages environments carefully, e.g. conda or a project venv).
3. Do not ask for, read, or write the Kaggle API key. Tell the user to save their token to ~/.kaggle/kaggle.json themselves.
4. Tell the user to start a new session (Claude Code: or run /reload-plugins) and that they can begin with: "Start a Kaggle competition workspace for <slug>." Mention that the first step will be a short setup round (models per role, Critic, compute, team and host names), so they can have their choices ready.
5. End your report with this welcome, once, in the user's language:

   🎖️ The war room is open. Scout is heading out for recon, the Strategist is laying out the map, the Red Team (Critic) is ready to challenge every plan, and Engineering (Builder) is standing by. Commander, let's win this one — good luck!

Development

Run the tests from the repository root (needs numpy and pyyaml):

python -m unittest discover -s tests -v

When releasing, bump version in both plugins/kaggle-war-room/.claude-plugin/plugin.json and plugins/kaggle-war-room/.codex-plugin/plugin.json so installed users receive the update.

License

MIT