SkillLearnBench

SkillLearnBench

SkillLearnBench: A benchmark for continual learning methods that generate agent skills for real-world tasks. 20 skill-dependent tasks · 15 sub-domains · 100 verified instances

🔥Exciting news!🔥 Our paper has been accepted to COLM 2026! 🎉🎉

Installation

pip install anthropic openai rich tomli dataclaw json-repair   # tomli is only required on Python < 3.11
dataclaw --help                                                # verify that the dataclaw CLI is on your PATH
cp .env.example .env                                           # fill in your API keys (some tasks also require extra variables such as GH_TOKEN — see .env.example)

Docker is a hard requirement, since every agent trial runs inside a container. Install it from docs.docker.com/get-docker.

Quick Start of Evaluation

# Always dry-run first — preview what will run, no execution
python evaluate_skills.py court-form-filling github-repo-analytics --dry-run

# Evaluate with human-authored skills (default, `skills/human_authored`) on two tasks (court-form-filling, github-repo-analytics)
python evaluate_skills.py court-form-filling github-repo-analytics

# The committed `skills/<method>/` tree holds pre-generated skills for all baselines,
# e.g., compare method: one-shot (claude-sonnet-4-6) vs. human-authored vs. no-skill baseline.
# Warning: This command will evaluate these 3 methods across all 20 tasks in SkillLearnBench and requires some time to complete.
python evaluate_skills.py --skill-path skills/b1-one-shot-claude-sonnet-4-6 skills/human_authored none

When running evaluate_skills.py, the code will load the correspondent skills from skill-path, and evaluate them in the tasks of SkillLearnBench. Evaluation results are written to output/evaluation_reports/<method>/<task>/, a report.csv record the average results of each metrics.

⚠️ Keep --max-workers ≤ 50 to avoid API rate limits.

(1) Tasks

SkillLearnBench contains 20 tasks across 6 real-world categories, with 100 instances in total.

(2) Evaluation Dimensions

The solving agent is powered by Claude Sonnet 4.6, and the LLM-as-judge uses GPT-5-mini.

Baselines

We implement four continual learning methods based on skill generation.

The agent can be powered by any LLM. In our codebase we provide results for Claude and Gemini (claude-haiku-4-5, claude-sonnet-4-6, and claude-opus-4-6; gemini-3.1-flash-lite-preview, gemini-3-flash-preview, and gemini-3.1-pro-preview).

Run a baseline to generate skills:

python generate_skills.py --tasks court-form-filling --methods b1-one-shot --models claude-sonnet-4-6

(1) Evaluate other LLMs with the existing four baselines

You can plug additional LLMs into the four baselines above. See BASELINES.md for more details.

(2) Evaluate other continual learning methods

To evaluate your own continual learning method, use the tasks folder, which provides the verifier (in each tests subfolder) and the instance data for every task. Feed the instance-1 information together WITHOUT its verifier into your method to generate skills. The generated skills should follow the same layout as any subfolder in skills. Then evaluate your method with python evaluate_skills.py --skill-path your_skill_path.

See CONTRIBUTING.md for the full set of options and instructions on adding new methods.

Citation

@article{zhong2026skilllearnbench,
  title={SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks},
  author={Zhong, Shanshan and Lu, Yi and Ning, Jingjie and Wan, Yibing and Feng, Lihan and Ao, Yuyi and Ribeiro, Leonardo F. R. and Dreyer, Markus and Ammirati, Sean and Xiong, Chenyan},
  journal={arXiv preprint arXiv:2604.20087},
  year={2026},
  url={https://arxiv.org/abs/2604.20087}
}