Skip to content
Conduit

mcp server

DynamicMCPBench

ITMO-NSS-team/DynamicMCPBenchcommunitylistedunreviewed
Use tool

Set up

Runs on your computer. Pick your agent and operating system.

Before you start

  • uv (`curl -LsSf https://astral.sh/uv/install.sh | sh`)
~/Library/Application Support/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "dynamicmcpbench": {
      "command": "uvx",
      "args": [
        "dmcp"
      ]
    }
  }
}

same format Cursor, GitHub Copilot CLI, Junie, JetBrains AI Assistant, Augment (Auggie CLI), Factory Droid, Kiro, Devin CLI +12

Values in angle brackets are placeholders. Restart Claude Desktop after editing its config.Claude Desktop docs

README

synced Oct 25, 2026

DynamicMCPBench

Trace-grounded benchmark generation for LLM agents over live Model Context Protocol (MCP) servers.

The benchmark is built by observing successful agent trajectories on real MCP servers, distilling them into path-agnostic effect checkpoints, and grading candidate agents on whether they recreate those effects — never on string-matching a final answer. This makes it robust to dynamic data (live web, live prices, live wikis) and removes the ground-truth-tool-list noise that plagues graph-sampling benchmarks. The design rationale is in docs/CONCEPT.md.

Pipeline

MCP Registry crawl  →  goal-gen  →  forward exploration  →  distill
                                          (live MCP)         (LLM-driven)

                                                                  ↓
                                                            TaskSpec JSONL
                                                                  ↓
                                            evaluate (replay, Tier-1 + Tier-2)
                                                                  ↓
                                                          markdown leaderboard

CLI: dmcp crawl / goal-gen / explore / distill / generate / eval / refresh / report / record.

Install

uv pip install -e ".[servers]"        # dmcp + the substrate MCP servers
cp .env.example .env                   # then set OPENROUTER_API_KEY (see the file)

Optional extras: .[studio] (the DMCP Studio web app), .[dev] (ruff + pytest), .[annotate]. See docs/SETUP.md for standing up the docker-compose substrate and the credentialed server tier.

Quick start

# Smoke a single server (no benchmark run)
uv run dmcp record .venv/bin/wikipedia-mcp -t stdio -s wiki \
  --tool search_wikipedia --args '{"query":"Alan Turing"}'

# Generate a benchmark
uv run dmcp goal-gen --manifest manifests/local.json --per-server 3 --cross-pairs 12 -o goals/auto.json
uv run dmcp generate goals/auto.json --traces-out traces/run.jsonl --specs-out specs/run.jsonl

# Evaluate a candidate against the generated benchmark, deterministically
uv run dmcp eval specs/run.jsonl --replay --reference-traces traces/run.jsonl \
  --model anthropic/claude-haiku-4.5 -o evals/run_haiku45.jsonl
uv run dmcp report --specs specs/run.jsonl --evals evals/run_haiku45.jsonl -o reports/leaderboard.md

dmcp eval/explore/generate/crawl make paid LLM/network calls — run them deliberately, not as casual checks.

Components

  • dmcp/ — the benchmark framework and dmcp CLI (crawl → generate → eval → report), with deterministic Tier-1 effect scoring and an optional Tier-2 fuzzy/LLM-equivalence layer.
  • benchmark_advisor/ — a pre-run statistical planner that scopes a defensible benchmark (minimum detectable effect, power) before generation.
  • dmcp-studio/ — an interactive web studio that drives the pipeline (collect → explore → distill → score) and flips a candidate's verdict between effect-based and answer-based grading. See its README.md.

Repo layout

dmcp/                 the benchmark framework + `dmcp` CLI
  trace.py            schema: Trace / Step / fingerprints
  recorder.py         live MCP capture (stdio/SSE/HTTP)
  replay.py           deterministic replay + Tier-2 fuzzy cache
  manifest.py         servers + dynamism + sandbox validation
  explorer.py         goal-seeded forward exploration
  goal_gen.py         auto goal-gen from tool surfaces
  distiller.py        LLM-driven trace → TaskSpec
  spec.py             TaskSpec + discriminated checkpoint union
  evaluator.py        Tier-1 deterministic scorer
  judge.py            Tier-2 LLM effect-equivalence judge
  refresh.py          re-run reference traces, classify drift
  report.py           markdown leaderboard
  cli.py              typer CLI entrypoint
benchmark_advisor/    pre-run statistical planner
dmcp-studio/          interactive studio (FastAPI + React)
manifests/local.json  substrate manifest (sandboxed + public-API servers)
docs/CONCEPT.md       design rationale and method catalogue
docs/SETUP.md         substrate + credentialed-tier setup

License

Apache-2.0.

↑↓ navigate openesc close