{"id":"6473a083-f387-4b43-9ce3-bf2684991eca","slug":"mcpeval","kind":"mcp","name":"mcpeval","summary":"Lint, test, and optimize your MCP tool schemas before shipping. Catches the bugs that make LLMs pick the wrong tool.","owner":"jashshah999","repoUrl":"https://github.com/jashshah999/mcpeval","license":"MIT","licenseTier":"permissive","trust":"community","stars":3,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-05-05T05:31:53.000Z","addedAt":"2026-10-21T06:02:53.816Z","logoUrl":"https://avatars.githubusercontent.com/u/49280550?u=e3f3d238f7b5314b7da429b43e7afcee361bde1a&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["developer-tools","memory-knowledge"],"tags":["ai-agents","cli","developer-tools","llm-tools","schema-validation","testing"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T06:05:17.235Z","sample":false,"url":"https://conduits.dev/tools/mcpeval","language":"Python","transports":["stdio"],"install":{"pypi":"mcpeval"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# mcpeval\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![Tests](https://img.shields.io/badge/tests-23%20passed-green.svg)]()\n\n**Lint, test, and optimize your MCP tool schemas before shipping.**\n\nYou built an MCP server. But will Claude/GPT actually use your tools correctly? `mcpeval` tells you before your users find out.\n\n```\n$ mcpeval check my_server.json\n\n  Schema Score: 34/100 (F)\n  Errors: 3  Warnings: 12\n\n  ✗  perform_delete    Tool has no description. LLMs cannot understand when to use it.\n  !  do_search         Parameter 'data' is too generic. LLMs won't know what to pass.\n  !  update_record     Parameter 'status' looks like it should have enum values.\n  !  search/find       'search_users' and 'find_users' have 89% similar descriptions.\n```\n\n## Why?\n\nBuilding MCP servers is easy. Building MCP servers that **LLMs actually use correctly** is hard:\n\n- **Ambiguous descriptions** → LLM picks the wrong tool\n- **Generic param names** (`data`, `value`, `input`) → LLM passes garbage\n- **Missing enums** → LLM invents invalid values\n- **Similar tools** → LLM can't distinguish between them\n- **Schema bloat** → wastes 40% of your context window\n\nYou find these bugs in production, from user complaints. `mcpeval` catches them in CI.\n\n## Install\n\n```bash\npip install mcpeval\n```\n\nFor LLM simulation (optional):\n```bash\npip install mcpeval[gemini]    # uses Gemini Flash (cheapest)\npip install mcpeval[openai]    # uses GPT-4o-mini\npip install mcpeval[anthropic] # uses Claude Sonnet\n```\n\n## Quick Start\n\n### 1. Check your schema (no API key needed)\n\n```bash\n# From MCP tools JSON\nmcpeval check my_server.json\n\n# From OpenAPI spec\nmcpeval check api_spec.yaml\n\n# Generate a sample to try\nmcpeval init\nmcpeval check mcpeval.json\n```\n\n### 2. Simulate LLM tool selection\n\n```bash\n# Test if an LLM actually picks the right tool for natural language queries\nmcpeval check my_server.json --simulate --provider gemini\n\n  Simulation Results\n  Accuracy: 75% (6/8)\n\n  ✓  \"Find users named John\"        → search_users     ✓\n  ✗  \"Remove the old account\"       → delete_user      ✗ (got: deactivate_user)\n  ✓  \"Update email to new@test.com\" → update_user      ✓\n```\n\n### 3. Get AI-powered fix suggestions\n\n```bash\nmcpeval check my_server.json --suggest\n\n  AI Suggestions\n\n  1. do_search (naming)\n     - do_search\n     + search_customers\n     Prefix 'do_' is redundant and the noun clarifies what's being searched.\n\n  2. perform_delete (description)\n     - (empty)\n     + Permanently delete a customer record by ID. Cannot be undone.\n     LLMs need clear descriptions to know when to select a tool.\n```\n\n### 4. Use in CI\n\n```bash\n# Exit code 1 on errors (blocks PR)\nmcpeval check my_server.json --ci\n\n# JSON output for programmatic use\nmcpeval check my_server.json --json-output\n```\n\n### 5. Token budget analysis\n\n```bash\nmcpeval tokens my_server.json\n\n  Token Budget\n  Tool                    Tokens   % Budget\n  update_customer_info…      196       50%   ████████████░░░░░░░\n  search_customers            77       20%   ████░░░░░░░░░░░░░░░\n  get                         40       10%   ██░░░░░░░░░░░░░░░░░\n  Total                      393      100%\n```\n\n## What It Checks\n\n### Static Analysis (free, instant)\n\n| Check | What it catches |\n|-------|----------------|\n| `NO_DESCRIPTION` | Tool has no description — LLM can't know when to use it |\n| `SHORT_DESCRIPTION` | Description too brief for LLM to understand |\n| `LONG_DESCRIPTION` | Description wastes context tokens |\n| `SPACE_IN_NAME` | Spaces break MCP clients |\n| `LONG_NAME` | Names >64 chars confuse LLMs |\n| `REDUNDANT_PREFIX` | `do_`, `perform_`, `execute_` prefixes add nothing |\n| `TOO_MANY_PARAMS` | >10 params — LLMs struggle with complex inputs |\n| `PARAM_NO_DESC` | Parameter has no description |\n| `MISSING_ENUM` | Looks like it needs enum values (status, type, mode) |\n| `AMBIGUOUS_PARAM` | Generic names: `data`, `value`, `input`, `item` |\n| `SIMILAR_TOOLS` | Two tools with >70% similar descriptions |\n| `TOKEN_HOG` | Single tool uses >1000 tokens |\n| `GENERIC_NAME` | Tool named `search` or `get` in a server with 5+ tools |\n\n### LLM Simulation (requires API key)\n\n- Sends natural language queries to the LLM with your tool schemas\n- Measures: does it pick the right tool? Pass the right args?\n- Tests positive cases (should trigger tool) and negative cases (should not)\n- Reports accuracy score and identifies failure cases\n\n## Input Formats\n\nmcpeval accepts:\n\n**MCP Tools JSON** (most common):\n```json\n{\n  \"name\": \"my-server\",\n  \"tools\": [\n    {\n      \"name\": \"search_docs\",\n      \"description\": \"Search documents by query\",\n      \"inputSchema\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"query\": { \"type\": \"string\", \"description\": \"Search term\" }\n        },\n        \"required\": [\"query\"]\n      }\n    }\n  ]\n}\n```\n\n**OpenAPI/Swagger spec** (YAML or JSON):\n```bash\nmcpeval check openapi.yaml\n```\n\n**Custom test cases** (for simulation):\n```yaml\n# tests.yaml\n- query: \"Find all active users\"\n  expected_tool: search_users\n- query: \"What's the weather?\"\n  expected_tool: null  # should NOT trigger any tool\n```\n\n```bash\nmcpeval check server.json --simulate --cases tests.yaml\n```\n\n## pytest Integration\n\n```python\n# test_my_mcp_server.py\n\ndef test_schema_quality(assert_mcp_quality):\n    \"\"\"Fail if schema score drops below 80 or has errors.\"\"\"\n    assert_mcp_quality(\"mcp_tools.json\", min_score=80)\n\ndef test_no_token_bloat(mcp_schema):\n    \"\"\"Ensure total schema tokens stay reasonable.\"\"\"\n    assert mcp_schema.total_tokens < 2000\n\ndef test_all_tools_documented(mcp_schema):\n    \"\"\"Every tool must have a description.\"\"\"\n    for tool in mcp_schema.tools:\n        assert tool.description, f\"{tool.name} has no description\"\n```\n\n```bash\npytest --mcpeval-schema mcp_tools.json\n```\n\n## GitHub Action\n\n```yaml\nname: MCP Schema Check\non: [push, pull_request]\njobs:\n  mcpeval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-python@v5\n        with:\n          python-version: \"3.11\"\n      - run: pip install mcpeval\n      - run: mcpeval check mcp_tools.json --ci\n```\n\nAutomatically posts a schema report as a PR comment when schema files change (see `.github/workflows/pr-comment.yml`).\n\n## Schema Diffing (catch regressions)\n\n```bash\n# Compare before/after and block CI on regressions\nmcpeval diff schema_v1.json schema_v2.json --ci\n\n  Schema Diff: schema_v1.json → schema_v2.json\n  Score: 72 → 45 (-27)\n  Tokens: +340\n  - Removed: get_user\n  ~ search: description changed; added params: filter; tokens: +85\n\n  Regressions:\n    ✗ Score dropped: 72 → 45\n    ✗ Removed tools: get_user\n    ✗ 3 new schema issues introduced\n```\n\n### Watch mode (live development)\n\n```bash\n# Re-runs analysis every time you save the file\nmcpeval watch my_server.json\n```\n\n### PR comment reports\n\n```bash\n# Generate markdown for PR comments\nmcpeval report my_server.json -f markdown -o report.md\n\n# Get a badge URL\nmcpeval report my_server.json -f badge\n# → https://img.shields.io/badge/mcpeval-85%2F100%20(B)-brightgreen\n```\n\n### Auto-fix\n\n```bash\n# Fix common issues automatically (renames, adds inferred descriptions)\nmcpeval fix my_server.json\n  → Renamed 'do_search' → 'search'\n  → Renamed 'perform_delete' → 'delete'\n  → Added inferred description to 'delete'\n```\n\n### Connect to live MCP servers\n\n```bash\n# Analyze a running MCP server directly via stdio\nmcpeval connect npx -y @modelcontextprotocol/server-filesystem /tmp\n```\n\n## CLI Reference\n\n```\nmcpeval check <spec>       Analyze schema (add --simulate, --suggest, --ci)\nmcpeval diff <v1> <v2>     Compare versions, detect regressions\nmcpeval fix <spec>         Auto-fix common issues\nmcpeval watch <spec>       Live re-analysis on file changes\nmcpeval report <spec>      Generate markdown/badge reports\nmcpeval tokens <spec>      Show token usage breakdown\nmcpeval improve <spec>     Get AI suggestions (requires API key)\nmcpeval gen <spec>         Generate test cases from schema\nmcpeval connect <cmd...>   Connect to live MCP server via stdio\nmcpeval init               Create sample mcpeval.json\n```\n\n## Why Not Just Use the MCP Inspector?\n\nThe [MCP Inspector](https://github.com/modelcontextprotocol/inspector) (9.6k stars) is for **manual visual testing** — you send requests and look at responses.\n\nmcpeval is for **automated quality assurance**:\n- Catches schema anti-patterns without running your server\n- Simulates how LLMs will interpret your schemas\n- Runs in CI to prevent regressions\n- Gives a score and actionable fixes\n\nThey're complementary: Inspector for debugging, mcpeval for quality gates.\n\n## License\n\nMIT\n"}