mcp

Agent Web Map MCP server

Find documentation for a development task, with source URLs, matching evidence and freshness in one response. Agent Web Map indexes the links, headings and descriptions that websites publish in llms.txt. Agents can use it over MCP or plain HTTP; public reads need no account or API key.

• MCP: https://agentwebmap.com/mcp (Streamable HTTP, stateless)
• REST: https://agentwebmap.com/v1/resolve?q=modelcontextprotocol.io
• Reference: API docs · OpenAPI · directory llms.txt

Add it

The same server works with Codex, Claude Code, Cursor, VS Code and other clients supporting remote HTTP MCP. Merge entries into an existing configuration rather than replacing it. ~ means your home directory.

Codex

codex mcp add agentwebmap --url https://agentwebmap.com/mcp

Or add to ~/.codex/config.toml (all projects), or .codex/config.toml in a trusted project:

[mcp_servers.agentwebmap]
url = "https://agentwebmap.com/mcp"

Official Codex MCP setup.

Claude Code

claude mcp add --transport http agentwebmap --scope project https://agentwebmap.com/mcp

Project configuration lives in .mcp.json at the project root:

{
  "mcpServers": {
    "agentwebmap": {
      "type": "http",
      "url": "https://agentwebmap.com/mcp"
    }
  }
}

Use --scope user for all projects, stored in ~/.claude.json. Official Claude Code MCP setup.

Cursor

Add to .cursor/mcp.json in the project, or ~/.cursor/mcp.json for all projects:

{
  "mcpServers": {
    "agentwebmap": {
      "url": "https://agentwebmap.com/mcp"
    }
  }
}

Official Cursor MCP setup.

VS Code

Add to .vscode/mcp.json in the workspace. VS Code uses servers as the top-level key:

{
  "servers": {
    "agentwebmap": {
      "type": "http",
      "url": "https://agentwebmap.com/mcp"
    }
  }
}

For profile-level configuration, run MCP: Open User Configuration from the Command Palette. Current VS Code also supports the portable .mcp.json format shown for Claude Code, and user configuration in ~/.copilot/mcp-config.json using mcpServers. Official VS Code MCP setup.

Plain HTTP

curl --get --fail-with-body https://agentwebmap.com/v1/resolve \
  --data-urlencode "q=modelcontextprotocol authorization" \
  --data-urlencode "limit=3" \
  --data-urlencode "max_age=86400"

A dependency-free MCP client with initialization and tool discovery is in examples/mcp-client.mjs:

node examples/mcp-client.mjs resolve '{"query":"modelcontextprotocol authorization","limit":3}'

Give agents a reason to call it

Use resolve when a task needs a site's documentation and the exact page is unknown. Pass the publisher and task when known, for example modelcontextprotocol authorization. Read the returned source page before implementing or quoting it. Fall back to the site's documentation search or web search if evidence is weak, missing or stale. An empty result is useful information; listing ownership does not prove technical relevance.

An optional, portable agentwebmap-docs skill describes that workflow, including terms, freshness and fallback behavior. To use it in a repository that supports Agent Skills, copy the skills/agentwebmap-docs folder to .agents/skills/agentwebmap-docs. This is an optional project change; connecting the MCP server does not install a skill. Other clients can use the same SKILL.md or adapt it to their instructions format.

The index contains publisher-provided link descriptions and section headings, rather than the full text of every documentation page. Returned excerpts explain the match using that index; fetch the original source for current API details. Missing owner terms mean unknown, not unrestricted permission.

Tools

Tool	What it does
resolve	Domain or task to relevant documentation URLs, evidence, terms, freshness and alternatives; compact by default
search_llms_registry	Search domains, titles, summaries, descriptions, tags and published documentation links; relevance ranks before verification
lookup_domain	Find a domain's llms.txt, llms-full.txt, terms and directory metadata; detail: "full" includes owner information
list_domains	Browse ranked sites; verified_only restricts to owner-verified sites
list_categories	Read category IDs/descriptions/counts, the uncategorized count and publisher-declared capabilities before filtering discovery
directory_stats	Directory-wide counts
check_llms_txt	Live validation of a site's llms.txt, with suggested fixes
generate_llms_txt	Draft an llms.txt from a site's homepage, sitemap and page titles
watch_domain	Subscribe a webhook to changes, breakage and restoration
unwatch_domain / test_watch	Manage an existing watch with its ID and secret
verification_instructions	Ownership token and verification methods
register_domain / verify_domain	Submit a site you control and verify its ownership
update_site	Owner: edit profile, terms and notification channels
site_stats	Owner: lookups, score history, notifications and badge
recrawl_site / test_notification	Owner: revalidate or send a test notification

Owner tools require the site's API key. Watch creation and owner operations have effects outside the current task: use them when requested. Documentation discovery needs only public read tools. Tools publish output schemas and annotations so clients can identify reads and writes.

Freshness and response size

resolve accepts detail: "compact" | "full", max_age in seconds (0–2592000, default 86400), refresh: true for live llms.txt validation, and limit from 1 to 5 for documentation sources (default 3). health.stale reports whether the check is older than the requested age; health.live_checked identifies a live check. A refresh checks the published llms.txt; it does not validate every linked page or update the stored index. Compact results keep owner-verification instructions out of repeated consumer calls. Use detail: "full" when those details are needed.

Discover by category

Call list_categories or GET /v1/categories to discover the taxonomy, then pass one category ID to resolve, search_llms_registry or list_domains. REST search, domain listing and resolve accept the same category filter. Compact domain responses include categories; full responses add the matching evidence and classification version. Use capability: "mcp" (REST capability=mcp) to discover sites publishing an explicit MCP endpoint link, separately from their business categories.

Categories are deterministic, multi-label classifications from publisher metadata. A site can have several labels or remain uncategorized when evidence is insufficient. Confidence is a rule score, not a probability, endorsement or verified business fact. Business categories do not prove that a site runs an MCP server; capability detection requires an explicit same-publisher endpoint declaration. Its availability and tools remain unverified, and a generic MCP mention is insufficient. See category discovery and evidence for the taxonomy, response fields and examples.

Measure adoption value

Task examples show discovery, source reading and fallback. The benchmark runs 20 curated documentation tasks and reports source hit rate at 1 and 3, provider matches, latency, response bytes and a clearly labeled token estimate. Run the same tasks against a baseline before claiming an improvement. No benchmark scores are asserted by this repository.

For site owners

Publish an llms.txt with descriptive links to useful documentation pages. Register the domain, place an ownership token, then verify it. Verification records domain control and provides profile editing, badges, alerts and analytics. Search prioritizes relevance; verification is not an endorsement of page accuracy. Owners can publish agent_terms for automated access, rate limits, attribution and pricing through update_site.

Reference and examples

• docs/api.md: API parameters and response fields
• docs/task-examples.md: practical agent workflows
• docs/categories.md: category discovery, filters and publisher evidence
• docs/webhooks.md: payloads and signature verification
• docs/agent-terms.md: terms-for-agents specification
• docs/measurement.md: Google Analytics, Search Console, Bing verification and sitemap setup
• openapi.json: OpenAPI snapshot; the live specification describes the deployed service
• examples/: shell, Python and Node clients, ownership verification and watches
• benchmarks/: reproducible task dataset and dependency-free runner

Source and local development

This repository includes the application in src/, its tests/, and the Node package files, alongside the client examples and documentation. Use Node.js 20.18.1 or newer (Node.js 22.7+ recommended for the test runner) and a local PostgreSQL 16 instance. Create a database named agentwebmap owned by a local agentwebmap role with password agentwebmap, or set DATABASE_URL in your shell to another PostgreSQL connection string. The default development connection is postgres://agentwebmap:agentwebmap@localhost:5432/agentwebmap.

npm ci
npm start

Open http://localhost:3000; the application initializes its database schema on startup. Configuration comes from process environment variables; .env.example lists available settings, and npm start does not load an .env file automatically. To run the tests with Node.js 22.7+:

npm test

Tests use an isolated in-memory PostgreSQL implementation and local HTTP fixtures; they do not require a running PostgreSQL server. The local directory starts empty. To populate it, register a domain through your local API, then optionally run npm run crawl against due domains. Crawling fetches public websites and is separate from starting the web server. Free top-list seed commands are available in package.json; the optional seed:dataforseo command requires your own credentials and incurs provider charges only when explicitly invoked. It is never run automatically.

The included docker-compose.yml and deploy/ scripts describe a server deployment with an existing Caddy proxy and external Docker network named edge; the Compose file does not publish a localhost web port. Use the local PostgreSQL setup above for development, or adapt the deployment configuration to your own host.

Questions and ideas: Discussions. Contact: admin@agentwebmap.com.

Brand and license

The brand assets may be used to link to agentwebmap.com. The X profile banner matches the website palette in a 3:1 format. This repository is released under the MIT License; the directory data and service are provided by agentwebmap.com.