{"id":"1c5f3830-9477-4e27-a502-2e16c4928eda","slug":"charley-forey-web-search-mcp","kind":"mcp","name":"web-search-mcp","summary":"Multi-provider web search MCP with reranking, SSRF-safe fetch, and eval harness.","owner":"charley-forey","repoUrl":"https://github.com/charley-forey/web-search-mcp","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-07-15T03:37:07.000Z","addedAt":"2026-10-21T21:32:01.214Z","logoUrl":"https://avatars.githubusercontent.com/u/183576736?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["search","browser-web"],"tags":[],"runtime":"node","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-23T08:25:04.395Z","sample":false,"url":"https://conduits.dev/tools/charley-forey-web-search-mcp","language":"TypeScript","transports":["stdio"],"install":{"npm":"web-search-mcp"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# web-search-mcp\n\nBest-in-class web search MCP server for grounded agent retrieval.\n\n## Features\n\n- Multi-provider search (`tavily`, `brave`, `exa`, `serper`, `searxng`, `duckduckgo`)\n- Provider fallback chain and optional ensembling\n- Optional local semantic rerank (`RERANKER=local`) with `@xenova/transformers`\n- SSRF-safe URL fetching with robots policy support\n- Injection defense: `<untrusted_content>` wrapping, hidden-char stripping, heuristic detection\n- HTML, PDF, RSS, and YouTube transcript extraction\n- Reranking, dedupe/canonicalization, chunk anchors for citation\n- Persistent search/page memory with SQLite (`STATE_BACKEND=sqlite`)\n- MCP-native capabilities: tools, resources, prompts, completions, sampling, elicitation\n- Evaluation harness (`Recall@5`, `MRR@10`) and provider A/B compare CLI\n- Production endpoints: `/livez`, `/readyz`, `/metrics`, `/health`\n\n## Install\n\n```bash\nnpm i\nnpm run build\n```\n\n## Configure via `.env`\n\nCopy and fill the template:\n\n```bash\ncp .env.example .env\n```\n\nSupported keys:\n\n- `SEARCH_PROVIDER=tavily|brave|exa|serper|duckduckgo`\n- `TAVILY_API_KEY`, `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`\n- `SEARCH_PROVIDER_FALLBACK=brave,duckduckgo`\n- `HOST=127.0.0.1`\n- `STATE_BACKEND=memory|sqlite|redis`\n- `SQLITE_PATH=.cache/web-search-mcp.sqlite`\n- `DOMAIN_ALLOWLIST=*.gov,*.edu`\n- `DOMAIN_BLOCKLIST=internal.example.com`\n- `HTTP_RATE_LIMIT_RPS=30`\n- `FETCH_MAX_BODY_BYTES=5000000`\n- `PORT=8080`\n- `MCP_BASE_PATH=/mcp` (set `/mcp/search` if desired)\n- `CORS_ORIGIN=*` (or a specific origin)\n- `MCP_AUTH_TOKEN=<secret>` (optional bearer token for MCP endpoint)\n- `DEBUG=0|1`\n\n## Run locally (default streamable MCP)\n\n```bash\nnpm run dev\n```\n\nThis auto-loads `.env` and starts a streamable MCP endpoint at:\n\n- `http://127.0.0.1:8080/mcp` (or `HOST`/`PORT` + `MCP_BASE_PATH` from `.env`)\n\nIf you need stdio mode explicitly:\n\n```bash\nnpm run dev:stdio\n```\n\n## MCP client configuration\n\n### Streamable MCP (recommended)\n\nUse URL-based config:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-search\": {\n      \"url\": \"http://127.0.0.1:8080/mcp\"\n    }\n  }\n}\n```\n\nIf you set `MCP_BASE_PATH=/mcp/search`, use:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-search\": {\n      \"url\": \"http://127.0.0.1:8080/mcp/search\"\n    }\n  }\n}\n```\n\nIf `MCP_AUTH_TOKEN` is enabled, include:\n\n- `Authorization: Bearer <token>`\n\n### Stdio fallback (`.cursor/mcp.json`)\n\nIf your client requires command/stdio mode, use:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-search\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"web-search-mcp\", \"--stdio\"],\n      \"env\": {\n        \"SEARCH_PROVIDER\": \"tavily\",\n        \"TAVILY_API_KEY\": \"YOUR_KEY\",\n        \"SEARCH_PROVIDER_FALLBACK\": \"brave,duckduckgo\"\n      }\n    }\n  }\n}\n```\n\n### Claude Desktop (stdio fallback)\n\n```json\n{\n  \"mcpServers\": {\n    \"web-search\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"web-search-mcp\", \"--stdio\"],\n      \"env\": {\n        \"SEARCH_PROVIDER\": \"tavily\",\n        \"TAVILY_API_KEY\": \"YOUR_KEY\"\n      }\n    }\n  }\n}\n```\n\n### VS Code MCP (stdio fallback)\n\n```json\n{\n  \"servers\": {\n    \"web-search\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"web-search-mcp\", \"--stdio\"],\n      \"env\": {\n        \"SEARCH_PROVIDER\": \"duckduckgo\"\n      }\n    }\n  }\n}\n```\n\n## Tools\n\n- `search`\n- `fetch_url`\n- `search_and_extract`\n- `news_search`\n- `academic_search`\n- `get_youtube_transcript`\n- `extract_pdf`\n- `check_config`\n\n## Prompts\n\n- `research_topic`\n- `fact_check`\n- `latest_news`\n\n## Resources\n\n- `web-search://page/{id}`\n- `web-search://history/{id}`\n\n## CLI\n\n```bash\nnpm run compare -- --query \"model context protocol\"\nnpm run eval\n```\n\n## Inspector validation\n\n```bash\nnpx @modelcontextprotocol/inspector npm run dev\n```\n\n## Smoke test\n\n```bash\ncurl -s http://127.0.0.1:8080/health\ncurl -s http://127.0.0.1:8080/livez\ncurl -s http://127.0.0.1:8080/readyz\ncurl -s http://127.0.0.1:8080/metrics\n```\n\nExpected:\n\n```json\n{ \"status\": \"ok\", \"version\": \"0.1.0\", \"provider\": \"...\" }\n```\n\nWith auth enabled:\n\n```bash\ncurl -i http://127.0.0.1:8080/mcp\n# 401 unauthorized (without Bearer token)\n```\n\n## Docker\n\nBuild:\n\n```bash\ndocker build -t web-search-mcp .\n```\n\nRun with env file:\n\n```bash\ndocker run --env-file .env -p 8080:8080 web-search-mcp\n```\n\nThen validate each tool:\n\n- `search`\n- `fetch_url`\n- `search_and_extract`\n- `news_search`\n- `academic_search`\n- `get_youtube_transcript`\n- `extract_pdf`\n- `check_config`\n\nFor nginx/Kubernetes deployment guidance see [`docs/NGINX.md`](docs/NGINX.md) and manifests in `deploy/k8s` plus Helm chart in `charts/web-search-mcp`.\n\n## Build and release\n\n```bash\nnpm test\nnpm run build\n```\n\nPublish:\n\n```bash\nnpm publish --provenance\n```\n"}