Skip to content
Conduit

mcp server

web-search-mcp

Multi-provider web search MCP with reranking, SSRF-safe fetch, and eval harness.

charley-forey/web-search-mcpcommunitylistedunreviewed
Use tool

Set up

Runs on your computer. Pick your agent and operating system.

Before you start

  • Node.js 20 or newer
~/Library/Application Support/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "charley-forey-web-search-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "web-search-mcp"
      ]
    }
  }
}

same format Cursor, GitHub Copilot CLI, Junie, JetBrains AI Assistant, Augment (Auggie CLI), Factory Droid, Kiro, Devin CLI +12

Values in angle brackets are placeholders. Restart Claude Desktop after editing its config.Claude Desktop docs

README

synced Oct 23, 2026

web-search-mcp

Best-in-class web search MCP server for grounded agent retrieval.

Features

  • Multi-provider search (tavily, brave, exa, serper, searxng, duckduckgo)
  • Provider fallback chain and optional ensembling
  • Optional local semantic rerank (RERANKER=local) with @xenova/transformers
  • SSRF-safe URL fetching with robots policy support
  • Injection defense: <untrusted_content> wrapping, hidden-char stripping, heuristic detection
  • HTML, PDF, RSS, and YouTube transcript extraction
  • Reranking, dedupe/canonicalization, chunk anchors for citation
  • Persistent search/page memory with SQLite (STATE_BACKEND=sqlite)
  • MCP-native capabilities: tools, resources, prompts, completions, sampling, elicitation
  • Evaluation harness (Recall@5, MRR@10) and provider A/B compare CLI
  • Production endpoints: /livez, /readyz, /metrics, /health

Install

npm i
npm run build

Configure via .env

Copy and fill the template:

cp .env.example .env

Supported keys:

  • SEARCH_PROVIDER=tavily|brave|exa|serper|duckduckgo
  • TAVILY_API_KEY, BRAVE_API_KEY, EXA_API_KEY, SERPER_API_KEY
  • SEARCH_PROVIDER_FALLBACK=brave,duckduckgo
  • HOST=127.0.0.1
  • STATE_BACKEND=memory|sqlite|redis
  • SQLITE_PATH=.cache/web-search-mcp.sqlite
  • DOMAIN_ALLOWLIST=*.gov,*.edu
  • DOMAIN_BLOCKLIST=internal.example.com
  • HTTP_RATE_LIMIT_RPS=30
  • FETCH_MAX_BODY_BYTES=5000000
  • PORT=8080
  • MCP_BASE_PATH=/mcp (set /mcp/search if desired)
  • CORS_ORIGIN=* (or a specific origin)
  • MCP_AUTH_TOKEN=<secret> (optional bearer token for MCP endpoint)
  • DEBUG=0|1

Run locally (default streamable MCP)

npm run dev

This auto-loads .env and starts a streamable MCP endpoint at:

  • http://127.0.0.1:8080/mcp (or HOST/PORT + MCP_BASE_PATH from .env)

If you need stdio mode explicitly:

npm run dev:stdio

MCP client configuration

Streamable MCP (recommended)

Use URL-based config:

{
  "mcpServers": {
    "web-search": {
      "url": "http://127.0.0.1:8080/mcp"
    }
  }
}

If you set MCP_BASE_PATH=/mcp/search, use:

{
  "mcpServers": {
    "web-search": {
      "url": "http://127.0.0.1:8080/mcp/search"
    }
  }
}

If MCP_AUTH_TOKEN is enabled, include:

  • Authorization: Bearer <token>

Stdio fallback (.cursor/mcp.json)

If your client requires command/stdio mode, use:

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "tavily",
        "TAVILY_API_KEY": "YOUR_KEY",
        "SEARCH_PROVIDER_FALLBACK": "brave,duckduckgo"
      }
    }
  }
}

Claude Desktop (stdio fallback)

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "tavily",
        "TAVILY_API_KEY": "YOUR_KEY"
      }
    }
  }
}

VS Code MCP (stdio fallback)

{
  "servers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "duckduckgo"
      }
    }
  }
}

Tools

  • search
  • fetch_url
  • search_and_extract
  • news_search
  • academic_search
  • get_youtube_transcript
  • extract_pdf
  • check_config

Prompts

  • research_topic
  • fact_check
  • latest_news

Resources

  • web-search://page/{id}
  • web-search://history/{id}

CLI

npm run compare -- --query "model context protocol"
npm run eval

Inspector validation

npx @modelcontextprotocol/inspector npm run dev

Smoke test

curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/livez
curl -s http://127.0.0.1:8080/readyz
curl -s http://127.0.0.1:8080/metrics

Expected:

{ "status": "ok", "version": "0.1.0", "provider": "..." }

With auth enabled:

curl -i http://127.0.0.1:8080/mcp
# 401 unauthorized (without Bearer token)

Docker

Build:

docker build -t web-search-mcp .

Run with env file:

docker run --env-file .env -p 8080:8080 web-search-mcp

Then validate each tool:

  • search
  • fetch_url
  • search_and_extract
  • news_search
  • academic_search
  • get_youtube_transcript
  • extract_pdf
  • check_config

For nginx/Kubernetes deployment guidance see docs/NGINX.md and manifests in deploy/k8s plus Helm chart in charts/web-search-mcp.

Build and release

npm test
npm run build

Publish:

npm publish --provenance
↑↓ navigate openesc close