web-search-mcp

web-search MCP server

A self-hosted web search MCP server for Claude Code / any MCP client. Multi-engine (CN-direct + international-via-proxy auto-routing), three tools, and research-backed optimizations: RRF rank fusion, SimHash near-duplicate dedup, per-host rate limiting, frequency-aware caching, and TLS/JA3 fingerprint impersonation fallback.

中文文档

---

Installation

Prerequisites

• Node.js ≥ 18

• Python 3 + curl_cffi (optional, for TLS impersonation on protected sites):

  pip install curl_cffi
  

• A proxy on 127.0.0.1:7890 (e.g) — set PROXY_URL="" to disable.

Build

git clone git@github.com:nuoyax/web-search-mcp.git
cd web-search-mcp
npm install

Wire into Claude Code

User-level (available in all projects):

claude mcp add web-search -s user -e PROXY_URL=http://127.0.0.1:7890 \
  -- node /absolute/path/to/web-search-mcp/index.js

Or add to .mcp.json (project-level):

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/absolute/path/to/web-search-mcp/index.js"],
      "env": { "PROXY_URL": "http://127.0.0.1:7890" }
    }
  }
}

Verify

node index.js          # start the MCP server (stdio)
node test-smoke.js     # smoke test every engine + fetch

(Recommended) Disable Claude Code's built-in WebFetch

Claude Code ships a built-in WebFetch tool that fetches via claude.ai's server-side domain-safety check. On a restricted network it errors with Unable to verify if domain … is safe to fetch … blocking claude.ai, and it can't use your proxy. Since this server's fetch_url already fetches through 127.0.0.1:7890 (and TLS-impersonates on 403), disable the built-in one so Claude always uses fetch_url.

Add WebFetch to permissions.deny in ~/.claude/settings.json (global, all projects):

{
  "permissions": {
    "deny": ["WebFetch"]
    // also common: "WebSearch" if you want web_search to fully replace it
  }
}

Or, project-only — D:\agents\web_search\.claude\settings.local.json (gitignored):

{ "permissions": { "deny": ["WebFetch"] } }

deny rules stack across the user → project → local layers and are honored even in bypassPermissions mode. Restart Claude Code after editing. Verify with /permissions.

---

Highlights

• 8 engines — DuckDuckGo / Brave / Wikipedia / Bing (international, via proxy) + Bing CN / Baidu / Sogou / 360 (direct). Auto-selected by query language.
• 3 tools — web_search, fetch_url, deep_research (multi-engine fan-out → dedup → rank → fetch → cited report)
• Auto proxy routing — CN hosts go direct, international hosts go through 127.0.0.1:7890; re-decided per redirect hop.
• RRF rank fusion (k=60) across engines — robust without score normalization.
• SimHash dedup — 64-bit, Hamming ≤ 3, merges syndicated copies across hosts.
• Per-host token bucket + exponential backoff — defeats frequency-based anti-bot detection.
• Disk cache with frequency-aware TTL (search 30 min / news 1 h / docs 7 d).
• TLS/JA3 impersonation via curl_cffi for Cloudflare-protected sites (e.g. docs.anthropic.com).

---

Architecture

flowchart TB
    Client["Claude Code / MCP Client\n(stdio · JSON-RPC)"]

    subgraph Server["web-search MCP Server  (index.js)"]
        T1["web_search"]
        T2["fetch_url"]
        T3["deep_research"]
        Cache["cache.js\n disk KV · frequency-aware TTL"]
    end

    subgraph Core["Core layer"]
        Eng["engines.js\n8 engine adapters"]
        Fetch["fetcher.js\nHTML → markdown\n+ TLS fallback dispatch"]
        Res["research.js\nRRF fusion + SimHash dedup"]
    end

    subgraph HTTP["http.js  (undici primary path)"]
        Proxy["Proxy routing\nCN direct / Intl → 7890\nre-routed per hop"]
        Bucket["Token bucket\n1 concurrent / host\n+120ms interval"]
        Backoff["Exponential backoff\n429 / 5xx retry\n+ Retry-After"]
        Undici["decompress interceptor\n+ manual 3xx loop"]
        Sim["simhash.js\n64-bit Charikar near-dup"]
    end

    TLS["tlsbypass.js → curl_cffi (Python)\nbrowser JA3/JA4 impersonation"]

    CN["CN engines (direct)\nbaidu · bingcn · sogou · so"]
    Intl["International engines (proxy 127.0.0.1:7890)\nduckduckgo · bing"]

    Client <--> Server
    T1 --> Core
    T2 --> Core
    T3 --> Core
    T1 -.cache.-> Cache
    T2 -.cache.-> Cache
    T3 -.cache.-> Cache
    Core --> HTTP
    HTTP -- "403 / 429\nor hard-case host" --> TLS
    HTTP --> CN
    HTTP --> Intl

deep_research pipeline

flowchart TB
    Q(["query"]) --> S1["1. Engine selection\nCJK → CN first · Latin → Intl first\nor engines=[...] / all"]
    S1 --> S2["2. Fan-out search\nall engines in parallel · fault-tolerant"]
    S2 --> S3["3. Baidu redirect resolution\nbaidu.com/link?url= → real URL"]
    S3 --> S4["4. Two-stage dedup\nstage 1: exact URL · stage 2: SimHash Hamming ≤ 3"]
    S4 --> S5["5. RRF ranking\nscore = Σ 1/(60 + rank)"]
    S5 --> S6["6. Fetch top-K\nmax 4 concurrent · token bucket + backoff\nTLS → curl_cffi on 403"]
    S6 --> S7["7. Synthesize report\ncited markdown"]
    S7 --> R(["cited report"])

---

Principles

1. Auto proxy routing (CN direct / international via proxy)

src/http.js keeps a list of CN-domain suffixes (baidu.com / bing.com / so.com …). On each request the host suffix decides:

• CN domain hit → direct (proxying CN sites is slower and trips their risk control)
• International host → through 127.0.0.1:7890 (works around host-network issues reaching foreign sites)
• On redirect, the route is re-decided per hop: a 301 from an international site to a CN CDN stops using the proxy
• engine.search() may override with forceProxy / forceDirect (e.g. DuckDuckGo must be proxied)
• Set PROXY_URL="" to disable the proxy (e.g. when a global VPN is active)

2. RRF rank fusion (Reciprocal Rank Fusion)

src/research.js. Each result's score = Σ_engine 1/(k + rankᵢ), with k=60.

• Unsupervised: no score normalization needed; robust to engines returning different counts
• Cross-engine consensus: results that appear in multiple engines near the top naturally surface
• Query-term-in-title is only a 1e-4-scale tiebreaker; it never overrides cross-engine consensus
• k=60 is the standard default from Cormack et al. (2009) and ranx.fuse

3. SimHash near-duplicate dedup

src/simhash.js. 64-bit SimHash (Charikar), tokenized with FNV-1a hashing + unigram/bigram.

• Two stages: exact normalized-URL merge first, then SimHash Hamming ≤ 3 as near-duplicate
• The fingerprint excludes the host, so syndicated copies across different hosts merge (the same article republished by different portals, or the same result wrapped differently by Baidu vs Bing)
• Verified: identical text under different hosts → distance 0; reworded → 11; unrelated → 36 — discriminative

4. Per-host token bucket + exponential backoff (anti frequent-access detection)

src/http.js. Per host:

• Concurrency 1 + ≥120 ms between requests (HOST_BUCKETS)
• Transient errors (429/502/503/504) + network failures: exponential backoff 500·2^attempt + jitter, up to 2 retries
• Honors the Retry-After response header
• Different hosts still run in parallel (no throughput loss)

This directly targets the main cause of "frequent-access detection" — high request rate to a single host.

5. Result caching + frequency-aware TTL (incremental)

src/cache.js. Disk JSON KV with TTLs scaled by site update frequency:

Site category	TTL	Rationale
Search-engine result pages (baidu/bing/ddg…)	30 min	result ordering shifts fast
News aggregators (163/sina/cctv/reuters…)	1 h	high-churn
Docs/API/encyclopedia (docs./wikipedia/arxiv…)	7 d	stable content
Other	6 h	default

• web_search / fetch_url / deep_research return cached results on hit (tagged _cached (age Nmin, ttl Mmin)_)
• Only caches when there are results (empty results aren't cached, so the next call retries)
• CACHE_DISABLED=1 disables; CACHE_DIR overrides the cache directory
• Rationale: frequency-aware incremental crawling (2010) — refresh interval scaled by page change frequency

6. TLS/JA3 fingerprint impersonation fallback (curl_cffi)

src/tlsbypass.js + src/fetcher.js. Node undici's TLS ClientHello differs from a real browser, so heavily-protected sites (Cloudflare-protected docs.anthropic.com etc.) return 403. curl_cffi (Python) can impersonate a real browser's JA3/JA4 fingerprint + HTTP/2 settings.

Strategy:

• Fast path: undici by default (fast, native)
• Known hard-case hosts (docs/platform.anthropic.com etc.): go straight to curl_cffi
• Generic fallback: on undici 403/429 or network error → auto-fallback to curl_cffi; if both fail, report honestly (with impersonated flag)
• Reports tag TLS: impersonated (curl_cffi) or native (undici)
• Requires Python + pip install curl_cffi; without it, degrades to undici-only (main flow unaffected)
• Rationale: TLS Beyond the Browser (ACM IMC 2019) — non-browser TLS clients are fingerprintable; impersonation closes that gap

7. Engine adapters

Engine	Region	Route	Notes
duckduckgo	international	proxy	html.duckduckgo.com/html/ no-key HTML endpoint
brave	international	proxy	search.brave.com
wikipedia	international	proxy	en.wikipedia.org search API
bing	international	proxy	setmkt=en-US&cc=US; reconstructs real URL from <cite> (bypasses the ck/a redirect wrapper)
bingcn	CN	direct	cn.bing.com
baidu	CN	direct	Detects CAPTCHA interstitial and errors out; reads mu attr for the real URL
sogou	CN	direct	Mobile + desktop dual fallback to dodge anti-bot
so (360)	CN	direct	www.so.com

A query with CJK characters defaults to CN engines first; pure-Latin queries go international first. engine='all' fans out to every engine.

8. Key undici v7 handling

• interceptors.decompress(): auto-decompress gzip/br/deflate (Bing international returns brotli)
• Manual 3xx redirect loop: undici v7's request() no longer accepts maxRedirections, and the redirect interceptor alone isn't enough (per-request opts default to 0 and short-circuit). The manual loop also lets us re-route proxy/direct per hop.

---

Tools

web_search

{ "query": "anthropic claude api pricing", "num": 8, "engine": "auto" }

• engine: auto (default) | all | duckduckgo | bing | bingcn | baidu | sogou | so

fetch_url

{ "url": "https://example.com/page", "max_chars": 16000 }

Fetch a URL, strip boilerplate, return markdown. Auto-routes proxy/direct by domain; falls back to TLS impersonation on 403.

deep_research

{
  "query": "中国空间站 最新进展",
  "engines": ["bingcn", "baidu"],
  "num_per_engine": 8,
  "fetch_top_k": 4,
  "fetch_chars": 6000
}

Multi-engine fan-out → dedup & RRF rank → fetch top-K → cited markdown report.

---

Environment variables

Variable	Default	Effect
PROXY_URL	http://127.0.0.1:7890	proxy endpoint; empty string disables proxying
CACHE_DISABLED	0	set 1 to disable result caching
CACHE_DIR	./cache	cache directory

---

Referenced papers

Papers referenced in the implementation (DOIs verifiable; retrieved via the OpenAlex academic API + DuckDuckGo/Bing):

Result fusion / ranking

Paper	Year	Used for	DOI
ranx.fuse: A Python Library for Metasearch	2022 (CIKM)	RRF implementation reference; 25 fusion algorithms	10.1145/3511808.3557207
Comparing Rank and Score Combination Methods for Data Fusion in IR	2005	Comparison of CombMNZ etc.; basis for RRF k	10.1007/s10791-005-6994-4
The use of MMR, diversity-based reranking	1998 (Carbonell)	Foundational diversity reranking (not yet implemented)	10.1145/290941.291025
Fusion-based methods for result diversification in web search	2018 (Information Fusion)	Combining fusion and diversity	10.1016/j.inffus.2018.01.006

Near-duplicate detection

Paper	Year	Used for	DOI
A Review for Weighted MinHash Algorithms	2018	MinHash/SimHash survey	10.48550/arxiv.1811.04633
Improved Near-Duplicate Detection for Aggregated and Paywalled News	2025 (NAACL)	Recent advances in near-dup detection	10.18653/v1/2025.naacl-industry.73
Effective and Fast Near Duplicate Detection via Signature-Based	2016	Signature-based dedup engineering	10.1155/2016/3919043

Parallel / incremental crawling / rate limiting

Paper	Year	Used for	DOI
BUbiNG: Massive Crawling for the Masses	2016	Per-host politeness queue, linear scaling	10.48550/arxiv.1601.06919
SIMHAR — Smart Distributed Web Crawler for the Hidden Web	2020 (IEEE Access)	Distributed queue + SIM+Hash dedup	10.1109/access.2020.3004756
Design of a Priority Based Frequency Regulated Incremental Crawler	2010	Frequency-aware incremental crawling (caching)	10.5120/23-131
On the Feasibility of Geographically Distributed Web Crawling	2008	Geo-distributed crawling reduces latency	10.4108/icst.infoscale2008.3550

Anti-bot / blocking evasion

Paper	Year	Used for	DOI
TLS Beyond the Browser	2019 (ACM IMC)	TLS/JA3 fingerprint exposure (P1)	10.1145/3355369.3355601
FP-Crawlers: Studying the Resilience of Browser Fingerprinting	2020	Browser fingerprinting detection	10.14722/madweb.2020.23010
A First Look at User-Installed Residential Proxies	2024 (CNSM)	Residential-proxy ecosystem (proxy pool, planned)	10.23919/cnsm62983.2024.10814519

Full optimization plan and gap analysis: docs/optimization-research.md.

---

Project structure

web-search-mcp/
├── index.js              # MCP server entry (registers 3 tools, stdio)
├── src/
│   ├── http.js           # HTTP layer + proxy routing + token bucket + backoff
│   ├── engines.js        # 8 engine adapters
│   ├── fetcher.js        # HTML → markdown + TLS fallback dispatch
│   ├── research.js       # RRF fusion + SimHash dedup pipeline
│   ├── simhash.js        # 64-bit Charikar SimHash
│   ├── cache.js          # disk KV with frequency-aware TTL
│   └── tlsbypass.js      # curl_cffi TLS impersonation bridge
├── test-smoke.js         # smoke test
├── docs/
│   ├── optimization-research.md
│   ├── architecture.svg
│   └── pipeline.svg
├── .mcp.json             # project-level MCP registration
└── package.json

---

Known limitations

• All network requests time out in 20–25 s; nothing hangs.
• Baidu occasionally returns a CAPTCHA interstitial; that engine then errors and deep_research falls back to the other engines.
• Bing international wraps result links in bing.com/ck/a; the real URL is reconstructed from the result's <cite>.
• DuckDuckGo uses the no-key html.duckduckgo.com/html/ endpoint, which requires the proxy.
• TLS impersonation depends on Python + curl_cffi. Without it, falls back to undici-only and hard-case sites (e.g. docs.anthropic.com) return 403; with it, impersonation is automatic.
• The cache expires by TTL without background refresh; the next call after expiry re-fetches. Set CACHE_DISABLED=1 to disable.

---

License

MIT