{"id":"d2aec775-b24b-49fc-8d42-0eeadb935282","slug":"nuoyax-web-search-mcp","kind":"mcp","name":"web-search-mcp","summary":"Self-hosted web search MCP server (Node). 6 engines (CN-direct + Intl-via-proxy auto-routing), RRF fusion, SimHash dedup, per-host rate limiting, frequency-aware cache, TLS/JA3 impersonation fallback.","owner":"nuoyax","repoUrl":"https://github.com/nuoyax/web-search-mcp","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-08-08T03:04:35.000Z","addedAt":"2026-10-21T04:57:04.711Z","logoUrl":"https://avatars.githubusercontent.com/u/38398739?u=dc20baca5d2d3802ebcca8bf156e2c643b6ef7ee&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["search","browser-web","cloud-infra"],"tags":["ai-agents","caching","deep-research","javascript","llm","nodejs","rate-limiting","rrf"],"runtime":"node","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T11:02:45.591Z","sample":false,"url":"https://conduits.dev/tools/nuoyax-web-search-mcp","language":"JavaScript","transports":["stdio"],"install":{"npm":"web-search-mcp"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# web-search MCP server\n\n[![MCP](https://img.shields.io/badge/MCP-server-blue)](https://modelcontextprotocol.io)\n[![Node](https://img.shields.io/badge/Node-%3E%3D18-green)](https://nodejs.org)\n[![License](https://img.shields.io/badge/license-MIT-brightgreen)](#license)\n\nA self-hosted **web search MCP server** for Claude Code / any MCP client. Multi-engine (CN-direct + international-via-proxy auto-routing), three tools, and research-backed optimizations: RRF rank fusion, SimHash near-duplicate dedup, per-host rate limiting, frequency-aware caching, and TLS/JA3 fingerprint impersonation fallback.\n\n[中文文档](./README.zh.md)\n\n---\n\n## Installation\n\n### Prerequisites\n\n- **Node.js ≥ 18**\n- **Python 3** + `curl_cffi` (optional, for TLS impersonation on protected sites):\n\n  ```bash\n  pip install curl_cffi\n  ```\n\n- A proxy on `127.0.0.1:7890` (e.g) — set `PROXY_URL=\"\"` to disable.\n\n### Build\n\n```bash\ngit clone git@github.com:nuoyax/web-search-mcp.git\ncd web-search-mcp\nnpm install\n```\n\n### Wire into Claude Code\n\nUser-level (available in all projects):\n\n```bash\nclaude mcp add web-search -s user -e PROXY_URL=http://127.0.0.1:7890 \\\n  -- node /absolute/path/to/web-search-mcp/index.js\n```\n\nOr add to `.mcp.json` (project-level):\n\n```json\n{\n  \"mcpServers\": {\n    \"web-search\": {\n      \"command\": \"node\",\n      \"args\": [\"/absolute/path/to/web-search-mcp/index.js\"],\n      \"env\": { \"PROXY_URL\": \"http://127.0.0.1:7890\" }\n    }\n  }\n}\n```\n\n### Verify\n\n```bash\nnode index.js          # start the MCP server (stdio)\nnode test-smoke.js     # smoke test every engine + fetch\n```\n\n### (Recommended) Disable Claude Code's built-in `WebFetch`\n\nClaude Code ships a built-in `WebFetch` tool that fetches via claude.ai's server-side domain-safety check. On a restricted network it errors with `Unable to verify if domain … is safe to fetch … blocking claude.ai`, and it can't use your proxy. Since this server's `fetch_url` already fetches through `127.0.0.1:7890` (and TLS-impersonates on 403), disable the built-in one so Claude always uses `fetch_url`.\n\nAdd `WebFetch` to `permissions.deny` in `~/.claude/settings.json` (global, all projects):\n\n```jsonc\n{\n  \"permissions\": {\n    \"deny\": [\"WebFetch\"]\n    // also common: \"WebSearch\" if you want web_search to fully replace it\n  }\n}\n```\n\nOr, project-only — `D:\\agents\\web_search\\.claude\\settings.local.json` (gitignored):\n\n```jsonc\n{ \"permissions\": { \"deny\": [\"WebFetch\"] } }\n```\n\n`deny` rules stack across the user → project → local layers and are honored even in `bypassPermissions` mode. Restart Claude Code after editing. Verify with `/permissions`.\n\n---\n\n## Highlights\n\n- **8 engines** — DuckDuckGo / Brave / Wikipedia / Bing (international, via proxy) + Bing CN / Baidu / Sogou / 360 (direct). Auto-selected by query language.\n- **3 tools** — `web_search`, `fetch_url`, `deep_research` (multi-engine fan-out → dedup → rank → fetch → cited report)\n- **Auto proxy routing** — CN hosts go direct, international hosts go through `127.0.0.1:7890`; re-decided per redirect hop.\n- **RRF rank fusion** (`k=60`) across engines — robust without score normalization.\n- **SimHash dedup** — 64-bit, Hamming ≤ 3, merges syndicated copies across hosts.\n- **Per-host token bucket + exponential backoff** — defeats frequency-based anti-bot detection.\n- **Disk cache** with frequency-aware TTL (search 30 min / news 1 h / docs 7 d).\n- **TLS/JA3 impersonation** via `curl_cffi` for Cloudflare-protected sites (e.g. `docs.anthropic.com`).\n\n---\n\n## Architecture\n\n```mermaid\nflowchart TB\n    Client[\"Claude Code / MCP Client\\n(stdio · JSON-RPC)\"]\n\n    subgraph Server[\"web-search MCP Server  (index.js)\"]\n        T1[\"web_search\"]\n        T2[\"fetch_url\"]\n        T3[\"deep_research\"]\n        Cache[\"cache.js\\n disk KV · frequency-aware TTL\"]\n    end\n\n    subgraph Core[\"Core layer\"]\n        Eng[\"engines.js\\n8 engine adapters\"]\n        Fetch[\"fetcher.js\\nHTML → markdown\\n+ TLS fallback dispatch\"]\n        Res[\"research.js\\nRRF fusion + SimHash dedup\"]\n    end\n\n    subgraph HTTP[\"http.js  (undici primary path)\"]\n        Proxy[\"Proxy routing\\nCN direct / Intl → 7890\\nre-routed per hop\"]\n        Bucket[\"Token bucket\\n1 concurrent / host\\n+120ms interval\"]\n        Backoff[\"Exponential backoff\\n429 / 5xx retry\\n+ Retry-After\"]\n        Undici[\"decompress interceptor\\n+ manual 3xx loop\"]\n        Sim[\"simhash.js\\n64-bit Charikar near-dup\"]\n    end\n\n    TLS[\"tlsbypass.js → curl_cffi (Python)\\nbrowser JA3/JA4 impersonation\"]\n\n    CN[\"CN engines (direct)\\nbaidu · bingcn · sogou · so\"]\n    Intl[\"International engines (proxy 127.0.0.1:7890)\\nduckduckgo · bing\"]\n\n    Client <--> Server\n    T1 --> Core\n    T2 --> Core\n    T3 --> Core\n    T1 -.cache.-> Cache\n    T2 -.cache.-> Cache\n    T3 -.cache.-> Cache\n    Core --> HTTP\n    HTTP -- \"403 / 429\\nor hard-case host\" --> TLS\n    HTTP --> CN\n    HTTP --> Intl\n```\n\n### `deep_research` pipeline\n\n```mermaid\nflowchart TB\n    Q([\"query\"]) --> S1[\"1. Engine selection\\nCJK → CN first · Latin → Intl first\\nor engines=[...] / all\"]\n    S1 --> S2[\"2. Fan-out search\\nall engines in parallel · fault-tolerant\"]\n    S2 --> S3[\"3. Baidu redirect resolution\\nbaidu.com/link?url= → real URL\"]\n    S3 --> S4[\"4. Two-stage dedup\\nstage 1: exact URL · stage 2: SimHash Hamming ≤ 3\"]\n    S4 --> S5[\"5. RRF ranking\\nscore = Σ 1/(60 + rank)\"]\n    S5 --> S6[\"6. Fetch top-K\\nmax 4 concurrent · token bucket + backoff\\nTLS → curl_cffi on 403\"]\n    S6 --> S7[\"7. Synthesize report\\ncited markdown\"]\n    S7 --> R([\"cited report\"])\n```\n\n---\n\n## Principles\n\n### 1. Auto proxy routing (CN direct / international via proxy)\n\n`src/http.js` keeps a list of CN-domain suffixes (`baidu.com` / `bing.com` / `so.com` …). On each request the host suffix decides:\n\n- CN domain hit → direct (proxying CN sites is slower and trips their risk control)\n- International host → through `127.0.0.1:7890` (works around host-network issues reaching foreign sites)\n- On redirect, **the route is re-decided per hop**: a 301 from an international site to a CN CDN stops using the proxy\n- `engine.search()` may override with `forceProxy` / `forceDirect` (e.g. DuckDuckGo must be proxied)\n- Set `PROXY_URL=\"\"` to disable the proxy (e.g. when a global VPN is active)\n\n### 2. RRF rank fusion (Reciprocal Rank Fusion)\n\n`src/research.js`. Each result's score = Σ_engine `1/(k + rankᵢ)`, with `k=60`.\n\n- **Unsupervised**: no score normalization needed; robust to engines returning different counts\n- **Cross-engine consensus**: results that appear in multiple engines near the top naturally surface\n- Query-term-in-title is only a 1e-4-scale tiebreaker; it never overrides cross-engine consensus\n- `k=60` is the standard default from Cormack et al. (2009) and ranx.fuse\n\n### 3. SimHash near-duplicate dedup\n\n`src/simhash.js`. 64-bit SimHash (Charikar), tokenized with FNV-1a hashing + unigram/bigram.\n\n- Two stages: exact normalized-URL merge first, then SimHash Hamming ≤ 3 as near-duplicate\n- The fingerprint **excludes the host**, so syndicated copies across different hosts merge (the same article republished by different portals, or the same result wrapped differently by Baidu vs Bing)\n- Verified: identical text under different hosts → distance 0; reworded → 11; unrelated → 36 — discriminative\n\n### 4. Per-host token bucket + exponential backoff (anti frequent-access detection)\n\n`src/http.js`. Per host:\n\n- Concurrency 1 + ≥120 ms between requests (`HOST_BUCKETS`)\n- Transient errors (429/502/503/504) + network failures: exponential backoff `500·2^attempt + jitter`, up to 2 retries\n- Honors the `Retry-After` response header\n- Different hosts still run in parallel (no throughput loss)\n\nThis directly targets the main cause of \"frequent-access detection\" — high request rate to a single host.\n\n### 5. Result caching + frequency-aware TTL (incremental)\n\n`src/cache.js`. Disk JSON KV with TTLs scaled by site update frequency:\n\n| Site category | TTL | Rationale |\n|---|---|---|\n| Search-engine result pages (baidu/bing/ddg…) | 30 min | result ordering shifts fast |\n| News aggregators (163/sina/cctv/reuters…) | 1 h | high-churn |\n| Docs/API/encyclopedia (docs./wikipedia/arxiv…) | 7 d | stable content |\n| Other | 6 h | default |\n\n- `web_search` / `fetch_url` / `deep_research` return cached results on hit (tagged `_cached (age Nmin, ttl Mmin)_`)\n- Only caches when there are results (empty results aren't cached, so the next call retries)\n- `CACHE_DISABLED=1` disables; `CACHE_DIR` overrides the cache directory\n- Rationale: frequency-aware incremental crawling (2010) — refresh interval scaled by page change frequency\n\n### 6. TLS/JA3 fingerprint impersonation fallback (curl_cffi)\n\n`src/tlsbypass.js` + `src/fetcher.js`. Node undici's TLS ClientHello differs from a real browser, so heavily-protected sites (Cloudflare-protected `docs.anthropic.com` etc.) return 403. `curl_cffi` (Python) can impersonate a real browser's JA3/JA4 fingerprint + HTTP/2 settings.\n\nStrategy:\n\n- **Fast path**: undici by default (fast, native)\n- **Known hard-case hosts** (`docs/platform.anthropic.com` etc.): go straight to curl_cffi\n- **Generic fallback**: on undici 403/429 or network error → auto-fallback to curl_cffi; if both fail, report honestly (with `impersonated` flag)\n- Reports tag `TLS: impersonated (curl_cffi)` or `native (undici)`\n- Requires Python + `pip install curl_cffi`; without it, degrades to undici-only (main flow unaffected)\n- Rationale: *TLS Beyond the Browser* (ACM IMC 2019) — non-browser TLS clients are fingerprintable; impersonation closes that gap\n\n### 7. Engine adapters\n\n| Engine | Region | Route | Notes |\n|---|---|---|---|\n| duckduckgo | international | proxy | `html.duckduckgo.com/html/` no-key HTML endpoint |\n| brave | international | proxy | `search.brave.com` |\n| wikipedia | international | proxy | `en.wikipedia.org` search API |\n| bing | international | proxy | `setmkt=en-US&cc=US`; reconstructs real URL from `<cite>` (bypasses the `ck/a` redirect wrapper) |\n| bingcn | CN | direct | `cn.bing.com` |\n| baidu | CN | direct | Detects CAPTCHA interstitial and errors out; reads `mu` attr for the real URL |\n| sogou | CN | direct | Mobile + desktop dual fallback to dodge anti-bot |\n| so (360) | CN | direct | `www.so.com` |\n\nA query with CJK characters defaults to CN engines first; pure-Latin queries go international first. `engine='all'` fans out to every engine.\n\n### 8. Key undici v7 handling\n\n- `interceptors.decompress()`: auto-decompress gzip/br/deflate (Bing international returns brotli)\n- **Manual 3xx redirect loop**: undici v7's `request()` no longer accepts `maxRedirections`, and the `redirect` interceptor alone isn't enough (per-request opts default to 0 and short-circuit). The manual loop also lets us re-route proxy/direct per hop.\n\n---\n\n## Tools\n\n### `web_search`\n\n```jsonc\n{ \"query\": \"anthropic claude api pricing\", \"num\": 8, \"engine\": \"auto\" }\n```\n\n- `engine`: `auto` (default) | `all` | `duckduckgo` | `bing` | `bingcn` | `baidu` | `sogou` | `so`\n\n### `fetch_url`\n\n```jsonc\n{ \"url\": \"https://example.com/page\", \"max_chars\": 16000 }\n```\n\nFetch a URL, strip boilerplate, return markdown. Auto-routes proxy/direct by domain; falls back to TLS impersonation on 403.\n\n### `deep_research`\n\n```jsonc\n{\n  \"query\": \"中国空间站 最新进展\",\n  \"engines\": [\"bingcn\", \"baidu\"],\n  \"num_per_engine\": 8,\n  \"fetch_top_k\": 4,\n  \"fetch_chars\": 6000\n}\n```\n\nMulti-engine fan-out → dedup & RRF rank → fetch top-K → cited markdown report.\n\n---\n\n\n## Environment variables\n\n| Variable | Default | Effect |\n|---|---|---|\n| `PROXY_URL` | `http://127.0.0.1:7890` | proxy endpoint; empty string disables proxying |\n| `CACHE_DISABLED` | `0` | set `1` to disable result caching |\n| `CACHE_DIR` | `./cache` | cache directory |\n\n---\n\n## Referenced papers\n\nPapers referenced in the implementation (DOIs verifiable; retrieved via the OpenAlex academic API + DuckDuckGo/Bing):\n\n### Result fusion / ranking\n\n| Paper | Year | Used for | DOI |\n|---|---|---|---|\n| ranx.fuse: A Python Library for Metasearch | 2022 (CIKM) | RRF implementation reference; 25 fusion algorithms | [10.1145/3511808.3557207](https://doi.org/10.1145/3511808.3557207) |\n| Comparing Rank and Score Combination Methods for Data Fusion in IR | 2005 | Comparison of CombMNZ etc.; basis for RRF k | [10.1007/s10791-005-6994-4](https://doi.org/10.1007/s10791-005-6994-4) |\n| The use of MMR, diversity-based reranking | 1998 (Carbonell) | Foundational diversity reranking (not yet implemented) | [10.1145/290941.291025](https://doi.org/10.1145/290941.291025) |\n| Fusion-based methods for result diversification in web search | 2018 (Information Fusion) | Combining fusion and diversity | [10.1016/j.inffus.2018.01.006](https://doi.org/10.1016/j.inffus.2018.01.006) |\n\n### Near-duplicate detection\n\n| Paper | Year | Used for | DOI |\n|---|---|---|---|\n| A Review for Weighted MinHash Algorithms | 2018 | MinHash/SimHash survey | [10.48550/arxiv.1811.04633](https://doi.org/10.48550/arxiv.1811.04633) |\n| Improved Near-Duplicate Detection for Aggregated and Paywalled News | 2025 (NAACL) | Recent advances in near-dup detection | [10.18653/v1/2025.naacl-industry.73](https://doi.org/10.18653/v1/2025.naacl-industry.73) |\n| Effective and Fast Near Duplicate Detection via Signature-Based | 2016 | Signature-based dedup engineering | [10.1155/2016/3919043](https://doi.org/10.1155/2016/3919043) |\n\n### Parallel / incremental crawling / rate limiting\n\n| Paper | Year | Used for | DOI |\n|---|---|---|---|\n| BUbiNG: Massive Crawling for the Masses | 2016 | Per-host politeness queue, linear scaling | [10.48550/arxiv.1601.06919](https://doi.org/10.48550/arxiv.1601.06919) |\n| SIMHAR — Smart Distributed Web Crawler for the Hidden Web | 2020 (IEEE Access) | Distributed queue + SIM+Hash dedup | [10.1109/access.2020.3004756](https://doi.org/10.1109/access.2020.3004756) |\n| Design of a Priority Based Frequency Regulated Incremental Crawler | 2010 | Frequency-aware incremental crawling (caching) | [10.5120/23-131](https://doi.org/10.5120/23-131) |\n| On the Feasibility of Geographically Distributed Web Crawling | 2008 | Geo-distributed crawling reduces latency | [10.4108/icst.infoscale2008.3550](https://doi.org/10.4108/icst.infoscale2008.3550) |\n\n### Anti-bot / blocking evasion\n\n| Paper | Year | Used for | DOI |\n|---|---|---|---|\n| TLS Beyond the Browser | 2019 (ACM IMC) | TLS/JA3 fingerprint exposure (P1) | [10.1145/3355369.3355601](https://doi.org/10.1145/3355369.3355601) |\n| FP-Crawlers: Studying the Resilience of Browser Fingerprinting | 2020 | Browser fingerprinting detection | [10.14722/madweb.2020.23010](https://doi.org/10.14722/madweb.2020.23010) |\n| A First Look at User-Installed Residential Proxies | 2024 (CNSM) | Residential-proxy ecosystem (proxy pool, planned) | [10.23919/cnsm62983.2024.10814519](https://doi.org/10.23919/cnsm62983.2024.10814519) |\n\n> Full optimization plan and gap analysis: [`docs/optimization-research.md`](./docs/optimization-research.md).\n\n---\n\n## Project structure\n\n```\nweb-search-mcp/\n├── index.js              # MCP server entry (registers 3 tools, stdio)\n├── src/\n│   ├── http.js           # HTTP layer + proxy routing + token bucket + backoff\n│   ├── engines.js        # 8 engine adapters\n│   ├── fetcher.js        # HTML → markdown + TLS fallback dispatch\n│   ├── research.js       # RRF fusion + SimHash dedup pipeline\n│   ├── simhash.js        # 64-bit Charikar SimHash\n│   ├── cache.js          # disk KV with frequency-aware TTL\n│   └── tlsbypass.js      # curl_cffi TLS impersonation bridge\n├── test-smoke.js         # smoke test\n├── docs/\n│   ├── optimization-research.md\n│   ├── architecture.svg\n│   └── pipeline.svg\n├── .mcp.json             # project-level MCP registration\n└── package.json\n```\n\n---\n\n## Known limitations\n\n- All network requests time out in 20–25 s; nothing hangs.\n- Baidu occasionally returns a CAPTCHA interstitial; that engine then errors and `deep_research` falls back to the other engines.\n- Bing international wraps result links in `bing.com/ck/a`; the real URL is reconstructed from the result's `<cite>`.\n- DuckDuckGo uses the no-key `html.duckduckgo.com/html/` endpoint, which requires the proxy.\n- TLS impersonation depends on Python + `curl_cffi`. Without it, falls back to undici-only and hard-case sites (e.g. `docs.anthropic.com`) return 403; with it, impersonation is automatic.\n- The cache expires by TTL without background refresh; the next call after expiry re-fetches. Set `CACHE_DISABLED=1` to disable.\n\n---\n\n## License\n\nMIT\n"}