my-websearch

🔍 MyWebSearch

Keyless Multi-Engine AI Web Search & High-Purity Content Extraction Engine
MCP Server · CLI · Local HTTP Daemon · Skill-Guided Agent Workflows

🇨🇳 简体中文 | 🇺🇸 English

---

One server, four ways to use it: connect an AI client over MCP, run a one-shot CLI query, keep a local daemon alive, or use the built-in diagnostics before going live.

Jump to: Quick start · MCP tools · CLI · Configuration · Diagnostics · Development

✨ Features

my-websearch is a full-stack search and technical documentation engine built specifically for AI coding agents (Claude Desktop, Cursor, Cherry Studio, Windsurf, Cline, ZCode, etc.). It requires zero paid API keys:

• 🌐 13-Engine Federated Search & Smart Routing
  Aggregates direct domestic engines (Bing, Baidu, CSDN, Juejin, Sogou), global engines (DuckDuckGo, Brave, Startpage, Exa), and specialized developer platforms (GitHub, StackOverflow, Hacker News, arXiv). Features automatic language-aware routing (auto), parallel multi-query execution (queries: string[]), URL deduplication, and automatic minResults cascading fallback.
• ⚡ Native Anti-Bot & Proof-of-Work Solvers (Zero-Browser Fast Path)
  Built on wreq-js with native Chrome TLS/HTTP2 handshake emulation and persistent session cookies. Includes pure-JS/Rust solvers for Startpage's Anubis SHA-256 PoW challenge and DuckDuckGo's d.js (isJsaChallenge) HTML5/arithmetic challenge, bypassing anti-bot walls in milliseconds without launching a 400MB browser. Real URL decryption for Sogou redirect links, Baidu Location, and Bing Base64 links.
• 📄 High-Purity Markdown Article Extraction & Dedicated Site Purifiers
  Combines Mozilla Readability with intelligent chrome noise stripping. Features dedicated purification rules for WeChat Official Accounts (mp.weixin.qq.com), StackOverflow questions & accepted answers, and transparent internal routing for CSDN & Juejin. Removes <nav>, <aside>, <footer>, and breadcrumbs while faithfully preserving article titles, code blocks, and GFM tables. Supports seamless pagination (startIndex).
• 📚 Unified Context7 Library Docs Lookup (queryDocs)
  Directly query official documentation with natural package names (e.g. "next.js", "vue") or Context7 IDs without needing a separate resolve step. Includes automatic candidate ranking, blind-guess 404 self-healing, and related alternatives discovery.
• 🐙 Extended GitHub Resource Fetcher
  fetchGithubReadme supports GitHub/Gitee repository READMEs, raw code blob files (/blob/...), and complete issue discussions with comments (/issues/...).
• 🌏 Zero-Config Network Detection & Clash Fake-IP Ready
  Auto-detects OS system proxies (Windows Registry / macOS scutil / Linux env). Automatically routes overseas engines through proxy while keeping domestic engines on fast direct lines. Pre-whitelists 198.18.0.0/15 for Clash TUN / Fake-IP compatibility while enforcing strict SSRF security bounds.
• 🔌 Versatile Deployment Modes
  Works seamlessly across MCP STDIO (Claude Desktop/Cursor), Streamable HTTP / SSE (Cherry Studio/remote clients), CLI one-shot commands, and a Local HTTP Daemon with connection pool reuse and Prometheus /metrics monitoring.

---

🏗️ Architecture

flowchart TB
    subgraph Clients["🤖 Entrypoints"]
        MCP["MCP Server<br/>(STDIO / Streamable HTTP / SSE)"]
        CLI["CLI Commands<br/>(my-websearch search / fetch-*)"]
        Daemon["Local Daemon<br/>(127.0.0.1:3210 · /health · /metrics)"]
    end

    subgraph Core["🧠 Search & Fetch Orchestrator"]
        Router["Language-Aware Auto Router<br/>ZH → Baidu/Sogou | EN/Tech → Bing + DuckDuckGo"]
        Cascade["minResults Cascade & Circuit Breaker<br/>(Auto Fallback + 5min TTL LRU Cache)"]
        Ranker["Cross-Engine URL Deduplication & BM25 Relevance Ranking"]
    end

    subgraph Transport["🛡️ Anti-Bot & Security Transport"]
        Wreq["wreq-js Chrome TLS/H2 Fingerprint<br/>+ Automatic Session Cookie Jars"]
        Solvers["Millisecond Challenge Solvers<br/>Startpage PoW | DDG JSA Solver"]
        PW["Playwright Stealth Browser Fallback<br/>(Budget Bounds + Cross-Process Locking)"]
        Guard["SSRF Guard & Clash Fake-IP Support<br/>(PROXY_URL Split Routing + 198.18.0.0/15)"]
    end

    subgraph Engines["🌍 13 Search Engines + 6 Content/Docs Tools"]
        CN["🇨🇳 Direct Domestic Engines<br/>Bing · Baidu · CSDN · Juejin · Sogou"]
        INTL["🌐 Global & Tech Engines<br/>DuckDuckGo · Brave · Startpage · Exa<br/>GitHub · StackOverflow · HackerNews · arXiv"]
        Docs["📚 Content & Official Docs<br/>fetchWebContent · queryDocs · GitHub/Gitee · CSDN/Juejin"]
    end

    Clients --> Core
    Core --> Transport
    Transport --> Engines

---

🌍 13 Search Engines

Engine	Mainland China Connectivity	API Key	Core Technology	Best For
bing	🇨🇳 Direct	None	Chrome TLS impersonation + Base64 real URL resolution + stealth fallback	General technical search, mixed EN/ZH queries
baidu	🇨🇳 Direct	None	Concurrent Location redirect resolution + anti-bot detection	Chinese news, documentation, domestic forums
csdn	🇨🇳 Direct	None	WAF cookie jar persistence + empty response auto-retry	Chinese error logs, debugging tutorials
juejin	🇨🇳 Direct	None	Direct official API querying for clean long-form articles	Modern frontend/backend technical blogs
sogou	🇨🇳 Direct	None	Desktop & mobile dual parsing + redirect link decoding + ad removal	WeChat ecosystem links & long-tail queries
duckduckgo	🌐 Proxy in CN	None	Preload d.js + built-in isJsaChallenge JS solver	English technical search & open-source discussions
startpage	🌐 Proxy in CN	None	Built-in Anubis SHA-256 PoW solver + cookie session (no browser needed)	Google-backed high-privacy results
brave	🌐 Proxy in CN	None	Native TLS fingerprinting + strict sponsored ad stripping	Independent English index & technical blogs
exa	🌐 Direct API	Optional Free Key	Official semantic search API (skipped if key is unset)	Semantic similarity, AI research & papers
github	🌐 Proxy in CN	None	Official GitHub Search REST API for issues & repositories	GitHub issues, bug reports, and repositories
stackoverflow	🌐 Proxy in CN	None	Official StackExchange 2.3 API with HTML entity decoding	Developer troubleshooting, stack traces & answers
hackernews	🌐 Proxy in CN	None	Algolia HN Search API with points, comments & author metadata	Tech discussions, startup launches & engineering news
arxiv	🌐 Proxy in CN	None	Official arXiv Atom XML API parsing titles, authors & abstracts	Scientific papers, machine learning & AI research

---

🛠️ 4 Core MCP Tools (Minimal & Token-Efficient)

Tool Name	Purpose	Key Highlights
search	Multi-engine federated search (13 engines)	Supports single query or parallel queries: string[], engines, limit, and minResults cascading; includes GitHub, StackOverflow, HackerNews & arXiv developer engines
fetchWebContent	Web page extraction & Markdown conversion	All-in-one content extractor: Built-in smart routing for CSDN, Juejin, WeChat, StackOverflow, and GitHub; supports pagination (startIndex) and focus snippets (focus)
fetchGithubReadme	Comprehensive GitHub / Gitee content reader	Fetches repository READMEs, branch-specific source files (fenced code blocks with syntax highlighting), ASCII directory trees, and Issue / PR discussions (with reactions & changed files)
queryDocs	All-in-one official library documentation	Directly resolves library names (e.g. "next.js", "vue"), auto-heals 404s, returns alternative libraries & versioned code

💡 Backward Compatibility: Legacy fetchCsdnArticle and fetchJuejinArticle are automatically routed under fetchWebContent. Legacy resolveLibraryId is merged into queryDocs (which supports directly passing library name or legacy libraryId alias). Set REGISTER_LEGACY_ARTICLE_TOOLS=true if your LLM requires fetchCsdnArticle / fetchJuejinArticle explicitly exposed as separate tools.

---

🚀 Quick Start

1. Run with NPX (Zero Configuration)

No installation required:

npx -y my-websearch@latest

💡 Automated Proxy Discovery: The server automatically detects your operating system's proxy settings. If a local proxy (Clash, v2ray, Surge) is active, overseas engines automatically use the proxy while domestic engines stay on fast direct lines. 198.18.0.0/15 is whitelisted out of the box for Clash Fake-IP.

2. Configure in AI Clients

🔹 Claude Desktop / Cursor / Windsurf / Cline (STDIO Mode)

Add to your MCP settings file:

{
  "mcpServers": {
    "my-websearch": {
      "command": "npx",
      "args": ["-y", "my-websearch@latest"],
      "env": {
        "MODE": "stdio"
      }
    }
  }
}

(For Windows environments where npx is not in the system path, set command to cmd and args to ["/c", "npx", "-y", "my-websearch@latest"])

🔹 Cherry Studio (Streamable HTTP Mode)

Start the server in your terminal with npx my-websearch@latest (default port 3211), then configure:

{
  "mcpServers": {
    "web-search": {
      "name": "MyWebSearch",
      "type": "streamableHttp",
      "baseUrl": "http://localhost:3211/mcp"
    }
  }
}

---

💻 CLI & Local Daemon

my-websearch can be used directly from your terminal as a standalone CLI tool or persistent daemon. Use --json in automation and keep the daemon on 127.0.0.1 unless you explicitly need remote access:

# Global install
npm install -g my-websearch

# Start persistent local HTTP daemon (port 3210 by default)
my-websearch serve

# One-shot terminal search (reuses running daemon automatically)
my-websearch search "Model Context Protocol best practices" --limit 5 --min-results 5 --json

# Extract web page as clean Markdown
my-websearch fetch-web "https://blog.vuejs.org/posts/vue-3-5" --max-chars 15000 --json

# View daemon status and Prometheus metrics (/health, /status, /metrics)
my-websearch status --json

# Clear in-memory search cache
my-websearch cache-clear

---

🩺 Diagnostics

Use the built-in doctor command before troubleshooting a proxy, browser, or engine:

# Local configuration and browser checks
my-websearch doctor

# Real proxy/engine probes (may contact upstream services)
my-websearch doctor --network

# Limit network probes to selected engines
my-websearch doctor --network --engine bing,duckduckgo --json

# Reset and clear cached engine circuit breaker / health state
my-websearch doctor --reset-health

The report separates configuration, proxy TCP reachability, HTTP reachability, application parsing, browser launch, and anti-bot challenges. Network mode performs real requests and should not be used as a high-frequency health check.

🌐 Proxy Troubleshooting & Upstream IP Risk Control

When configuring or diagnosing proxy connectivity, refer to these common edge cases:

1. Startup log shows 🌐 Proxy: disabled even though proxy.url is set?

   • Check useProxy flag: In ~/.my-websearch/config.json, setting "useProxy": false takes precedence and explicitly overrides and disables url (setting PROXY_URL='false'). Change "useProxy": true or simply omit the field (providing a valid url automatically enables the proxy).
   • No system proxy active without PROXY_URL: If no system proxy (Clash/v2ray) is running, overseas engines will be blocked by default. Explicitly declare PROXY_URL (e.g. http://127.0.0.1:7897 or 7890).

2. Proxy is configured properly, but overseas engines fail or timeout? (Upstream IP Blocking, NOT a Config Error)

   • DuckDuckGo returns HTTP 202 / challenge: DDG detected excessive traffic from the proxy node's egress IP and served an anti-bot challenge.
   • Brave returns HTTP 429 Too Many Requests: Brave Search rate-limits noisy datacenter IP ranges.
   • Startpage times out or Anubis PoW difficulty spikes: Startpage dynamically raises Anubis PoW difficulty to 6+ for flagged IP addresses, exceeding single-engine execution budgets.
   • 💡 Resolution: These symptoms indicate that the proxy node's public IP has been flagged by upstream search providers, not a bug in my-websearch. Switch to a cleaner proxy server/node in your proxy client, fallback to domestic/direct engines, or supply an official API key such as BRAVE_API_KEY for guaranteed SLA.

⚙️ Configuration & Environment Variables

Global Configuration (~/.my-websearch/config.json)

To share settings across all MCP clients without duplicate configuration:

{
  "apiKeys": {
    "context7": "ctx7sk-...",
    "exa": "..."
  },
  "proxy": {
    "url": "http://127.0.0.1:7897",
    "useProxy": true,
    "engines": ["duckduckgo", "exa", "brave", "startpage"]
  }
}

⚠️ Note on useProxy: When "url" is configured, proxy is enabled automatically. Setting "useProxy": false will explicitly override and disable url. If you want proxy enabled, set "useProxy": true or omit the field entirely.

📋 Exhaustive Environment Variables Reference

Can be set via the global configuration file (~/.my-websearch/config.json), local .env, or system environment variables:

1. 🔍 Search & Routing Orchestration

Variable	Default	Allowed Values / Description
DEFAULT_SEARCH_ENGINE	auto	Default engine: auto (smart query routing: Chinese baidu,sogou, English/foreign bing,duckduckgo). Supports all 13 engines: auto, bing, baidu, sogou, duckduckgo, brave, csdn, juejin, startpage, exa, github, stackoverflow, hackernews, arxiv (vertical developer engines can be configured as default, though auto or general web engines are recommended for global default)
ALLOWED_SEARCH_ENGINES	all	Comma-separated whitelist of allowed search engines (e.g. bing,baidu,duckduckgo)
BING_BASE_URL	https://cn.bing.com/search	Base URL for Bing queries. Overseas users or reverse proxy nodes can set to https://www.bing.com/search (legacy alias OPEN_WEBSEARCH_BING_HOST supported)
DEFAULT_MIN_RESULTS	5	Cascade threshold: triggers fallback engines when usable results are fewer than this value
DEFAULT_SEARCH_LIMIT	10	Default result limit when not explicitly specified in tool calls (max 50)
SEARCH_DEADLINE_MS	30000	Global hard deadline for multi-engine search execution (milliseconds)
SEARCH_AUTHORITY_DOMAINS	empty	Additional authority domain whitelist for result re-ranking (comma-separated, appended to built-in list)
SEARCH_MODE	auto	Search mode and browser policy. Supports single mode, multi-engine per mode, and compound multi-mode combinations:• Global mode: auto (default fallback), request/none (pure HTTP), playwright (force browser)• Single mode, multiple engines: playwright:bing,startpage, auto:bing,startpage, bing,startpage• Compound multiple modes: playwright:bing;auto:startpage (bing forces browser, startpage uses auto fallback), auto;request:startpage (global auto but startpage uses pure HTTP)
AUTO_ROUTE_EN_ENGINES	bing,duckduckgo	Comma-separated engines for default English auto-routing
AUTO_ROUTE_ZH_ENGINES	baidu,sogou	Comma-separated engines for default Chinese auto-routing

2. 🌐 Proxy & Network Security

Variable	Default	Allowed Values / Description
PROXY_URL	empty (off)	Consolidated upstream proxy configuration (merges previous USE_PROXY and PROXY_ENGINES). Not needed in TUN mode.• Default proxy: passing empty PROXY_URL="" or PROXY_URL=true activates default http://127.0.0.1:7897 (common port for Clash Verge; use 10809 for v2rayN or 7890 for legacy Clash; overseas engines use proxy, domestic engines stay direct)• Custom address: PROXY_URL=http://127.0.0.1:10809• Custom engines: PROXY_URL="http://127.0.0.1:7897;duckduckgo,brave" or all engines PROXY_URL=";all"
FAKE_IP_CIDRS	198.18.0.0/15	Whitelisted CIDRs for Clash Fake-IP mode to avoid SSRF false-positive blocking
IMPERSONATE_BROWSER	chrome_149	wreq-js browser TLS/H2 fingerprint profile (e.g. chrome_149, safari_18)
IMPERSONATE_OS	windows	Operating system platform for fingerprint impersonation (windows, macos, linux)
BING_IMPERSONATE	true	Toggle Chrome TLS impersonation for Bing HTTP mode (auto-trips circuit breaker after consecutive empty results)
ANUBIS_MAX_POW_DIFFICULTY	5	Maximum difficulty threshold for Startpage Anubis PoW solver; skips/falls back when difficulty exceeds this
DIRECT_FIRST_PROBE_TIMEOUT_MS	3500	Timeout in ms for direct-first connection probes before proxy fallback
FETCH_WEB_INSECURE_TLS	false	Disable TLS certificate verification for fetchWebContent only (for broken legacy site certs)
GITHUB_README_CDN_FIRST	false	When true, forces skipping raw.githubusercontent.com direct probe and queries jsDelivr CDN first (domestic network acceleration)

3. 🔐 Credentials & API Keys

Variable	Default	Allowed Values / Description
CONTEXT7_API_KEY	empty	Context7 API Key (strongly recommended for proxy/TUN users to bypass shared IP quotas)
EXA_API_KEY	empty	Exa Free API Key (only required if using the exa semantic search engine)
BRAVE_API_KEY	empty	Brave Search API Key (optional; uses official high-reliability API when set)
GITHUB_TOKEN	empty	GitHub Personal Access Token (optional; raises rate limit to 5,000 req/hr)
STACKOVERFLOW_API_KEY	empty	Stack Apps Key (optional; raises quota to 10,000 req/day)

4. 🖥️ Server Modes, Concurrency & Session Reaper

Variable	Default	Allowed Values / Description
MODE	both	MCP transport mode: stdio (stdin/stdout only), http (HTTP/SSE only), both (concurrent)
PORT	3211	MCP HTTP/SSE listen port (local persistent daemon uses 3210 by default)
MAX_CONCURRENT_SEARCHES	20	Max concurrent searches in daemon mode (0 for unlimited)
MCP_MAX_SESSIONS	100	Max retained MCP HTTP sessions before returning HTTP 429 Too Many Requests
MCP_SESSION_TTL_MS	1800000	Idle session expiration duration before reaping (default 30 min)
MCP_SESSION_REAPER_MS	300000	Session reaper scan interval (default 5 min)
LOG_LEVEL	info	Log level: quiet (completely silent), debug, info, warn, error
SECURITY_AUDIT	false	Enable SSRF blocking and host header security audit logging
OPEN_WEBSEARCH_ALLOWED_HOSTS	loopback	Host header whitelist for HTTP server (DNS Rebinding protection, comma-separated)

5. 🧭 Playwright Browser & Stealth Controls

[!NOTE] BING_PLAYWRIGHT_FALLBACK and STARTPAGE_PLAYWRIGHT_FALLBACK have been consolidated into SEARCH_MODE (e.g. SEARCH_MODE=bing or SEARCH_MODE=request). Backward compatibility is automatically preserved.

Variable	Default	Allowed Values / Description
PLAYWRIGHT_CDP_ENDPOINT	empty	Remote Chrome/Edge CDP endpoint URL (e.g. http://127.0.0.1:9222 or ws://...)
PLAYWRIGHT_EXECUTABLE_PATH	auto	Absolute path to browser executable (auto-detects system Edge/Chrome by default)
PLAYWRIGHT_HEADLESS	true	Run headless (on Windows, automatically utilizes an isolated hidden desktop window)
PLAYWRIGHT_NAVIGATION_TIMEOUT_MS	20000	Hard page navigation timeout for Playwright (milliseconds)

6. 🛠️ Application Directory & Tool Customization

Variable	Default	Allowed Values / Description
MYWEBSEARCH_DATA_DIR	~/.my-websearch	Root directory for configuration and persistent cookie files
OPEN_WEBSEARCH_RESPONSE_CAP_BYTES	60000	Maximum character limit returned to LLMs to prevent prompt context explosion
MCP_TOOL_SEARCH_NAME	search	Custom registration name for the search tool in MCP
MCP_TOOL_FETCH_WEB_NAME	fetchWebContent	Custom registration name for the content extraction tool
MCP_TOOL_FETCH_GITHUB_NAME	fetchGithubReadme	Custom registration name for the GitHub README tool

💡 Zero-Config Upgrades (Redundant Variables Removed):

• CORS Switches (ENABLE_CORS/CORS_ORIGIN): CORS is disabled by default; enable it explicitly and restrict origins when browser clients need cross-origin access.
• Playwright Package Selection (PLAYWRIGHT_PACKAGE/PLAYWRIGHT_MODULE_PATH): Now 100% automated via intelligent client and system binary discovery.
• Remote Browser Endpoint (PLAYWRIGHT_WS_ENDPOINT): Unified into standard PLAYWRIGHT_CDP_ENDPOINT.
• Emoji Switch (OPEN_WEBSEARCH_NO_EMOJI): Automatic console code page detection smoothly falls back to ASCII tags on legacy terminals.
• Fragmented Debug Flag (OPEN_WEBSEARCH_DEBUG): Unified into standard LOG_LEVEL=debug.
• Cookie Persistence TTL (MYWEBSEARCH_COOKIE_TTL_DAYS): Solidified to 30 days with automated sliding renewal.

---

🤝 Contributing & Tests

Pull requests and issues are welcome! To run the test suite locally:

npm install
npm run build
npm test              # Run the Vitest suite (45 files, 401 tests at the current baseline)
npm run test:live     # Run bounded-concurrency live integration tests
npm run eval:live     # Run 23-case live STDIO JSON-RPC evaluation suite

Acknowledgements

Author: wtznicy

This project evolved from Open-WebSearch (originally created by Aas-ee) — special thanks to the original author. Thanks also to:

• wreq-js for native Chrome TLS/HTTP2 fingerprint impersonation
• Context7 for powering official documentation retrieval
• Mozilla Readability & Turndown for content extraction and GFM Markdown formatting