stealth-scraper-mcp

Stealth Web Scraper MCP

An MCP server that fetches any web page and returns clean, token-efficient Markdown for AI agents.

• Tiered fetching, cheapest first: direct HTTP, browser, HTTP via residential proxy, browser via proxy. Each tier runs only when the previous one is blocked.
• Fast path: curl_cffi with real browser TLS fingerprints (Chrome, Safari, Firefox, Edge), rotated on retry.
• Browser fallback: when a page is blocked or needs JavaScript, it is rendered in your installed Google Chrome (or Playwright's Chromium).
• Fewer tokens: pass query to get only the passages that answer a question, ranked with BM25. Long pages are paginated.
• Safe by default: refuses localhost and private-network URLs, binds to 127.0.0.1, optional API key.

Benchmark

Measured on 30 real sites (docs, reference, news, shops, JavaScript apps and bot-protected sites), with no proxy. Each site was fetched once. Full results are in benchmark/RESULTS.md; re-run with PYTHONPATH=src python benchmark/run.py.

	Result
Success rate, no proxy	24/30 (80%)
Tokens vs raw HTML, clean Markdown	97.9% fewer (median 90,222 → 1,901)
Tokens vs raw HTML, with query	99.6% fewer (median 341)
Median time	2.6 s HTTP, 11.4 s browser

The 6 misses: three sites behind strict bot protection that need a residential proxy (Etsy, G2, Glassdoor), and three image-heavy homepages with little text.

Install

Claude Code:

claude mcp add stealth-scraper -- uvx stealth-scraper-mcp

Claude Desktop, Cursor, VS Code and other clients (mcpServers config):

{
  "mcpServers": {
    "stealth-scraper": {
      "command": "uvx",
      "args": ["stealth-scraper-mcp"]
    }
  }
}

The browser fallback uses Google Chrome if it is installed. Otherwise download Chromium once:

uvx stealth-scraper-mcp --install-browser

Tool: scrape_url_stealth

Parameter	Default	Purpose
url	required	Page to fetch
query	none	Return only the passages relevant to this question
top_k	5	Passages to return with query
output_format	markdown	markdown, text, html or links
response_format	concise	detailed adds engine, metadata and cache diagnostics
max_chars / offset	15000 / 0	Pagination; pass next_offset back as offset
max_age_seconds	1 hour	Cache freshness; 0 forces a fresh fetch, -1 reads only the cache
render_js	auto	always or never force or skip the browser
proxy	auto	auto uses the residential proxy only after a block; never or always
css_selector	none	Only return matching elements
wait_for_selector, scroll	none / false	Browser options for dynamic and lazy-loaded pages

Errors come back as [code] message Hint: ..., e.g. blocked, timeout, url_not_allowed, selector_not_found.

Running over HTTP

uvx stealth-scraper-mcp --transport http --port 8080

The endpoint is http://127.0.0.1:8080/mcp, with /health for checks. To expose it on a network, set an API key first:

SCRAPER_API_KEY=change-me SCRAPER_HOST=0.0.0.0 uvx stealth-scraper-mcp --transport http

Clients then send Authorization: Bearer change-me.

Configuration

Variable	Default	Purpose
SCRAPER_PROXY_URL	none	Residential proxy, used only after a direct fetch is blocked
SCRAPER_API_KEY	none	Require a bearer token on HTTP
SCRAPER_HOST / SCRAPER_PORT	127.0.0.1 / 8080	HTTP bind address
SCRAPER_CACHE_TTL	3600	Default cache freshness in seconds
SCRAPER_CACHE_MAX_ENTRIES	256	Cache size
SCRAPER_BROWSER_CHANNEL	chrome	Browser to launch; empty for bundled Chromium
SCRAPER_BROWSER_HEADLESS	true	false runs a visible, harder-to-detect browser
SCRAPER_BROWSER_LOCALE / SCRAPER_BROWSER_TIMEZONE	en-US / system	Keep these consistent with your proxy's location
SCRAPER_ALLOW_PRIVATE_NETWORKS	false	Allow localhost and private IPs
SCRAPER_FETCH_TIMEOUT, SCRAPER_MAX_RETRIES	20, 2	Request timeout and retries

Plugins

Packages can extend the server through the stealth_scraper_mcp.plugins entry point group. They can add usage metering, extra HTTP routes and CLI commands; see src/stealth_scraper_mcp/plugins.py. Without plugins the server is free and unmetered.

Responsible use

Respect each site's terms of service and robots.txt, and avoid heavy request rates.

License

AGPL-3.0-or-later. You can use, modify and self-host this freely. If you offer a modified version to others as a network service, you must publish your changes under the same license.