{"id":"25de1f87-798e-4794-b7cf-0c33e8cf8dae","slug":"certainlyscraped","kind":"mcp","name":"certainlyscraped","summary":"A hybrid scraper with escalating effort. Cheapest fetch first, escalates only when blocked, reports exactly what it got and why it stopped. CLI + MCP server for agents.","owner":"vhsgreed","repoUrl":"https://github.com/vhsgreed/certainlyscraped","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-08-29T15:26:44.000Z","addedAt":"2026-11-01T13:30:18.043Z","logoUrl":"https://avatars.githubusercontent.com/u/77639664?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["browser-web","files-documents","cloud-infra"],"tags":["anti-bot","csv","curl-cffi","mit-license","python","scraping","tls-impersonation","web-scraping"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-11-01T13:36:08.528Z","sample":false,"url":"https://conduits.dev/tools/certainlyscraped","language":"Python","transports":["stdio"],"install":{"pypi":"certainlyscraped"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# certainlyscraped\n\nA hybrid scraper with escalating effort.\n\nIt starts with the cheapest fetch that works, escalates only when a page is\nblocked or still failing, and reports exactly what it got and why it stopped.\nIt reads the page, writes a config for you, checks that every field filled, and\nturns a listing into rows instead of a promise.\n\n[![tests](https://github.com/vhsgreed/certainlyscraped/actions/workflows/tests.yml/badge.svg)](https://github.com/vhsgreed/certainlyscraped/actions/workflows/tests.yml)\n[![live check](https://github.com/vhsgreed/certainlyscraped/actions/workflows/live.yml/badge.svg)](https://github.com/vhsgreed/certainlyscraped/actions/workflows/live.yml)\n\n## Quickstart: 20 seconds\n\n```bash\npip install \"certainlyscraped[mcp]\"\n\ncscrape init https://books.toscrape.com/catalogue/page-1.html --name books\ncscrape run books.yaml\n```\n\n`init` probes the site, finds the block that repeats on the page, guesses the\nfields and the next-page link, and prints how many items each field filled.\n`run` fetches the pages, writes `out.csv`, and fails loudly if it gets fewer\nrows than `--min-rows` (default 1) instead of reporting success.\n\nFor an agent, add the MCP server to your client. With [uv](https://docs.astral.sh/uv/)\nthere is nothing to install first:\n\n```json\n{\n  \"mcpServers\": {\n    \"certainlyscraped\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"certainlyscraped[mcp]\", \"certainlyscraped\", \"mcp\"]\n    }\n  }\n}\n```\n\nWithout uv, `pip install \"certainlyscraped[mcp]\"` and use `\"command\": \"certainlyscraped\", \"args\": [\"mcp\"]`.\n\nThat gives the agent `probe`, `init`, `try_selector` and `scrape`; the full\ncontract is under [MCP server (for agents)](#mcp-server-for-agents).\n\n![certainlyscraped demo: init, try, run](docs/demo.gif)\n\nReal output from the daily live check:\n\n```\n[probe] https://books.toscrape.com/catalogue/page-1.html -> tier 1 (static page)\n[init] wrote books.yaml: 20 items on this page (20 repeats, median text 51 chars, each has a link, each has an image)\nfield coverage on this page:\n  title           20/20 (100%)\n  url             20/20 (100%)\n  image           20/20 (100%)\n  price           20/20 (100%)\n  availability    20/20 (100%)\n```\n\nThe config is plain YAML you can read and edit:\n\n```yaml\nname: books\nurl: \"https://books.toscrape.com/catalogue/page-1.html\"\ntier: 1\nmax_pages: 5\nrate_limit_sec: 2.0\npaginate:\n  selector: \"li.next a\"\nitems:\n  selector: \"article.product_pod\"\nfields:\n  title:        { selector: \"h3 a\", attr: title }\n  url:          { selector: \"h3 a\", attr: href, absolute: true }\n  image:        { selector: \"img.thumbnail\", attr: src, absolute: true }\n  price:        { selector: \"p.price_color\", attr: text, type: float }\n  availability: { selector: \"p.instock.availability\", attr: text }\noutput: csv\n```\n\n## The escalation ladder\n\nEvery page starts at the probed tier (or the config's `tier`) and climbs only\nwhen that rung fails.\n\n| What came back | What happens next |\n|---|---|\n| rate limited (429, or 419 without a challenge) | try the page through another proxy group; if there is none, wait once (`Retry-After`, else `cooldown_sec`) and retry. Prints `[ladder] <host>: rate limited, waiting <n>s` |\n| challenge page (Cloudflare, PerimeterX, DataDome markers) | keep the proxy, raise the tier by one, up to `max_tier` |\n| plain 403 | try another proxy group first; with none left, raise the tier |\n| 5xx | an error page keeps the proxy: retry once at the same tier, then raise the tier. A short 503 (body under 20 kB) is classified as a block instead, not an error, so it gets no same-tier retry: it buys another proxy group when one exists, otherwise it raises the tier |\n| no response (refused, DNS, TLS, reset) | with a pool, try another proxy group once; then, pool or not, retry once at the same tier, then raise the tier |\n\nEvery group switch also prints `[ladder] <host>: <reason>, trying another proxy\ngroup`, so a rotation is visible while it happens.\n\nOnce a site escalates on a challenge or a 403, later pages on that site start\nat the new level. A level reached only by an error climb is not remembered.\n`steps` in the page report shows the rungs and the leased group, for example\n`[\"t1@dc-a\", \"t1@dc-b\", \"t2@dc-b\"]` (a 403 from one group, a challenge from the\nother, real HTML at tier 2).\n\n### Tiers\n\n| Tier | What it is | What it costs |\n|---|---|---|\n| 1 | `httpx`, plain HTTP | milliseconds per page, almost no install |\n| 2 | `curl_cffi`, TLS impersonation | a real Python dependency; a site that rejects Python's TLS fingerprint may still serve it real HTML |\n| 3 | a browser stack (`scrapling`, `cloakbrowser` or `playwright`) | an optional, heavy extra; used only with `max_tier: 3` and never promised for a specific site |\n\n`tier: auto` probes first and picks the lowest tier that returns a real page.\nWith a proxy pool configured, `auto` skips the probe and starts at tier 1: a\nprobe is a direct request, and with a pool no request may leave from the origin\nIP. The ladder climbs from there. A tier is not a promise: it is where the\nladder starts.\n\n## Outcomes and exit codes\n\nOne record per page, available with `--report report.jsonl` and in the MCP\nresponse's `pages` field:\n\n```json\n{\"url\": \"https://httpbin.org/status/403\", \"outcome\": \"blocked\", \"status\": 403, \"rows\": 0, \"attempts\": 2, \"steps\": [\"t1\", \"t2\"], \"proxy_group\": null, \"elapsed_ms\": 1358, \"note\": \"max_tier 2 reached\"}\n```\n\n`outcome` is one of `ok`, `empty`, `blocked`, `rate_limited`, `not_found`,\n`error`. A run also prints one summary line:\n\n```\n[done] 2 pages: 1 ok, 1 blocked (HTTP 403 on /status/403)\n```\n\nA page with no response has no HTTP status to quote, so it reads\n`[done] 1 page: 1 error (no response on /list)`.\n\n| Exit code | Meaning |\n|---|---|\n| 0 | complete: at least `--min-rows`, no incomplete page |\n| 1 | config, I/O or fetch error; with `--min-rows 0`, a run whose every page errored |\n| 2 | fewer than `--min-rows` rows (2 wins over 4); an all-error run at the default `--min-rows 1` has no rows, so it is 2 |\n| 3 | `probe --fail-blocked`: the target is walled |\n| 4 | incomplete: at least `--min-rows`, but one or more pages ended `blocked`, `rate_limited` or `error` after all retries |\n\nExit 1 for a page-level failure is reachable only when `--min-rows 0` asks for\nno rows. At the default `--min-rows 1`, a run that lost every page to errors\nfails the row check first and exits 2.\n\nExit 4 is narrow on purpose. A trailing `not_found` or `empty` page past the\nend of a range does not count, and neither does a page that succeeded on retry.\n`--accept-partial` maps 4 to 0 for scripts that only check for zero. A run that\nloses one page still writes the rows it did get, and the report says which page\nis missing.\n\n## MCP server (for agents)\n\n```bash\npip install \"certainlyscraped[mcp]\"\ncertainlyscraped mcp\n```\n\nClient configuration:\n\n```json\n{\n  \"mcpServers\": {\n    \"certainlyscraped\": {\n      \"command\": \"certainlyscraped\",\n      \"args\": [\"mcp\"]\n    }\n  }\n}\n```\n\nFour tools, each a thin call into the same API:\n\n| Tool | Returns |\n|---|---|\n| `probe(url)` | tier verdict, block markers, embedded JSON found |\n| `init(url)` | config YAML plus field coverage |\n| `try_selector(url, selector, within?)` | match count, sample values, hints |\n| `scrape(config, min_rows?, max_rows?)` | `status`, rows (up to `max_rows`, default 50), the true row count, a dataset path, and per-page outcomes |\n\n`config` is inline YAML or JSON text, never a path. Config keys `proxy`,\n`proxies`, `cookies`, `since`, `snapshot` and `fetcher` are refused, and so are\nunknown keys such as `json` and `headers` (embedded JSON goes under\n`items.json`). `status` says what the run\ngot: `complete` (at least `min_rows` rows and no incomplete page), `incomplete`\n(enough rows, but a page ended blocked, rate-limited or errored),\n`too_few_rows` (fewer than `min_rows` rows), `blocked` (zero rows and every\npage was walled or rate-limited) and `error` (zero rows and every page\nerrored). These are not the CLI's exit codes: a blocked-only `run` exits 2,\nexit 3 is only `probe --fail-blocked`, and exit 1 is a config, I/O or fetch\nexception (or, with `--min-rows 0`, an all-error run). A bad request is still a\nstatus dict: a tool never raises into the JSON-RPC channel.\n\nPrivate addresses are refused by default: loopback, RFC 1918, link-local\n(including `169.254.169.254`) and unique-local IPv6, checked after DNS\nresolution and on every redirect hop. `--allow-private` turns that off for\nlocal development. The CLI does not refuse private addresses; only the MCP\nserver does. A refusal is terminal for the page: one attempt, no retry, no\nclimb, outcome `error` with the refusal in the note.\n\nProxies come only from the server's own `--proxies proxies.yaml`, never from a\ntool argument, and the server never reads `CERTAINLYSCRAPED_PROXIES`. `probe`,\n`init` and `try_selector` lease one proxy from that pool for the call; if the\npool has nothing free they return `blocked` with `no proxy available for\n<host>` and make no request. Without `--proxies` the server has no pool and\nfetches directly. File reads and datasets stay inside the working directory.\n\n## Proxies\n\nOne proxy still works and becomes a pool of one:\n\n```yaml\nproxy: \"http://10.1.2.3:8000\"\n```\n\nA pool names groups so a 403 or a 429 can try a different network before paying\nfor a heavier tier:\n\n```yaml\nproxies:\n  list: proxies.txt          # one URL per line, optional \"# group\" label\n  gateway:                   # one generic provider gateway\n    template: \"http://{user}-country-{country}-session-{session}:{password}@gw.example:7000\"\n    user: myuser\n    password_env: GATEWAY_PASSWORD\n    country: se\n    session: rotate          # or sticky\n  per_proxy_rate_sec: 1.0\n  cooldown_sec: 300\n  max_retries: 3\n```\n\n`proxies.txt`:\n\n```\nhttp://user:pass@10.1.2.3:8000 # dc-a\nhttp://10.1.9.9:8000           # dc-b\nhttp://10.1.1.7:8000\n```\n\nGroups are the label after `#`, else a neutral label `net-1`, `net-2`, ... in\norder of first appearance, one per IPv4 `/16`, IPv6 `/48` or hostname. A label\nnever contains a proxy address or hostname.\nA proxy that is walled for one host is parked for that host for\n`cooldown_sec`; it is not removed from the pool. `session: rotate` generates a\nnew token per lease: one per page, and a 403 or 429 that switches group leases\nagain. `sticky` keeps one token per host and worker.\n\nPasswords never belong in the main config. Put them in a dedicated proxies\nfile passed with `--proxies`, or in the environment through `password_env`.\nA password inside an inline `proxy:`/`proxies:` URL in the committed YAML is\nrefused with a message naming both alternatives. Page outcomes and MCP\nresponses carry the group label, never the proxy URL or password. A page that\nfailed on a dead proxy keeps the transport error as its note, for example\n`ConnectionError: Failed to perform, curl: (7) Failed to connect to\nbooks.toscrape.com:443 over proxy *** ...` (the target is kept, the proxy\naddress is `***`); an unresolvable host reads\n`DNSError: ... Could not resolve host: no-such-host.invalid ...`. No log line\nquotes a proxy address, so the group label and the error text are the\ndiagnosis. A private address refused by the MCP guard is terminal: one\nattempt, no retry, no climb.\n\nPrecedence for `run`: `--no-proxy` > `--proxies FILE` >\n`CERTAINLYSCRAPED_PROXIES` > the config. `--no-proxy` ignores every configured\nproxy and fetches directly. MCP has no such flag, never reads\n`CERTAINLYSCRAPED_PROXIES`, and without `--proxies` has no pool at all.\n\nSpeed comes from many hosts or a proxy pool, not from threads alone. The ladder\npaces per (host, proxy) at `per_proxy_rate_sec` (it falls back to\n`rate_limit_sec`; default 1.0).\nWith no pool, one host gets at most one request per interval no matter the\n`concurrency`, so a single-host `pages:` job is still paced by the site. With a\npool, N proxies to one host give N slots: eight pages at 1.0 s on one host took\n7.1 s with one proxy and 1.1 s with four, in one derived `net-1` group or four\nlabelled groups alike.\n\n## Parallel pages\n\n```yaml\npages:\n  template: \"https://news.ycombinator.com/news?p={n}\"\n  from: 1\n  to: 10\nconcurrency: 4\n```\n\nOr an explicit list:\n\n```yaml\nurls:\n  - \"https://example.com/a\"\n  - \"https://example.com/b\"\n```\n\nRows are emitted in page order regardless of which response arrived first.\nA `pages:` range stops after any two consecutive `not_found` or `empty` pages,\nnot only at the end of the range: a gap in the middle ends the run early. Those\npages are not incomplete, so the run can still exit 0.\n\n## More commands\n\n```bash\ncscrape try <url> \"p.price_color\" --in \"article.product_pod\"   # test a field\ncscrape fetch <url> --out page.html                             # save once, iterate offline\ncscrape probe <url>                                             # tier verdict\ncscrape verify data.csv --min-rows 10                           # check a dataset after the fact\n```\n\n`try` also flags pages rendered by JavaScript, points at embedded JSON, and\nwarns about brittle DevTools \"Copy selector\" paths. [docs/selectors.md](docs/selectors.md)\nis the full guide.\n\n## What it will not do\n\n- Log in, get past login walls, or use credentials that are not yours.\n- Promise that a site works today because it worked on another day.\n- Beat fingerprint or JavaScript challenges (Cloudflare managed challenge,\n  PerimeterX, DataDome) with proxies. Those pages report `blocked`; a proxy you\n  bring gets past IP bans, IP-range bans and rate limits, not a challenge.\n- Ship scraped data. It ships the tool.\n\n## What it reaches\n\n[`COVERAGE.md`](COVERAGE.md) is a table of 40 popular data targets, probed\nnightly and published with a date: which returned rows, which returned a\nJavaScript shell, which returned a repeated block that is not the data\n(tracking JSON, a cookie dialog, a priceless carousel on a shop), and which\nwere walled. Each row lists the fields it found, so you can judge the verdict\nyourself. It is generated from\n[`targets/popular.yaml`](targets/popular.yaml) and it updates itself, so a\nsite that starts refusing traffic shows up there instead of staying a claim in\nthis file. A site behind a login is walled here by definition: tiers 1 to 3 do\nnot have your password, and do not ask for it.\n\n## Why trust the numbers\n\nEvery run records one outcome per page, and the daily live check scrapes the\nsame example sites asserted at exit 0, so a quiet regression shows up as a red\ncheck rather than a friendly empty CSV.\n\n## The old name\n\nThe `scrapeforge` command still works until 0.5.0 and prints one deprecation\nnotice on stderr; `import scrapeforge` also works and raises a\n`DeprecationWarning`, which Python shows when the import runs in `__main__`\n(the usual script case) and hides inside imported modules. New code should use\n`certainlyscraped` or `cscrape`; the old name is removed in 0.5.0.\n\n## Development\n\n```bash\ngit clone https://github.com/vhsgreed/certainlyscraped && cd certainlyscraped\npip install -e \".[dev]\"\npython -m pytest -q\n```\n\nTests run offline against saved pages in `tests/fixtures/`. CI runs them on\nPython 3.11 to 3.14. The live check runs the examples against real sites daily.\nMIT licensed. Part of the [vhsgreed](https://vhsgreed.win) toolset.\n\n<!-- mcp-name: io.github.vhsgreed/certainlyscraped -->\n"}