{"id":"860f4031-c8cc-4f06-aa44-a42cff1801cb","slug":"pinchtab-webgraph","kind":"mcp","name":"pinchtab-webgraph","summary":"🕸️ Crawl any website into a queryable navigation + content graph and find the shortest click-path to any action — deterministic, no-LLM browser automation","owner":"egouilliard","repoUrl":"https://github.com/egouilliard/pinchtab-webgraph","license":"MIT","licenseTier":"permissive","trust":"community","stars":5,"match":null,"fewStars":false,"format":"mcp","createdAt":"2026-07-02T10:14:12.000Z","addedAt":"2026-10-21T17:38:11.639Z","logoUrl":"https://avatars.githubusercontent.com/u/91854688?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["browser-web","developer-tools","memory-knowledge"],"tags":["accessibility","ai-agents","aria","browser-automation","cli","cytoscape","graph","knowledge-graph"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T17:42:30.672Z","sample":false,"url":"https://conduits.dev/tools/pinchtab-webgraph","language":"Python","transports":["stdio"],"install":{"pypi":"pinchtab-webgraph"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# 🕸️ pinchtab-webgraph\n\n***Turn any website into a queryable navigation + content graph, then answer \"how do I do X?\" as the shortest click-path — deterministically, with no LLM in the runtime.***\n\n![License](https://img.shields.io/badge/license-MIT-3DA639?style=flat-square)\n![Python](https://img.shields.io/badge/python-3.10+-3776AB?style=flat-square&logo=python&logoColor=white)\n![No LLM at runtime](https://img.shields.io/badge/runtime-no%20LLM-6E56CF?style=flat-square)\n![Browser automation](https://img.shields.io/badge/driver-PinchTab-FF6B00?style=flat-square)\n![Graph viewer](https://img.shields.io/badge/viewer-Cytoscape.js-F7A81B?style=flat-square)\n\n`pinchtab-webgraph` drives a **real, JavaScript-rendered browser** (through the [PinchTab](#-requirements) automation CLI) to map an entire web app — every page, SPA state, button, tab, menu, form, and data collection — into a structured **navigation + content graph**. It then answers questions *offline* against that graph in milliseconds: shortest click-path between two views, \"how do I create a X\" (with the target form's fields read straight from the live UI), and \"where does this data live and how do I reach it\".\n\nThe whole pipeline is **deterministic** — structural heuristics only (ARIA roles, repeated-sibling detection, URL grouping), no model in the loop, no per-run cost, reproducible output. It works on *any* site: there is no app-specific vocabulary anywhere in the crawler.\n\n## ✨ Highlights\n\n- **Graph from anything** — one crawl records each state's full control inventory (links / buttons / tabs / menus) **and** its data collections (tables, grids, trees, lists, feeds, virtualized/scroll-loaded content). A complete nav + content graph of any site, all from structural signals.\n- **Offline \"how-to\" in milliseconds** — BFS over the crawled graph returns the shortest click-path to any action *plus* the fields of the form it opens, in ~60–130 ms with zero browser calls.\n- **Runnable answers, not just directions** — every how-to also compiles the click-path + terminal action into a copy-pasteable **PinchTab command block** (`pinchtab nav … → click … → download/upload/fill …`). So \"how do I download the Q3 report\" returns both the route *and* a `pinchtab download <url> -o q3.pdf` you can run. Form answers become a `fill`/`select`/`check` script with the submit line left **commented out** (safety). See [Runnable command blocks](#️-runnable-command-blocks).\n- **Declarative automation flows — that you author by *describing* them.** A how-to answers \"how do I do X\" and `perform` does it *once, in a straight line*. A **flow** is the layer above: a JSON **document** — not a script — executed by a step VM with `for_each` / `paginate` / `collect`, so \"download all 20 PDFs on this page, then do it again for each of the 12 pages, and only keep the files I don't already have\" is 12 lines of JSON. And you don't type them: **say it to an agent**, which drafts the document grounded in the *real crawled graph* (it looks the trigger up instead of inventing a selector) onto an **editable visual canvas**. It can only **propose** — you Save and you Run. No code executes, so a scheduler or an HTTP handler can safely run one; every download is content-hashed against a **persistent dedupe ledger**, which turns a dumb poller into a change detector. See [Automation flows](#-automation-flows).\n- **Download / export discovery** — download and export affordances become read-only `download` nodes: a link that resolves to a file (a `download` attribute, a file-extension path, or a `blob:`/`data:` URL → tagged with the file URL) or a button whose label is a download/export verb (JS-triggered). Like uploads, they are **never clicked** during the crawl — detected, recorded, and turned into a `pinchtab download`/`click` command on demand.\n- **File-upload discovery** — the crawl also finds where you can upload a document. File inputs (including ones hidden behind a styled `<label>`/button) and `ondrop` dropzones become read-only `upload` nodes tagged with the file types they accept (e.g. `.pdf,.docx`, `image/*`), so \"how do I upload a … ?\" is answerable — and the crawler never clicks them (that would pop a native OS file dialog).\n- **Safe by construction** — discovery opens and reads forms, then presses Escape. It never submits, saves, or deletes anything. Destructive-looking controls are skipped and recorded, not clicked.\n- **Never loses progress** — atomic checkpoints every N states plus a SIGINT/SIGTERM handler; a crash, OOM, or Ctrl-C keeps the partial graph. `meta.stopped` always says *why* a crawl ended (complete vs. truncated) — no silent truncation.\n- **Knows what it has already crawled** — a state is identified by *what the screen is*, not only by its URL. Apps that keep UI state in the query string (switch the main tab and the previous sub-tab's `?…Tab=` params stay in the URL) render one screen under dozens of URLs; the interaction crawler fingerprints each rendered screen and records a new URL that renders a known screen as an **alias** — no re-expansion, no re-capture, no form re-read. Persistent chrome (a tab bar) whose target is already proven is not re-clicked from every tab. See [Screen dedupe](#-screen-dedupe-already-crawled-detection).\n- **Spans app boundaries** — `--cross-host` follows links and `iframe[src]` into other hosts as graph nodes, so an embedded/linked app becomes part of the same graph. App-shell SPAs (e.g. Teams-style apps that swap views without changing the URL) are **auto-detected** and driven in single-URL mode — no flag needed; `--single-url` / `--no-single-url` force it either way. Their create-forms are read **in place** (a JS-dispatch click, never a navigation that would blank the shell; a trigger that can't open without navigating is recorded form-less and the crawl recovers in place).\n- **Cache-first workflow** — `ask.py` answers from a per-host cache when it can, falls back to a live discovery on a miss, and writes the result back so the next ask is an offline hit. A cache *hit* answers in milliseconds (measured: **0.4s**); a cold *miss* stays fast too — a shallow \"how do I X\" (trigger 0–2 clicks deep) discovers in **under 10s** end-to-end (measured against a real bridge: **7.5s** cold, including the live crawl and the cache write-back) because each state costs one bridge round-trip to settle, not a poll loop of them. `scripts/bench-discovery.sh` regression-guards that offline with no live app — compare its `ROUNDTR` column, which is load-independent, rather than its wall-clock, which is inflated by the fake bridge's per-call python fork. `recipe.py --time-budget SECONDS` caps the worst case for deep, blind multi-step discovery.\n- **No LLM in the runtime** — indexing and path-finding are pure Python + the PinchTab CLI. Predictable, reproducible, free to re-run.\n- **Three ways to call it** — the exact same graph queries are reachable from a full **CLI**, a **Model Context Protocol (MCP)** server, and a **Universal Tool Calling Protocol (UTCP)** manual — all over one shared, importable core API. See [Three ways to call it](#-three-ways-to-call-it).\n\n## 📑 Table of Contents\n\n- [Why a web-navigation graph?](#-why-a-web-navigation-graph)\n- [Requirements](#-requirements)\n- [Quickstart](#-quickstart)\n- [The tools](#️-the-tools)\n- [Runnable command blocks](#️-runnable-command-blocks)\n- [Automation flows](#-automation-flows)\n- [Self-test & report](#-self-test--report)\n- [Regression audit (10 public sites)](#-regression-audit-10-public-sites)\n- [Three ways to call it](#-three-ways-to-call-it)\n- [MCP server](#-mcp-server)\n- [UTCP interface](#-utcp-interface)\n- [Documentation](#-documentation)\n- [How interaction crawling works](#-how-interaction-crawling-works)\n- [Architecture](#️-architecture)\n- [Graph shape](#-graph-shape)\n- [Screen dedupe (already-crawled detection)](#-screen-dedupe-already-crawled-detection)\n- [Safety model](#️-safety-model)\n- [Authenticated apps (login)](#-authenticated-apps-login)\n- [Importing into Neo4j](#️-importing-into-neo4j-optional)\n- [Roadmap](#️-roadmap)\n- [Contributing](#-contributing)\n- [Star History](#-star-history)\n- [License](#-license)\n\n## 🧠 Why a web-navigation graph?\n\nAutomating or documenting a web app usually means one of two brittle things: hand-writing selectors that rot on every redesign, or asking an LLM to \"figure out the UI\" live on every request (slow, non-deterministic, and expensive).\n\n`pinchtab-webgraph` takes a different stance: **crawl the UI once into a graph, then query the graph.**\n\n- **How-to guides & onboarding** — \"how do I create a template / an invoice / a new team?\" becomes a shortest-path query that returns the exact clicks *and* the form fields, in milliseconds.\n- **Change detection & QA** — snapshot the full control + content graph, then diff two crawls to see what moved, appeared, or disappeared.\n- **Site maps for humans and agents** — a structured, low-noise map of an app's real navigation, far cheaper than replaying a browser for every question an agent asks.\n- **Content discovery** — `--find TEXT` searches every view's captured data (rows / files / messages / cards) and returns what matched, which view it's in, and the click-path to get there. Add `--all-hosts` to search **every crawled host at once**, offline — results are merged, ranked by reach, and labeled by origin host.\n\n## 📦 Requirements\n\n- **Python 3.10+** — the tools are pure Python, no third-party dependencies.\n- **The [PinchTab](https://github.com/) browser-automation CLI** available on your `PATH` as `pinchtab`. Every tool drives the live browser through it; you run an **isolated** PinchTab bridge (own profile, own port) so a \"click-everything\" crawl never touches a browser holding a live session you care about.\n\n## 📦 Install\n\nPure Python (stdlib only) — the one runtime prerequisite is the external [PinchTab](#-requirements) CLI.\n\n```bash\n# from GitHub (no PyPI account needed):\npipx install git+https://github.com/egouilliard/pinchtab-webgraph\n#   or:  uv tool install git+https://github.com/egouilliard/pinchtab-webgraph\n#   or:  pip install git+https://github.com/egouilliard/pinchtab-webgraph\n```\n\nThis installs the **`pinchtab-webgraph`** command (short alias **`pwg`**) with subcommands\n`crawl · howto · ask · recipe · linkcrawl · paths · test`. Run `pinchtab-webgraph --help` for the map.\n\n## 🚀 Quickstart\n\n```bash\n# 1. Start the isolated crawl browser (own profile/port). Leave it running.\n#    (a PinchTab bridge — see Requirements; a helper script lives in the repo)\n\n# 2a. Full interaction + content graph of an app (the main tool):\npinchtab-webgraph crawl https://app.example.com/dashboard --out out/app     # (pwg crawl …)\n\n# 2b. …or a page→page link graph + interactive Cytoscape viewer:\npinchtab-webgraph linkcrawl https://docs.example.com --interaction-depth 0 --out out/docs\nxdg-open out/docs.html\n\n# 3. Ask the graph, offline, in milliseconds:\npwg howto out/app.json --goal \"create template\"     # shortest click-path + form spec\npwg howto out/app.json --find \"invoice\"             # where does this data live + how to reach it\npwg howto out/app.json --list-content               # per-view data inventory\n```\n\nGraphs and screenshots default to the gitignored `out/` directory (e.g. `out/webgraph.json`, `out/recipe.png`); pass `--out <path>` to change it — parent dirs are created for you. From a git checkout you can also run any tool without installing: `python3 -m pinchtab_webgraph.cli crawl …`.\n\nThe `scripts/run-*.sh` wrappers forward the bridge auth token automatically and point at the isolated browser. Copy `crawl-config.example.json` to `crawl-config.json` and set a real token (`openssl rand -hex 24`) before the first run — `crawl-config.json` is gitignored because it holds that token.\n\n## 🛠️ The tools\n\n| Tool | What it does |\n| --- | --- |\n| `interaction_crawl.py` / `scripts/run-crawl-interactions.sh <url>` | **The core.** Crawls the live UI once into an interaction graph: states + action edges + every create-trigger's form spec. Full **capture-all is the default** — control inventory *and* data collections per state. Atomic checkpoints (never loses progress), explicit truncation reasons in `meta.stopped`. Modes: app-shell SPA mode is auto-detected (override with `--single-url` / `--no-single-url`), `--cross-host` (follow links + iframes to other hosts). Safe: opens and reads forms, never submits. |\n| `howto.py <graph.json>` | **Offline** BFS over a crawled graph → shortest click-path + form spec in ms, no browser. `--goal \"…\"` for actions; `--find TEXT` searches captured data → what matched, which view, and the path to it; `--list-content` = per-view data inventory. |\n| `ask.py` / `scripts/run-ask.sh` | **Cache-first** entry point. Routes by host to a per-host cache, answers offline via `howto.py`; on a miss runs a live discovery, then writes the result back so the next ask is an offline hit. `--verify` re-checks live. Content queries (`--find` / `--list-content`) also take **`--all-hosts`** to search *every* cached host in one offline pass — results merged, ranked by reach, and labeled by origin host. |\n| `recipe.py` / `scripts/run-recipe.sh` | **Live** how-to finder: priority-BFS over the running UI to a goal's trigger, opens the form, reads the fields, never submits. The live fallback for cache misses. |\n| `perform.py` (`pinchtab-webgraph perform`) | **PERFORM** a how-to: resolve it OFFLINE (from a crawled cache/graph), then RUN the compiled block live through the bridge — navigate the path, then download / upload / fill. Safe by default: navigation + downloads run; a form field with no supplied value is skipped (`--set 'Label=value'`, `--file <path>`); the submit runs only with `--allow-submit`; `--dry-run` previews. See [Runnable command blocks](#️-runnable-command-blocks). |\n| `flow_cmd.py` (`pinchtab-webgraph flow`) | **Run a declarative automation FLOW** — a JSON document (not a script) executed by a step VM: `goto` / `do` / `click` / `fill` / `download` / `collect` plus the control-flow ops **`for_each`** and **`paginate`**. Downloads are content-hashed against a **persistent dedupe ledger** (a re-run reports `dupe`, not a result). Safe by default: a write runs only if the flow *declares* the capability **and** the caller *grants* it. `flow run \\| validate \\| schema`. See [Automation flows](#-automation-flows) and [`docs/flows.md`](docs/flows.md). |\n| `crawl.py` / `scripts/run-crawl.sh <url>` | Page→page **link graph** → `<out>.json` + a self-contained Cytoscape.js `<out>.html` viewer. |\n| `paths.py` | Offline shortest / all click-paths over a crawled link graph (`--from`, `--to`, `--structural`, `--all`). |\n| `login.py` (`pinchtab-webgraph login`) | Open a persistent browser session and sign in to a host (credentials from the OS keyring) so subsequent crawls run authenticated. Needs the optional `login` extra (`keyring`). |\n| `cache_cmd.py` (`pinchtab-webgraph cache`) | Inspect / manage the per-host interaction-graph caches `ask.py` writes back: `cache list`, `cache path <host>`, `cache show <host>`, `cache clear <host>` / `--all` (destructive, dry-run unless `--yes`). |\n| `query_cmd.py` (`pinchtab-webgraph query`) | **Machine-readable** twin of `howto.py` / `paths.py`: runs the offline `api.*` queries (`graph_summary`, `howto`, `find_content`, `list_content`, `list_forms`, `link_paths`, plus the cross-host `find_content_hosts` / `list_content_hosts`) and prints the result as JSON on stdout. Takes `--host` (cache) or `--graph` (path); the cross-host ops take neither and search every cached host. The substrate the UTCP manual shells out to. |\n| `utcp_manual.py` (`pinchtab-webgraph manual`) | Build / print / serve the [UTCP](https://www.utcp.io) tool-calling manual so external tool-callers can invoke the `query` (and live `crawl`/`ask`) surface by running the CLI directly — no wrapper server. `manual --out FILE` / `manual --serve`. |\n| `selftest.py` (`pinchtab-webgraph test`) | **Self-improvement loop.** Interactively throw your hardest \"how do I do X?\" goals at a crawled graph — each is answered **offline** via `api.howto`, you judge whether it's right, and every miss/wrong answer becomes a captured gap. Writes a self-contained HTML report; with `--repo OWNER/NAME` it can (after you confirm) file the report as a GitHub issue. See [Self-test & report](#-self-test--report). |\n| `commands.py` | The deterministic **path → executable** compiler shared by every how-to surface. Turns a click-path + terminal action into a runnable `pinchtab` command block (`nav`/`click`/`download`/`upload`/`fill`/`select`/`check`). Pure, stdlib-only, no browser. See [Runnable command blocks](#️-runnable-command-blocks). |\n\n## ▶️ Runnable command blocks\n\nEvery how-to answer — from `howto.py`, `recipe.py`, `ask.py`, and the `api`/MCP/UTCP surfaces — now carries a `commands` block: the shortest click-path **and** its terminal action, compiled into a copy-pasteable sequence of [PinchTab](#-requirements) CLI commands that reproduces it. The route tells a human what to click; the command block lets an agent (or you) *run* it.\n\nThe terminal action is chosen structurally from the graph — no LLM, no app-specific vocabulary:\n\n| Action kind | How it's recognized | Emitted command |\n| --- | --- | --- |\n| **download** (direct) | a link/anchor that resolves to a file (`download` attr, file-extension path, or `blob:`/`data:` URL) | `pinchtab download '<url>' -o '<file>'` |\n| **download** (JS) | a control whose label is a download/export verb | `pinchtab click --css '<selector>' --wait-nav` — the browser session captures the file |\n| **upload** | a file input / styled `<label>` / `ondrop` dropzone | `pinchtab upload '<FILE>' -s '<selector>'` (accepted types annotated) |\n| **form** | a create-style trigger that opens a form | one `fill`/`select`/`check` per field (placeholder values); the **submit line is commented out** |\n\n```console\n$ pinchtab-webgraph query --host app.example.com howto --goal \"download the q3 report\"   # (or howto.py / ask.py)\n\n=== HOW TO: DOWNLOAD THE Q3 REPORT ===\nShortest route — 2 clicks:\n  1. Go to https://app.example.com/home\n  2. Click \"Reports\"\n  3. Download via \"Download report\"\nThis downloads a file: https://app.example.com/files/q3-report.pdf\n\nRun it with PinchTab:\n  pinchtab nav 'https://app.example.com/home'\n  pinchtab nav 'https://app.example.com/reports'   # Reports\n\n  # --- download ---\n  pinchtab download 'https://app.example.com/files/q3-report.pdf' -o 'q3-report.pdf'\n```\n\n**Safe by construction.** The click-path only ever navigates — it never traverses a write/destructive control (those are `skipped` nodes with no selector recorded). Download and upload affordances are detected but **never clicked during the crawl** (a JS download can pop a native OS save dialog; a file input opens a native picker). And the form terminal fills fields but leaves the submit **commented out** unless you deliberately uncomment it. Nothing a compiled block does mutates data on its own.\n\nThe structured surfaces (`api.howto`, MCP, UTCP) return the same thing under `commands`, plus `action_kind` (`form` / `download`) and, for a direct download, `download_url`.\n\n### …and run it — `perform` (the opt-in)\n\nThe block above is copy-pasteable, but you don't have to copy-paste it. `pinchtab-webgraph perform` **resolves the how-to offline and then runs the compiled block through the bridge** — so \"download the Q3 report\" is one command end-to-end:\n\n```console\n$ pinchtab-webgraph perform --host app.example.com --goal \"download the q3 report\"\n=== PERFORM: DOWNLOAD THE Q3 REPORT ===  (download, ran)\n  ✓ pinchtab nav 'https://app.example.com/home'\n  ✓ pinchtab nav 'https://app.example.com/reports'   # Reports\n  ✓ pinchtab download 'https://app.example.com/files/q3-report.pdf' -o 'q3-report.pdf'\n```\n\nThe **same safety rules are enforced at execution**, not just in the printed text:\n\n- **navigation + downloads run** (downloading is the point); `--out-dir <dir>` chooses where the file lands.\n- a **form field with no value is skipped**, never filled with placeholder junk — pass real values with `--set \"Name=Acme\" --set \"Plan=Pro\"`, and a file with `--file ./doc.pdf`.\n- a form's **submit never runs** unless you add `--allow-submit`.\n- `--dry-run` prints exactly what *would* run and touches nothing; `--json` emits a structured per-step result (run / skipped / error).\n\nResolution is offline, so `perform` needs a crawled cache/graph (`--host <h>` or `--graph <file>`) — crawl or `ask` the site first. Only execution needs the bridge. The same capability is exposed as the MCP `perform` tool and the UTCP `perform` manual entry.\n\n**Bridge requirements for the two download kinds:**\n\n- **JS-triggered downloads** (a `click` on an export button) work whenever the bridge config has `security.allowDownload = true`; the file lands in the browser profile's download directory.\n- **Direct downloads** (`pinchtab download <url>`) also need `allowDownload = true`, but note PinchTab's `download` performs a **server-side fetch guarded against SSRF** — it refuses `internal or blocked host` URLs (e.g. `localhost`/`127.0.0.1`/link-local). That only affects fetching *internal* hosts; a normal `https://app.example.com/…/file.pdf` is fine. (The crawl bridge ships `allowDownload = false` on purpose — that config is for read-only discovery — so point `perform` at a bridge that enables it.) A rejected download surfaces the bridge's error verbatim.\n\n> **Verified end-to-end** against a real site crawled through the live browser: `crawl → howto → perform` navigates the path and the browser actually writes the file to disk (JS-export path), and the form path fills + submits with real values. See the walkthrough in [`docs/perform-live-test.md`](docs/perform-live-test.md).\n\n## 🔁 Automation flows\n\n`perform` runs a **straight line**: nav, click, fill, submit — one how-to, once. Every real automation needs more: *download all 20 PDFs on this page, then do it again for each of the 12 pages, and only keep the files I don't already have.* That is a **flow**.\n\nA flow is a **JSON document, not a script**, executed by a step VM. That is load-bearing: no arbitrary code runs, so a scheduler or an HTTP handler can safely execute one; a step names its target **semantically** (`goal` / `match`) as well as structurally, so it re-resolves against a re-crawled graph instead of snapping on a stale selector; and `inputs` derives a **JSON Schema**, so a saved flow can become a typed endpoint / MCP tool with no hand-written wrapper.\n\n**You don't hand-write it — you describe it.** JSON is the *format*, not the interface. In the [web UI](#-three-ways-to-call-it)'s **Flows tab** you say what you want, and an agent drafts the document **grounded in the site you actually crawled**:\n\n> **you:** download every report PDF across all the pages\n\nIt calls `find_content` / `howto` / `graph_summary` over the **real interaction graph** to find the actual trigger and click-path — so it *resolves* a real control instead of guessing at `.btn-download`, which is the whole advantage over a generic AI flow builder. Then it hands you this:\n\n```json\n{\n  \"name\": \"download-all-invoices\",\n  \"host\": \"app.example.com\",\n  \"inputs\": { \"since\": { \"type\": \"string\", \"required\": false } },\n  \"capabilities\": { \"allow_download\": true },\n  \"steps\": [\n    { \"op\": \"goto\", \"goal\": \"invoices\" },\n    { \"op\": \"paginate\", \"max_pages\": 50, \"body\": [\n      { \"op\": \"for_each\", \"match\": { \"kind\": \"download\" }, \"as\": \"item\", \"body\": [\n        { \"op\": \"download\", \"href\": \"${item.href}\", \"name\": \"${item.text}.pdf\" }\n      ]}\n    ]}\n  ]\n}\n```\n\n**The agent can only PROPOSE — you Save and you Run**, and that is *structural*, not a promise: its one flow tool (`propose_flow`) is a pure validate-and-echo (no disk, no browser, no subprocess — a test poisons `open`/`os.replace`/`subprocess` to prove it), and **no tool anywhere on the MCP surface can save or run a flow**. The draft lands in **three synchronized views of one document** — the chat, an editable **visual canvas** (`for_each`/`paginate` visibly *wrap* their children), and the JSON — each of which you can edit, with validation on every change. Your hand edits survive the agent's next revision, because every turn ships it your *live* document.\n\nThe same document runs unchanged from the CLI:\n\n```bash\npwg flow validate ./invoices.json                                  # structure + every ${var} + capabilities\npwg flow schema   ./invoices.json                                  # the `inputs` block as a JSON Schema\npwg flow run      ./invoices.json --host app.example.com --dry-run # print what WOULD run; touch nothing\npwg flow run      ./invoices.json --host app.example.com --input since=2026-01-01\n```\n```console\n=== FLOW: DOWNLOAD-ALL-INVOICES ===  (live)\n  ▸ run       flow=download-all-invoices\n  ✓ goto      goal=invoices target=Invoices\n  · paginate  page=1\n  ✓ for_each  match={\"kind\": \"download\", \"limit\": 200} found=2\n  ✓ download  name=Download report A.pdf via=fetch\n  …\n--- ok: 14 steps, 5 new file(s), 0 duplicate(s), 6.1s\n```\n\nNothing in the document is app-specific. `goal: \"invoices\"` resolves **offline** against the crawled graph (the same resolver `howto`/`perform` use); `match: {\"kind\": \"download\"}` is the crawler's structural download classifier applied to the live DOM; `paginate` finds the next-page control structurally (`rel=next` / `aria-label` / `aria-disabled` first, a UI-verb regex second) and stops when it's exhausted — or when the page stops changing.\n\n**Safe by default, twice over.** The effective capability is the **AND** of what the flow *declares* and what the caller *grants* — either side vetoes. Downloading is read-only and on by default (`--no-allow-download` withdraws it); a form **submit** and a file **upload** write to the site, so both need `capabilities.allow_submit`/`allow_upload` in the document **and** `--allow-submit`/`--allow-upload` on the command. A document that performs a write it didn't declare is rejected at *validation* time — so a scheduled run can never half-execute.\n\n**Re-running a flow is a change detector.** \"Download the report every 10s\" is really \"*tell me when a NEW report appears*\", so every downloaded file is sha256'd into a content-addressed store with a **dedupe ledger that persists across runs**: a file whose bytes were seen before comes back as a `dupe`, not a result. Downloads take the **in-session fetch** path first (a `fetch()` inside the page — it inherits the session's cookies, so authenticated apps just work), falling back to the `pinchtab download` CLI for cross-origin hrefs.\n\n> **Verified end-to-end** with nothing mocked (real bridge, real headless Chrome): crawl a 3-page fixture site → `paginate` all 3 pages → download **5 real files** (`via=\"fetch\"`, 5 distinct sha256s) → re-run on the same ledger → **0 new, 5 dupes**.\n\n**The Flows tab, in full.** Beyond the [agent + canvas + JSON](docs/flows.md#authoring-a-flow-with-the-ai-agent) workbench: the host's saved automations; validation on every keystroke (a typo'd `${var}` is caught, **with its path in the document**, before a browser is ever leased — and the offending canvas box lights up); **resolvability warnings** — a `goal` that matches nothing on the crawled site used to validate green and abort at run time, so it now warns in amber, *\"did you mean “Add Report”?\"*, checked through the **same resolver the runner uses**; a run panel where the **safety model is visible** — Allow-submit / Allow-upload are *disabled unless the flow itself declares that capability*, and dry-run is checked by default; a streaming run log with a live **`N new · M dupe`** counter; run history; and the flow's all-time artifact ledger. **Authoring is always available; only *running* is opt-in**, because unlike a crawl a flow *can* write to the site:\n\n```bash\nPINCHTAB_WEBGRAPH_ENABLE_FLOWS=1 pinchtab-webgraph-ui        # then open the Flows tab\n```\n\n> **AI authoring verified end-to-end**, nothing mocked: asked *\"download every report PDF across all the pages\"*, the agent queried the graph, proposed a real `goto` → `paginate` ⊃ `for_each` ⊃ `download`, the draft landed **live** on the canvas — saved and run, it fetched **5 new** files across 3 pages; re-run, **0 new / 5 dupe**.\n\nFull format (every op and its args), [AI authoring](docs/flows.md#authoring-a-flow-with-the-ai-agent) (the `propose_flow` tool, the `flow_draft` frame, the propose-only safety model, the [resolvability warnings](docs/flows.md#resolvability-warnings)), the capability model, the download constraints, the ledger, an authoring walkthrough, and the [web-UI surface](docs/flows.md#running-flows-from-the-web-ui) (storage layout, caps, REST + WS frames, the subprocess/cancel design): **[`docs/flows.md`](docs/flows.md)**.\n\n## 🧪 Self-test & report\n\nRight after you crawl a site, sanity-check that the graph actually answers the questions you care about — and turn every gap into feedback:\n\n```bash\n# Interactive: describe your hardest goals, judge each answer, keep going until done.\npwg test --start https://app.example.com/dashboard\n\n#   Scenario #1 — describe a hard goal (blank to finish): create a team\n#     ✓ graph found a path (3 clicks): … / form: 4 fields at the trigger\n#   Is that correct / what you expected? [Y/n] y\n#   Test another scenario? [Y/n] y\n#   …blank line finishes → writes test-report-app.example.com-<ts>.html\n\n# Non-interactive (CI / scripting): seed goals, get the HTML report unattended.\npwg test --graph app.json --goal \"create role\" --goal \"add invoice\" --out report.html\n\n# Opt-in issue: only with an explicit --repo, and only after you confirm the (public!) preview.\npwg test --start https://app.example.com/dashboard --repo egouilliard/pinchtab-webgraph\n```\n\nEach scenario is answered **offline** against the graph (no browser, deterministic) — a \"miss\" *is* the finding: it means the crawl didn't capture that path. The report groups scenarios by your verdict (pass / fail / unrated) with the returned click-path, form field count, and your \"what's wrong\" notes. **The report can contain target-app labels, URLs and form details** — issue creation is therefore off by default, requires you to name `--repo`, and shows the full body plus a PUBLIC-repo warning before posting.\n\n## 🎯 Regression audit (10 public sites)\n\nA repeatable extraction-quality gate over ten structurally different public sites (Hacker News, books/quotes.toscrape, python.org, getbootstrap, MDN, gov.uk, Wikipedia, stripe, automationexercise). It crawls each, then scores it **browser-free**, so a change to the crawler or the query layer is measured the same way every time.\n\n```bash\n# Copy the audit config and set a token, then run on the host (bridge reachable):\ncp crawl-config.audit.example.json crawl-config.audit.json   # set server.token\nscripts/site-audit.sh                 # crawl all 10 + score  (--gate dup, default)\n\n# Score already-crawled graphs without a browser:\npython3 tests/audit/check.py --graphs .audit-graphs\n```\n\nThe runner wipes the Chrome profile **and** the pinchtab `stateDir` per site (pinchtab restores open tabs from `stateDir` across restarts — otherwise a prior crawl's tabs leak into the next graph) and crawls with `--max-restarts 0`. `tests/audit/check.py` reports two things per site from `tests/audit/sites.json`:\n\n- **`dup_ratio`** — states that share a normalized URL. **0 means no over-noding** (the deterministic-identity guarantee above); it is the hard gate (`--gate dup`).\n- the **50 hard-question goals** (5 per site) run through the offline `howto` API — an informational scoreboard that rises as the crawler improves.\n\nThe scorer is unit-tested (`tests/audit/test_check.py`), and the identity guarantee has its own browser-free guards in `tests/test_state_identity.py`.\n\n## 🔌 Three ways to call it\n\nThe same crawl-once-query-offline capability is reachable through three interfaces, all layered over one importable core (`pinchtab_webgraph.api` — typed, print-free functions that return structured dicts). Pick whichever fits your consumer; they all resolve to the exact same graph queries, so their answers never disagree.\n\n| Interface | For | How | Extra dep |\n| --- | --- | --- | --- |\n| **CLI** | humans, shell/CI scripts | `pwg query howto --host app.example.com --goal \"create a team\"` → JSON on stdout (or the human-readable `pwg howto …`). `pwg --help` lists every subcommand. | none (pure stdlib) |\n| **MCP server** | LLM agents / MCP hosts (Claude, IDEs) | `pinchtab-webgraph-mcp` over stdio — 6 offline query tools, 2 live tools (`crawl`, `ask_howto`) with streamed progress, and `graph://…` resources. See [MCP server](#-mcp-server). | `pip install 'pinchtab-webgraph[mcp]'` |\n| **UTCP manual** | any UTCP-aware tool-caller | a static [UTCP](https://www.utcp.io) manual (`pwg manual`, `--out`, or `--serve`) whose `cli` call templates invoke `pwg` directly — no wrapper server in the call path. See [UTCP interface](#-utcp-interface). | none to use (`[utcp]` only validates it) |\n\nFor a point-and-click front end there's also an **optional local web UI** — a browser app with a **Workspace | Graph | Explore | Flows** view switcher: a Workspace of a \"how do I…\" chat agent + a live headless-browser pane, an interactive **Graph view** that renders the crawled interaction graph (states as blue circles, form-triggers as green diamonds) right in the browser, and an **Explore view** to search / browse everything the crawl captured — plus a read-only REST API over the same queries, behind the `pinchtab-webgraph-ui` script and the `[ui]` extra:\n\n| Interface | For | How | Extra dep |\n| --- | --- | --- | --- |\n| **Web UI** | humans, at a browser | `pinchtab-webgraph-ui` serves a loopback-only two-pane SPA + `/api/*` REST over the offline graph. See [docs/ui.md](docs/ui.md). | `pip install 'pinchtab-webgraph[ui]'` |\n\nThe chat pane can reach Claude two ways: the **Anthropic API** (`ANTHROPIC_API_KEY`, the default when a key is set) or your **locally-logged-in Claude Code** with no API key (add the separate `[ui-claude-code]` extra + a logged-in `claude` CLI). Both are locked to the same six offline graph tools. See [Chat backends](docs/ui.md#chat-backends).\n\nChats are **persistent and multiple**: the chat pane's chip bar holds several **named chats per host**, each saved to disk (`<home>/sessions/<host>/<id>.json`) and restored on reconnect — new / switch / rename / delete right from the bar. The `api` backend continues a reopened chat; the `claude_code` backend restores it for display only in v1. See [Chat sessions](docs/ui.md#chat-sessions).\n\nA \"how do I get to X\" chat answer also offers a **\"Show me How\"** button: a guided tour that highlights each step directly on the live browser pane and, on **Next**, performs the real click to advance — stopping at the target form without ever submitting it. See [Show Me How guided tour](docs/ui.md#show-me-how-guided-tour).\n\nThe **Graph view** renders the host's cached interaction graph entirely offline (via `GET /api/hosts/{host}/graph`), with search, an adjacency-highlight on node click, a detail panel, and an \"Ask in chat\" button that prefills the chat with the click-path question. Its Cytoscape libraries are lazy-loaded on first open so the SPA stays light. See [Graph view](docs/ui.md#graph-view).\n\nThe UI can also **crawl a new URL and store it**: a sidebar **\"New crawl\"** form spawns the interaction crawler over a WebSocket, streams live progress, and atomically promotes the resulting graph into the cache so the new host appears in the sidebar and is instantly usable by the Graph view + chat. It is **opt-in** (off unless `PINCHTAB_WEBGRAPH_ENABLE_CRAWL` is set) because a crawl drives a real browser through the whole target app and opens every Create form (it never submits). See [New crawl](docs/ui.md#new-crawl-get-wscrawl-opt-in).\n\nThe **Explore view** is a read-only browser over everything the crawl captured, in three sub-tabs: **Search** (full-text search of captured page data, each hit showing reachable/click-count badges, the click-path, and the matched items), **Forms** (the create-form inventory + a free-text goal path-finder — each form has a **\"Show me how\"** button that reuses the live guided tour), and **Content** (the per-view inventory of captured collections). A **Ctrl/Cmd-K command palette** launches over the whole UI — switch host, jump view, new chat / new crawl, manage credentials, and a free-text \"search content for …\" hand-off into Explore. See [Explore view](docs/ui.md#explore-view).\n\nThe **Flows view** is the UI for the [automation flow layer](#-automation-flows) — a workbench of **three synchronized views of one document**: an **[AI agent](docs/flows.md#authoring-a-flow-with-the-ai-agent)** you describe the automation to (it grounds every step in the real crawled graph, and can only *propose* — you Save and you Run), an editable **visual canvas** (`for_each`/`paginate` visibly wrap their children; every edit form is derived from `GET /api/flows/op_schema`, so there's no second copy of the DSL to drift), and the **JSON**, validated as you type. Plus a run panel where the **safety model is visible** (Allow-submit / Allow-upload are *disabled unless the flow declares that capability*; dry-run is on by default), a streaming run log with a live **`N new · M dupe`** dedupe counter, run history, and the flow's all-time artifact ledger. Each run is a **subprocess**, which is what makes **Cancel** work at all. Authoring always works; **running** is **opt-in** (`PINCHTAB_WEBGRAPH_ENABLE_FLOWS=1`) because — unlike a crawl, which never submits — a flow *can* write to the site. See [Flows view](docs/ui.md#flows-view-opt-in).\n\nOnly the **base install** (`pip install pinchtab-webgraph`, pure stdlib) is needed for the CLI and the UTCP manual; the MCP server and the web UI each live behind an optional extra (`[mcp]` / `[ui]`) so the base package stays dependency-free.\n\n> On externally-managed Python (Debian/Ubuntu, PEP 668) install the extras into a venv, or use `pip install --user --break-system-packages 'pinchtab-webgraph[mcp]'`.\n\n## 🔌 MCP server\n\nAn optional [Model Context Protocol](https://modelcontextprotocol.io) server exposes\nthe same offline queries — plus two live browser-driven tools — to any MCP client\n(Claude Desktop, Claude Code, …). It's a thin binding onto the `api.py` query surface,\nso answers are identical. The base install stays **mcp-free**: the server lives behind\nits own extra and console script, and nothing in the base package imports it.\n\n```bash\npip install 'pinchtab-webgraph[mcp]'   # on Ubuntu/PEP-668: add --user --break-system-packages, or use a venv\n```\n\n```json\n{ \"mcpServers\": { \"pinchtab-webgraph\": { \"command\": \"pinchtab-webgraph-mcp\" } } }\n```\n\n- **Offline tools** (`graph_summary`, `howto`, `find_content`, `list_content`,\n  `list_forms`, `link_paths`) take either `host=` (cache routing) or `graph=` (a path);\n  no browser, no network. `find_content_hosts` / `list_content_hosts` take neither —\n  they search **every** cached host at once and label each result by origin host.\n- **Resources** `graph://hosts`, `graph://{host}/summary`, `graph://{host}` browse the\n  interaction-graph cache.\n- **Live tools** `crawl` (replaces a host's cache) and `ask_howto` (cache-first,\n  merges) need a running PinchTab bridge; offline tools don't. The crawler's\n  restart/login shell hooks are **operator-only** (env/config), never tool parameters.\n\nFull inventory, env vars, and `.mcp.json` example: **[docs/mcp-server.md](docs/mcp-server.md)**.\n\n## 🔌 UTCP interface\n\nPrefer to call the CLI directly, with **no server running**? `pinchtab-webgraph` also\nships a [UTCP](https://www.utcp.io) manual: a description of each tool's JSON-schema\ninputs/outputs plus the exact `pwg …` command to run, with args injected as\n`UTCP_ARG_<name>_UTCP_END`. A UTCP-aware caller runs the command itself — the same\n`api.py` queries as MCP, no wrapper process. Manual generation is **pure stdlib**.\n\n```bash\npwg manual                        # print the manual JSON\npwg manual --out utcp-manual.json # write it (a committed copy lives at repo root)\npwg manual --serve                # serve at /utcp + /.well-known/utcp (default :9872)\n\npwg query howto --host app.example.com --goal \"create role\"   # the substrate, prints JSON\n```\n\nThe exposed surface is a deliberate **subset** — required core args only, `--host`\nrouting only — so every command string is placeholder-free. Full tool table, exit-code\nconvention, and endpoints: **[docs/utcp.md](docs/utcp.md)**.\n\n## 📚 Documentation\n\nDeep-dive guides live in **[`docs/`](docs/README.md)** — start at the **[documentation index](docs/README.md)**, which links everything below:\n\n| Guide | What it covers |\n| --- | --- |\n| **[Automation flows](docs/flows.md)** | The flow document format (every op + its args), **[authoring a flow with the AI agent](docs/flows.md#authoring-a-flow-with-the-ai-agent)** (the three synchronized views, the [`propose_flow` tool + `flow_draft` frame](docs/flows.md#propose_flow-and-the-flow_draft-frame), [why propose-only is *structural*](docs/flows.md#the-agent-can-only-propose--and-that-is-structural), the [resolvability warnings](docs/flows.md#resolvability-warnings)), the [capability / safety model](docs/flows.md#the-capability--safety-model), the [download strategy](docs/flows.md#downloads-in-session-fetch-first-cli-fallback) (in-session fetch first, CLI fallback) and its constraints, the [dedupe ledger](docs/flows.md#the-dedupe-ledger), a [by-hand authoring walkthrough](docs/flows.md#authoring-a-flow--a-walkthrough), [running flows from the web UI](docs/flows.md#running-flows-from-the-web-ui) (storage, caps, REST + WS frames, the subprocess/cancel design, the artifact-scope caveat), and the [gotchas](docs/flows.md#gotchas). |\n| **[`perform` live test](docs/perform-live-test.md)** | The real-browser proof of `crawl → howto → perform`: a local test site, a downloads-enabled bridge, and the two bugs the live run caught. |\n| **[MCP server](docs/mcp-server.md)** | Run `pinchtab-webgraph-mcp`: the `[mcp]` extra, `.mcp.json` registration, the tool + resource inventory, env vars, and the live-tool safety model. |\n| **[UTCP interface](docs/utcp.md)** | The `pwg query` JSON surface + the `pwg manual` / `--serve` UTCP manual, the 8 tools, the scope subset, and the exit-code convention. |\n| **[Web UI](docs/ui.md)** | The optional local web UI (`pinchtab-webgraph-ui`, `[ui]` extra): the Workspace/[Graph](docs/ui.md#graph-view)/[Explore](docs/ui.md#explore-view)/[Flows](docs/ui.md#flows-view-opt-in) view switcher + [command palette](docs/ui.md#command-palette), the REST API + vault endpoints, the chat + screencast WebSockets, [persistent named chats](docs/ui.md#chat-sessions), the opt-in [New crawl](docs/ui.md#new-crawl-get-wscrawl-opt-in) + [flow-run](docs/ui.md#flows-view-opt-in) endpoints, env vars, the loopback-only security model, and the [operational notes](docs/ui.md#operational-notes-developing--e2e-testing-the-ui) for driving the UI itself. |\n| **[Authenticated login](docs/authenticated-login.md)** | Crawl behind a login safely: hand-login vs. keyring automation, the threat model, sandbox/bot-account isolation, and how to test it. |\n| **[Contributing](CONTRIBUTING.md)** | Branch model, Conventional Commits, the stay-generic rule, safety, security, and PRs. |\n\n## 🔎 How interaction crawling works\n\nFor each state the crawler reads every link and clickable widget (stable structural CSS selectors, not framework-generated refs), plus the state's data collections. The clickable set spans `button` / `[role=\"button\"]` / tabs / menu items / `summary` / `[onclick]` **and** upload affordances — `input[type=\"file\"]` (including a file input hidden behind a styled `<label>`/button) and `[ondrop]` dropzones — each carrying its `accept` attribute (accepted file types). Then, for each non-skipped widget, it **re-materializes** the state (replay the click-path from a known start), clicks the widget, and classifies the result:\n\n- **navigated** (URL changed) → a page edge; enqueue the new page.\n- **DOM changed, same URL** → a new SPA/state node + edge, recursed into up to the interaction depth.\n- **create-trigger** → the form/modal is opened, its fields are read (label / type / required / options / accepted file types / confirm button), then Escape — nothing is persisted.\n- **upload affordance** → recorded as a read-only `upload` node (with its `accept` file types) and a skipped action edge, but **never clicked** — clicking a file input opens a native OS file dialog the crawler can't dismiss, so uploads are documented, not activated.\n- **no change** → ignored.\n\n> A dropzone whose drop handler is attached via `addEventListener` (not an inline `ondrop` attribute) and that wraps no file input can't be seen from the DOM; the nested-file-input heuristic covers the common case.\n\nTwo kinds of state become **trigger targets** for the offline `howto` query: a control whose label carries a create-verb (`create` / `add` / `new` / …), **and** a state that *structurally is a form* — it renders real `input`/`select`/`textarea` fields plus a submit control — even when nothing on it carries a create-verb. That second, structural signal (`--capture-form-states`, on by default) is how sign-in / sign-up / contact pages become answerable (e.g. \"how do I sign in\" → `/login` with its email + password form), and it's fully generic — form shape, no app vocabulary. On the query side, a matched trigger whose form has **no fields** is treated as low-confidence and `howto` prefers `no_match` over surfacing it, so a nav link that merely shares a verb (say \"Find a **new** job\" for \"post a job\") is never returned as a confident match.\n\nRe-materializing per probe keeps every click starting from a known state and avoids stale element references across reloads. **State identity is URL-primary in normal navigation** — one normalized URL (with `#fragment` and generic tracking params like `utm_*`/`gclid` stripped, remaining query kept) is exactly one state, so control-count / feed-content jitter between reads can never mint duplicate states for the same page. The converse — one screen under many URLs (sticky query params) — is handled by the [screen fingerprint](#-screen-dedupe-already-crawled-detection): a new URL that renders an already-crawled screen on the same path becomes an alias of it. Single-URL app-shells (e.g. MS Teams), whose URL never changes as views swap in place, instead key on a structural signature that folds in ARIA view markers so same-shell views stay distinct. This mode is **auto-detected at crawl start**: the crawler exercises a few representative nav controls on the live page and watches the URL — a control that changes the URL *path* proves URL-primary routing (nav mode), while one that swaps the visible control set / ARIA view *without* changing the URL proves an app-shell (single-URL mode). The probe is purely structural (control counts + URL + ARIA view — no app vocabulary) and non-destructive; on anything ambiguous it defaults to nav mode. Force it either way with `--single-url` / `--no-single-url` (the escape hatch when detection guesses wrong). (Trade-off: hash-router SPAs that route only via `#/…` are detected as single-URL — which keeps their views distinct — rather than collapsing to one state as they would in plain nav mode.)\n\n## 🏗️ Architecture\n\n```\n   Any website              PinchTab (real browser)          Graph                 Query\n ┌──────────────┐      ┌─────────────────────────┐    ┌────────────────┐    ┌──────────────┐\n │ pages        │ ───► │ read controls + content  │──► │ states         │──► │ howto.py     │\n │ SPA states   │      │ click widgets            │    │ + action edges │    │  (offline    │\n │ forms        │      │ open forms (read-only)    │    │ + form specs   │    │   BFS, ms)   │\n │ tables/grids │      │ scroll virtualized data   │    │ + collections  │    │ ask.py cache │\n └──────────────┘      └─────────────────────────┘    └───────┬────────┘    │ paths.py     │\n   structural signals    isolated bridge, safe             checkpointed      │ Cytoscape UI │\n   only (ARIA, siblings) never submits/saves               (atomic write)    └──────────────┘\n```\n\nEverything runs locally against your own isolated browser bridge. The pipeline is deterministic — no LLM in the indexing or path-finding path — and every crawl flushes atomic checkpoints so a kill never loses work.\n\n## 🧩 Graph shape\n\nThe JSON graph is `{ nodes, edges, meta }`:\n\n- **Nodes** — **pages** (by normalized URL) and **SPA/modal states** (same URL, changed DOM). Cross-host mode adds `external` / `iframe` nodes. File-upload affordances become a distinct `upload` node carrying an `accept` field (the accepted file types); download/export affordances become a distinct `download` node carrying `dlKind` (`direct`/`js`) and, for a direct download, `dlHref` (the file URL). Each node can carry its **control inventory** and its **content collections**.\n- **Edges** — **links** (navigation) and **actions/clicks**. Destructive-looking actions that were deliberately skipped — and upload/download affordances, which are recorded but never clicked — are stored as dashed (skipped) edges so you can see what was avoided.\n- **meta** — crawl parameters plus `meta.stopped`: `frontier-exhausted` (complete) vs. `hit-max-*` / `wedge` (truncated). Truncation is always explicit. Additive `meta.uploads` / `meta.downloads` count the upload / download affordances found. The interaction graph additionally carries `meta.dedupe` (see [Screen dedupe](#-screen-dedupe-already-crawled-detection)).\n- **Viewer** — the Cytoscape HTML viewer renders `upload` nodes distinctly (cyan, \"tag\" shape) and `download` nodes distinctly (violet, \"vee\" shape, a \"download / export\" legend entry) alongside the \"SPA / modal state\" and \"skipped\" legends; the `Uploads` / `Downloads` stats are guarded, so older graphs without them still render. The viewer is **truly self-contained and offline** — its six Cytoscape/layout libraries are vendored inline (no CDN, no network), so it opens and lays out with nothing but a browser. It uses a fast `fcose` layout by default (a big graph of ~2,500 edges lays out in well under a second) with a **High quality** button for an on-demand higher-fidelity relayout, and shows a \"Laying out graph…\" indicator while a layout runs.\n\n## ♻️ Screen dedupe (already-crawled detection)\n\n`interaction_crawl.py` keys nav-mode states by normalized URL. On its own that breaks on apps that keep UI state in **sticky query params**: on 2026-09-22 a LeytonGo crawl found 243 distinct `/settings?…` URLs for ~20 real tabs, spent 194 of its 208 states on that one page, took 2h21m and still stopped at `hit-max-visits(1800)` before reaching the rest of the app. The crawler now also asks \"have I already crawled this *screen*?\":\n\n- **Screen fingerprint** (`screen_fingerprint`) — page path + every in-page control (role, tag, label, href with query pairs merely echoed from the current URL removed) + the active ARIA markers (selected tab, pressed toggle, current item) + visible headings + the first 60 visible data items + visible form fields (name + non-secret value). It is gathered in the same eval as the controls, so it costs no extra round-trip. Two URLs on the **same path** with the same fingerprint are one state: the new URL is recorded as an **alias** (edge, `state_index` entry, `aliases` on the state) and is not expanded, content-captured or form-read again.\n- **What it deliberately ignores** — the query string itself; site **chrome**, learned at runtime as any control seen on 3 or more different paths (sidebar and header entries render late and would otherwise split one screen in two); and the *value* shown by a select-like control (its presence still counts).\n- **Why it can't merge different screens** — anything that differs *inside* the screen changes the fingerprint: a different selected tab, other rows (so `?page=1` and `?page=2` with the same toolbar stay apart), another heading, an extra in-page control, a loaded record's field values. Different paths are never merged. When in doubt it under-merges: a volatile clock inside the data just means no alias, i.e. the old behaviour.\n- **Persistent-chrome prediction** (`--chrome-evidence N`, default 2) — a click-only nav control (same selector + label) that led to the same state from N different parents, with no disagreement ever seen, is chrome: from further parents its edge is recorded (`\"predicted\": true`) without clicking. One disagreement disables it for that control.\n- **Guard** — `meta.dedupe` reports `screenAliases`, `aliasesPerState`, `predictedEdges`, `formReads`, `contentCaptures`, `inertParams` (query params that did not change the screen, per page — diagnostic only, never used to rewrite unvisited URLs, because a param inert on one tab matters on another) and `hotPaths`. If one path holds ≥50% of the states (and ≥40 of them, tunable with `--hot-path-share` / `--hot-path-min`) the crawl prints `⚠ HOT PATH` during the run and in the final summary, and adds it to `meta.dedupe.warnings` — the signature of this failure mode recurring on a site the fingerprint can't collapse.\n- `--no-screen-dedupe` restores the URL-only identity; single-URL app-shell mode is unaffected (its identity is already structural). Tests: `tests/test_screen_dedupe.py` drives the real crawl loop against an in-process sticky-param fixture (`tests/fixtures/sticky_site.py`).\n\n## 🛡️ Safety model\n\n- **Same-origin by default** — the crawler won't wander off the target site unless you pass `--cross-host`.\n- **Never mutates data** — discovery opens and reads forms, then Escapes. Create / save / delete / submit controls are skipped by default and recorded, not clicked. **File-upload and download/export affordances are recorded but never clicked** (a file input opens a native OS picker; a JS download can pop a native save dialog the crawler can't dismiss), so the read-only contract holds. The runnable [command blocks](#️-runnable-command-blocks) inherit this: their path only navigates, downloads/uploads are separate explicit commands, and a form's submit line ships commented out. **Never run a \"click everything\" crawl in an authenticated session you care about** — that's exactly why the isolated bridge exists.\n- **Hard caps** on states, actions-per-state, interaction depth, and a global action budget prevent the classic SPA state explosion.\n- **Secrets stay out of git** — `crawl-config.json` (bridge token) and `.instance/` (live browser profile/session) are gitignored. Commit explicit source files only.\n- **OS-level sandbox (opt-in)** — run under [Claude Code's built-in sandbox](#-run-it-under-a-sandbox-recommended) to confine the crawler at the OS level: no keyring/SSH/cloud-cred reads, localhost-only egress, and no `sudo` escape. A ready posture ships in [`.claude/settings.json`](.claude/settings.json).\n\n## 🔐 Authenticated apps (login)\n\nCrawling a site that needs a login is **built in and opt-in** — two ways to authenticate,\nthen a sandbox to run the whole thing safely. Full reference, threat model, config, and\ntest steps: **[docs/authenticated-login.md](docs/authenticated-login.md)**.\n\n### Two ways to log in (safest first)\n\n1. **Log in by hand once (recommended, zero config).** Open the persistent bridge\n   profile, sign in, and crawl — the session cookie lives in `.instance/` (gitignored)\n   and the crawler reuses it. **Your password never touches this toolkit.** This is the\n   safest path and needs nothing below.\n\n2. **Automated login (opt-in), for unattended / long-running crawls.** When the bridge\n   may restart mid-crawl or you're running on a schedule, enable keyring-backed login:\n\n   ```bash\n   pip install 'pinchtab-webgraph[login]'          # optional dependency, only for this\n   cp login-config.example.json login-config.json  # gitignored — ROUTING only, no password\n   keyring set pinchtab-webgraph you@example.com    # the password lives in the OS keyring\n   interaction_crawl --start https://app.example.com/home --login-config login-config.json\n   ```\n\n   The password is read from the **OS keyring at runtime** — never from a file, the graph\n   JSON, or logs (only its length is ever printed). `login-config.json` holds per-host\n   routing (`url`, `username`, optional field selectors); login form fields are\n   auto-detected from standard HTML (input `type` / `autocomplete` / DOM order), so most\n   apps need no selectors. The same login is reused to re-authenticate after a bridge wedge.\n   See **[docs/authenticated-login.md](docs/authenticated-login.md)** for the full config\n   reference, security properties, limits (SSO/2FA are not automated), and test steps.\n\n### 🧱 Run it under a sandbox (recommended)\n\nOn its own, keyring is only **at-rest** hygiene: any process running as your user — an AI\nagent included, and certainly one with `sudo` — can read a keyring secret with one command.\nThe fix that actually confines the automation, **without a VM or container**, is to run the\ncrawler inside **[Claude Code's](https://claude.com/claude-code) built-in sandbox** (this\nfeature *requires* Claude Code). It uses OS primitives (`bubblewrap` on Linux, Seatbelt on\nmacOS) to enforce, at the OS level, what commands can read and which hosts they can reach —\nand a sandboxed process runs in an unprivileged user namespace, so it **can't `sudo` out**.\n\nThis repo ships a locked-down posture in [`.claude/settings.json`](.claude/settings.json):\nsandbox on, network egress limited to **localhost**, and reads **denied** for the OS keyring,\n`~/.ssh`, cloud creds, and GitHub/npm tokens — so a compromised crawl can't read your secrets\nor phone them home.\n\n```bash\n# Linux/WSL2 deps (macOS needs nothing extra):\nsudo apt-get install bubblewrap socat\nnpm install -g @anthropic-ai/sandbox-runtime   # optional: seccomp unix-socket blocking\n```\n\nThen run `/sandbox` in Claude Code and crawl as usual. Add your target app's domain to\n`.claude/settings.local.json` (gitignored) or approve it on the first prompt.\n\n**Recommended safe combo:** the shipped default **denies the keyring**, so pair the sandbox\nwith **hand-login / session-reuse** (option 1) — the agent drives the authenticated session\nbut can't read your credentials at all. If you want automated keyring login, remove the\nkeyring deny locally and use a **dedicated bot account**.\n\n> ⚠️ A strong risk-reducer, not a perfect wall: the proxy allow-lists by hostname without\n> TLS inspection, localhost egress stays open (the crawler needs the local bridge), and a\n> denylist is never exhaustive. Keep the allow-list tight and keep a **bot account** as your\n> backstop. Full model: [threat model + sandbox setup](docs/authenticated-login.md#threat-model--read-this-before-trusting-keyring).\n\n## 🗄️ Importing into Neo4j (optional)\n\nThe JSON maps directly to a property graph:\n\n```cypher\n// after: WITH the json loaded as $g\nUNWIND $g.nodes AS n\n  MERGE (p:Page {id:n.id}) SET p.url = n.url, p.title = n.title, p.type = n.type;\nUNWIND $g.edges AS e\n  MATCH (a:Page {id:e.source}), (b:Page {id:e.target})\n  MERGE (a)-[r:NAV {label:e.label, kind:e.kind}]->(b);\n```\n\n## 🛣️ Roadmap\n\n- ✅ ~~Auto-detect single-URL app-shell mode (no `--single-url` flag).~~ Shipped — the crawler now detects app-shell SPAs structurally at crawl start; `--single-url` / `--no-single-url` force the mode either way.\n- ✅ ~~Form-reading inside single-URL apps.~~ Shipped — create-forms are opened in place via JS-dispatch clicks with a shell-blank guard; a trigger that can't open without navigating is recorded form-less and the crawl recovers in place.\n- ✅ ~~Sub-10s cold-start live discovery for cache misses.~~ Shipped — one bridge round-trip per state to settle (not a poll loop); measured 7.5s cold end-to-end, regression-guarded offline by `scripts/bench-discovery.sh`.\n- ✅ ~~Richer content queries surfaced through `ask.py` (cross-host collections).~~ Shipped — `ask.py --find/--list-content --all-hosts` searches every cached host offline, ranked and labeled by origin (also `pwg query find_content_hosts` / `list_content_hosts` and the matching MCP tools).\n\n## 🤝 Contributing\n\nPRs welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)** for the full guide (branch model, commit conventions, PR checklist, issue reporting). The short version:\n\n- Cut feature branches from **`dev`** and open PRs against it. `main`, `release`, and `hotfix` are protected — every PR into them needs a Code Owner review (see [`CODEOWNERS`](CODEOWNERS)); force-pushes and deletions are blocked.\n- **The one hard rule: stay generic** — no hardcoded app routes, labels, or vocabulary in the crawler; structural heuristics only.\n- Discovery stays **safe** (never submits) and **secrets stay out of git**. Please open an issue before a large refactor.\n\n**Development.** The runtime is pure stdlib; tests use `pytest`, added via the `test` extra:\n\n```bash\npip install -e '.[test]'   # editable install + pytest\npytest                     # runs the suite in tests/\n```\n\n## ⭐ Star History\n\n<a href=\"https://star-history.com/#egouilliard/pinchtab-webgraph&Date\">\n <picture>\n   <source media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=egouilliard/pinchtab-webgraph&type=Date&theme=dark\" />\n   <source media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=egouilliard/pinchtab-webgraph&type=Date\" />\n   <img alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=egouilliard/pinchtab-webgraph&type=Date\" />\n </picture>\n</a>\n\n## 📄 License\n\nLicensed under the [MIT License](LICENSE). Copyright © 2026 Edouard Gouilliard.\n"}