mcp server

hyperbolic-gateway

Kuang Grade Mark Eleven: a self-hosted MCP browser agent - a real Chromium your LLM can drive. Local-first, no cloud, no API keys, no telemetry. A DIY alternative to browser-use, Playwright agents and

hyperbolictonic/hyperbolic-gatewaycommunitylistedunreviewed

Use tool

Set up

Runs on your computer. Pick your agent and operating system.

This project does not publish an install method we can turn into a config. Follow the README instead.

README

synced Oct 28, 2026

⚡ Hypagon Gateway

The local browser for AI agents.

"I'll hack the Gibson"
— Dade Murphy (AKA Crash Override), Circa 1995

🧭 Index

About — What it is · A punk-ethos alternative to browser clouds
The deck — Theme & Cyberdeck UI · Ghost cursor telemetry · Focus-independent input · Multi-tab browsing
Reading the web — Orthogonal primitive layer · Construct harvester · JEV page compaction · Set-of-Marks
Driving the web — Shield-breaker · Form filler · Human-in-the-loop
Memory & data — Cortex · Downloads · Media download integration · Session vault · Script vault · Timelapse recorder
Inspection — Network & API streams · WebSocket & SSE
Operating it — System tray · MCP tools (16 advertised · +2 decision · +3 JEV) · Agent skills · REST wrappers
Building on it — Architecture lowdown · Codebase & docs · Quickstart · Testing · OKF knowledge
Optional decision layer — decide / drive design notes · Control layer: targeting, consent, human takeover · Shadow DOM & iframes: what Hypagon Gateway can see and click · Shadow Walk: sealed-content plan · Field notes from dogfooding · Changelog
Release notes — What shipped · All releases
Lore — Neuromancer inspiration

🕶️ What it is

Hypagon Gateway is a visible, stateful web browser that any MCP agent can drive.

Most agent browsers are a library you integrate, or a cloud you rent. Hypagon Gateway is a desktop application you own: a real Chromium window on your machine, controlled over the Model Context Protocol. Your agent runs where you already run it — local or remote — and Hypagon Gateway supplies the browser, the input, and the memory.

A real browser, not a simulation. Visible Chromium with hardware acceleration, persistent cookies, real tabs, downloads, and a human-sized viewport. No headless fingerprint.
Structured first, pixels when needed. observe returns a compact, addressable view of the page (markdown, text, accessibility tree, or Set-of-Marks). take_screenshot is there when layout matters.
Input that looks human. Clicks and keys travel over the Chrome DevTools Protocol with Bézier paths and Gaussian typing cadence — and they land even when another application owns the foreground.
State that survives. LanceDB memory, bookmark and session vaults, download harvesting and a script vault mean the next session starts where the last one stopped.
Self-hosted by construction. stdio MCP plus a localhost bridge. No account, no dashboard, and no telemetry leaving the machine.

Recorded from the running app: Google → “hypagon definition” → AI Mode. Open the MP4 if your browser does not animate WebP.

[!WARNING] EXPERIMENTAL PROTOTYPE: Hypagon Gateway is currently under active, experimental development. Tool contracts, telemetry tuning, and database schemas may evolve rapidly. Built for developers experimenting with multimodal local AI agents.

[!IMPORTANT] RESEARCH PURPOSES ONLY: This software is provided strictly for educational, testing, and local AI agent research purposes. It is not intended for malicious use, automated credential stuffing, unauthorized exploitation, or violating third-party websites' Terms of Service. Users are responsible for complying with all applicable laws and terms.

Installers: grab them from Releases — Windows Setup/Portable/zip, macOS dmg/zip (x64 + arm64), Linux AppImage/deb/tar.gz. macOS is unsigned for now: right-click → Open (or xattr -dr com.apple.quarantine "/Applications/Hypagon Gateway.app"). Built by GitHub Actions; see docs/BUILDING.md.

🤘 A Punk-Ethos, DIY Alternative to browser-use, Playwright Agents & Hosted Browser Clouds

Most "AI browser" tooling today is a library you integrate, or a cloud you rent. browser-use is a popular MIT-licensed Python framework that drives Playwright with its own agent loop (plus an optional hosted cloud); Playwright / Puppeteer / Selenium are general-purpose drivers where you write the loop; and managed browser clouds run a remote Chromium and bill per session. They're good tools — this isn't a rivalry, it's a different shape.

Hypagon Gateway is the other road: a single, self-hosted desktop app you own (Electron, no account, no dashboard). It runs a real, visible Chromium on your own machine and speaks MCP over stdio (+ a localhost bridge), so it drops into whatever agent runner and whichever model you already use — local or remote, with no cloud account, no API key, no seat pricing, and no telemetry leaving your box.

That's the punk ethos: one operator, one machine, a tool you can read, fork and keep. Built by one person, not a company. No VC, no roadmap-by-committee, no dashboard.

How it differs, honestly

	`browser-use` (Python / Playwright)	Playwright · Puppeteer agents	Hosted browser clouds	Hypagon Gateway
Shape	Library with its own agent loop	Driver — you write the loop	Managed remote browser	MCP server any MCP client can call
Browser	Playwright Chromium (often headless)	Playwright / Puppeteer	Remote Chromium	Visible Chromium (`headless: false`) in an Electron chassis
Where it runs	Your machine, or their cloud	Your machine / your CI	Their datacentre	Your machine, always — local-first
Model	Bring your own (usually a cloud LLM)	Bring your own	Bring your own	Bring your own — local or remote
Input	Playwright-driven events	Playwright-driven events	CDP over the wire	Focus-independent CDP input + Bézier ghost cursor + Gaussian typing (`isTrusted: true`)
Memory	Framework-level, BYO store	none	Their store	Local LanceDB "Cortex" — semantic recall of everything it harvests
Cost	MIT + your LLM (+ optional cloud)	Free + your LLM	Per-session billing	MIT. No cloud, no metering, no telemetry.

What that buys you in practice

Self-hosted and local-first. The browser, the memory, the logs and the model all stay on the box.
Built for a machine you're also using. Input goes over CDP and is focus-independent — clicks and typing land while your editor owns the foreground — and occluded windows keep hydrating and scrolling, so long harvests keep running behind your other apps.
Tier-graded tool surface, not a token bonfire. Reading is observe(as:"markdown") with inline [#id] marks (no screenshot tax); interaction is one act; [TIER 1/2/3] prefixes plus a tools/recommended resource tell the agent what to prefer.
Honest about its limits. Detached reads are gated, evaluate_javascript is "use deliberately" rather than banned, and failures are machine-coded (STALE_REF, TIMEOUT, NO_TARGET, BAD_REQUEST, NOT_SUPPORTED) instead of vibes.
No framework lock-in. Because the interface is MCP, your runner stays yours: swap the agent, swap the model, keep the tool.

New here? Plain-English FAQ

What is this, in one sentence? A web browser that an AI can drive for you: it opens real pages, reads them, clicks, types, fills in forms, and remembers what it found.

Is it a browser I use myself? It's a real browser window. You can watch it, and you can take over at any time — most people just let their AI do the driving.

Do I need to be a programmer? To use it with an AI assistant you already have: no — you paste one line of configuration into that assistant. To change how it behaves internally: yes, it's a code project.

What's an "MCP server", in plain terms? MCP is a standard plug shape for AI tools. If your AI app accepts MCP plugins, it can use this one. Claude Desktop, Cursor and several other apps speak it.

Can I install an agent skill too? Yes. The hb-browser and hb-wrap skills are at skills/.

hermes skills install https://raw.githubusercontent.com/hyperbolictonic/hypagon/main/skills/hb-browser/SKILL.md
hermes skills install https://raw.githubusercontent.com/hyperbolictonic/hypagon/main/skills/hb-wrap/SKILL.md

npx skills add hyperbolictonic/hypagon --skill hb-browser
npx skills add hyperbolictonic/hypagon --skill hb-wrap

Add -g -y to the npx commands for a non-interactive global install.

Which AI can drive it? Any assistant that supports MCP — a cloud service, or a model running on your own computer. You choose, and you aren't locked to one provider.

Do I need to pay for anything? No subscription and no account. The software is free and open source. The only cost is whatever you already pay for your AI, if anything.

Where does my data go? Nowhere. The pages it reads, the notes it keeps and the logs all stay on your computer. There's no company server in the middle and no usage tracking.

Could a website tell it's a bot? It's built to be hard to spot: a normal visible browser, mouse movement and typing that look human, and it clears cookie banners politely. Nothing is unbeatable, but it isn't the obvious "automated script" that gets blocked.

What happens at a CAPTCHA? It doesn't solve them. It asks you to step in (it can send a notification and bring the window forward), then continues once you're done.

Does it run headless on a server? Not really, and that's on purpose. It runs a visible browser on a normal desktop, which is exactly why it looks human. For large-scale server-side scraping, a different tool fits better.

Is using it allowed? It's for personal research and testing. You still have to follow each website's rules and the law — running it yourself doesn't change that. Don't use it to break into accounts or ignore terms of service.

What do I need to run it? A normal Windows desktop (that's the tested target). If your computer can run a browser and an AI assistant, it can run this.

Is it finished? No. It's an experimental project by one developer — expect rough edges and things that change.

FAQ for developers — browsers, memory, and the tricky bits

Is there a self-hosted, open-source alternative to browser-use? Yes — MIT-licensed and entirely local. But note the shape: this is a tool surface, not an agent framework. If you want a Python framework that owns the loop, browser-use is the right tool; if you want a self-hosted browser that your own agent drives, this is it.

Can I use it with Claude Desktop, Cursor, or a local LLM? Yes. It's an MCP server, so any MCP client works - point your client at bin/mcp-server.cjs and the hb-browser tool namespace appears (legacy kuang alias still works).

Does it need a cloud account, API key, or per-session billing? No. There is no cloud tier. The only network calls are the ones your agent makes to the pages it visits (plus, optionally, a search engine via search_web).

Is it headless? Deliberately the opposite. It runs a visible Chromium so a human can watch, take over at a CAPTCHA, and so the session stays an honest, real browser. Headless-at-scale scraping is a different job; this is a single operator's console.

Will Cloudflare block it like Puppeteer / Playwright? It's built for exactly that fight: real CDP input with isTrusted: true, cubic Bézier pointer kinematics, Gaussian typing cadence, a genuine cookie jar, and an autonomous consent-modal breaker. No promises — anti-bot is an arms race — but it is not naive headless emulation.

Can I point it at my existing Playwright test suite? No. This isn't a test driver; Playwright remains excellent for deterministic suites. Grade Eleven is an agent-facing browser an LLM operates.

Where does memory live? Locally, in LanceDB ("Cortex"): harvested pages and vaulted scripts are embedded and semantically recallable, so your agent keeps research context across sessions without a hosted vector store.

Is it production-ready? No — it's an experimental prototype for local AI research. That's rather the point.

Punk ethos, not carte blanche. Owning your tooling doesn't change the law: the RESEARCH PURPOSES ONLY terms at the top of this README still apply. The point is ownership and readability, not immunity.

🎯 Core Architecture Principles

100% Real Browser Chassis: No headless emulation, synthetic user-agent spoofing, or sandboxed scraping. Hypagon Browser runs a full-fledged, visible Chromium instance (headless: false) with hardware acceleration, audio/video capabilities, and authentic cookie/session persistence.
Multimodal Vision Integration: Specifically tailored for multimodal models (e.g. Bonsai 2 27B). The agent takes high-res viewport screenshots (take_screenshot) to visually understand layouts, buttons, and dynamic canvas states, and can act directly by coordinate, selector, or mark badge (act).
Full Developer Console & Script Execution: Gives the agent full console access (get_console_logs) to inspect runtime errors and API output, plus arbitrary JavaScript evaluation (evaluate_javascript) to interact with complex SPAs.
Behavioral Bot Detection Bypass: Bypasses Cloudflare, Akamai, Datadome, and behavioral bot sensors by avoiding synthetic element.click() events (isTrusted: false). Instead, it dispatches real input events over the Chrome DevTools Protocol (Input.dispatchMouseEvent / Input.dispatchKeyEvent / mouseWheel) accompanied by organic cubic Bézier pointer movement and human Gaussian typing cadence. CDP delivery is focus-independent — it lands even when another application owns the OS foreground — with webContents.sendInputEvent kept only as a fallback.

🎨 Theme & Cyberdeck Interface

Built with Tailwind CSS v4 and DaisyUI 5:

Custom Theme: mark-eleven
Color Inspiration: Antigravity electric cyan (oklch(75% 0.18 215)), VS Code workspace slate (oklch(21% 0.012 250)), and Photoshop deep charcoal neutrals.
DaisyUI Components:
- navbar & mockup-browser address bar with real-time navigation and SSL/protocol status.
- status and badge indicators reflecting live agent lifecycle (IDLE, TRAVERSING, SIMULATING_INPUT, HARVESTING).
- tabs (tabs-border) for switching between Telemetry Kinematics, Construct Terminal, and MCP Matrix Logs.
- stats / stat blocks tracking coordinates, Bézier velocity, tokens saved, and interaction counters.
- mockup-code for viewing synthesized Markdown memory constructs with token downsampling ratios.

🕹️ Ghost Cursor Telemetry Engine

Organic Bézier Trajectories: Natural cubic Bézier curve calculation with quintic ease-in/ease-out acceleration, micro-jitter, and target deceleration.
Focus-Independent CDP Delivery: Input events are dispatched over the tab's already-attached Chrome DevTools Protocol debugger (Input.dispatchMouseEvent / Input.dispatchKeyEvent / Input.mouseWheel), with Emulation.setFocusEmulationEnabled so focus-gated page JS still responds. Clicks, keystrokes, and momentum wheel scrolling land regardless of which window owns the OS foreground. webContents.sendInputEvent remains only as a fallback (and only that path needs the window focused). The active transport is reported by bridge_health → input.transport (cdp | native).
Anti-Bot Defense Bypass: Avoids heuristic bot triggers by behaving like an authentic human operator; every interaction is isTrusted: true because it is genuine browser input, not a synthetic DOM .click().

🎛️ Focus-Independent Input & Background Rendering

Two real-desktop failure modes are handled explicitly so long harvests and HIL flows stay reliable:

Input is not tied to OS focus. Synthetic input is delivered over CDP and therefore reaches the renderer even while you work in another application. If the debugger is unavailable the simulator falls back to webContents.sendInputEvent and brings the window forward first (that path is the only one that steals the foreground). bridge_health exposes input.transport so an agent can tell which path is live.
Occluded windows keep rendering. Browsing tabs run with backgroundThrottling: false, and the app boots with --disable-backgrounding-occluded-windows, --disable-renderer-backgrounding, and --disable-background-timer-throttling. Without these, Chromium reports the page hidden, throttles timers to ≥1s, and stops producing frames — so scroll events never fire and IntersectionObserver/rAF hydration stalls (the classic "lazy list stops growing" bug).
Viewport gating is explicit. Real input needs a non-zero viewport; a fresh or hidden tab can briefly report 0x0. Call set_window_visibility({ visible: true }) and wait for a non-zero viewport before acting. Showing the window is otherwise only needed to make it visible to a human (HIL).

🧩 Orthogonal Primitive Layer

The older tool list is being consolidated onto a few orthogonal verbs - one observation model, one action model, one wait model - with the flat tools becoming projections. All primitives run on the TIER-1 humanlike path (real input events; no page DOM mutation beyond Hypagon Browser's own data-mark-id stamp).

Primitive	Job	Replaces
`observe(as, region, budget, cursor)`	One budgeted, cursor-paged observation model (`markdown`/`text`/`ax`/`som`/`list`/`table`/`app_data`)	`get_page_markdown` + `snapshot_rendered_page` + `list_marks`
`act(mark_id \| selector \| x+y, action, text, secret, await_conditions, dry_run)`	Unified humanlike click/focus/hover/type/press on a mark ref, CSS selector, or coordinate + optional post-conditions + delta report	`click_mark`/`type_mark`/`click_element`/`type_text` plus a following wait
`wait_for(conditions, require, timeout_ms)`	Declarative readiness gate — the primitive that replaces blind sleeps	every `sleep`/"did it load?" guess
`expect(conditions)`	Immediate assertion (no polling) with per-condition evidence	agent-side guessing
`find(role, name, contains, regex, tag, selector)`	Semantic element search → refs (mark ids)	hand-written CSS/XPath
`extract(kind, ref, selector)`	Structured reads: repeating `list` rows, `table` records, embedded `app_data`	regex over HTML
`collect_all(selector, scroll, max_scrolls)`	Scroll a lazy/virtualized list to the end and dedupe rows in one call	a manual scroll-and-harvest sweep
`page_state()` / `diff(since)`	Cheap monotonic change signature + field-by-field diff for loops	re-reading the whole page
`bridge_health()`	Liveness, `capabilities[]`, `revision`, `window`, `input.transport`	assuming the bridge is up
`export_har(pattern, save)` · `context_*`	HAR 1.2 export · named cookie/storage profiles	ad-hoc replay · manual session juggling

Refs are marks. A ref is a Set-of-Marks badge id — find returns rows carrying the mark, and act(mark_id=…) consumes it (re-resolving the live centre each time). When there is no badge, the same act also takes selector="…" or x=/y= — one action tool, three address modes.

Wait vocabulary: selector · gone · text/regex · url · role_name · count(min/equals/max) · network_idle · stable(marks|height|text|nodes) · revision.

Structured errors: new endpoints return a machine code (HTTP 400) — STALE_REF (re-scan and re-issue), TIMEOUT, NO_TARGET, BAD_REQUEST, NOT_SUPPORTED.

Full recipes (R1–R5), the wait table, and the error catalogue live in the OKF partition hypagon_docs/partitions/primitives_and_workflows.md, also readable as the MCP resource mark-eleven://okf/partitions/primitives-and-workflows.

📄 Token-Efficient Construct Harvester

Pure Node.js: Powered by Mozilla Readability and Turndown. No external Python or platform-specific binaries required.
Composed-DOM capture: Serialisation walks through open shadow roots and bakes live form state (value, checked, selected) into attributes, so web-component forms — signup and login pages especially — are actual content instead of empty custom-element tags. Password values are recorded as a length, never a value.
Semantic Flattening: Strips ads, navigation sidebars, and script clutter. Links and images resolve to absolute URLs.
Honest, stable envelope. Front-matter carries only what changes the reading of the body — no timestamp (so two harvests of an unchanged page are byte-identical and can be diffed or paged), no byte-ratio vanity metric:
```
---
url: "https://example.com"
title: "Example Domain"
reader: "readability"
extracted_chars: 2140
page_chars: 5410
shadow_roots: 2
---
```
reader names the extraction path; extracted_chars and page_chars are the same unit (characters of page text), so coverage is comparable. A partly-visible page says so, and names the alternates, instead of looking short.
Pageable by default: observe(as:"markdown") returns a 25,000-character view; budget + next_cursor page through the rest, and budget: 0 returns the whole construct. On very large pages, JEV adds a section index on top.

🗜️ JEV Page Compaction

Just-Enough-View (JEV) keeps a large page out of the agent's context window. Instead of a full Markdown dump, the agent gets a compact section index — headings, token weights and one-line digests — and fetches only the sections it needs from an in-memory depth store.

Where: Settings → Decision Model → JEV page compaction. Optional, not experimental; a master switch plus a read mode. Both are disabled until a decision model is configured. Persisted to jev.json over IPC.
prefer (default): jev_index, read_section and search_page lead the tool surface; full-page reads still work.
enforce: unscoped observe(as:"markdown"|"text") and GET /markdown return JEV_REQUIRED (HTTP 409). Region-scoped reads, budgeted reads (budget > 0 or cursor), full: true and pages under 1,500 tokens still work. Every other projection — som, components, ax, list, table, app_data, find/extract/act, screenshots — is untouched.
A funnel, not a trap: a cheap precheck refuses an obviously oversized read before the harvest (~15 ms). After two refusals for the same URL in 60 seconds, the next read is served with a note rather than blocking the turn. Harvest errors and unhelpful indexes fail open.
Measured: a 14,823-token Wikipedia article becomes a ~396-token index; index plus one section is ~2,045 tokens (86–97% saved).
Free when off: the three tools are not advertised at all, so their schemas (~1k tokens per request) are not billed.

Tools: jev_index, read_section, search_page. Full contract and limits: docs/JEV.md.

🧠 LanceDB Persistent Cortex & Semantic Recall

Hypagon Gateway features an embedded, serverless vector database (LanceDB) combined with Tantivy-powered Full-Text Search (BM25) stored locally under ./data/cortex/.

Auto-Harvest on Navigation: Every web page visited is automatically chunked, embedded, and indexed into the page_history table in the background.
Hybrid Semantic Recall: Models can invoke recall_memory to execute hybrid vector + BM25 search across past research sessions, retrieving verbatim snippets, page titles, and relevance match scores.
Persistent Bookmarks: Save critical research nodes, documentation hubs, or matrix targets with semantic tags and research notes.
DaisyUI 5 Cortex HUD: Open the Cortex modal (🧠 CORTEX) from the navbar to browse recent history, execute live semantic queries, or manage bookmarked cards.

📑 Hardware-Accelerated Multi-Tab Browsing

Electron WebContentsView Isolation: Independent navigation state, separate history stacks, and zero-flicker view swapping.
Automatic Popup Trapping: Links configured with target="_blank" or window.open are cleanly trapped into managed child tabs rather than escaping as detached OS windows.
Dynamic DaisyUI 5 Tab Bar Strip: Real-time loading spinners, active indicators, title truncations, and memory reclamation upon tab closure.

🛡️ Autonomous Shield-Breaker (Cookie & GDPR Destroyer)

Zero-Token Modal Clearance: Saves 1–2 LLM tool calls on almost every new domain.
Preload Framework Detection: Detects OneTrust, Cookiebot, Google GDPR (#L2AGLb), TrustArc, and generic consent frameworks.
Bézier Dismissal: Dispatches authentic Bézier cursor movement and hardware clicks to "Accept All" or "Agree" before notifying the agent that the page is ready.
Manual Directive: Agents can also manually call dismiss_consent whenever unexpected overlays block the viewport.

🎯 Set-of-Marks (SoM) Visual Grounding

Zero Coordinate Hallucination: Completely eliminates hallucinated pixel coordinates or fragile CSS selectors.
Shadow-DOM aware: Badges are collected through open shadow roots, so controls inside web components (login/signup forms, design-system dialogs) get marks like any light-DOM element — occlusion is tested with the deepest hit and composed containment, not elementFromPoint alone. Only mode:"closed" roots stay opaque.
High-Contrast Neon Badges: Ephemerally injects high-contrast amber neon numbered badges ([1], [2], [3], etc.) over every visible, interactable DOM element (<a>, <button>, <input>, select, [role="button"]).
Precision Multimodal Actuation:
1. take_screenshot({ marks: true }) captures the badge overlay and returns an element lookup table ({ id: 1, tag: "BUTTON", text: "Submit", bbox: [...] }).
2. act(1) moves the Bézier ghost cursor to the element's centroid with sub-pixel accuracy and clicks.
3. act(2, action: "type", text: "cyberdeck") focuses the input and types text with human cadence.

One tool, three cost tiers via take_screenshot({ marks: true, detail: … }) (pick the cheapest that fits your model):

`detail`	Payload	Use when
`"full"` (default)	Marks table + full-resolution PNG	Vision model, fine visual detail matters
`"economy"`	Marks table + 640px-wide JPEG (q45), roughly 10-20x smaller	Vision model, tokens/bandwidth matter
`"markdown"`	Marks table only, no image	Text-only models (e.g. local GGUFs) or the cheapest page scan

All three return the same mark IDs, so act(mark_id, …) works after any of them. Text-only models must not be sent images: a base64 PNG inlined into the prompt can be 100k+ tokens and will stall local inference servers.

📝 Single-Turn Batch Form Filler

Instantaneous Multi-Input Population: Fills complex checkout forms, flight booking screens, or multi-field search bars in a single turn without round-trip LLM latency.
Human Typing Cadence: Simulates variable human inter-key delays (60–240ms) with Gaussian jitter between fields.
Flexible Field Types: Supports text inputs, password fields, dropdown <select> menus, checkboxes, and radio buttons with optional automatic form submission (submit_selector).

📥 Download Manager & Cortex Harvester

Workspace-Local Routing: Intercepts Chromium's will-download session events, routing all files into ./data/downloads/ rather than the OS user folder.
Automated Text Extraction: Automatically extracts clean Markdown text from downloaded PDFs (pdf-parse), CSVs, and JSON files.
LanceDB Vector Ingestion: Automatically ingests downloaded document text into the LanceDB Cortex vector store for instant conversational recall.
MCP Inspection: list_downloads and read_download(filename) allow agents to inspect and read downloaded files on demand.

📥 Media download integration

Preview before committing: media preview returns the title, duration, a transcript excerpt and the poster image without downloading the media. media transcript returns the full captions as text, paged with max_chars/cursor, and indexes them into Cortex so memory recall can answer from the video later.
Queue, don't block: media download starts a job and returns an id; media jobs reports percent, bytes, speed, ETA and output files; media cancel kills the process tree and removes the partial output.
Managed binary: the downloader is pinned in the build, verified with SHA-256, and updates itself on launch and on an interval (default 60 minutes, configurable). It runs on Electron's own Node runtime, so no extra runtime is installed.
Signed-in downloads are opt-in: off by default. A site that needs a session returns LOGIN_REQUIRED and raises a desktop notification; a human enables Settings → Media → "Use browser session" and retries. Cookies are scoped to the requested host and deleted after the run.
ffmpeg when you want it: auto uses a configured binary, a managed copy or PATH; path and managed are explicit. Managed installs are checksum-verified and not shipped in the installer.
Save folder: a picker in Settings → Media; empty keeps the app's media folder.

Read docs/MEDIA.md for the design notes and threat model.

🎥 Timelapse Recorder

Records the deck itself: the REC button (next to SETTINGS) captures the Cyberdeck window as a timelapse — frames every intervalMs, played back at a chosen fps. One minute of work becomes two seconds of video at the 1 fps / 30 fps defaults.
True re-timing, not a sped-up screen recording: the renderer uses WebCodecs VideoEncoder with explicit frame timestamps and muxes VP9 into WebM with webm-muxer (MediaRecorder cannot retime frames).
Settings → Recording: enable/disable, output directory with BROWSE…, capture interval, playback fps, max width and bitrate; finished files are listed with COPY PATH.
Agent access is a separate, human-only switch: "Let agents use the recording tools" removes record_start, record_stop and list_recordings from tools/list and makes the bridge refuse them with RECORDING_TOOLS_DISABLED. Only the Settings panel can flip it — no tool or HTTP route writes that flag.
Programmatic hand-off: record_start → record_stop returns the absolute .webm path (plus size, frames and speed-up), and list_recordings lists recent files newest-first. That path is a plain string you can type into a file-upload field; if the field only opens a native picker, hand it to a human.
Why a path matters: upload forms usually cannot be driven by setting a value - the browser requires a real file selection. Hypagon Gateway gives the agent (or the human) the exact file to choose instead of guessing.

🔐 Session & Cookie Jar Vault

Stateful Auth Persistence: Captures and restores active session cookies, localStorage, and sessionStorage into encrypted/structured snapshots under ./data/sessions/<name>.json.
Bypass 2FA & Logins: Resume authenticated states on GitHub, airline portals, news sites, or internal dashboards across agent sessions without triggering repetitive logins or CAPTCHAs.
MCP Tools: context_save(name), context_open(name), context_list(), and context_delete(name).
Legacy aliases: save_session(name) / restore_session(name) / list_sessions() still work (same vault) but are hidden from tools/list.

📡 Live Network & API Stream Interceptor

Direct Chrome DevTools Protocol Wire: Attaches directly via CDP (Network.enable) to monitor all incoming and outgoing network traffic.
Direct JSON Extraction: Bypasses DOM scraping entirely for modern Single-Page Applications (SPAs like Twitter, Reddit, Skyscanner). Captures completed XHR/Fetch response bodies (Network.getResponseBody).
MCP Inspection: get_network_traffic filters requests by URL pattern or method, and query_json_data({ source: "network_payload", source_id }) returns parsed JSON (sliced by path/view) directly into the LLM context.
Action-Correlated Capture: get_network_traffic({ since: 'last_action' }) returns only the requests fired by the most recent mutating action (also accepts an epoch-ms timestamp or a page revision). export_har writes a HAR 1.2 log of captured traffic for offline replay.

⚡ Real-Time WebSocket & SSE Interceptor

Duplex Stream Surveillance: Captures duplex frames across persistent wss:// WebSockets and Server-Sent Events (text/event-stream).
Auto-Parsed Payloads: Automatically detects and parses JSON streaming frames from live crypto tickers, chat platforms, financial feeds, and streaming LLM APIs.
In-Memory Ring Buffer: Rolling buffer of 500 frames with filtering by stream ID, direction (sent / received), and message limit.
MCP Tools: list_realtime_streams and clear_capture({ scope: "streams" }); inspect frames with query_json_data({ source: "stream_messages", source_id }).

💾 Script Vault & LanceDB Semantic Macro Engine

Reusable Automation Macros: Store, recall, and execute custom JavaScript DOM extraction and automation scripts under ./data/scripts/<name>.json.
Vector-Indexed Semantic Recall: Every vaulted script is automatically embedded into the LanceDB scripts table. Models can recall scripts using conversational natural language queries (list_scripts({ query: "extract table rows as json" })).
Parameter Injection: run_script(name, args) executes vaulted scripts inside the active tab with dynamic runtime arguments and closure isolation.

🗔 System Tray & Background Minimization ("Out of the Way")

Unobtrusive Background Operation: Keeps the Cyberdeck running completely out of the way in the Windows notification tray during long-running automated tasks, batch scrapes, and research pipelines.
Single-Click Summon: Left-clicking or double-clicking the system tray icon toggles window visibility instantly.
Close-to-Tray Protection: Closing the window (X) minimizes to the system tray rather than killing active background tasks and MCP sessions.
MCP Tools: set_window_visibility({ visible?, focus? }) and configure_system_tray({ close_to_tray }).

🚨 Human-In-The-Loop (HIL) Intervention Protocol

Graceful Human Collaboration: When the agent hits Cloudflare Turnstile, CAPTCHAs, 2FA/SMS OTPs, or payment confirmations, it invokes request_human_intervention.
Multi-Channel Alerting:
1. OS Native Desktop Notification: Fires an urgent desktop notification banner with sound.
2. Window Summon & Taskbar Flash: Automatically restores the window from the tray, brings it to the front, and flashes the Windows taskbar amber (flashFrame(true)).
3. Interactive In-Page Cyberdeck Banner: Injects an amber neon cyberdeck banner (#__kuang_hil_banner) directly into the active webpage with a "RESUME AGENT (DONE)" button.
4. HUD Action Indicator: Changes the status badge to AWAITING_INPUT with live prompt text.
Instant Unpause: When the user completes the action and clicks "DONE", the agent immediately resumes its autonomous turn.
MCP Tools: request_human_intervention, resolve_human_intervention, get_human_intervention_status, and non-blocking send_notification.

🔌 Consolidated Semantic MCP Tools (16 advertised · +2 decision · +3 JEV)

Advertised surface: 17 grouped tools, plus drive and decide when a decision model is configured, and jev_index, read_section and search_page when JEV page compaction is on. The old flat names remain callable by name but are hidden from tools/list; they are listed below as the underlying catalog.

Tool manager: Settings → Tools controls enablement, order, per-tool instructions, per-action instructions for grouped tools, and Core/Research/Automation/Full presets. The default order starts with navigate → observe → act → wait_for → find → extract.

Tier map: TIER 1 · prefer: navigate, observe, act, wait_for, find, extract; TIER 2 · fallback: capture; TIER 3 · deliberate: evaluate_javascript. drive and decide are Tier 1 when a decision model is configured.

Tool	Actions / scope
`navigate`	`open`, `back`, `forward`, `reload`, `consent`, `scroll`, `recipe`
`observe`	markdown, text, AX, Set-of-Marks and components projections
`find`	Semantic target search returning mark refs
`act`	click, focus, hover, type, press, scroll and multi-field fill
`wait_for`	Declarative readiness gate
`extract`	list, table(s), app data, schema.org and lazy-list `collect`
`capture`	viewport/Set-of-Marks screenshot or rendered snapshot
`evaluate_javascript`	Deliberate page JS execution
`devtools`	state, diff, console, network, streams, HAR, JSON query, fingerprint, health
`human_in_loop`	request/status/resolve HIL, activity/resume, notify, window and tray
`collaborate`	highlight, ask, read, reply and resolve human page threads
`browser_workspace`	tabs, downloads, contexts, recordings and scripts
`media`	status, info, preview, transcript, download, jobs, cancel, list
`shadow_iframes`	`frames`, `focus_walk`
`research`	read, search, LLM surface/fetch, crawl, topic research, prerender
`memory`	recall, browsing history and bookmarks
`help`	Categorized operational cheatsheet
`jev_index` (JEV)	Section index for the active page — the preferred read when JEV is on
`read_section` (JEV)	Fetch one indexed section, optionally with children
`search_page` (JEV)	Keyword/regex search across all section text
`drive` (optional)	Goal-driven observe → decide → act loop
`decide` (optional)	Typed decision model

Underlying catalog (hidden aliases)

The table below documents the flat catalog entries. Most are hidden from tools/list behind the groups above; they remain callable by name for compatibility and are useful when editing tool instructions.

Category	Tool	Parameters	Description
⌥ Optional	⌥ `drive`	`goal`, `values?`, `max_steps?`, `threshold?`, `dry_run?`	Listed first when a decision model is configured. Runs observe → guarded decide → verified act in one call and returns a typed stop reason (`done`/`blocked`/`human_required`/`needs_agent`/`failed`/`max_steps`). Absent otherwise.
⌥ Optional	⌥ `decide`	`state`, `questions`, `model?`, `guard?`, `eliminate?`, `include_page_text?`, `include_screenshot?`	Listed first when configured. One typed decision (`choice`/`score`/`noul`) over a state, answers forwarded verbatim, with an optional guard + elimination. Absent otherwise.
Primitives & Abstraction	`observe`	`as?: 'markdown' \| 'text' \| 'ax' \| 'som'`, `region?: number \| string`, `budget?: number`, `cursor?: string`, `include_offscreen?: boolean`, `limit?: number`	The page read (TIER 1). One budgeted observation model with projections. Use `budget` + `next_cursor` to page a large page instead of truncating. Absorbed `get_page_markdown` / `harvest_markdown` / `list_marks`; structured reads live on `extract`.
	`act`	`mark_id?: number`, `selector?: string`, `x?: number`, `y?: number`, `action?: 'click' \| 'focus' \| 'hover' \| 'type' \| 'press'`, `text?: string`, `secret?: boolean`, `clear_first?: boolean`, `await_conditions?: object[]`, `timeout_ms?: number`, `dry_run?: boolean`	The interaction entry point: one humanlike action addressed by mark ref, CSS selector, or coordinates, with optional post-conditions and a delta report. Selectors pierce open shadow roots and resolve a custom-element host to the control a mouse would hit. `secret: true` redacts typed text from logs; `dry_run: true` resolves without acting. Typing appends by default - pass `clear_first: true` on a pre-filled field (origins, search boxes, comboboxes) to replace instead (trusted Ctrl+A + Backspace). Typing is verified: a provable no-op returns `ok: false` with `error.code: "TYPE_NO_EFFECT"` and `typing` evidence (key counts and value lengths - never the text).
	`wait_for`	`conditions: object[]`, `require?: 'any' \| 'all'`, `timeout_ms?: number`, `poll_ms?: number`	Declarative readiness gate — polls until every/any condition holds, then returns. The primitive that replaces blind sleeps.
	`find`	`role?: string`, `name?: string`, `contains?: string`, `regex?: string`, `tag?: string`, `selector?: string`, `limit?: number`, `include_offscreen?: boolean`	Semantic element search returning Set-of-Marks refs (mark ids) instead of pixels. Feed ids straight into `act`.
	`extract`	`kind?: 'list' \| 'table' \| 'tables' \| 'app_data' \| 'schema_org'`, `ref?: number`, `selector?: string`, `type?: string`, `limit?: number`	Structured, read-only extraction: repeating rows, one table, every table, embedded app JSON (`__NEXT_DATA__`/JSON-LD), or typed schema.org items (`kind:"schema_org"` flattens `@graph`; `type` filters to e.g. `"Recipe"`). Article extraction is `observe(as:"markdown")` (Readability).
	`collect_all`	`selector?: string`, `ref?: number`, `scroll?: 'container' \| 'window'`, `step?: number`, `delay_ms?: number`, `max_scrolls?: number`, `max?: number`	Scrolls a virtualized/lazy list to the end and collects every distinct row (deduped by text+href) in one call, with scroll diagnostics.
	`expect`	`conditions: object[]`	Immediate assertion (no polling) over the same vocabulary as `wait_for`; returns pass/fail plus per-condition evidence.
	`page_state`	none	Cheap monotonic state signature (`revision`, `url`, `marks`, `nodeCount`, `height`, `scrollY`, `textLen`, `textHash`) for change detection without re-reading the page.
	`diff`	`since?: number`	Field-by-field diff between the current state and a previously observed revision — shows exactly what changed.
	`llm_surface`	`url?: string`	Checks whether a site publishes an LLM-friendly representation (`/llms.txt` / `/llms-full.txt`, `/sitemap.xml`, a `<link rel="alternate">` or a real `<a href="….md">` page link, or Markdown content negotiation) and returns `recommended` — `"llms_txt" \| "llms_full" \| "markdown" \| "text"`, or `null` when there is none (never `"rendered"`). Also `scope` (`"origin"` \| `"page"`) and `next` — the unambiguous next step (`llm_fetch(as)` or `observe({as:"markdown"})`). Well-known paths only; cached per origin.
	`read_url`	`url?: string`, `max_chars?: number`, `cursor?: string \| number`, `as?: 'llms_txt' \| 'llms_full' \| 'markdown' \| 'text'`	Read any URL as Markdown in ONE call. Reads the site's own representation when one covers the URL asked for (a page-level alternate for a page); otherwise a detached render distilled to Markdown (your active tab is never hijacked). A site-wide index is never returned in place of a page — the page is rendered, and `reason` says why. Always returns text, so you never branch on `recommended`. Reports `via`, `scope`, `scope_mismatch` and `reason`; paged with `max_chars` + `next_cursor`.
	`llm_fetch`	`url?: string`, `as?: 'llms_txt' \| 'llms_full' \| 'markdown' \| 'text'`	Fetches that advertised Markdown/Text representation (~5–10× cheaper than a render). Gated: refuses (`NO_LLM_SURFACE`) unless the site advertises a surface. Reports `scope` + `scope_mismatch` (page requested, site-wide index returned). Per-page alternates load via a real navigation, not a synthetic fetch.
	`search_web`	`query: string`, `engine?: 'google' \| 'duckduckgo' \| 'bing' \| 'brave' \| 'startpage' \| 'custom'`, `engines?: string[]`, `url_template?: string`, `max_results?: number`	Multi-engine web search run in a detached view (your active tab is untouched) with automatic fallback when an engine bot-checks. Returns ranked `{title, url, snippet}` to feed `navigate_to`/`collect_all`. `custom` takes a `{query}` template (e.g. SearXNG).
	`deep_crawl`	`url: string`, `max_depth?: number`, `max_pages?: number`, `match_patterns?: string[]`, `same_origin?: boolean`, `prefer_llm?: boolean`, `harvest_to_cortex?: boolean`, `selector?: string`, `delay_ms?: number`	Autonomous BFS crawl from a seed: follows same-origin links within depth/page budgets, distills each page to Markdown, and indexes it into LanceDB Cortex. Each page takes the cheap path first (content-negotiated Markdown/Text, else a detached render). `match_patterns` scopes by glob or `/regex/`.
	`research_topic`	`query: string`, `max_sources?: number`, `engine?`, `engines?: string[]`, `url_template?: string`, `excerpt_chars?: number`, `harvest_to_cortex?: boolean`, `prefer_llm?: boolean`	Autonomous research mode: searches, gathers the top sources (cheap Markdown path first), distills them, and indexes them into Cortex. Returns extractive, cited evidence — each highlight is a real paragraph mentioning the query terms. It does not write prose; you synthesize the answer.
	`bridge_health`	none	Liveness + capability discovery: `version`, `revision`, `capabilities[]`, `window {visible, focused}`, `input {transport}`, `lastActionAt`.
	`context_save`	`name?: string`	Saves the current browsing context (cookies + storage) as a named profile.
	`context_open`	`name: string`	Restores (switches to) a named browsing context/profile.
	`context_list`	none	Lists saved browsing contexts (cookie/storage profiles).
	`context_delete`	`name: string`	Deletes a saved browsing context/profile.
	† `focus_window`	none	Deprecated — use `set_window_visibility({ visible: true })`. Brings the Cyberdeck window to the foreground (HIL visibility, or the legacy `native` input fallback).
Navigation	`navigate_to`	`url: string`, `auto_dismiss_consent?: boolean`	Navigates active tab to URL. Auto-dismisses GDPR/cookie overlays by default. Supports 0ms instant elevation if pre-rendered.
	† `go_to_url`	`url: string`, `auto_dismiss_consent?: boolean`	Primary alias for `navigate_to`. Dispatches visible window navigation.
	`navigate_history`	`action: 'back' \| 'forward' \| 'reload'`	Browser history navigation and reload controls.
	† `get_page_info`	none	Returns current URL, title, and load state without full extraction.
Shield & Defense	`dismiss_consent`	none	Autonomously detects and dismisses GDPR/cookie banners via Bézier mouse click.
Interaction	† `click_element`	`selector?: string`, `x?: number`, `y?: number`	Deprecated — folded into `act(selector: …)` / `act(x: …, y: …)`. Clicks element by CSS selector or direct viewport (x, y) coordinates.
	† `type_text`	`text: string`, `selector?: string`	Deprecated — folded into `act(action: "type", text: …, selector: …)`. Types text with human Gaussian cadence variance (60–240ms).
	`fill_form`	`fields: Record<string, string \| boolean \| number>`, `submit_selector?: string`	Fills multiple form inputs, selects, and checkboxes in a single turn.
	`press_key`	`key: string`	Presses specific key (e.g. `Enter`, `Tab`, `Escape`, `Backspace`).
	`scroll_page`	`direction: 'up' \| 'down'`, `amount?: number`	Incremental momentum wheel scrolling.
Observation & Vision	`take_screenshot`	`marks?: boolean`, `detail?: 'full' \| 'economy' \| 'markdown'`, `include_offscreen?: boolean`, `max_width?: number`, `format?: 'jpeg' \| 'png'`, `quality?: number`	Viewport capture. Plain mode (default) returns a bounded 768px fast JPEG (~15ms encode, tiny wire payload); `marks: true` returns Set-of-Marks with `detail: 'full'` (PNG), `'economy'` (640px JPEG), or `'markdown'` (marks table, no image). Returns file path text & native MCP image block. Saved to `data/screenshots/` with automatic retention pruning.
	† `get_page_markdown`	none	Downsamples active DOM to clean token-efficient Markdown with YAML metadata.
	† `harvest_markdown`	none	Primary alias for `get_page_markdown`. Extracts clean markdown from the active tab.
	`snapshot_rendered_page`	`format?: 'ax_tree' \| 'markdown' \| 'html' \| 'unrendered_html'`, `wait_until_idle_ms?: number`, `pierce_shadow_dom?: boolean`, `force_rerender?: boolean`, `save_to_disk?: boolean`	Rendered Point-in-Time Snapshot. Freezes dynamic SPAs, pierces Shadow DOM, and extracts ultra-token-efficient Accessibility (AX) Trees, rendered Markdown, or unrendered source HTML.
	`prerender_links`	`urls?: string[]`, `selector?: string`, `max_pages?: number`, `harvest_to_cortex?: boolean`, `wait_until_idle_ms?: number`	Speculative Pre-rendering Engine. Pre-hydrates candidate links in background views for 0ms instant navigation, and auto-indexes into LanceDB Cortex memory.
	`evaluate_javascript`	`script: string`	Evaluates custom JS expression in page context and returns serialized JSON. Accepts `script`, `code`, or `expression`. A powerful tool that can read and mutate — reach for it deliberately (`observe`/`act` cover the common cases, and injected JS is a bot-detection signal).
	`get_console_logs`	`limit?: number`	Returns recent web page and script console logs, warnings, and errors.
	`get_fingerprint_profile`	none	Returns active stealth host browser fingerprint and spoofed headers.
Multi-Tab	`new_tab`	`url?: string`	Spawns a new independent browser tab instance.
	`switch_tab`	`tab_id: string`	Activates and focuses the specified tab.
	`close_tab`	`tab_id?: string`	Closes tab and reclaims memory resources.
	`list_tabs`	none	Returns structured array of all open tabs (IDs, titles, URLs, active status).
LanceDB Memory	`recall_memory`	`query: string`, `limit?: number`, `domain?: string`	Performs hybrid vector and BM25 full-text search across visited pages and markdown.
	`browsing_history`	`limit?: number`, `domain?: string`	Chronological browsing history from persistent LanceDB cortex.
	`browsing_history_clear`	`older_than?: string`	Prunes or purges browsing history from persistent storage.
	`add_bookmark`	`url?: string`, `title?: string`, `notes?: string`, `tags?: string[]`	Saves active or specified page with tags and notes in LanceDB.
	`list_bookmarks`	`query?: string`, `tag?: string`, `limit?: number`	Retrieves or filters saved bookmarks by query terms (title/notes/URL/tags) or exact tag.
	`delete_bookmark`	`url_or_id: string`	Deletes bookmark by URL or unique ID.
Downloads & Docs	`list_downloads`	none	Lists all files downloaded into `data/downloads/` with sizes, state, and timestamps.
	`read_download`	`filename_or_id: string`	Parses downloaded file (PDF, CSV, JSON, text) into token-efficient markdown.
Timelapse Recording	`record_start`	`name?: string`	Starts a timelapse of the Cyberdeck window itself: frames are captured every few hundred ms and played back faster than real time. The finished `.webm` path becomes an ordinary value you can type into a file-upload field.
	`record_stop`	none	Stops the timelapse and saves it (VP9/WebM). Returns the absolute path, size, frame count and speed-up factor.
	`list_recordings`	`limit?: number`	Lists finished timelapses as absolute paths, newest first — ready for upload fields.
Session Vault	† `save_session`	`name?: string`	Captures active cookies and localStorage state into persistent JSON vault.
	† `restore_session`	`name: string`	Rehydrates cookies and localStorage state into browser session.
	† `list_sessions`	none	Lists all stored session snapshots with timestamps and cookie counts.
Network Intelligence	`get_network_traffic`	`url_pattern?: string`, `method?: string`, `resource_type?: string`, `limit?: number`, `since?: number \| 'last_action'`	Inspects live HTTP/XHR/Fetch requests captured via Chrome DevTools Protocol (returns compact 1-line markdown summaries). `since` narrows to requests after an epoch-ms timestamp, a page revision, or the last mutating action.
	† `get_api_payload`	`request_id: string`	Retrieves and parses the full raw JSON or text response body for an intercepted request.
	`query_json_data`	`source: string`, `source_id?: string`, `path?: string`, `view?: string`, `filter_predicate?: string`, `max_items?: number`, `max_string_length?: number`	Token-Efficient JSON Navigator. Queries large payloads/streams/files via dot/bracket paths (`items[0..3].price`), extracts schemas, searches keys, and applies filters without context-window blowup.
	† `clear_network_traffic`	none	Deprecated — use `clear_capture({ scope: "network" })`. Resets the in-memory network traffic ring buffer.
	`export_har`	`pattern?: string`, `limit?: number`, `save?: boolean`	Exports captured traffic as a HAR 1.2 log (request/response headers + body previews). `save: true` writes `data/har/har_<ts>.har` and returns the path.
	`list_realtime_streams`	`type?: 'websocket' \| 'sse'`, `active_only?: boolean`	Lists active WebSocket connections and open SSE channels in compact 1-line format.
	† `get_stream_messages`	`stream_id?: string`, `type?: string`, `direction?: string`, `limit?: number`	Inspects captured incoming/outgoing frames from WebSockets and SSE token streams.
	† `clear_stream_messages`	`stream_id?: string`	Deprecated — use `clear_capture({ scope: "streams" })`. Clears the in-memory WebSocket and SSE message frame buffer.
	`clear_capture`	`scope?: 'network' \| 'streams' \| 'all'`	Clears the in-memory capture ring buffers: HTTP traffic (`"network"`), WebSocket/SSE frames (`"streams"`), or both (`"all"`, default).
Visual Grounding	† `take_annotated_screenshot`	none	Tags all visible interactive elements with numbered neon badges (`[1]`, `[2]`, etc.). Returns dual content: Set-of-Marks table text & native MCP image block (`type: 'image'`). Saved to `data/screenshots/`.
	† `take_economy_screenshot`	none	Lightweight variant: same marks table, but the image is downscaled to 640px wide and JPEG-encoded (q45, `image/jpeg`). Saved to `data/screenshots/` as `annotated_eco_*.jpg`.
	† `list_marks`	none	Text-only variant: returns the marks table with no image and no file written. Ideal for text-only models.
	† `click_mark`	`mark_id: number`	Deprecated — folded into `act(mark_id: …)`. Moves humanized Bézier ghost cursor to mark #ID and clicks with 100% accuracy.
	† `type_mark`	`mark_id: number`, `text: string`, `clear_first?: boolean`	Deprecated — folded into `act(mark_id: …, action: "type", text: …)`. Focuses mark #ID and types with variable human typing cadence.
Shadow DOM & Frames	`frames`	`include_controls?`, `screenshots?`, `selector?`, `max_frames?`, `max_controls?`, `include_offscreen?`, `max_side?`	Lists iframes as first-class content: frame rect, URL and controls with viewport-offset coordinates. Opt into per-frame screenshots for cross-origin widgets and embedded checkouts.
	`focus_walk`	`selector?`, `x?`, `y?`, `direction?`, `max_steps?`, `restore?`	Enters a closed shadow root like a keyboard user: focus the host, send trusted Tab/Shift+Tab steps, and read the accessibility tree to confirm the landed control.
Script Vault & Macros	`save_script`	`name: string`, `code: string`, `description?: string`, `domain?: string`, `tags?: string[]`	Saves and embeds a custom JavaScript macro into LanceDB Cortex.
	† `recall_script`	`query?: string`, `domain?: string`, `tag?: string`, `limit?: number`	Deprecated — use `list_scripts({ query })`. Semantically searches saved scrapers & macros using natural language.
	`run_script`	`name: string`, `args?: object`	Executes a saved script in active tab with dynamic runtime arguments.
	`list_scripts`	`query?: string`, `tag?: string`, `limit?: number`	Lists or semantically searches registered macros in the persistent vault (`query` ranks by relevance).
	`delete_script`	`name: string`	Removes a script from the persistent vault and vector index.
Desktop & System Tray	`set_window_visibility`	`visible?: boolean`, `focus?: boolean`	Shows/restores the window (and focuses it by default) to make it visible to a human, or minimizes it to the Windows notification tray.
	`configure_system_tray`	`close_to_tray?: boolean`	Configures tray options such as close-to-tray background operation.
Human-In-The-Loop & Alert	`request_human_intervention`	`reason: string`, `timeout_seconds?: number`, `bring_to_front?: boolean`	Halts agent, triggers native OS alert, unhides window, and injects interactive banner for CAPTCHAs/credentials.
	`resolve_human_intervention`	`id?: string`, `notes?: string`	Resolves active intervention request and resumes agent execution.
	`get_human_intervention_status`	none	Checks active HIL request status and history. Does not report the passive takeover window; use `get_human_activity`.
	`get_human_activity`	none	Reports the passive human-takeover window that blocks synthetic input: `active`, remaining pause and the last human event.
	`resume_agent`	none	Clears a passive takeover pause after the human confirms. Does not resolve a pending `request_human_intervention`.
	`send_notification`	`title: string`, `message: string`, `urgency?: string`, `sound?: boolean`	Emits non-blocking OS native desktop banner notification to user.
	† `send_desktop_notification`	`message: string`, `title?: string`, `urgency?: string`, `sound?: boolean`	Dedicated HIL notification tool to summon the human operator with high-priority desktop sound and alert banner.
Navigation Recipes	`recipe`	`action?: 'list' \| 'save' \| 'replay' \| 'delete'`, `id?`, `url?`, `name?`, `path?`, `steps?`, `expect_timeout_ms?`	Compiles a page path into descriptor steps and replays it with zero model calls. Scoped to origin + path; retires after three failed replays.
Assistance & Discovery	`help`	`category?: 'all' \| 'recommended' \| 'navigation' \| 'interaction' \| 'primitives' \| 'inspection' \| 'tabs' \| 'memory' \| 'macros' \| 'network' \| 'system'`	Returns a categorized operational cheatsheet of all available browser automation capabilities (served directly in OKF).

🔌 REST wrappers (`hb-rest`)

A second, separate MCP server for turning a REST endpoint into a saved, typed tool. It is deliberately not mixed into the browser server: hb-rest ships its own binary, its own stdio process, its own config entry and its own bridge token.

	Browser	REST wrappers
Binary	`bin/mcp-server.cjs`	`bin/wrap-server.cjs`
Server name	`hb-browser` (legacy: `kuang`)	`hb-rest` (legacy: `kuang-wrap`)
Surface	16 advertised browser tools (+2 optional decision tools)	10 `wrap_*` meta-tools + up to 40 generated `wrap_<id>` (configurable via `HB_WRAP_MAX_TOOLS` or legacy `KUANG_WRAP_MAX_TOOLS`)
Env	`HB_BRIDGE_URL` (legacy: `KUANG_BRIDGE_URL`)	`HB_WRAP_TOKEN_FILE` (legacy: `KUANG_WRAP_TOKEN_FILE`)

What it is for: agent access to JSON APIs (dashboards, internal tools, SaaS). The wrapper author — human or agent — defines the request once: method, URL, typed arguments, a response recipe, and a credential name. The agent then calls flat arguments; it never assembles a URL, a header set, or a body envelope, and it never sees a credential value.

Credentials are human-only and host-bound: a static value, or an OAuth2 client-credentials client that mints and caches a token. Secrets are encrypted with the OS keychain (safeStorage), injected only on an approved host, and never enter the MCP process.

// Settings → MCP Clients → REST wrappers generates an entry like this
{
  "mcpServers": {
    "hb-rest": {
      "command": "node",
      "args": ["<path-to-repo>/bin/wrap-server.cjs"],
      "env": { "HB_WRAP_TOKEN_FILE": "<userData>/wrap/token" },
    },
  },
}

Meta-tools: wrap_list, wrap_get, wrap_secret_list, wrap_derive, wrap_create, wrap_update, wrap_delete, wrap_run, wrap_test, wrap_reload.

[!WARNING] Read the limits before relying on it. No human-approval gate for mutating calls · OAuth2 is client_credentials only · the wrap server does not raise HIL itself (a rejected credential returns AUTH_REQUIRED) · exposeAsTool has no Settings control · 40-tool cap · DNS-rebinding residual risk · the token URL is outside allowedHosts. The full, blunt breakdown — what works, what doesn't, what is untested, what is risky — is docs/HYPAGON_REST.md.

Tests: npm run test:wrap (50 offline) · npm run test:wrap:mcp (11 lifecycle) · npm run test:wrap:live (46 against DummyJSON).

🗺️ Codebase Directory Structure & Documentation

Each folder contains a dedicated README.md detailing its responsibilities, dependencies, and internal workflows:

Directory	Documentation Link	Description
`docs/`	docs/CODEBASE.md	Architecture lowdown, field notes, decision-model notes, build and packaging docs.
`scripts/`	scripts/README.md	Test harnesses, release helpers (`check-release-tag`, `release-notes`), the MCP bundle build, and one-off tools.
`public/`	public/README.md	Static web root and the built-in offline Cyberdeck Manual (`welcome.html`).
`public/assets/`	public/assets/README.md	Tactical high-resolution brand insignia (legacy asset names retained for compatibility).
`src/`	src/README.md	Root source directory organizing process-isolated TypeScript modules.
`src/main/`	src/main/README.md	Electron main process, `BrowserChassis` coordinator, and HTTP bridge (`:4111`).
`src/main/cortex/`	src/main/cortex/README.md	Embedded LanceDB vector database, 128-d embedder, chunker, and hybrid semantic recall.
`src/main/tabs/`	src/main/tabs/README.md	Hardware-accelerated `WebContentsView` multi-tab pooling and popup trapping.
`src/main/shield/`	src/main/shield/README.md	Autonomous Shield Breaker engine auto-dismissing cookie banners and GDPR modals.
`src/main/downloads/`	src/main/downloads/README.md	Deterministic workspace file download routing and automatic document-to-Cortex ingestion.
`src/main/session/`	src/main/session/README.md	Persistent cookie jar and session snapshot vault for cross-run auth persistence.
`src/main/network/`	src/main/network/README.md	Chrome DevTools Protocol wire interceptor capturing background XHR/Fetch JSON streams.
`src/main/marks/`	src/main/marks/README.md	Set-of-Marks visual grounding engine injecting numbered neon badges for zero-ambiguity LLM actuation.
`src/main/primitives/`	src/main/primitives/README.md	Orthogonal primitive layer (`observe` · `act` · `wait_for` · `expect` · `find` · `extract` · `collect_all` · `page_state`/`diff` · `export_har`) with structured error codes and injected DOM probes.
`hypagon_docs/`	hypagon_docs/okf.json	OKF (Open Knowledge Format) bundle `hypagon_browser_docs` — playbook/reference partitions exposed as MCP resources.
`src/main/scripts/`	src/main/scripts/README.md	Persistent script vault and LanceDB semantic macro recall engine with parameter injection.
`src/main/tray/`	src/main/tray/README.md	Windows System Tray integration and close-to-tray background minimization.
`src/main/hil/`	src/main/hil/README.md	Human-In-The-Loop cooperative intervention protocol, OS notifications, and in-page alert banners.
`src/main/telemetry/`	src/main/telemetry/README.md	Organic cubic Bézier mouse kinematics, Gaussian typing cadence, and stealth fingerprinting.
`src/main/harvester/`	src/main/harvester/README.md	Mozilla Readability & Turndown token-efficient markdown distillation.
`src/preload/`	src/preload/README.md	Sandboxed ContextBridge IPC exposing typed `window.electronAPI` to the DOM.
`src/renderer/`	src/renderer/README.md	DaisyUI 5 Cyberdeck HUD, phosphor radar trail canvas, and Cortex modal dialogs.
`src/shared/`	src/shared/README.md	Shared TypeScript contracts, telemetry events, and construct schemas.
`src/mcp/`	src/mcp/README.md	In-process TypeScript MCP server definition for zero-overhead unified execution.
`src/main/decide/` + `drive/`	docs/DECISION_MODELS.md	Optional typed decision client, guard, human gate detection, and the observe → decide → act loop.
`src/main/recording/`	src/main/recording/recording-service.ts	Timelapse configuration, window capture handshake, file writing, and the recordings registry.
`src/main/search/` · `crawl/` · `research/`	src/mcp/README.md	Multi-engine search, breadth-first crawl, and cited research.
`src/main/updates/`	docs/BUILDING.md	`electron-updater` wiring and release feed handling.

🚀 Quickstart

Development Mode (with Live Reloading)

pnpm dev

One bridge per port, one profile per process. A dev run and an installed build derive the same userData directory, so they share Electron's single-instance lock and the default bridge port 4111. To run both at once, give the dev run its own profile and port:
HB_USER_DATA_DIR="$APPDATA/hypagon-gateway-dev" HB_BRIDGE_PORT=4113 pnpm dev
Then point a second MCP client config at HB_BRIDGE_URL=http://127.0.0.1:4113 and HB_WRAP_TOKEN_FILE=<that profile>/wrap/token. The dev window's Settings → MCP Clients tab generates the config for you, with the right port and profile paths.

pnpm dev:isolated does all of that in one step: it starts the dev build on its own profile and port 4113, and prints a ready-to-paste MCP config.

Production Build & Launch

pnpm run build
pnpm start

MCP Client Configuration (OpenCode, Claude, Cursor, Antigravity, Atomic Chat)

The HUD's Settings → MCP Clients tab generates a correct, copy-ready config for this installation — OpenCode, Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Google Antigravity, Gemini CLI, Atomic Chat, Zed, and the generic mcpServers shape. It fills in the absolute server path and bridge port, and offers two runtimes:

Bundled (default) - runs the server on Hypagon Gateway's own Electron runtime with ELECTRON_RUN_AS_NODE=1, so Node does not need to be installed. Installed builds unpack bin/ out of the asar precisely so this works.
System Node — node bin/mcp-server.cjs, requiring Node 20+ on PATH.

A minimal Claude-style entry looks like this (the tab generates the real paths):

{
  "mcpServers": {
    "hb-browser": {
      "command": "<path-to-app>/Hypagon Gateway.exe",
      "args": ["<userData>\\bin\\mcp-server.cjs"],
      "env": { "ELECTRON_RUN_AS_NODE": "1" }
    }
  }
}

Note: the Cyberdeck must be running (pnpm run dev or pnpm start) so the stdio server can reach the browser instance on http://127.0.0.1:4111. If you run on another port with HB_BRIDGE_PORT, set HB_BRIDGE_URL for the MCP server — the generated snippet includes it automatically. Installed builds copy the server into userData/bin/ (refreshed on update), because resources/ is temporary for portable and AppImage builds.

Agent Skills

The MCP servers provide the tools; the skills teach the agent which grouped tool to use and give exact recipes. Install either skill directly from GitHub:

hermes skills install https://raw.githubusercontent.com/hyperbolictonic/hypagon/main/skills/hb-browser/SKILL.md
hermes skills install https://raw.githubusercontent.com/hyperbolictonic/hypagon/main/skills/hb-wrap/SKILL.md

Or with the Skills CLI:

npx skills add hyperbolictonic/hypagon --skill hb-browser
npx skills add hyperbolictonic/hypagon --skill hb-wrap

For a non-interactive global install, add -g -y. The source files are skills/hb-browser/SKILL.md and skills/hb-wrap/SKILL.md.

hb-browser covers the grouped tools plus the JEV read path (jev_index → read_section, including the enforce refusal); hb-wrap covers wrapper discovery, wrap_test dry-runs, recipes and credential hand-off.

🧪 Testing

All suites run against the live Cyberdeck on http://127.0.0.1:4111 (start the app first):

Command	Suite	What it proves
`npm run test:harvest`	Harvest regression (4 checks)	Markdown extraction, marks, off-screen marks still work end-to-end.
`npm run test:primitives`	Offline primitives (25 checks)	`wait_for`, `extract`, `observe` budget/cursor, `collect_all`, `find`+`act`+delta, `expect`, `page_state`/`diff`, `bridge_health` capabilities.
`npm run test:mcp:primitives`	MCP stdio primitives (14 checks)	The same primitives through a real MCP client — tool schemas, viewport gating, `find`→`act`, contexts, `focus_window`.
`npm run test:focus`	Focus independence (4 checks)	Hands the OS foreground to another app (Notepad) and asserts typing/clicks/wheel still land and that `input.transport === 'cdp'`.
`npm run test:mcp-http`	MCP HTTP host (6 checks)	Bearer auth, stateful and stateless MCP clients, session eviction, non-MCP path rejection, and idle expiry.
`npm run test:navigation`	Navigation error mapping	DNS, refusal, timeout, certificate and aborted failures map to `400` with stable `NAVIGATION_*` codes.
`npm run test:cloudflared-tunnel`	Named tunnel manager (3 checks)	Registration parse, startup timeout, process failure, and clean shutdown using a fake cloudflared fixture.
`npm run test:cloudflared`	cloudflared manager (4 checks)	Release selection, SHA-256 verification, atomic managed install, checksum rejection, unsupported-platform errors, and pure-Node macOS tar extraction.
`npm run test:mcp-oauth`	MCP OAuth server (4 checks)	Discovery metadata, forwarded HTTPS origin, authorization code + PKCE, token/refresh, bad verifier and disallowed redirect rejection.
`npm run test:devlogs`	Dev logs MCP smoke	Lists/read/searches allowlisted logs and confirms token redaction.
`npm run test:remote`	Remote gateway smoke (experimental)	Builds `bin/remote-server.cjs`, starts it with a token file, rejects unauthenticated calls, lists the browser tools, and tests the app service start/stop/token rotation/tunnel handoff.

Type-check and bundles: npx tsc --noEmit, npm run build, npm run build:mcp.

📚 Documentation & Knowledge (OKF)

developer/ — the standard OKF v0.2 developer bundle: MCP setup, browser and REST APIs, wrapper/recipe formats, and examples.
CHANGELOG.md — release history and migration-relevant changes.
docs/ — the documentation hub: concept lore, subsystem specs, field notes, and the abstraction roadmap.
docs/PROPOSED_ABSTRACTIONS.md — the primitive-layer design and its status (A1–A6, B, C, D).
docs/FIELD_NOTES_2026-10-18.md - root-caused findings from hardening the TIER-1 path on real SPAs.
docs/FIELD_NOTES_2026-10-20_SIGNUP.md - the signup dogfood run: three silent no-ops, and the plan they produced.
docs/DECISION_MODELS.md - the optional decision layer: decide / drive, guard + elimination, gates, screenshots, and who decides what.
docs/JEV.md - Just-Enough-View: the surface index, prefer / enforce modes, refusals and the preferred reading path.
docs/BUILDING.md - installers for Windows/macOS/Linux, signing (macOS unsigned first), GitHub Releases, and auto-update.
hypagon_docs/ — the OKF bundle (hypagon_browser_docs): playbook/reference partitions, also served to agents as standard MCP resources:
- mark-eleven://skills/guide — the full operational manual.
- mark-eleven://okf/manifest — the bundle manifest.
- mark-eleven://okf/partitions/selector-and-som · captcha-and-anti-bot · browser-navigation · primitives-and-workflows.

📦 Release notes

[!NOTE] What's new in 2.1.1 — Cortex recall for video, and a VS Code-style Settings. media transcript now indexes the full transcript into Cortex under the video URL, so memory recall can answer from a video later (pass index:false to skip). Settings navigation moved from a wrapping top tab strip to a scrollable left rail: the nav scrolls on its own, the active panel scrolls independently, and narrow windows collapse the rail into a horizontal strip. Earlier releases are in CHANGELOG.md.

📄 License

🌌 Neuromancer Inspiration & Cyberpunk Canon

"A Kuang Grade Mark Eleven penetration program. Icebreaker. The military stuff, the heavy artillery."
— William Gibson, Neuromancer (1984)

In William Gibson's genre-defining cyberpunk masterpiece Neuromancer, console cowboys Case and the ROM personality construct of the Dixie Flatline jack into the Matrix using an Ono-Sendai Cyberspace 7 deck. Their ultimate weapon: the Kuang Grade Mark Eleven—a mythical military-grade icebreaker designed to slice through the impenetrable corporate ICE surrounding the Tessier-Ashpool mainframe.

Hypagon Gateway began as a project inspired by that fictional lineage and has been renamed since; the current name is its own. The quote and the original Kuang Grade Mark Eleven reference are preserved here as inspiration; Hypagon Gateway is an independent project and is not affiliated with William Gibson, Neuromancer, or the fictional device.

It carries that inspiration into the modern web:

Piercing the Modern Web's "ICE": Headless automation (Puppeteer, Playwright) is routinely flagged and blocked by Cloudflare, Datadome, and behavioral bot sensors. Hypagon Browser provides a tactile, visible Chromium cyberdeck that operates with authentic human telemetry.
The Ghost Cursor as Neural Trodes: Dispatches organic cubic Bézier pointer movement and Gaussian typing cadences, behaving like an authentic human operator seated at the console.
The LanceDB Cortex as ROM Personality Construct: Provides persistent semantic recall and memory traces so your local AI agent retains research context and bookmarks across browsing sessions.