{"id":"5c1c7554-b035-4f4e-9ea5-694658e913e4","slug":"glovebox-mcp","kind":"mcp","name":"glovebox-mcp","summary":"Sandboxed computer-use MCP — drive a real browser + desktop apps in a nested X11 window.","owner":"segentic-lab","repoUrl":"https://github.com/segentic-lab/glovebox-mcp","license":"MIT","licenseTier":"permissive","trust":"community","stars":13,"match":null,"fewStars":false,"format":"mcp","createdAt":"2026-07-01T03:14:43.000Z","addedAt":"2026-10-14T05:20:48.740Z","logoUrl":"https://avatars.githubusercontent.com/u/285443774?u=05701c40a2eab0d7de4474d692a627f0a676ccbe&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["browser-web","memory-knowledge"],"tags":["ai-agents","automation","browser-automation","computer-use","sandbox","xephyr"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T02:51:11.466Z","sample":false,"url":"https://conduits.dev/tools/glovebox-mcp","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# glovebox-mcp\n\nA **sandboxed computer-use MCP server** — let an AI agent drive a real browser and desktop apps\n(mouse, keyboard, screenshots, clipboard, vision grounding), confined to a **nested X11 window** so\nits input and vision never touch your real screen or the windows of your other apps.\n\nLike a lab **glovebox**: the agent reaches in and manipulates real applications, sealed off from\nyour desktop. Bring the sandbox up, log into whatever sites or apps you want to automate *inside\nthat window*, and the agent operates only there — you can watch it live and close it instantly.\n\n> Speaks the [Model Context Protocol](https://modelcontextprotocol.io), so it works with MCP clients\n> like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.\n\n![glovebox-mcp — an agent filling a sign-up form inside the sandbox](assets/demo.gif)\n\n<sub>An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (`Nadja Kovačič`), typing, and submitting. All confined to a nested X11 window.</sub>\n\n## Why a nested X11 sandbox?\n\n- Most desktop automation (xdotool, PyAutoGUI) is **X11-only**, but many modern desktops run **Wayland**.\n- **Xephyr** provides a real X server *inside a single window* (`DISPLAY :1`). Everything the agent\n  does — clicks, typing, screenshots — is **confined to that window**, not your real desktop.\n- You stay in control: watch it live, `pkill Xephyr` to close everything.\n\nThe isolation is **display-level** (see [Safety](#safety)): the apps themselves run as your user with\nyour filesystem. For OS-level isolation run glovebox inside a VM or container.\n\n## Requirements\n\n- **Linux** — the sandbox nests a real X server (Xephyr), so it works even on **Wayland** hosts\n  (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.\n- **Python 3.10+** and [`uv`](https://docs.astral.sh/uv/) (used for the virtualenv).\n- **System packages** — `xserver-xephyr` (Xephyr), `openbox`, `scrot`, `x11-utils`, `xdotool`,\n  `wmctrl`, `xclip` (+ `tesseract-ocr` for `basic`). On Debian/Ubuntu the installer **auto-installs**\n  them via `apt` (sudo); on Fedora/Arch it prints the matching `dnf`/`pacman` command. The MCP server\n  itself is distro-agnostic — any Linux with these tools works. `openbox` is not optional in\n  practice: without a window manager Xephyr ignores synthetic pointer motion, so clicks would land in\n  the screen center (`launch_app` warns when it is missing).\n- A **browser** in the sandbox (Chromium, Chrome, Brave, Edge, Vivaldi or Opera).\n- **NVIDIA GPU (≥6 GB VRAM)** — only for the `local` vision mode. The server checks this itself\n  (`status()` → `vision.gpu`) and `auto` falls back to `basic`/`none` when it is not met.\n\n## Install\n\nPick a vision backend and run its one-liner (clone → install). Each one installs the system packages\n(auto via `apt` on Debian/Ubuntu) and the Python deps for that mode, and writes a ready-to-paste\n`mcp-config.json` with your paths.\n\n**`none`** — no local models; your agent reads screenshots itself (lightest, instant):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh none\n```\n\n**`basic`** — Tesseract OCR grounding (`parse_screen` → text lines + coordinates, CPU-only):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh basic\n```\n\n**`local`** — OmniParser on an NVIDIA GPU (`parse_screen` → text **+ icons**, pixel-precise; ~4 GB weights, ≥6 GB VRAM):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh local\n```\n\nYour choice is written to `.vision-mode` (override per run with the `GLOVEBOX_VISION` env var).\nWithout either, the server runs in **`auto`** and picks the best backend that is actually usable.\n`python vision.py check` prints the same diagnosis `status()` returns (GPU, weights, tesseract).\n\n### Works with any MCP client / harness\n\nClaude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host.\nTwo compatibility notes: **`basic`/`local`** return element **coordinates as text**, so they work even\nwith text-only agents; **`none`** relies on the client passing the tool's **screenshots** to a\nmultimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients). In any mode,\n`clipboard()` gets exact text out of an app without vision at all.\n\n## Quickstart\n\n1. **Start the sandbox** (leave it running):\n   ```bash\n   ./start-display.sh              # 1440×900 Xephyr window with a browser\n   ./start-display.sh 1920x1080    # …or pass a screen size (or set $RES)\n   ```\n   Log into any sites or apps you want to automate **in that window**.\n2. **Register the server** with your MCP client. `install.sh` already wrote **`mcp-config.json`** with\n   your real install path — copy its `glovebox` block into your client's MCP config:\n   ```jsonc\n   { \"mcpServers\": { \"glovebox\": {\n       \"command\": \"/abs/path/to/glovebox-mcp/.venv/bin/python\",   // filled in by install.sh\n       \"args\":    [\"/abs/path/to/glovebox-mcp/server.py\"],\n       \"env\":     { \"DISPLAY\": \":1\" }\n   } } }\n   ```\n   Restart the client so it loads the server.\n3. Ask the agent to screenshot / click / type — it operates **only** on the `:1` window.\n\n> **Driving it with an AI agent?** Paste [`AGENTS.md`](AGENTS.md) into the agent's system prompt — it\n> teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.\n\n## Tools (23)\n\n| Tool | What |\n|---|---|\n| `status()` | Server + sandbox status in one read-only call: version, the vision diagnosis (requested vs. effective backend and *why*, GPU name/VRAM, OmniParser code/weights, tesseract languages), live instances, and which system deps are present. Run it first — and paste it into bug reports. |\n| `screenshot(instance, region?, scale?, format?)` | `[json, image]`. `region=\"x,y,w,h\"` zooms into a rectangle, `scale=0.5` shrinks, `format=\"jpeg\"` compresses — cut payload or read small text. |\n| `parse_screen(instance, granularity?)` | Vision grounding → JSON of detected elements (id, type, label, interactive, pixel-center; capped at 300 per call, flagged via `\"truncated\"`) + a numbered Set-of-Mark image. `basic` merges OCR words into **lines** (`\"Sign in\"` = one target; `granularity=\"word\"` for single words). |\n| `click_element(id)` | Click an element from the last `parse_screen` (no coordinate guessing). |\n| `click(x,y)` · `double_click` · `move_mouse` · `scroll` · `drag(…, steps?)` | Pointer ops. Coordinates outside the display are **rejected** (xdotool would silently clamp them). `drag` emits intermediate motion events so canvases and sliders register it. |\n| `type_text(text)` | **Unicode-safe** typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps). |\n| `press_keys(\"ctrl+a\" / \"Return\" / \"ctrl+a Delete\")` | Keys, combos, and space-separated **sequences** (xdotool syntax). Invalid names error — nothing is pressed. |\n| `clipboard(action=\"get\"\\|\"set\", text?)` | Read the instance's clipboard (select + `ctrl+c`, then read **exact** text — no OCR) or set it (then `ctrl+v`). Per-instance. |\n| `wait_for_stable(instance, timeout_ms?, threshold?)` | Wait until the screen **stops changing** (page loaded, app drew) instead of guessing a delay. Reports `stable`/`timed_out` honestly. |\n| `window_info(instance)` | Active window title + geometry and the visible windows — confirm navigation or that an app opened, without an image. |\n| `upload_file(filepath, selector?, page_url_contains?)` | Attach a local file to a page's `<input type=file>` via the **Chrome DevTools Protocol**. The nested X11 file picker is invisible to automation **and hangs the renderer**, so use this for all uploads — never click an upload button expecting a dialog. Works on Chromium-family browsers started by `launch_app`/`start-display.sh` (per-instance `--remote-debugging-port`, `9222+N`). **Browser file inputs only** — for native apps see `open_file`. |\n| `open_file(filepath, app?)` | Open a local file in a **native** app on the instance's display (e.g. `app=\"gimp\"`) or via `xdg-open`. Fails honestly if the handler exits immediately (no app registered for that type). |\n| `list_files()` | The instance's staging folder `files/<N>/` + entries with size, mtime and a `partial` flag for in-progress downloads. |\n| `app_log(instance, lines?)` | Tail of the launched app's stderr — read it when an app crashed or drew nothing. |\n| `launch_app(command, name?, size?)` · `list_instances()` · `close_instance(n)` | Multi-instance control (see below). |\n| `wait_ms(ms)` · `get_screen_size()` | Timing / sandbox size + effective vision backend. |\n\nEvery control tool takes **`instance=N`** and optional **`observe`** / **`settle_ms`** (see below).\nIn `local` mode OmniParser is **lazy-loaded** on first `parse_screen` (~6 s once, then ~2 s/parse),\nafter a GPU preflight (VRAM present and free).\n\n### Structured, honest responses\n\nEvery tool returns JSON: `{\"ok\": true, \"action\": \"click\", \"instance\": 1, \"detail\": \"clicked (10,20) button 1\", …}`.\nFailures come back with the MCP `isError` flag set and the same JSON shape embedded in the error\ntext — `{\"ok\": false, \"error\": \"…\", \"fix\": \"…\"}` — where **`fix` names the call that unblocks you**\n(e.g. a click on a dead instance says to run `list_instances()` / `launch_app()`; an unknown element\nid says to re-run `parse_screen()`; a GPU that is too small says which mode to switch to). Silent\nno-ops are treated as failures too: an invalid keysym, a zero scroll, an off-screen click, closing an\ninstance that isn't running, an app that exits on launch, or non-ASCII typing without `xclip` all\nerror instead of pretending success. Clamped waits are flagged (`clamped` / `settle_clamped`).\nImage-returning calls (`screenshot()`, `observe=\"screenshot\"|\"parse\"`) return `[json, image]`.\nNo subprocess can hang a call: xdotool/xdpyinfo/xclip run under timeouts and a frozen display is\nreported as such.\n\n### Vision backend (selectable, self-diagnosing)\n\n`GLOVEBOX_VISION` env var, or the `.vision-mode` file, or default **`auto`**:\n\n| Mode | `parse_screen` | Needs | When |\n|---|---|---|---|\n| `auto` | the best **ready** backend below (`local` → `basic` → `none`); `status()` says which and why | — | default |\n| `none` | disabled (returns a note) — use `screenshot()` + reason, `clipboard()` for exact text | nothing (mcp, mss, xdotool) | lightest; let the agent's own vision do grounding |\n| `basic` | Tesseract OCR → text **lines** + coords (+ confidence) | `tesseract-ocr` + `pytesseract` | no GPU; text-only grounding |\n| `local` | OmniParser → text **+ icons** + coords | torch + CUDA + OmniParser weights + NVIDIA GPU ≥ 6 GB | best grounding |\n\nIn `local` mode the models are loaded by **one shared vision-service process per machine**\n(`vision_service.py`), not by every agent's server: the first `parse_screen` starts it, parallel\nagents queue on it (`queued_ms` in the result), and it exits after 15 idle minutes so a forgotten\nservice never pins the GPU. If it can't be reached the server loads the models in-process instead —\nthe service is an optimization, never a new failure mode. `status()` shows it under\n`vision.omniparser.service`; `python vision_service.py status|stop` from the shell.\n\nAn explicitly requested `basic`/`local` that is not usable **fails `parse_screen` with the fix named**\n(OmniParser not installed, no/too-small GPU, GPU out of memory, missing tesseract language data) —\nit never silently falls back; only `auto` does, and it says so. Tesseract languages default to\n`slv+eng` filtered to what is installed (`GLOVEBOX_TESS_LANG` overrides). Switch anytime with\n`./install.sh <mode>` (installs only what that mode needs).\n\n### Multi-instance (a fleet of app windows)\n\nEvery control tool takes **`instance=N`** (default 1 = the `start-display.sh` sandbox). Spin up more —\neach its own Xephyr display/window on the host desktop:\n\n- `launch_app(command, name?, size?)` → starts the next free `:N` running **any** GUI app\n  (`chromium`, `gimp`, `inkscape`, `xterm`, …). Chromium-family browsers auto-get X11 flags, a\n  per-instance profile, a remote-debugging port, and downloads routed to `files/<N>/`; other GTK apps\n  get D-Bus isolation. Returns the instance id, size and any warnings.\n- `list_instances()` · `close_instance(n)` · `app_log(n)`.\n\nBecause each display has its **own cursor**, multiple agents can drive different instances **in parallel** —\none window each. Each agent's server process allocates display numbers under a **cross-process lock**,\nso simultaneous launches don't collide. The GPU is shared through the vision service (one model\nload, requests queue; see above). The host display for new windows is `GLOVEBOX_HOST_DISPLAY`\n(default `:0`); `XAUTHORITY` is auto-discovered. Instances **outlive the server process** by default\n(another agent or session can pick them up via `list_instances`); set `GLOVEBOX_CLEANUP_ON_EXIT=1` to\nclose the ones a server launched when it exits.\n\n### One-call action + observe\n\n`click` · `click_element` · `type_text` · `press_keys` · `scroll` · `drag` · `double_click` ·\n`upload_file` · `wait_for_stable` take **`observe`** (`none` default · `screenshot` · `parse`) and\nmost take **`settle_ms`**. With `observe=\"screenshot\"` the action returns its result **and** the\nresulting screen in a single call — no separate screenshot round-trip. Use `settle_ms` when you know\nthe delay, `wait_for_stable(observe=\"screenshot\")` when you don't. Default `none` keeps routine steps\ncheap; opt into `screenshot`/`parse` on the steps that change the page (navigations, submits).\n\n### Files & uploads\n\nEach instance gets a staging folder **`files/<N>/`** inside the install dir — a stable place to drop\nfiles for that instance (readable by native apps and, since it's under `$HOME`, by snap Chromium too).\n`list_files(instance)` shows the folder and its contents, with sizes and a `partial` flag for\nin-progress downloads.\n\n- **Browser** `<input type=file>` → **`upload_file(path, instance)`** (via CDP; `page_url_contains`\n  picks the tab when several are open). The nested file picker is invisible to automation and hangs\n  snap Chromium, so never click an upload button expecting a dialog.\n- **Native apps** (GIMP, Inkscape, editors) → **`open_file(path, instance, app=\"gimp\")`**, or just\n  drive the app's own **Open dialog** — unlike the browser's, it's a real visible window you can type a\n  path into (`Ctrl+L` in a GTK file chooser).\n- **Saving / downloads** → apps run as your user, so they can save anywhere you can write. `launch_app`\n  browser instances are pre-configured to **download and \"save as\" into `files/<N>/`**; point native\n  apps' Save dialogs there too, then `list_files(instance)` to see the results.\n\n### Environment variables\n\n| Variable | Default | Meaning |\n|---|---|---|\n| `GLOVEBOX_VISION` | `auto` (or `.vision-mode`) | `auto` · `none` · `basic` · `local` |\n| `GLOVEBOX_HOST_DISPLAY` | `:0` | host display where new Xephyr windows appear |\n| `GLOVEBOX_CLEANUP_ON_EXIT` | off | `1` → close the instances this server launched when it exits |\n| `GLOVEBOX_TESS_LANG` | `slv+eng` ∩ installed | tesseract language string for `basic` |\n| `GLOVEBOX_VISION_SERVICE` | `1` | `local` mode parses through the shared per-machine service; `0` = load models in every server process |\n| `GLOVEBOX_VISION_SOCKET` | `$XDG_RUNTIME_DIR/glovebox-vision.sock` | the service's Unix socket |\n| `GLOVEBOX_VISION_IDLE_MIN` | `15` | the service exits after this many idle minutes |\n| `GLOVEBOX_OCR_SCALE` | `2` | upscale factor before OCR in `basic` (1–4); UI text is too small for tesseract at 1x |\n| `DISPLAY` | `:1` | instance 1's display (set by `mcp-config.json`) |\n\n## Maintenance (`local` mode)\n\nThe OmniParser lifecycle lives in **`vision.py`** (the server owns it; `install.sh local` just calls\nit): `python vision.py install-omniparser` clones [OmniParser](https://github.com/microsoft/OmniParser),\ndownloads the v2 weights into `OmniParser/weights/`, and makes PaddleOCR optional (this uses easyocr).\n`python vision.py check` reports readiness. **`transformers` is pinned to 4.49.0** — newer releases\nbreak Florence-2's remote config. If you upgrade OmniParser manually, re-run `install-omniparser` to\nre-apply the patch.\n\n## Stop\n\n```bash\npkill Xephyr      # closes the sandbox (browser + WM + display)\n```\n\n## Safety\n\n- The agent's input and vision are **scoped to the sandbox display** — it does not see or control your real desktop.\n- The server process and the apps it launches run **as your user** (shell/file access, like any MCP\n  server); only the **GUI control** is sandboxed to the Xephyr window. Files the agent saves, opens or\n  uploads are your real files. For OS-level isolation from your files, run it inside a VM or container.\n- You can watch everything live and close it instantly with `pkill Xephyr`.\n- Automate responsibly — only sites and services you are authorized to use.\n\n## Files\n\n- `server.py` — the MCP server (all 23 tools).\n- `vision.py` — vision backends: selection rules, GPU preflight, tesseract grouping, OmniParser lifecycle (`check` / `install-omniparser` CLI).\n- `vision_service.py` — the shared OmniParser process (Unix socket, idle-exit, `status` / `stop` CLI).\n- `install.sh` — mode-aware installer (`none` / `basic` / `local`).\n- `start-display.sh` — launches the Xephyr sandbox (display + window manager + browser).\n- `AGENTS.md` — drop-in tool-usage instructions for the AI agent (paste into its system prompt).\n- `ROADMAP.md` — what is planned next (MCP SDK 2.x migration, …).\n- `mcp-config.json` — a ready-to-paste MCP client config snippet.\n\n## Credits\n\n`local` vision mode uses Microsoft's [OmniParser](https://github.com/microsoft/OmniParser)\n(cloned and weights downloaded by `vision.py`, under its own license). Screen capture uses\n[mss](https://github.com/BoboTiG/python-mss); input is driven with\n[xdotool](https://github.com/jordansissel/xdotool). Not affiliated with Microsoft.\n\n## Contributing\n\nShipped **as-is** under MIT. Issues and PRs are welcome, but this is maintained by one person —\n**no support or response time is guaranteed.** If it's useful to you, a ⭐ helps.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n"}