{"id":"d6256bbd-ac84-4c3d-a39a-ec85a75877c0","slug":"windows-computer-use","kind":"mcp","name":"windows-computer-use","summary":"Full-control local Windows computer-use plugin for Codex: UIA3, MCP, SendInput, capture, and OCR.","owner":"BraveCowNoFear","repoUrl":"https://github.com/BraveCowNoFear/windows-computer-use","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-07-29T04:53:07.000Z","addedAt":"2026-10-21T09:24:09.099Z","logoUrl":"https://avatars.githubusercontent.com/u/105468915?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["files-documents"],"tags":["codex","computer-use","desktop-automation","dotnet","uiautomation","windows"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-23T03:58:04.487Z","sample":false,"url":"https://conduits.dev/tools/windows-computer-use","language":"C#","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Windows Computer Use\n\n[简体中文](README.zh-CN.md)\n\nA local Codex plugin for controlling Windows like a human, with semantic UI Automation first and native mouse, keyboard, capture, and OCR fallbacks. It is delivered as the combination the platform is designed for: a Codex skill, a local MCP server, and a native C# Windows broker.\n\nThe plugin runs in **full-control mode**. It does not maintain its own app allowlist, action blocklist, or confirmation matrix. It can address any interactive desktop window available to the current Windows user. Codex host permissions, process integrity, UAC secure desktop, and Windows policy remain outside the plugin and cannot be bypassed by it.\n\n## What works today\n\n- UI Automation 3 inspection through FlaUI, including names, AutomationIds, types, bounds, focus, Value/read-only/selection/toggle/expand/scroll states, document text, and selected text.\n- Hierarchical UIA descriptors with parent/depth/child metadata, path-stable control ids, automatic stale-element re-location, and incremental observation diffs.\n- Semantic `invoke`, explicit `perform_secondary_action` focus/selection/toggle/expand/collapse/scroll actions, and Unicode `enter_text`, with native SendInput fallback and automatic before/after visual evidence.\n- Physical pointer-position observation, smooth move/hover, five-button mouse click/drag/wheel, tracked cross-action mouse-down/mouse-up holds, repeated keyboard chords with implied modifiers, tracked key-down/key-up holds, app launching with automatic desktop visual evidence, window activation, and virtual-desktop coordinates.\n- Physical multi-monitor topology with per-display bounds, work area, effective DPI, primary flag, and scale percentage.\n- Picker-free window PNG capture through native Windows Graphics Capture plus full virtual-desktop screen copy; both return fresh screenshot ids that can ground later physical input.\n- Actionable `capture_region` can crop the exact pixels of an observed screenshot id (including nested crops), or acquire a fresh window/desktop frame, while retaining full-source identity for physical input and region-scoped visual waits.\n- DWM-visible-frame alignment and explicit `window` / `screen` / `screenshot` coordinate spaces, so WGC pixels map back to physical input without invisible-border offset.\n- Native `Windows.Media.Ocr` with line/word bounds; `ocr` and `find_text` can consume exact cached full/region screenshot pixels or acquire a fresh frame, preserving one screenshot identity through OCR-to-click workflows.\n- Local PNG/JPEG template matching over exact cached full/region screenshots or fresh frames, with bounded 0.25x-4x multi-scale search, coarse-to-fine sampled color scoring, cross-scale overlap suppression, and same-frame actionable coordinates.\n- Native Windows OLE clipboard text read/write plus session-local, all-direct-format backup tokens; atomic `paste_text` and `copy_text` use real Ctrl+V/Ctrl+C, semantic selection/Value verification, clipboard sequence tracking, restoration on both success and failure, and automatic before/after visual evidence.\n- Owned-window/root-owner metadata, `wait_for_window` for transient dialogs, and verified activate/minimize/maximize/restore/move/resize control with automatic whole-desktop visual evidence.\n- State-conditional waits instead of blind sleeps: semantic predicates for UIA, exact-PNG change/stability waits for pixel-only transitions, plus `compare_screenshots` for changed-pixel counts, exact union bounds, and tile-connected local regions between same-source frames.\n- Atomic UIA + image `snapshot` observations, timestamped screenshot ids and SHA-256 hashes, plus stale-coordinate rejection after a window moves, resizes, or ages out.\n- A local stdio MCP with 42 tools and a current-user named-pipe broker.\n- Compatibility with the global UI lock from `desktop-control-for-windows`.\n- A real WinForms end-to-end test covering MCP handshake, UIA discovery, rich state diffs, selected text, secondary actions, Chinese input, successful and deliberately failed atomic paste/copy restoration, semantic invoke, condition wait, window/desktop capture, desktop OCR-to-click, persistent key/mouse gestures, and cleanup.\n\nA local visual-language model is not yet implemented. Bounded multi-scale template matching covers known non-text targets across DPI/zoom variance; OCR and model-side image reasoning remain the interpretation layer for novel visuals and rotation. The extended compatibility gate covers Word, Excel, VS Code/Electron, WeChat, and SolidWorks; true multi-monitor/mixed-DPI and elevated-boundary runs still require suitable hardware/process state.\n\n## Architecture\n\n```mermaid\nflowchart LR\n    A[\"Codex task\"] --> B[\"windows-computer-use skill\"]\n    B --> C[\"Local stdio MCP\"]\n    C -->|\"current-user named pipe\"| D[\"C# native broker\"]\n    D --> E[\"UIA3 semantic tree\"]\n    D --> F[\"Windows Graphics Capture + Win32\"]\n    D --> G[\"Windows.Media.Ocr\"]\n    D --> H[\"SendInput mouse and keyboard\"]\n    E --> I[\"Target Windows app\"]\n    F --> I\n    G --> I\n    H --> I\n```\n\nThe calling agent chooses the highest reliable layer: application API/browser DOM, UIA3, OCR, then physical pixels. The broker activates the exact selected window, serializes live UI access through the shared lock, performs the action, and re-observes the result. See [architecture.md](plugins/windows-computer-use/docs/architecture.md) and the [tool reference](plugins/windows-computer-use/skills/windows-computer-use/references/tool-reference.md).\n\n## Build and test\n\nRequirements: Windows 10/11, PowerShell 5.1+, and the .NET 8 SDK. The repository build script also detects the SDK installed at `C:\\Users\\Clr\\.codex\\tools\\dotnet-sdk-8` on the development machine.\n\n```powershell\ncd \"C:\\path\\to\\windows-computer-use\\plugins\\windows-computer-use\"\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\test.ps1\n# New keyboard visual paths only (requires an existing Release build):\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario KeyboardVisual -RequireWgc\n# New semantic-action visual paths only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario SemanticVisual -RequireWgc\n# New atomic clipboard visual paths only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario ClipboardVisual -RequireWgc\n# New window-management visual paths only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario WindowVisual -RequireWgc\n# New application-launch visual path only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario LaunchVisual -RequireWgc\n# New automatic click visual paths only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario ClickVisual -RequireWgc\n# New automatic scroll/drag visual paths only:\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\e2e-test.ps1 -Scenario MotionVisual -RequireWgc\n```\n\n`test.ps1` first validates the Codex manifest, marketplace path, MCP launcher, skill/assets, and all PowerShell syntax. It then restores dependencies, builds and publishes the broker/MCP, runs the xUnit suite, opens the isolated WinForms test window, drives it through the real MCP, hard-gates hierarchical UIA/state diffs, Value and selected-text exposure, semantic secondary actions, actionable window/desktop region crops, region-scoped PNG change and continuous-stability waits, selected-window plus unchanged-current-foreground keyboard input, held/implied-modifier state, raw plus atomic Ctrl+V/Ctrl+C round trips (including forced paste/copy failure cleanup), Windows Graphics Capture, OCR and image-template grounding, snapshot freshness, stale-coordinate rejection, and owned-artifact cleanup, closes the test app, and exits non-zero on any failed gate. A separate fault-injection gate forces a 2000 ms broker deadline against a 5000 ms native wait, proves the native process is replaced without terminating MCP, releases a staged Ctrl/mouse hold, and verifies the next tool call. Redirected MCP diagnostics are drained asynchronously so expected-error coverage cannot fill the stderr pipe and deadlock the harness.\n\nFor a non-destructive real-app compatibility pass, run `scripts/real-app-smoke.ps1` after building. It opens isolated Notepad, File Explorer, and Settings windows, inspects/captures/OCRs them, and closes only windows whose ids were absent before the test.\n\nRun `scripts/extended-app-smoke.ps1` for isolated Word, Excel, and VS Code/Electron coverage. Add `-IncludeWeChat -IncludeSolidWorks` to cover those installed apps; the script skips any app family that was already running, reads UIA/WGC/OCR only, and closes only process groups created by the test. `-OptionalOnly` runs just the optional apps.\n\n## Local Codex plugin test\n\nBuild once, then add this repository root as a local marketplace:\n\n```powershell\ncd \"C:\\path\\to\\windows-computer-use\\plugins\\windows-computer-use\"\npowershell -NoProfile -ExecutionPolicy Bypass -File .\\scripts\\build.ps1\n\ncd ..\\..\ncodex plugin marketplace add .\n```\n\nInstall **Windows Computer Use** from the local **Brave Cow Windows Tools** marketplace in Codex. The plugin's `.mcp.json` starts `scripts/run-mcp.ps1`, which launches the local MCP and native broker. For protocol-only testing, start that script directly and send newline-delimited MCP JSON-RPC on stdin.\n\nBroker calls have a 180-second transport deadline so a stuck native UI provider cannot permanently wedge the MCP. Set `WCU_BROKER_CALL_TIMEOUT_MS` to another non-negative millisecond value, or `0` to disable the deadline completely. A timeout is never auto-replayed: the MCP replaces the broker, releases plugin-tracked held input, returns an explicit unknown-state error, and requires a fresh observation before another action.\n\n## MCP surface\n\nThe 42 tools are `list_windows`, `display_info`, `pointer_position`, `window_from_point`, `launch_app`, `wait_for_window`, `inspect_window`, `observe_changes`, `find_controls`, `invoke`, `perform_secondary_action`, `enter_text`, `paste_text`, `copy_text`, `wait_for_ui`, `wait_for_visual_change`, `wait_for_visual_stable`, `compare_screenshots`, `capture`, `capture_region`, `observe_desktop`, `snapshot`, `ocr`, `find_text`, `find_image`, `read_clipboard_text`, `write_clipboard_text`, `restore_clipboard`, `move_pointer`, `click`, `mouse_down`, `mouse_up`, `press_key`, `key_down`, `key_up`, `type_text`, `scroll`, `drag`, `set_window_state`, `set_window_bounds`, `activate_window`, and `end_session`.\n\nUse the exact window id returned by `list_windows`; the broker rehydrates that HWND directly while it is temporarily hidden or untitled. If an app recreates the HWND, the old id follows the unique replacement only when process id, native class, and non-empty title still match; otherwise it fails closed and requires a fresh selection. Use `wait_for_window` plus `owner_window_id` for transient dialogs. Prefer stable control ids from `inspect_window`/`find_controls`, inspect their current state fields, and pass an earlier `observation_id` to `observe_changes` after transitions. Use `perform_secondary_action` when the intended UIA action is more specific than a primary invoke. Otherwise use `find_text` for text, `find_image` for a known local template (optionally with a bounded scale range), or `snapshot` for model vision, then pass the returned `screenshot_id` with `coordinate_space: \"screenshot\"`. `capture`, `ocr`, `find_text`, and `find_image` accept `desktop: true`; their virtual-desktop screenshot ids can drive move/click/mouse-down/mouse-up/scroll/drag without selecting or foregrounding a window, and explicit `coordinate_space: \"screen\"` supports direct full-desktop input. Semantic/input actions invalidate older screenshots; the broker rejects ids if a target moved/resized, display topology changed, or `max_age_ms` elapsed. Restore minimized windows before window-scoped visual observation. Legacy selected-window coordinates remain physical window-relative pixels by default.\n\nEvery `move_pointer` invalidates prior screenshots and returns a fresh full-source `after_screenshot_id`; a screenshot-bound hover also returns top-level `visual_diff`. Every `click`, `scroll`, and self-contained `drag` now returns `data.visual_diff`: each reuses an authoritative screenshot when supplied, automatically captures the activated window before an unbound window action, or captures the whole desktop before a direct-screen action. Screenshot-bound `mouse_down` and `mouse_up` return equivalent evidence. It includes comparability, changed count/fraction, maximum channel delta, exact image/screen union bounds, and up to 20 localized regions. Chain each newest id through `mouse_down` → `move_pointer` → `mouse_up` to keep a long gesture grounded while held-button state remains tracked. If source geometry changes, the completed input remains successful but reports `comparable=false` with before/after bounds. Direct-screen or unbound moves still have no trusted baseline, so `visual_diff=null`, but they return a fresh post-action desktop/window id.\n\nFor whole-screen work, `observe_desktop` replaces four round trips with one read-only observation: it returns the virtual-desktop PNG plus monitor topology, visible top-level windows, and pointer position. Its desktop screenshot id is immediately actionable with screenshot-space pointer/mouse tools and it never selects or activates a window.\n\nWhen no reliable semantic predicate exists, pass a fresh window or desktop `screenshot_id` to `wait_for_visual_change`. It repeatedly captures that exact source, rejects stale geometry/topology, compares exact PNG SHA-256 content, and returns a new actionable PNG whether it matched or timed out. Prefer `wait_for_ui` when possible: animations, clocks, blinking carets, and any unrelated desktop pixel can legitimately satisfy an exact visual-change wait.\n\nAfter a change starts, `wait_for_visual_stable` requires fresh captures to keep the same exact PNG hash for `stable_ms` continuously. A changed sample resets the timer; a late or impossible-to-stabilize source returns `stable: false` plus the latest image at `timeout_ms`. It observes stability only after the call begins, not from the age of the source screenshot. Dynamic clocks, carets, video, and desktop notifications can prevent full-frame stability.\n\nUse `compare_screenshots` when a hash change must be localized. Both fresh ids must represent the same window/desktop source, geometry, and exact crop. The broker compares decoded BGRA pixels, returns exact changed-pixel count/fraction and union bounds, then groups changed tiles into bounded local regions with image and physical screen bounds. `channel_threshold=0` is exact; raise it only to ignore known minor rendering noise. The tool does not capture, activate, or send input.\n\nUse `capture_region` when only one image-relative rectangle matters. Pass a fresh `screenshot_id` to crop the exact already-observed pixels without a second capture; that id is authoritative, so window/desktop selectors cannot be mixed into the call. Without an id, the tool acquires a fresh selected window or desktop frame. `x/y/width/height` are relative to the chosen source image, including an earlier crop. The returned PNG reports physical `bounds`, `(0,0)` maps to that physical origin, and nested crops accumulate the original full-source offset. Visual change/stability waits re-capture only the final region while still rejecting full window geometry or desktop-topology drift.\n\nPass that same fresh full or region `screenshot_id` to `ocr` or `find_text` to recognize exactly the pixels already returned to the model. The broker validates age and original geometry/topology, writes only a temporary local PNG for the Windows OCR adapter, deletes it after recognition, and returns the unchanged screenshot id/bounds/hash. `find_text` centers therefore drive screenshot-space input against the same frame. Cached sources are mutually exclusive with `desktop`/window selectors; `ocr path` is a separate authoritative, non-actionable existing-image mode.\n\n`find_image` accepts the same cached screenshot contract without writing it to disk: the local matcher decodes the cached PNG in memory, searches exact or bounded scales, and returns centers plus physical bounds bound to the unchanged screenshot id/hash. This is the preferred template path after `snapshot`, `observe_desktop`, or `capture_region`; omit the id only when a deliberately fresh frame is required.\n\nBefore acting on an ambiguous screen pixel, `window_from_point` can map physical x/y to both the actual native child HWND/class/title under that pixel and its top-level root window descriptor. This gives the agent a stable `window_id` for UIA inspection or confirms which app a later mouse action would reach, without moving the pointer, activating, or clicking.\n\n`press_key`, `key_down`, `key_up`, and `type_text` capture before and after every input and return `data.after_screenshot_id`, `data.visual_changed`, and exact `data.visual_diff` while retaining their focus/held-key verification. Window mode compares the selected window; `desktop: true` sends directly to the unchanged current foreground and compares the whole virtual desktop, so it is mutually exclusive with `window_id`, `title`, and `app`. Prefer explicit window mode when the destination is known, and use desktop mode for system shortcuts or an already-established foreground/focus chain. A desktop `key_up` still sends the release if Windows temporarily has no foreground window.\n\n`invoke`, `perform_secondary_action`, and `enter_text` preserve their UIA re-observation verification while also returning `data.after_screenshot_id`, `data.visual_changed`, and exact `data.visual_diff` for the selected window. If a completed semantic action closes its source window, the action remains successful and returns a fresh virtual-desktop screenshot with `comparable=false` and `reason=source-window-unavailable`, so the caller can continue from a real post-action observation.\n\n`launch_app` runs under the same shared UI lock as other desktop mutations, waits for initial input readiness up to `wait_ms`, and returns its process id plus a fresh whole-desktop `after_screenshot_id`, `visual_changed`, and exact `visual_diff`. Use `wait_for_window` with that process id when the next step requires a specific top-level window; visual change alone is evidence of the observed desktop, not proof that a particular window is ready.\n\n`activate_window`, `set_window_state`, and `set_window_bounds` capture the whole virtual desktop before and after their Win32 operation and return a fresh top-level or `data.after_screenshot_id` with exact visual-diff evidence while retaining native foreground/state/bounds verification. The stable desktop coordinate system keeps minimize and geometry changes comparable. Bounds use physical virtual-desktop pixels rather than title-bar drag guesses; negative origins are accepted for left/upper monitors. A bounds change restores minimized/maximized windows first and preserves the existing foreground unless `activate: true` is requested. True multi-monitor placement remains an external hardware-matrix item on this single-monitor development machine.\n\nFor transfer-only workflows, prefer `paste_text`/`copy_text`. Paste replaces or appends and waits for observable UIA Value; copy preserves the current selection or selects all, waits for a real clipboard sequence change, and performs at most one semantic-refocus retry when publication is missing or disagrees with the UIA selection. It then returns Unicode text and restores. Both preserve every direct format and restore before returning—even after verification or input failure—and now return `data.after_screenshot_id`, `data.visual_changed`, and exact `data.visual_diff` without weakening their clipboard/UIA verification. Clipboard publication/restoration tolerates bounded external ownership contention for up to ten seconds. Use the lower-level `write_clipboard_text`/`restore_clipboard` pair only when clipboard state must span multiple actions. Backup ids are broker-session-local; unsupported formats fail before mutation.\n\n## Repository layout\n\n```text\n.agents/plugins/marketplace.json       local marketplace\nplugins/windows-computer-use/\n  .codex-plugin/plugin.json            Codex plugin manifest\n  .mcp.json                            local MCP registration\n  skills/windows-computer-use/         agent workflow and references\n  src/WindowsComputerUse.Mcp/          MCP stdio host\n  src/WindowsComputerUse.Broker/       UIA3/Win32/SendInput broker\n  src/WindowsComputerUse.TestApp/      deterministic real-UI fixture\n  scripts/                             build, launch, OCR, and E2E gates\n  tests/                               protocol and native smoke tests\n```\n\n## License\n\nMIT\n"}