{"id":"35b5f659-1d08-4aee-b92d-0c60a5564753","slug":"usehid","kind":"skill","name":"usehid","summary":"🎮 Cross-platform virtual HID device library for AI agents","owner":"jiusanzhou","repoUrl":"https://github.com/jiusanzhou/usehid","license":"MIT","licenseTier":"permissive","trust":"community","stars":9,"match":null,"fewStars":false,"format":"skill","createdAt":"2026-02-20T07:48:25.000Z","addedAt":"2026-10-21T22:35:55.985Z","logoUrl":"https://avatars.githubusercontent.com/u/2942042?u=12a39dc0206de177c60ef6034b5d07d24177fc82&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["writing"],"tags":["zoe-lab"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T01:03:08.834Z","sample":false,"url":"https://conduits.dev/tools/usehid","language":"Rust","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# 🎮 useHID\n\n> Cross-platform virtual HID device library for AI agents\n\n[![Crates.io](https://img.shields.io/crates/v/usehid.svg)](https://crates.io/crates/usehid)\n[![PyPI](https://img.shields.io/pypi/v/usehid.svg)](https://pypi.org/project/usehid/)\n[![npm](https://img.shields.io/npm/v/usehid.svg)](https://www.npmjs.com/package/usehid)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![Platform](https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-lightgrey.svg)](#platform-support)\n\n**useHID** lets AI agents control computers like humans do — moving the mouse, typing on the keyboard, pressing gamepad buttons, taking screenshots, and reading UI elements. Built in Rust with bindings for Python, TypeScript, and Go.\n\n```python\nfrom usehid import AgentHID\n\nagent = AgentHID()\nagent.execute({\"action\": \"key_combo\", \"modifiers\": [\"cmd\"], \"key\": \"space\"})\nagent.execute({\"action\": \"type\", \"text\": \"Google Chrome\"})\nagent.execute({\"action\": \"key_press\", \"key\": \"enter\"})\n```\n\n---\n\n## ✨ Features\n\n- 🖱️ **Virtual Mouse** — Move, click, double-click, drag, scroll\n- ⌨️ **Virtual Keyboard** — Type text, press keys, key combinations\n- 🎮 **Virtual Gamepad** — Joysticks, buttons, triggers\n- 📸 **Screenshot** — Capture full screen or regions as PNG\n- 🌳 **UI Tree / Accessibility** — Read the OS accessibility tree (elements, roles, bounds)\n- 🤖 **Agent-Ready API** — Simple JSON interface for LLM agents\n- 🦀 **Pure Rust Core** — Safe, fast, zero-copy where possible\n- 🐍 **Multi-language** — Python, TypeScript, Go bindings\n\n---\n\n## 🚀 Quick Start\n\n### For AI Agents (Recommended)\n\nThe `AgentHID` class provides a JSON-based interface perfect for LLM tool-calling:\n\n```python\nfrom usehid import AgentHID\n\nagent = AgentHID()\n\n# Query screen info\nresult = agent.execute({\"action\": \"size\"})        # Returns: {width: 1920, height: 1080}\nresult = agent.execute({\"action\": \"position\"})    # Returns: {x: 100, y: 200}\n\n# Screenshot\nresult = agent.execute({\"action\": \"screenshot\"})  # Returns: {data: \"<base64 PNG>\"}\nresult = agent.execute({\"action\": \"screenshot_region\", \"x\": 0, \"y\": 0, \"width\": 400, \"height\": 300})\n\n# UI Tree (Accessibility)\nresult = agent.execute({\"action\": \"get_ui_tree\", \"depth\": 3})  # Returns: {tree: {role, title, children, ...}}\nresult = agent.execute({\"action\": \"find_ui_element\", \"role\": \"AXButton\", \"title\": \"Submit\"})\n\n# Mouse actions\nagent.execute({\"action\": \"mouse_move_to\", \"x\": 500, \"y\": 300})  # Absolute move\nagent.execute({\"action\": \"mouse_move\", \"x\": 100, \"y\": 50})      # Relative move\nagent.execute({\"action\": \"mouse_click\", \"button\": \"left\"})\nagent.execute({\"action\": \"mouse_drag_to\", \"x\": 600, \"y\": 400})  # Drag to position\nagent.execute({\"action\": \"mouse_scroll\", \"delta\": -3})\n\n# Keyboard actions\nagent.execute({\"action\": \"type\", \"text\": \"Hello, World!\"})\nagent.execute({\"action\": \"key_press\", \"key\": \"enter\"})\nagent.execute({\"action\": \"key_combo\", \"modifiers\": [\"ctrl\", \"shift\"], \"key\": \"s\"})\n```\n\n### Direct Functions (Python)\n\n```python\nimport usehid\n\n# Screen info\nwidth, height = usehid.size()       # Get screen dimensions\nx, y = usehid.position()            # Get mouse position\nusehid.move_to(500, 300)            # Move mouse to absolute position\n\n# Screenshot\npng_bytes = usehid.screenshot()              # Full screen → PNG bytes\npng_bytes = usehid.screenshot_region(0, 0, 800, 600)  # Region → PNG bytes\n\n# Accessibility / UI Tree\ntree = usehid.get_ui_tree(depth=3)           # Returns dict with role, title, children, bounds...\nelements = usehid.find_ui_element(role=\"AXButton\")  # Find matching elements\n```\n\n### Direct Device Control\n\nFor more control, use the device classes directly:\n\n```python\nfrom usehid import Mouse, Keyboard\n\nmouse = Mouse()\nmouse.move_by(100, 50)\nmouse.click(\"left\")\nmouse.drag(200, 100)   # Drag by relative offset\nmouse.scroll(-3)\n\nkeyboard = Keyboard()\nkeyboard.type_text(\"Hello!\")\nkeyboard.combo([\"cmd\"], \"s\")  # Cmd+S to save\n```\n\n---\n\n## 📋 Supported Actions\n\n### Screen/Query\n\n| Action | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `size` | — | `width`, `height` | Get screen dimensions |\n| `position` | — | `x`, `y` | Get current mouse position |\n\n### Screenshot\n\n| Action | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `screenshot` | — | `data` (base64 PNG) | Capture the full screen |\n| `screenshot_region` | `x`, `y`, `width`, `height` | `data` (base64 PNG) | Capture a screen region |\n\n**Usage:**\n```python\n# Full screen\nresult = agent.execute({\"action\": \"screenshot\"})\npng_base64 = result[\"data\"]\n\n# Region\nresult = agent.execute({\"action\": \"screenshot_region\", \"x\": 100, \"y\": 100, \"width\": 400, \"height\": 300})\n```\n\n**Platform backends:** macOS (CGWindowListCreateImage), Linux (ImageMagick `import`), Windows (GDI BitBlt)\n\n### Accessibility / UI Tree\n\n| Action | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `get_ui_tree` | `depth`?, `app`? | `tree` (UIElement) | Get accessibility tree from focused/named window |\n| `find_ui_element` | `role`?, `title`? | `elements` (UIElement[]) | Find matching elements in the focused window |\n\n**UIElement structure:**\n```json\n{\n  \"role\": \"AXButton\",\n  \"title\": \"Submit\",\n  \"value\": null,\n  \"description\": \"Submit the form\",\n  \"bounds\": {\"x\": 200, \"y\": 150, \"width\": 80, \"height\": 32},\n  \"children\": [],\n  \"actions\": [\"AXPress\"]\n}\n```\n\n**Usage:**\n```python\n# Get full UI tree (depth-limited)\nresult = agent.execute({\"action\": \"get_ui_tree\", \"depth\": 3})\ntree = result[\"tree\"]\n\n# Get UI tree for a specific app\nresult = agent.execute({\"action\": \"get_ui_tree\", \"app\": \"Safari\"})\n\n# Find all buttons\nresult = agent.execute({\"action\": \"find_ui_element\", \"role\": \"AXButton\"})\nbuttons = result[\"elements\"]\n\n# Find element by title\nresult = agent.execute({\"action\": \"find_ui_element\", \"title\": \"Save\"})\n```\n\n**Platform backends:** macOS (AXUIElement API), Linux (xdotool), Windows (UI Automation)\n\n> ⚠️ Requires **Accessibility** permissions on macOS (System Settings → Privacy & Security → Accessibility)\n\n### Mouse\n\n| Action | Parameters | Description |\n|--------|------------|-------------|\n| `mouse_move` | `x`, `y`, `duration`?, `tween`? | Move by relative offset |\n| `mouse_move_to` | `x`, `y`, `duration`?, `tween`? | Move to absolute position |\n| `mouse_click` | `button`? | Click (left/right/middle) |\n| `mouse_double_click` | `button`? | Double click |\n| `mouse_down` | `button`? | Press and hold |\n| `mouse_up` | `button`? | Release |\n| `mouse_scroll` | `delta` | Scroll wheel (+up/-down) |\n| `mouse_drag` | `x`, `y`, `button`?, `duration`?, `tween`? | Drag by relative offset |\n| `mouse_drag_to` | `x`, `y`, `button`?, `duration`?, `tween`? | Drag to absolute position |\n\n### Keyboard\n\n| Action | Parameters | Description |\n|--------|------------|-------------|\n| `type` | `text`, `interval`? | Type a string (interval in ms) |\n| `key_press` | `key` | Press and release a key |\n| `key_down` | `key` | Press and hold |\n| `key_up` | `key` | Release |\n| `key_combo` | `modifiers[]`, `key` | Key combination |\n\n**Supported modifiers:** `ctrl`, `shift`, `alt`, `cmd`/`meta`/`win`\n\n**Supported keys:** `a-z`, `0-9`, `enter`, `escape`, `backspace`, `tab`, `space`, `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`, `delete`, `insert`, `f1-f12`\n\n### Tween Functions\n\nFor smooth movement animations, use the `tween` parameter:\n\n| Tween | Description |\n|-------|-------------|\n| `linear` | Constant speed (default if duration=0) |\n| `ease_in` / `ease_in_quad` | Slow start |\n| `ease_out` / `ease_out_quad` | Slow end |\n| `ease_in_out` / `ease_in_out_quad` | Slow start and end (default) |\n| `ease_in_cubic` | Cubic slow start |\n| `ease_out_cubic` | Cubic slow end |\n| `ease_in_out_cubic` | Cubic slow start and end |\n| `ease_out_elastic` | Elastic bounce effect |\n| `ease_out_bounce` | Bounce effect |\n\n### Failsafe\n\nEmergency stop mechanism - moving mouse to any screen corner triggers failsafe and blocks further actions.\n\n| Action | Returns | Description |\n|--------|---------|-------------|\n| `failsafe_status` | `enabled`, `triggered` | Check failsafe state |\n| `failsafe_enable` | — | Enable failsafe (default) |\n| `failsafe_disable` | — | Disable failsafe |\n| `failsafe_reset` | — | Reset triggered state |\n\n**Usage:**\n```python\n# Check status\nresult = agent.execute({\"action\": \"failsafe_status\"})\n# {success: true, enabled: true, triggered: false}\n\n# If triggered, reset to continue\nagent.execute({\"action\": \"failsafe_reset\"})\n\n# Disable for specific operations (use with caution!)\nagent.execute({\"action\": \"failsafe_disable\"})\n# ... operations ...\nagent.execute({\"action\": \"failsafe_enable\"})\n```\n\n### Gamepad\n\n| Action | Parameters | Description |\n|--------|------------|-------------|\n| `gamepad_press` | `button` | Press button |\n| `gamepad_release` | `button` | Release button |\n| `gamepad_left_stick` | `x`, `y` | Left stick (0-255) |\n| `gamepad_right_stick` | `x`, `y` | Right stick (0-255) |\n| `gamepad_triggers` | `left`, `right` | Triggers (0-255) |\n\n**Buttons:** `a`, `b`, `x`, `y`, `lb`, `rb`, `back`, `start`, `guide`, `left_stick`, `right_stick`, `dpad_up`, `dpad_down`, `dpad_left`, `dpad_right`\n\n---\n\n## 🖥️ Platform Support\n\n| Platform | Mouse | Keyboard | Gamepad | Screenshot | UI Tree | Backend |\n|----------|:-----:|:--------:|:-------:|:----------:|:-------:|---------|\n| **macOS** | ✅ | ✅ | ⚠️ | ✅ | ✅ | CGEvent / CoreGraphics / AXUIElement |\n| **Linux** | ✅ | ✅ | ✅ | ✅ | ⚠️ | uhid / ImageMagick / xdotool |\n| **Windows** | ✅ | ✅ | ⚠️ | ✅ | ✅ | SendInput / GDI / UI Automation |\n\n> ⚠️ Gamepad notes:\n> - macOS: Requires code signing with `com.apple.developer.hid.virtual.device` entitlement\n> - Windows: Requires [ViGEmBus](https://github.com/ViGEm/ViGEmBus) driver\n\n### macOS Permissions\n\nuseHID requires **Accessibility** permissions for HID control and UI tree access:\n\n1. Go to **System Settings → Privacy & Security → Accessibility**\n2. Add your terminal app or the application using useHID\n\n> Screenshot capture works without Accessibility permissions.\n> UI Tree / Accessibility features require the permission to read element attributes.\n\n### Windows\n\nNo special permissions required. Uses Win32 `SendInput` API for mouse and keyboard.\n\n---\n\n## 📦 Installation\n\n### Rust\n\n```toml\n[dependencies]\nusehid = \"0.1\"\n```\n\n### Python\n\n```bash\npip install usehid\n```\n\n### TypeScript / Node.js\n\n```bash\nnpm install usehid\n```\n\n### Go\n\n```bash\ngo get go.zoe.im/usehid@latest\n```\n\n```go\nimport usehid \"go.zoe.im/usehid\"\n```\n\n---\n\n## 🧪 Examples\n\n### Open Chrome and Browse (Agent Test)\n\n```bash\ncargo run --release -p usehid --example agent_browse\n```\n\nThis example demonstrates an AI agent:\n1. Opening Spotlight with `Cmd+Space`\n2. Typing \"Google Chrome\" and pressing Enter\n3. Opening a new tab with `Cmd+T`\n4. Navigating to github.com\n5. Scrolling the page\n6. Performing a search\n\n### Basic Mouse Control\n\n```bash\ncargo run --release -p usehid --example mouse\n```\n\n### Keyboard Typing\n\n```bash\ncargo run --release -p usehid --example keyboard\n```\n\n### Screenshot Capture\n\n```bash\ncargo run --release -p usehid --example screenshot\n```\n\n### UI Tree (Accessibility)\n\n```bash\ncargo run --release -p usehid --example ui_tree\n```\n\n---\n\n## 🏗️ Architecture\n\n```\nusehid/\n├── usehid-core/          # Rust core library\n│   ├── src/\n│   │   ├── lib.rs            # Public API\n│   │   ├── mouse.rs          # Virtual mouse\n│   │   ├── keyboard.rs       # Virtual keyboard\n│   │   ├── gamepad.rs        # Virtual gamepad\n│   │   ├── screenshot.rs     # Screenshot capture (PNG)\n│   │   ├── accessibility.rs  # UI tree / accessibility API\n│   │   ├── agent.rs          # JSON API for agents\n│   │   ├── screen.rs         # Screen info & mouse position\n│   │   ├── failsafe.rs       # Emergency stop mechanism\n│   │   ├── tween.rs          # Smooth animation functions\n│   │   ├── hid.rs            # HID report descriptors\n│   │   └── platform/         # Platform backends\n│   │       ├── macos/        # CGEvent + IOHIDUserDevice\n│   │       ├── linux.rs      # uhid\n│   │       └── windows.rs\n│   └── examples/\n├── usehid-python/            # Python bindings (PyO3)\n├── usehid-node/              # Node.js bindings (napi-rs)\n└── usehid-go/                # Go bindings (CGO)\n```\n\n---\n\n## 🔧 Building from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/jiusanzhou/usehid\ncd usehid\n\n# Build the core library\ncargo build --release\n\n# Run tests\ncargo test\n\n# Build Python bindings\ncd usehid-python\nmaturin develop --release\n```\n\n---\n\n## 🤖 OpenClaw Integration\n\nuseHID can be integrated with [OpenClaw](https://github.com/openclaw/openclaw) to give AI agents full control over your computer.\n\n### Quick Setup\n\n1. **Install useHID:**\n```bash\npip install usehid\n```\n\n2. **Grant Permissions (macOS):**\nSystem Preferences → Security & Privacy → Privacy → Accessibility\n\n3. **Copy the skill to OpenClaw:**\n```bash\ncp -r skills/usehid ~/.openclaw/skills/\n```\n\n4. **Use it:**\n```\nYou: Click the Chrome icon on the dock\nOpenClaw: [executes mouse_click at dock position]\n\nYou: Type \"github.com\" and press enter  \nOpenClaw: [types text and presses enter]\n\nYou: Scroll down slowly\nOpenClaw: [executes mouse_scroll multiple times]\n```\n\n### Example Commands\n\n| User Request | useHID Action |\n|--------------|---------------|\n| \"Click here\" | `{\"action\": \"mouse_click\"}` |\n| \"Type hello world\" | `{\"action\": \"type\", \"text\": \"hello world\"}` |\n| \"Press Enter\" | `{\"action\": \"key_press\", \"key\": \"enter\"}` |\n| \"Save the file\" | `{\"action\": \"key_combo\", \"modifiers\": [\"cmd\"], \"key\": \"s\"}` |\n| \"Scroll down\" | `{\"action\": \"mouse_scroll\", \"delta\": -3}` |\n| \"Take a screenshot\" | `{\"action\": \"screenshot\"}` |\n| \"What's on screen?\" | `{\"action\": \"get_ui_tree\", \"depth\": 2}` |\n| \"Find the Submit button\" | `{\"action\": \"find_ui_element\", \"role\": \"AXButton\", \"title\": \"Submit\"}` |\n| \"Select all and copy\" | Two actions: Cmd+A then Cmd+C |\n\nSee [skills/usehid/SKILL.md](skills/usehid/SKILL.md) for full documentation.\n\n---\n\n## 🤝 Use Cases\n\n- **AI Agents** — Let LLMs control desktop applications\n- **Browser Automation** — Alternative to browser-specific APIs\n- **Game Bots** — Gamepad simulation for testing\n- **Accessibility Tools** — Alternative input methods\n- **Testing** — Automated UI testing without browser dependencies\n\n---\n\n## 🔄 Comparison with Other Libraries\n\n### Why useHID?\n\n| Feature | useHID | PyAutoGUI | enigo | autopilot-rs | RobotGo |\n|---------|:------:|:---------:|:-----:|:------------:|:-------:|\n| **Agent-Ready JSON API** | ✅ | ❌ | ❌ | ❌ | ❌ |\n| **Failsafe (Emergency Stop)** | ✅ | ✅ | ❌ | ❌ | ❌ |\n| **Multi-language Bindings** | Rust/Python/Node/Go | Python | Rust | Rust | Go |\n| **Gamepad Support** | ✅ | ❌ | ❌ | ❌ | ❌ |\n| **Smooth Tween Animations** | ✅ (10+ functions) | ✅ | ❌ | ❌ | ✅ |\n| **True HID-level Simulation** | ✅ (Linux uhid) | ❌ | ❌ | ❌ | ❌ |\n| **Zero C Dependencies** | ✅ (macOS/Windows) | ❌ | ❌ (libxdo) | ❌ | ❌ (CGO) |\n| **Screenshot** | ✅ | ✅ | ❌ | ✅ | ✅ |\n| **UI Tree / Accessibility** | ✅ | ❌ | ❌ | ❌ | ❌ |\n| **Image Recognition** | ❌ | ✅ | ❌ | ❌ | ✅ (OpenCV) |\n| **Window Control** | ❌ | ✅ (Win) | ❌ | ❌ | ✅ |\n\n### Key Advantages\n\n**🤖 Built for AI Agents**\n```python\n# Other libraries: function calls\npyautogui.moveTo(100, 200)\npyautogui.click()\n\n# useHID: JSON actions (perfect for LLM tool-calling)\nagent.execute({\"action\": \"mouse_move_to\", \"x\": 100, \"y\": 200})\nagent.execute({\"action\": \"mouse_click\"})\n```\n\n**🛡️ Safety First**\n- Built-in failsafe: move mouse to any screen corner to stop all automation\n- Query and control failsafe state via API\n- No accidental runaway scripts\n\n**🎮 Unique Gamepad Support**\n- Virtual gamepad for game automation and testing\n- Full controller emulation: buttons, sticks, triggers\n\n**🌐 True Multi-language**\n- Single Rust core with native bindings\n- No subprocess spawning or IPC overhead\n- Consistent API across Python, Node.js, and Go\n\n**⚡ Pure Rust Performance**\n- No C dependencies on macOS/Windows\n- Memory-safe by design\n- Fast and lightweight\n\n### When to Use Others\n\n| Library | Best For |\n|---------|----------|\n| **PyAutoGUI** | Screenshot + image recognition workflows |\n| **RobotGo** | Go projects needing OpenCV integration |\n| **enigo** | Rust projects needing Wayland support |\n| **autopilot-rs** | Simple Rust automation with screenshots |\n\n---\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n---\n\n## 🙏 Acknowledgments\n\n- Inspired by [foohid](https://github.com/unbit/foohid) for macOS virtual HID\n- Agent API design inspired by [browser-use](https://github.com/browser-use/browser-use)\n\n---\n\n<p align=\"center\">\n  Made with 🦀 by <a href=\"https://github.com/jiusanzhou\">Zoe</a>\n</p>\n"}