macos-ui-mcp

macOS UI MCP Server

A Model Context Protocol (MCP) server that provides AI agents with native macOS UI automation capabilities through the Accessibility API.

Author: Anton Starikov License: EUPL-1.2

Features

• Element Discovery: Navigate and inspect UI element hierarchies
• UI Actions: Click, type, set values, and perform accessibility actions
• Advanced Mouse: Scroll, right-click, double-click, hover, drag-and-drop
• Menu Navigation: Navigate menu paths and get menu structure
• Clipboard Operations: Read/write text and images to clipboard
• Window Management: Get/set window bounds, minimize, restore, raise
• Text Selection: Select text ranges, get selected text
• Element Inspection: Get all attributes, highlight elements, capture elements
• File Dialogs: Navigate, select files, set filename, confirm/cancel
• Application Management: Launch, activate, and quit applications
• Element Observation: Wait for elements and monitor state changes
• Keyboard Input: Type text and send keyboard shortcuts
• Screenshots: Capture windows, regions, or entire screens
• Permission Checking: Verify accessibility permissions

Requirements

• macOS 26.0 or later
• Swift 6.0+
• Accessibility permission granted in System Settings

Building

# Debug build
make build

# Release build (universal binary: ARM64 + x86_64)
make release

# Run tests (non-parallel to avoid UI conflicts)
make test

# Run the server
make run

# Install to /usr/local/bin
make install

# Show all available commands
make help

The universal release binary will be at .build/universal/macos-ui-mcp.

Granting Accessibility Permission

1. Open System Settings > Privacy & Security > Accessibility
2. Click the + button
3. Navigate to the built binary or your terminal application
4. Enable the toggle

MCP Configuration

Add to your Claude Code MCP configuration (~/.claude/claude_desktop_config.json):

{
  "mcpServers": {
    "macos-ui": {
      "command": "/path/to/macos-ui-mcp"
    }
  }
}

Available Tools

Element Discovery

Tool	Description
get_element_tree	Get UI element hierarchy for an application
find_elements	Search for elements by role, title, or identifier

UI Actions

Tool	Description
click	Click on an element
set_value	Set the value of an input element
perform_action	Perform an accessibility action (press, confirm, cancel, etc.)
click_and_type	Click an element then type text
click_and_wait	Click an element and wait for a condition

Application Management

Tool	Description
list_applications	List running GUI applications
launch_application	Launch an application by bundle ID
activate_application	Bring an application to the foreground
quit_application	Quit an application

Element Observation

Tool	Description
get_element_state	Get current state of an element
wait_for_element	Wait for an element to match a condition

Keyboard Input

Tool	Description
type_text	Type text into the frontmost application
send_shortcut	Send a keyboard shortcut

Screenshots

Tool	Description
capture_window	Capture a window screenshot
capture_region	Capture a screen region
capture_screen	Capture the entire screen

Permissions

Tool	Description
check_permission	Check if accessibility permission is granted

Advanced Mouse Interactions

Tool	Description
scroll	Scroll in any direction (up, down, left, right)
right_click	Right-click on an element to show context menu
double_click	Double-click on an element
hover	Hover over an element
drag_drop	Drag an element to a target element or coordinates

Menu Navigation

Tool	Description
navigate_menu	Navigate a menu path (e.g., "File > Save As...")
get_menu_structure	Get the full menu hierarchy for an application

Clipboard Operations

Tool	Description
read_clipboard	Read text or image from the clipboard
write_clipboard	Write text or image to the clipboard

Window Management

Tool	Description
get_window_bounds	Get position and size of a window
set_window_bounds	Move and/or resize a window
list_windows	List all windows for an application
minimize_window	Minimize a window
restore_window	Restore a minimized window
raise_window	Bring a window to the front

Text Selection

Tool	Description
select_text_range	Select a range of text by start/end index
get_selected_text	Get the currently selected text
select_all	Select all text in an element

Element Inspection

Tool	Description
get_all_attributes	Get all accessibility attributes of an element
highlight_element	Show a visual highlight overlay on an element
capture_element	Capture a screenshot of a specific element

File Dialog Handling

Tool	Description
navigate_dialog	Navigate to a path in an Open/Save dialog
select_file	Select a file in an Open dialog
set_filename	Set the filename in a Save dialog
confirm_dialog	Click the Save/Open button
cancel_dialog	Click the Cancel button

Example Usage

Calculator Automation

1. Launch Calculator:
   launch_application(bundle_id: "com.apple.calculator")

2. Type a calculation:
   type_text(bundle_id: "com.apple.calculator", text: "7+3")

3. Press equals:
   send_shortcut(bundle_id: "com.apple.calculator", key: "return", modifiers: [])

4. Get the result:
   get_element_tree(bundle_id: "com.apple.calculator", max_depth: 5)

5. Capture the result:
   capture_window(bundle_id: "com.apple.calculator")

Finding and Clicking Elements

1. Find a button:
   find_elements(bundle_id: "com.apple.calculator", role: "AXButton", title: "5")

2. Click using the returned path:
   click(bundle_id: "com.apple.calculator", path: {...})

Menu Navigation

1. Open a menu item:
   navigate_menu(bundle_id: "com.apple.TextEdit", menu_path: "File > Save As...")

2. Get all menu items:
   get_menu_structure(bundle_id: "com.apple.TextEdit")

File Dialog Workflow

1. Navigate to Save As dialog:
   navigate_menu(bundle_id: "com.apple.TextEdit", menu_path: "File > Save As...")

2. Navigate to Desktop:
   navigate_dialog(bundle_id: "com.apple.TextEdit", path: "~/Desktop")

3. Set filename:
   set_filename(bundle_id: "com.apple.TextEdit", filename: "mydocument.txt")

4. Save:
   confirm_dialog(bundle_id: "com.apple.TextEdit")

Window Management

1. List all windows:
   list_windows(bundle_id: "com.apple.finder")

2. Move and resize:
   set_window_bounds(bundle_id: "com.apple.finder", x: 100, y: 100, width: 800, height: 600)

3. Minimize:
   minimize_window(bundle_id: "com.apple.finder")

Clipboard Operations

1. Copy text to clipboard:
   write_clipboard(text: "Hello, World!")

2. Read clipboard:
   read_clipboard()

Element Paths

Elements are identified using paths that describe their location in the UI hierarchy:

{
  "segments": [
    {"role": "AXApplication", "title": "Calculator"},
    {"role": "AXWindow", "title": "Calculator"},
    {"role": "AXButton", "identifier": "Five"}
  ]
}

Each segment can match by:

• role: Accessibility role (AXButton, AXTextField, etc.)
• title: Element title
• identifier: Accessibility identifier
• index: Position among siblings

Error Codes

Code	Description
ELEMENT_NOT_FOUND	Element does not exist at path
ELEMENT_STALE	Element is no longer valid
PERMISSION_DENIED	Accessibility permission not granted
APPLICATION_NOT_FOUND	Application is not running
APPLICATION_NOT_RESPONDING	Application is hung
OPERATION_TIMEOUT	Operation exceeded time limit
QUEUE_FULL	Too many pending operations
INVALID_INPUT	Invalid tool parameters
AX_ERROR	Accessibility API error
MENU_NOT_FOUND	Menu item not found in path
MENU_NOT_OPENED	Failed to open menu
CLIPBOARD_EMPTY	Clipboard has no content
WINDOW_NOT_FOUND	No window found for application
WINDOW_OPERATION_FAILED	Window operation failed
TEXT_SELECTION_FAILED	Text selection operation failed
INVALID_TEXT_RANGE	Selection range out of bounds
DIALOG_NOT_FOUND	No file dialog is currently open

Architecture

Sources/MacOSUIMCP/
├── main.swift              # Entry point
├── Server/
│   ├── MCPServer.swift     # JSON-RPC message handling
│   ├── StdioTransport.swift
│   └── Logger.swift
├── Tools/
│   ├── ElementTools.swift
│   ├── ActionTools.swift
│   ├── ApplicationTools.swift
│   ├── ObservationTools.swift
│   ├── KeyboardTools.swift
│   ├── ScreenshotTools.swift
│   ├── PermissionTools.swift
│   ├── MouseTools.swift        # scroll, right_click, double_click, hover, drag_drop
│   ├── MenuTools.swift         # navigate_menu, get_menu_structure
│   ├── ClipboardTools.swift    # read_clipboard, write_clipboard
│   ├── WindowTools.swift       # get/set_window_bounds, list_windows, minimize, restore, raise
│   ├── TextSelectionTools.swift # select_text_range, get_selected_text, select_all
│   ├── InspectionTools.swift   # get_all_attributes, highlight_element, capture_element
│   └── FileDialogTools.swift   # navigate_dialog, select_file, set_filename, confirm, cancel
├── Accessibility/
│   ├── AXElement.swift
│   ├── AXApplication.swift
│   ├── AXMenu.swift            # Menu traversal
│   └── ElementPathResolver.swift
├── Input/
│   ├── KeyboardSimulator.swift
│   └── MouseSimulator.swift    # Mouse event simulation
├── Screenshot/
│   └── ScreenCapture.swift
├── Clipboard/
│   └── ClipboardManager.swift  # NSPasteboard wrapper
├── Window/
│   └── WindowManager.swift     # Window AX operations
├── Highlight/
│   └── HighlightOverlay.swift  # Visual highlight overlay
└── Models/
    ├── Application.swift
    ├── Element.swift
    ├── ElementPath.swift
    ├── Errors.swift
    ├── ClipboardContent.swift
    ├── WindowBounds.swift
    ├── WindowInfo.swift
    ├── TextSelectionResult.swift
    ├── AttributeValue.swift
    ├── ElementAttributes.swift
    └── HighlightResult.swift

License

Copyright (c) 2025 Anton Starikov

Licensed under the European Union Public Licence (EUPL), Version 1.2. You may not use this work except in compliance with the Licence.

See the LICENSE file for the full licence text.