SoftAuto-MCP

What is SoftAuto?

SoftAuto is an open-source Windows RPA application and MCP server for AI agents. It locates desktop controls through Microsoft UI Automation (UIA), locates web elements through the Chrome DOM, and exposes click, type, read, highlight, and validation actions as Model Context Protocol (MCP) tools. It is designed for ERP automation, desktop automation, browser automation, GUI testing, and repeatable business workflows.

SoftAuto acts as a deterministic execution layer for Computer Use. Capture and validate an element once, then let an Agent invoke it by name without repeating screenshot analysis and coordinate clicking for every step.

The problem

Computer Use can operate almost any application, but repeatedly running the loop of screenshot, visual reasoning, coordinate clicking, and another screenshot is slow and sensitive to window movement or layout changes.

SoftAuto separates understanding from execution. A person captures and validates an element once. The Agent can then invoke that named element through MCP, using Windows UIA or browser DOM actions instead of repeating visual inference for every step.

	Traditional Computer Use	SoftAuto + MCP
Targeting	Re-analyze screenshots and coordinates	Reuse a validated UIA/DOM locator
Execution	Vision model → coordinate input	Agent → MCP → native element action
Window movement	Coordinates may fail	Stable properties are resolved again
Dynamic text	Requires another visual pass	Wildcards, prefixes, and MCP variables
Best fit	Unknown interfaces and exploration	ERP, line-of-business apps, repeatable workflows

flowchart LR
    A[Capture and validate once] --> B[(Project element library)]
    B --> C[SoftAuto MCP]
    D[Agent understands and decides] --> C
    C --> E[Windows UIA]
    C --> F[Browser DOM]
    E --> G[Desktop applications]
    F --> H[Web applications]

Highlights

• Capture and highlight desktop or web elements with Ctrl + left-click.
• Organize elements in project-scoped folder trees and move projects between computers.
• Edit recommended locator properties, wildcards, prefixes, and ${variable} values. Runtime ProcessId and NativeWindowHandle remain diagnostic-only and cannot be selected for matching.
• Resolve elements again after applications restart or windows move.
• Expose 19 MCP query and action tools, including find, highlight, click, input, focus, and read.
• Use Windows UIA and browser DOM backends from one element library; desktop capture automatically falls back through the bundled FlaUI UIA3 → UIA2 bridge when the primary provider cannot inspect a control.
• Switch between Simplified Chinese and English; responsive UI supports Windows DPI scaling.
• No arbitrary shell execution and no unrestricted coordinate clicking through MCP.

How the element-to-MCP workflow works

SoftAuto is intentionally split into two phases:

1. Capture phase (human/RPA-style): create a project, pick a desktop or web element with Ctrl + left-click, review the recommended properties, and press Validate. The element is stored in the current project's tree.
2. Execution phase (Agent/MCP): the Agent calls an element by name or path. SoftAuto resolves the saved locator and performs the native UIA or DOM action without another screenshot/vision loop.

The recommended locator properties are chosen for stability. AutomationId, control type, framework, window title, and an ancestor path are preferred when available. Volatile ProcessId and NativeWindowHandle values are diagnostics only and are deliberately excluded from matching, because they change after a restart. For dynamic labels, use a name prefix, */? wildcards, or a runtime variable such as ${reference}.

Each project owns its own folders and element library. Export the project JSON to move it to another Windows computer; after importing, validate the elements against the target application. For difficult legacy controls, use the bundled FlaUI bridge (UIA3 with UIA2 fallback) and the Deep Inspect action.

Agent execution and performance

Use run_sequence to send a group of dependent actions in one MCP request. This keeps the Agent in the decision loop while removing unnecessary per-click round trips:

{
  "steps": [
    {"action": "set_value", "element": "Staffnumber", "value": "admin"},
    {"action": "set_value", "element": "Password", "value": "admin"},
    {"action": "click", "element": "Sign In"}
  ],
  "confirm": true
}

Use confirm: true for action tools. Keep highlight_element and validate_saved_element for capture, diagnostics, and recovery; normal production runs should call the action directly. SoftAuto does not execute arbitrary shell commands or allow MCP to bypass the saved element library with unrestricted coordinates.

Example case: TrainingOrderSystem ERP batch entry

The repository includes a complete TrainingOrderSystem example. It models a common legacy ERP flow:

1. Log in with admin / admin.
2. Select Option 1 and click Go before each order.
3. Read Orders.csv, fill Product Code, Quantity, Unit Price, and Cost Centre, then submit.
4. Read the dynamically generated Order Reference.
5. Click Continue to return to the selection page and process the next row.

The example deliberately does not use an Option 2 recovery shortcut. It stops on the first failed stage and reports the row and action, so a missed order cannot be silently skipped. Its declarative workflow is in examples/training-order/workflow.json, and the latest benchmark is in benchmark-latest.json.

The latest five-row run completed in 45.493 seconds (about 8.7 seconds per row) without Computer Use. The v0.5.7 Release contains a ready-to-run SoftAuto-Demo-Bundle-0.5.7.zip with the installer, training application, CSV, captured element library, workflow, and Codex Skill.

Troubleshooting legacy applications

• Validation succeeds but a later run fails: re-check that the project points to the intended application window; do not select process IDs or native handles as locator properties.
• A changing label cannot be found: replace the full Name with a stable prefix or wildcard and pass dynamic text through variables.
• The first action is slow: use run_sequence for consecutive actions and avoid a preflight highlight on every step.
• A WinForms/ERP control is not exposed by the primary provider: run Deep Inspect or let the bundled FlaUI UIA3 → UIA2 fallback inspect it.
• MCP connects but actions are rejected: use the installed path shown by MCP Config and keep confirm: true; action execution is intentionally gated.

Quick start

Windows installer

1. Download Lingheyi-SoftAuto-Setup-*.exe from Releases.
2. Install SoftAuto and create a project and folder.
3. Click Pick Desktop Element or Pick Web Element, then press Ctrl + left-click on the target.
4. Adjust locator properties and click Validate.
5. Click MCP Config and paste the copied configuration into an MCP-compatible Agent.

Run from source

Requirements: Windows 10/11 and Python 3.11–3.13.

uv sync --extra dev
uv run softauto-inspector

Run the MCP server directly:

$env:SOFTAUTO_ALLOW_ACTIONS = "1"
uv run mcp run src/softauto/server.py:mcp

MCP configuration

The installed application can copy the exact configuration for the current computer. A typical configuration looks like this:

{
  "mcpServers": {
    "software-automation": {
      "command": "C:\\Users\\<user>\\AppData\\Local\\Programs\\SoftAuto\\mcp\\SoftAutoMCP.exe"
    }
  }
}

MCP follows the project currently selected in SoftAuto. Set SOFTAUTO_ELEMENT_LIBRARY when you want it to use an exported standalone element library instead.

Development

uv sync --frozen --extra dev
uv run ruff check src tests
uv run pytest -q

The current release has 47 automated tests. See CONTRIBUTING.md before sending a pull request.

Security and scope

SoftAuto is designed for systems and accounts you are authorized to automate. Saved locators may contain application titles, URLs, and control metadata, so review exported libraries before sharing them. See SECURITY.md for the action policy and vulnerability reporting process.

License

SoftAuto's original source code is available under the MIT License. Bundled and optional third-party components retain their own licenses; see THIRD_PARTY_NOTICES.md.