{"id":"f36a990a-3dda-4aae-8405-69aa716cd5bd","slug":"mobile-agent-bench","kind":"mcp","name":"mobile-agent-bench","summary":"A pre-registered, reproducible benchmark of agent-operated mobile-device tooling: 4 MCP tool stacks, 1 confined agent, 10 frozen tasks, 2 full grids, every transcript published — including the ones we","owner":"LincForge","repoUrl":"https://github.com/LincForge/mobile-agent-bench","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-07-02T02:13:23.000Z","addedAt":"2026-10-21T06:20:06.333Z","logoUrl":"https://avatars.githubusercontent.com/u/261543728?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["developer-tools","memory-knowledge"],"tags":["ai-agents","android","benchmark","mobile-testing"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T07:09:21.408Z","sample":false,"url":"https://conduits.dev/tools/mobile-agent-bench","language":"Python","transports":["stdio"],"install":{"pypi":"mobile-agent-bench"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# mobile-agent-bench\n\nA reproducible benchmark of **agent-operated mobile-device tooling**: four MCP\ntool stacks driven by the same confined AI agent, same pinned model, same\nphysical devices, on ten pre-registered tasks — plus raw transcripts for every\nrun.\n\nThree complete grids have been run and published: **v1**, **v2**, and **v3**,\n200 cells each (10 tasks × 5 runs × 4 tools = **600 cells total**).\n\n**→ Read the [combined v1+v2+v3 report](REPORT.md).**\n\n| Stack | Vendor |\n|---|---|\n| [nerve](https://github.com/LincForge/nerve) + [crucible](https://github.com/LincForge/crucible) + [spectra](https://github.com/LincForge/spectra) | LincForge (that's us — see the disclosure below) |\n| Maestro MCP | mobile.dev |\n| mobile-mcp | mobile-next (community) |\n| agent-device | Callstack |\n\n## Results at a glance\n\nCompletion over the 45 scored phone cells per tool (the Wear OS row is a\ncapability row, reported separately and never averaged), plus the **7-task UI +\nLogcat** phone subset (excluding the two JDWP debugger tasks `c10` and `c11`)\nand the **3-task UI-Only** subset (`a5, a6, c9`):\n\n| | v1 (45 cells) | v2 (45 cells) | v3 7-task UI+Logcat (`n=35`) | v3 3-task UI-Only (`n=15`) | v3 9-task Phone (`n=45`) |\n|---|---|---|---|---|---|\n| linc | 33/45 (73%) | **44/45 (98%)** | **34/35 (97%)** | **15/15 (100%)** | **44/45 (98%)** |\n| agent-device | 32/45 (71%) | 32/45 (71%) | **34/35 (97%)** | **15/15 (100%)** | 34/45 (76%) |\n| mobile-mcp | 25/45 (56%) | 28/45 (62%) | 18/35 (51%) | **15/15 (100%)** | 20/45 (44%) |\n| Maestro | 26/45 (58%) | 20/45 (44%) | 14/35 (40%) | 13/15 (87%) *(14/15 adj.)* | 14/45 (31%) |\n\nThe three-grid arc, and the reason all three grids are published together:\n\n1. **v1 was a loss.** Our stack came first by two points, was the *slowest*\n   column on the grid (7 timeouts, against 1–3 for everyone else), and scored\n   **0/5 on c10** — runtime state inspection, the one task our debugger uniquely\n   exists for. So did every other tool: c10 was 0/0/0/0.\n2. **v2 fixed what v1 exposed — with an overfitting caveat.** Both root causes\n   were filed as P0s against our own stack, fixed, pinned, and the\n   byte-identical grid was re-run under `v2-prereg`. `c10` went to 5/0/0/0 and\n   median wall time dropped to 138s (0 timeouts). Because we fixed defects on\n   tasks we had already watched fail, we named the objection ourselves: read v2\n   as *\"the vendor can fix what this benchmark measures,\"* and Remedy 2 was a\n   **v3 with unseen tasks** and updated competitor releases.\n3. **v3 tested unseen tasks against upgraded competitors and the frozen\n   `linc-nerve==0.1.0` PyPI release — scored once with zero post-hoc re-runs.**\n   On the **7-task UI + Logcat phone subset (`n=35`)**, Callstack's upgraded\n   **`agent-device 0.21.27` and `linc` match at 34/35 (97.1%)** (`agent-device`\n   is faster at the per-task median on 5 of 7 tasks, `64.1s` vs `80.6s` pooled\n   median; `linc` uses fewer uncached tokens on 6 of 7 tasks, `$1.01/pass` vs\n   `$1.41/pass`). The 10-cell difference on the 9-task phone grid (`44/45` vs\n   `34/45`) comes entirely from `c10` and `c11`, which require a JVM debugger\n   (`linc-spectra` goes `10/10`, while tools without JDWP support go `0/10` or\n   `2/10` via UI curve-fitting). All three packages in the `linc` column are now\n   open-source (`Apache-2.0`) on GitHub and PyPI (`linc-nerve`, `linc-crucible`,\n   `linc-spectra`); note that the published `v3` grid was scored once against the\n   frozen pre-release commits `crucible@cf04279` and `spectra@c471dad` paired\n   with `linc-nerve==0.1.0`.\n\n## Disclosure\n\nWe are the vendor of one of the stacks under test. That is exactly why this\nrepo exists in this shape: pre-registered criteria, vendor-recommended\nconfiguration for every competitor, harness + target app published as source,\nand a raw sanitized transcript behind every number. If you think a row is\nunfair, clone it and re-run it.\n\nEvery deviation from pre-registration is recorded with receipts in\n[`AMENDMENTS.md`](AMENDMENTS.md), including the interference we caught that\n*helped* us and the contaminated cells we voided that had already passed.\nVoided cells are kept in `results_invalid/` rather than deleted. In **v3**,\nin-harness lock detection (`is_device_locked` pre/post cell) ran across all 200\ncells with **0 locked cells and 0 amendments**.\n\n## Reading order\n\n1. [`REPORT.md`](REPORT.md) — the combined v1+v2+v3 report (start here)\n2. [`PREREGISTRATION.md`](PREREGISTRATION.md) — the v1 frozen contract (`v1-prereg` tag)\n3. [`PREREGISTRATION-v2.md`](PREREGISTRATION-v2.md) — the v2 frozen contract (`v2-prereg` tag)\n4. [`PREREGISTRATION-v3.md`](PREREGISTRATION-v3.md) — the v3 frozen contract (`v3-prereg` tag), incl. unseen task hashes & PyPI wheel SHA-256\n5. [`AMENDMENTS.md`](AMENDMENTS.md) — every deviation, with receipts\n6. [`docs/v1-deep-dive.md`](docs/v1-deep-dive.md) · [`docs/v2-deep-dive.md`](docs/v2-deep-dive.md) · [`docs/v3-deep-dive.md`](docs/v3-deep-dive.md) — per-task analysis\n\nThe `v1-prereg`, `v2-prereg`, and `v3-prereg` git tags are the integrity\nanchors: each was pushed *before* the first comparative cell of its grid ran.\nGit history is deliberately never rewritten in this repo.\n\n## Layout\n\n```\nREPORT.md              # combined v1+v2+v3 report\nPREREGISTRATION.md     # the frozen v1 contract (v1-prereg tag)\nPREREGISTRATION-v2.md  # the frozen v2 contract (v2-prereg tag)\nPREREGISTRATION-v3.md  # the frozen v3 contract (v3-prereg tag)\nAMENDMENTS.md          # every post-freeze deviation, with rationale + receipts\ntasks/                 # 10 frozen v1/v2 task definitions (prompt + reset + verification + APK pin)\ntasks_v3/              # 10 frozen v3 task definitions (4 v1/v2 anchors + 6 unseen tasks)\ntools/                 # 4 tool adapters (MCP config, vendor-recommended)\ntarget-app/            # BenchTarget: purpose-built open-source target app (SPEC.md + v3 additions)\nmobile_agent_bench/    # harness: runner, agent invocation, token accounting, lock guard, reset, sanitize\nresults/               # v1 raw run logs: <tool>/<task>/run-<n>/{transcript.jsonl,meta.json}\nresults_v2/            # v2 raw run logs, same shape\nresults_v3/            # v3 raw run logs, same shape\nresults_invalid/       # voided cells, kept as receipts (never deleted)\n```\n\n## Confinement\n\nThe agent gets the tool under test and **nothing else** — no shell, no\nfilesystem search, no web, no subagents, `--strict-mcp-config`, a neutral cwd,\nand a path-guarded `Read`. If a tool cannot see the UI tree, its agent is\nblind. That is the point: the benchmark measures the *tool surface*, not the\nmodel.\n\nClosing that confinement took three errata in v1 (all in `AMENDMENTS.md`) —\nmost notably a built-in tool that could run arbitrary shell, which some agents\nused to reach raw `adb` and `jdb`. The blast radius was measured per\ntranscript, the 8 affected cells were voided and re-run, and the remaining 112\nwere positively certified shell-free. That correction moved a *competitor's*\nnumbers, not ours.\n\n## Running it\n\nPrereqs: Python 3.12+ with [uv](https://docs.astral.sh/uv/), the Claude Code\nCLI (`claude`), adb + a connected Android device, and the tool stacks you want\nto test installed per their vendor docs (see `tools/*.yaml` notes).\n\n```bash\nuv sync\nexport BENCH_DEVICE_SERIAL=<your device serial>   # or bench.local.yaml (gitignored)\nuv run bench run --tool mobile-mcp --task a1 --runs 5\nuv run bench report\n\n# reproduce the v2 or v3 grid's scoreboard from the published rows\nBENCH_RESULTS_DIR=results_v2 uv run bench report\nBENCH_RESULTS_DIR=results_v3 BENCH_TASKS_DIR=tasks_v3 uv run bench report\n```\n\n- Machine-local MCP command paths go in `tools/<id>.local.mcp.json`\n  (gitignored); the committed configs assume vendor binaries on PATH.\n- `bench.local.yaml` (gitignored) also carries redaction literals — every\n  transcript is scrubbed of serials/hostnames/user paths at capture time.\n- The target app is built from `target-app/` (Gradle; `stock` flavor for all\n  tasks except c9, which uses `seeded`). Each task pins the exact APK by digest.\n\n## Status\n\n- [x] Harness, token accounting, scripted reset, confinement, in-harness lock guard\n- [x] Task suite + target-app spec frozen (`v1-prereg`)\n- [x] BenchTarget implementation matching the frozen spec\n- [x] **v1 grid complete** (200/200 cells) + report + deep-dive\n- [x] **v2 grid complete** (200/200 cells) under `v2-prereg` + combined report\n- [x] **v3 grid complete** (200/200 cells) under `v3-prereg` (6 unseen tasks, PyPI `linc-nerve==0.1.0` freeze, updated competitor versions, 0 locked cells)\n\n## If you build one of these tools\n\nRun it yourself and tell us where we're wrong. Open an issue with the cell and\nthe transcript. A row we cannot defend is a row we retract.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n"}