apkscope

apkscope

See and drive a running Android app as text, for an agent whose model rejects image input.

Most LLM-based coding agents cannot look at a screenshot: the model refuses image content, and the usual "describe this image" helpers call a hosted vision API that bills per request or rate-limits you to nothing. This MCP server exists so such an agent can still work on an Android UI — reading what is on screen, measuring it, and tapping it.

Everything is converted into characters the model can read.

Why it exists

Developed while building a Flutter app (SahayakAI) with an agent that had no image input at all. Two capabilities made UI work possible:

• Reading the semantics tree. Flutter publishes its accessibility tree only while an accessibility service is bound. With none bound, uiautomator dump against a Flutter screen returns a bare FrameLayout with no children. apkscope enables a stock system service for the duration of the read and restores the previous setting afterwards.
• Rendering pixels as glyphs. Enough to judge layout, spacing, colour and gross breakage — not a substitute for vision, but far better than nothing.

Install

pip install -e .

Requires adb on PATH, or $ANDROID_HOME / $ANDROID_SDK_ROOT set, or $APKSCOPE_ADB pointing at the binary.

MCP setup

{
  "mcp": {
    "apkscope": {
      "type": "local",
      "command": ["apkscope-mcp"],
      "environment": { "ANDROID_HOME": "C:\\Users\\you\\AppData\\Local\\Android\\Sdk" }
    }
  }
}

Or run it directly to check it starts:

apkscope-mcp

Tools

tool	mutates device	what it gives you
screen_tree	no	every labelled element: class, flags, pixel bounds, centre, label
audit_layout	no	tap targets under 48dp, overlapping controls, off-screen nodes
screenshot	no	the screen rendered as text, in one of three modes
tap	yes	taps the element whose label matches

screen_tree — start here

The primary way to read a Flutter UI. Labels, roles and exact bounds are precise and cheap; a pixel render is approximate and token-heavy.

View       [-FE--]   530x98    @(  540,  842)  Good morning
View       [CFE--]   477x283   @(  283, 1750)  Lesson Plan
Button     [CFE--]   126x126   @(  765,  202)  Network

Flags: C clickable, F focusable, E enabled, k checked, s selected. Only C means tappable. Flutter also marks plain text focusable, so F alone is not a control — that distinction is what keeps headings out of the tap list.

Pass tappable=True for just the tap-target table.

audit_layout — turn observation into findings

Reports TAP-SMALL (below Material's 48dp guidance, converted using the device's real density), OVERLAP (two controls sharing more than a third of the smaller one) and, with check_clip=True, CLIPPED off-screen nodes.

Full-bleed wrappers are excluded. Flutter routinely wraps a whole screen in one GestureDetector (a "tap anywhere to speak" surface); without that exclusion every other control is reported as overlapping it.

screenshot — pixels as text

Three modes:

mode	output	cost	good for
palette (default)	one glyph per pixel, plus a legend mapping glyph → true RGB	~2k tokens	layout, colour, shape
ramp	luminance ASCII (@%#*+=-:. )	cheap	structure only
ansi	truecolour half-blocks ▀, two pixels per character	very high (~40 chars/pixel)	detail crops only

Keep colors at 8 or below. On a light-themed app a large palette gets spent on imperceptible differences between near-identical backgrounds (#f3f1ec vs #f7f5f0) and the one meaningful hue is lost. Fewer bins force the quantiser to group the near-whites.

The legend always reports RGB from the unaltered image, so colour is never misrepresented.

crop accepts all, top, mid, bot, or 'a,b' fractions of height. A phone screen is tall and mostly empty near the bottom; cropping a band is the difference between a legible render and a hundred rows of background.

tap — drive the UI

Resolves a label to coordinates through the semantics tree, which survives layout changes and screen-size differences in a way hardcoded pixels do not.

Refuses an ambiguous label and lists the candidates, because a silent wrong tap navigates somewhere unexpected and costs more to debug than asking. Pass first=True when you know what you want.

CLI

The same operations without an MCP host:

apkscope tree --taps
apkscope audit --clip
apkscope shot --cols 72 --colors 8 --crop 0.3,0.6
apkscope shot --mode ramp --crop top
apkscope tap "Continue with Google"

audit exits non-zero when it finds something, so it works in a CI check.

Honest limits

A glyph render is not vision. It conveys structure, colour, alignment, spacing and obvious breakage. It does not reliably convey small body text, subtle typography, or whether a screen matches a design intent. For fine visual judgement a human still has to look.

screen_tree and audit_layout are the higher-value tools for UI work — exact, cheap, directly actionable. Reach for screenshot to catch layout problems, not to proofread.

The tree only shows what Flutter marks semantic. Text painted directly to a canvas is invisible to it, and to uiautomator generally.

Tests

pip install -e ".[dev]"
pytest

97 tests, no device required. They pin down the bugs this tool actually hit while being built: an inverted luminance ramp, a palette legend that crashed on low-colour crops, a whole-screen overlap false positive from Flutter's full-bleed wrapper, and unlabelled icon buttons being invisible to the tap-target filter.

License

MIT