{"id":"fec3a9a5-b2d6-4bd8-b136-5e095c7f75bc","slug":"mcp-mobile-agent","kind":"mcp","name":"mcp-mobile-agent","summary":"AI-agent + mobile-automation research artifact: Claude orchestrating Android devices via MCP, screenshot perception, and a low-latency Java input bridge.","owner":"mantotan","repoUrl":"https://github.com/mantotan/mcp-mobile-agent","license":"MIT","licenseTier":"permissive","trust":"community","stars":2,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-05-18T10:04:41.000Z","addedAt":"2026-10-21T08:54:05.556Z","logoUrl":"https://avatars.githubusercontent.com/u/30314235?u=98e44768ea748d8822b240a7ab2cff1937410803&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["design-media"],"tags":["adb","agent","android","claude","mobile-automation","python","research"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T21:45:36.420Z","sample":false,"url":"https://conduits.dev/tools/mcp-mobile-agent","language":"Python","transports":["stdio"],"install":{"pypi":"mcp-mobile-agent"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# mcp-mobile-agent\n\nA research artifact exploring how far you can push *AI-orchestrated mobile automation*: Claude (the model) is the decision-maker — perceiving the device via screenshots and on-screen text, choosing actions — and a small Python + Java stack is the executor that turns those decisions into actual ADB-mediated touches on a physical Android phone. The example domain in this repo is a TikTok account warmup loop, which happens to exercise the hardest constraints (anti-bot detection, low-latency input, hours-long session reliability), but the architecture generalizes to any \"Claude looks at the screen and acts on the phone\" task.\n\n> **Research artifact.** This repository demonstrates an architecture; it is not maintained as a TikTok automation product. Use against any specific platform — including TikTok — may violate that platform's Terms of Service; compliance is the user's responsibility. Not affiliated with, endorsed by, or built in cooperation with TikTok or ByteDance.\n\n## Why I built this\n\nI wanted to find out how much of a \"looks at a screen, decides what to do, then does it\" agent could be carried by a frontier model with screenshot perception, given a thin layer of MCP tools as its hands. TikTok was the testbed because (a) it has aggressive bot-detection signals that surface every flaw in the input layer, (b) sessions need to run for tens of minutes without quietly breaking, and (c) the perceptual cues a human uses (creator handle, hashtags, on-screen text, video genre) are the same ones a model can read from a screenshot. The interesting engineering question is what you have to build between Claude and the OS to make the loop work.\n\n## What's actually interesting\n\n### Claude as the decision-maker (MCP + screenshots)\n\nThe Python package exposes 11 tools and one workflow resource over [MCP](https://modelcontextprotocol.io). Claude reads the resource (the orchestration playbook for the engagement loop), then drives the session by calling tools: `start_session`, `take_screenshot`, `get_current_video_info`, `check_screen_state`, `scroll_to_next_video`, `watch_video`, `like_current_video`, `follow_creator`, plus a few session-management helpers. Perception is screenshot-first — when the dumpsys-based screen-state detector is ambiguous, Claude looks at the picture.\n\nThe server-side decisions worth pointing out:\n\n- **Daily caps are enforced server-side**, not trusted to Claude. The tool implementation checks the SQLite-backed counter and returns `daily_limit_reached` before firing the action. Claude can ask `like_current_video()` ten times in a row; nine of them will return `already_liked_this_video` or the daily cap.\n- **The workflow resource is version-controlled in code**, not in the prompt. Editing the `niche_engagement_workflow` function in [`mcp_server.py`](src/tiktok_warmup/mcp_server.py) changes how the agent behaves across all sessions, without touching any per-session prompt.\n- **Tools run blocking ADB work in an executor** so the FastMCP async loop stays responsive. The `_sessions` dict is server-lifetime state, so a single session survives across hundreds of tool calls without re-initialization.\n\nSee [`docs/MCP_USAGE.md`](docs/MCP_USAGE.md) for the registration recipe and what a session actually looks like from Claude's side.\n\n### Low-latency input via a custom Java DEX server\n\nThe naive way to do Android touch automation is `adb shell input tap x y`. Every call spawns a fresh shell process and a fresh `IInputManager` IBinder, end-to-end ~200-500ms per gesture. For a UI that responds to swipe velocity, that latency is the difference between a flick that scrolls a video and a slow drag that partial-scrolls.\n\nSo there's a small on-device Java server ([`InputBridge.java`](src/tiktok_warmup/server/InputBridge.java), ~230 lines) that binds a TCP socket, reads 8-byte binary commands, and injects them via `InputManager.injectInputEvent()` reached through reflection. Per-gesture latency drops to ~50-100ms. The server stays running for the whole session.\n\nThe interesting parts aren't the protocol — they're the platform-specific defenses:\n\n- **Phantom Process Killer (Android 12+)** kills `app_process` children when the parent shell exits. The fix is to hold the `adb shell` parent process alive via `subprocess.Popen` for the entire session, instead of using one-shot shell invocations.\n- **Low-memory killer** is happy to swap out a long-running JVM. Writing `-900` to `/proc/<server-pid>/oom_score_adj` keeps it pinned, with a retry loop because there's a race between `Popen` returning and the server writing its PID file.\n- **Heartbeat with broken-pipe wake** — the heartbeat thread waits on a threading event with a 4s timeout, but `_send_command()` sets the event immediately on `BrokenPipeError`, so a write failure triggers restart in <10ms rather than waiting up to 4s.\n- **Cross-platform socket nits** — Windows can't reuse a failed socket (fresh socket per connect attempt), macOS uses `TCP_KEEPALIVE` while Linux uses `TCP_KEEPIDLE`/`TCP_KEEPINTVL`/`TCP_KEEPCNT` (branch on `hasattr`).\n\nEach of these was an hour of debugging once, then a 2-line fix. [`docs/INPUTBRIDGE.md`](docs/INPUTBRIDGE.md) walks through the architecture end-to-end.\n\n### Statistically-literate behavior model\n\nIf you sample watch durations and pauses from uniform random, the resulting traffic looks like a bot within minutes. The behavior model in [`config/default.yaml`](config/default.yaml) samples from distributions chosen to match what real users do:\n\n- **Watch durations**: log-normal, `mu=1.6, sigma=0.8` — median ~5s, right-skewed tail to 90s. Most quick swipes, occasional full-watches.\n- **Pauses between scrolls**: gamma, `shape=2.0, scale=3.0` — mode at 3s, mean at 6s, long tail.\n- **Mood Markov chain**: three states (`rapid`, `casual`, `engaged`) with their own watch-time distributions and like-multipliers. Transitions sampled from a 3×3 matrix every 3-8 video checks.\n- **Imperfections**: 5% partial-scroll chance, 3% scroll-back, 5% replay. Deliberate deviation from the optimal path.\n- **Session-end taper**: sigmoid over session time, per-session variance in the spread and ceiling. Sessions end gradually rather than at a hard cliff.\n- **Phased warmup curriculum**: 5 phases over 14+ days, daily hard caps on likes/follows/comments, engagement multipliers that ramp from near-zero on day 1 to full on day 15+.\n- **Ban-signal circuit breaker**: 2 CAPTCHAs in a session flips an intensity-reduction flag for 48 hours; engagement multipliers get scaled by 0.5 on subsequent sessions.\n\nMost of these are config-tunable — the model is in YAML, not buried in code.\n\n## How it's built\n\nA standard Python package under `src/`, no heroics.\n\n- [`src/tiktok_warmup/cli/`](src/tiktok_warmup/cli/) — Typer-based CLI commands (`tiktok-warmup device list`, `account status`, …).\n- [`src/tiktok_warmup/config/`](src/tiktok_warmup/config/) — YAML loader + Pydantic schema for the behavior/safety params.\n- [`src/tiktok_warmup/core/`](src/tiktok_warmup/core/) — session lifecycle, the warmup phase machine, the per-session behavior model.\n- [`src/tiktok_warmup/device/`](src/tiktok_warmup/device/) — screen-state detection, ban monitor, device guard (lock-screen, app-foreground checks).\n- [`src/tiktok_warmup/adb/`](src/tiktok_warmup/adb/) — `AdbTransport` abstraction over `adbutils`.\n- [`src/tiktok_warmup/input/`](src/tiktok_warmup/input/) — gesture composition + the TouchServer client (`injector.py`).\n- [`src/tiktok_warmup/server/`](src/tiktok_warmup/server/) — `InputBridge.java` and its compiled DEX.\n- [`src/tiktok_warmup/persistence/`](src/tiktok_warmup/persistence/) — SQLAlchemy 2.0 models, session/account state, daily-limit counters.\n- [`src/tiktok_warmup/scheduling/`](src/tiktok_warmup/scheduling/) — time-window scheduler with break enforcement.\n- [`src/tiktok_warmup/mcp_server.py`](src/tiktok_warmup/mcp_server.py) — the MCP surface.\n- [`tests/`](tests/) — 13 pytest modules covering the warmup state machine, behavior model, ban monitor, persistence, screen-state detector, injector, session, scheduler, plus three on-device integration scripts for end-to-end verification.\n\n## Stack and reasoning\n\n- **Python 3.11+** — most of the ADB / Android surface is Python-native via `adbutils`. Matching the ecosystem.\n- **FastMCP** for the MCP server — async-first, decorators for tool registration. The stdio transport is what Claude Code expects.\n- **Pydantic v2** for config validation — Zod's Python cousin; nice error messages out of the box.\n- **SQLAlchemy 2.0** with the new typed ORM — relational state (accounts, sessions, daily counters) lives well in SQLite for a single-host tool.\n- **Typer** for the CLI — built on Click but with type-hint-driven parsing; the CLI is small and Typer is overkill but ergonomic.\n- **Standard library `logging`** with a per-module hierarchy and a [`SanitizingFormatter`](src/tiktok_warmup/utils/log_setup.py) that redacts device IPs from logs. Nothing fancy.\n- **Java + d8** for the input bridge — there's no Python alternative for `InputManager.injectInputEvent()`. Single `.java` file, single `.dex` artifact.\n- **pytest** with `pytest-cov`, `ruff` for lint, `mypy` for types.\n\n## Getting started\n\nThere are two ways to drive this:\n\n1. **As an MCP server for Claude Code** — Claude is the decision-maker; the Python package is the executor. This is the primary path and the reason the repo exists.\n2. **As a standalone Python script** — [`examples/run_session.py`](examples/run_session.py) runs the loop headless against a profile YAML, with niche-classification rules baked into Python instead of delegated to a model. Useful when you want the loop to run without a Claude session in the picture.\n\n### Prerequisites\n\n- Python 3.11+\n- An Android device (real or emulator) with USB debugging enabled\n- `adb` on `PATH` — `brew install android-platform-tools` on macOS\n- Optional: Android SDK with `d8` if you want to rebuild the DEX from `InputBridge.java`. The pre-built DEX is checked in, so this is only needed if you change the Java.\n\n### Setup (common to both paths)\n\n```bash\ngit clone https://github.com/mantotan/mcp-mobile-agent.git\ncd mcp-mobile-agent\n\npython -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev]\"\n\n# Connect a device\nadb devices                          # USB\nadb connect <device-ip>:<port>       # Wi-Fi\n\n# Verify the setup\ntiktok-warmup device list            # should list your connected device\n```\n\nIf `tiktok-warmup device list` doesn't show your phone, no MCP / Python session is going to work — fix `adb` first.\n\n### Path A — Drive it from Claude Code (recommended)\n\nThis is the headline path. The MCP integration runs on a **Claude subscription** ([Claude Code](https://claude.com/claude-code) wraps your Pro / Max plan) — there's no per-call Anthropic API cost for the reasoning. The agent loop's model turns all go through your subscription, and every \"action\" call (`take_screenshot`, `scroll_to_next_video`, etc.) is a local Python call against the device. The only metered cost is the proxy bandwidth or mobile data you'd be paying for anyway.\n\nA 30-minute warmup session is on the order of one Claude turn per video for several hundred videos, so a Pro subscription is fine for occasional one-shot sessions. If you're going to run sessions on a recurring schedule (`/loop`, see below), Max or higher gives you the headroom.\n\n**Register the MCP server.** Edit your Claude Code `settings.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"tiktok-warmup\": {\n      \"command\": \"tiktok-warmup-mcp\",\n      \"args\": []\n    }\n  }\n}\n```\n\nRestart Claude Code. The 11 tools and the `workflow://niche-engagement` resource should appear under the `tiktok-warmup` server.\n\n**Run a one-shot session.** Type something like:\n\n```\nRun a 30-minute warmup session on device 192.168.1.5:5555 for\naccount my_account. Niche: cooking, recipe, bake, foodie.\nLanguage filter: English. Stop on captcha, login prompt, or\ndaily-limit signals.\n```\n\nClaude reads the workflow resource, calls `start_session`, then loops through `check_screen_state` → `get_current_video_info` → decide-and-act → `scroll_to_next_video` until the deadline or a stop signal. See [`docs/MCP_USAGE.md`](docs/MCP_USAGE.md) for the full tool surface and what each call returns.\n\n**Run it on a recurring schedule.** Account warmup wants short, spaced sessions throughout the day, not one long burst. Claude Code's built-in `/loop` skill makes that one line:\n\n```\n/loop 8h Run a 30-minute warmup session on device 192.168.1.5:5555\nfor account my_account. Niche: cooking. Stop on captcha or daily-limit.\n```\n\nThat fires the same prompt every 8 hours until you stop the loop. Claude Code keeps the loop running locally — there's no remote infrastructure involved.\n\nThere's also a `/schedule` skill for Claude Cowork-style remote scheduled agents (cron-based, runs on Anthropic's infrastructure). **For this project, prefer `/loop`** — Cowork-scheduled runs execute remotely and can't reach a device sitting on your home LAN unless you expose the MCP server publicly, which you almost certainly do not want to do. `/schedule` is the right tool for cloud-only agents (Gmail, Calendar, etc.); `/loop` is the right tool for \"Claude Code on my laptop with reach to my devices.\"\n\n### Path B — Run it headless (no Claude needed)\n\nThe same primitives are exposed as a standalone Python script that takes a profile YAML. No model in the loop — niche classification is the dumb-but-fast substring matcher in [`examples/run_session.py`](examples/run_session.py).\n\n```bash\n# Copy a profile template and edit it\ncp examples/profiles/example_english.yaml examples/profiles/my_niche.yaml\n# Edit my_niche.yaml — set device_id, account_id, niche_keywords\n\n# Run a single session\npython examples/run_session.py --profile examples/profiles/my_niche.yaml\n```\n\nSee [`examples/profiles/README.md`](examples/profiles/README.md) for the profile schema.\n\n## What's not in here\n\nThis is a single-developer research tool, not a hosted product, so a few things are intentionally absent:\n\n- **iOS support.** None. The InputBridge is Android-specific.\n- **Multi-platform abstraction.** The screen-state detector, the TikTok-specific gestures, the gift-event schema — all TikTok. A \"BrowserAgent\" / \"GenericAndroidAgent\" abstraction would be the right way to generalize, but I haven't done it.\n- **GUI dashboard.** Session state is in SQLite. Queries are run directly or via the Typer CLI.\n- **Cloud / managed deployment.** Designed for a single host with USB/Wi-Fi reach to physical devices. The `_sessions` dict in the MCP server is process-local; running two MCP servers against the same device would race.\n- **Multi-finger gestures.** The protocol carries one pointer at a time. Pinch-zoom would require a small protocol revision.\n\n## A note on how this was built\n\nMost of the typing in this repo was driven through [Claude Code](https://claude.com/claude-code), Anthropic's CLI coding agent. The architectural calls — building a custom Java input server to escape `adb shell input` latency, treating the engagement loop as a state machine with screen-state recovery rather than a happy-path script, paying the per-PID-file dance to set `oom_score_adj` correctly, making Claude itself the decision-maker over screenshots rather than encoding niche-matching as Python rules — are mine, and so is the debugging when the Phantom Process Killer made the server vanish on every Android 12+ device. The agent made the implementation faster; the system design and the bugs-to-fix loop are the same engineering work as without it.\n\n## What I'd do differently\n\nThe biggest item is the in-process connection manager: it's straightforward to reason about for a single device, but once you have 3+ phones I'd push the session-orchestration layer out of the MCP server into a dedicated process (or a job queue like Redis-backed RQ) so a crash in the MCP layer doesn't take all the active sessions with it. The MCP tools would then talk to the session service over a local socket, and the MCP server itself would be stateless and trivially restartable.\n\nSecond item: the niche keyword matcher in `examples/run_session.py` is intentionally dumb (lowercase substring match). That's fine when you're driving the loop yourself, but when Claude is the decision-maker the keyword list ends up unused — Claude classifies directly from the screenshot + on-screen text. The right move is probably to drop the keyword matcher from the example and lean fully into Claude's judgment, with the keyword list staying around only as a fallback when running headless via `run_session.py`.\n\n## License\n\nMIT — see [`LICENSE`](LICENSE).\n"}