mcp-mobile-agent

mcp-mobile-agent

A research artifact exploring how far you can push AI-orchestrated mobile automation: Claude (the model) is the decision-maker — perceiving the device via screenshots and on-screen text, choosing actions — and a small Python + Java stack is the executor that turns those decisions into actual ADB-mediated touches on a physical Android phone. The example domain in this repo is a TikTok account warmup loop, which happens to exercise the hardest constraints (anti-bot detection, low-latency input, hours-long session reliability), but the architecture generalizes to any "Claude looks at the screen and acts on the phone" task.

Research artifact. This repository demonstrates an architecture; it is not maintained as a TikTok automation product. Use against any specific platform — including TikTok — may violate that platform's Terms of Service; compliance is the user's responsibility. Not affiliated with, endorsed by, or built in cooperation with TikTok or ByteDance.

Why I built this

I wanted to find out how much of a "looks at a screen, decides what to do, then does it" agent could be carried by a frontier model with screenshot perception, given a thin layer of MCP tools as its hands. TikTok was the testbed because (a) it has aggressive bot-detection signals that surface every flaw in the input layer, (b) sessions need to run for tens of minutes without quietly breaking, and (c) the perceptual cues a human uses (creator handle, hashtags, on-screen text, video genre) are the same ones a model can read from a screenshot. The interesting engineering question is what you have to build between Claude and the OS to make the loop work.

What's actually interesting

Claude as the decision-maker (MCP + screenshots)

The Python package exposes 11 tools and one workflow resource over MCP. Claude reads the resource (the orchestration playbook for the engagement loop), then drives the session by calling tools: start_session, take_screenshot, get_current_video_info, check_screen_state, scroll_to_next_video, watch_video, like_current_video, follow_creator, plus a few session-management helpers. Perception is screenshot-first — when the dumpsys-based screen-state detector is ambiguous, Claude looks at the picture.

The server-side decisions worth pointing out:

• Daily caps are enforced server-side, not trusted to Claude. The tool implementation checks the SQLite-backed counter and returns daily_limit_reached before firing the action. Claude can ask like_current_video() ten times in a row; nine of them will return already_liked_this_video or the daily cap.
• The workflow resource is version-controlled in code, not in the prompt. Editing the niche_engagement_workflow function in mcp_server.py changes how the agent behaves across all sessions, without touching any per-session prompt.
• Tools run blocking ADB work in an executor so the FastMCP async loop stays responsive. The _sessions dict is server-lifetime state, so a single session survives across hundreds of tool calls without re-initialization.

See docs/MCP_USAGE.md for the registration recipe and what a session actually looks like from Claude's side.

Low-latency input via a custom Java DEX server

The naive way to do Android touch automation is adb shell input tap x y. Every call spawns a fresh shell process and a fresh IInputManager IBinder, end-to-end ~200-500ms per gesture. For a UI that responds to swipe velocity, that latency is the difference between a flick that scrolls a video and a slow drag that partial-scrolls.

So there's a small on-device Java server (InputBridge.java, ~230 lines) that binds a TCP socket, reads 8-byte binary commands, and injects them via InputManager.injectInputEvent() reached through reflection. Per-gesture latency drops to ~50-100ms. The server stays running for the whole session.

The interesting parts aren't the protocol — they're the platform-specific defenses:

• Phantom Process Killer (Android 12+) kills app_process children when the parent shell exits. The fix is to hold the adb shell parent process alive via subprocess.Popen for the entire session, instead of using one-shot shell invocations.
• Low-memory killer is happy to swap out a long-running JVM. Writing -900 to /proc/<server-pid>/oom_score_adj keeps it pinned, with a retry loop because there's a race between Popen returning and the server writing its PID file.
• Heartbeat with broken-pipe wake — the heartbeat thread waits on a threading event with a 4s timeout, but _send_command() sets the event immediately on BrokenPipeError, so a write failure triggers restart in <10ms rather than waiting up to 4s.
• Cross-platform socket nits — Windows can't reuse a failed socket (fresh socket per connect attempt), macOS uses TCP_KEEPALIVE while Linux uses TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT (branch on hasattr).

Each of these was an hour of debugging once, then a 2-line fix. docs/INPUTBRIDGE.md walks through the architecture end-to-end.

Statistically-literate behavior model

If you sample watch durations and pauses from uniform random, the resulting traffic looks like a bot within minutes. The behavior model in config/default.yaml samples from distributions chosen to match what real users do:

• Watch durations: log-normal, mu=1.6, sigma=0.8 — median ~5s, right-skewed tail to 90s. Most quick swipes, occasional full-watches.
• Pauses between scrolls: gamma, shape=2.0, scale=3.0 — mode at 3s, mean at 6s, long tail.
• Mood Markov chain: three states (rapid, casual, engaged) with their own watch-time distributions and like-multipliers. Transitions sampled from a 3×3 matrix every 3-8 video checks.
• Imperfections: 5% partial-scroll chance, 3% scroll-back, 5% replay. Deliberate deviation from the optimal path.
• Session-end taper: sigmoid over session time, per-session variance in the spread and ceiling. Sessions end gradually rather than at a hard cliff.
• Phased warmup curriculum: 5 phases over 14+ days, daily hard caps on likes/follows/comments, engagement multipliers that ramp from near-zero on day 1 to full on day 15+.
• Ban-signal circuit breaker: 2 CAPTCHAs in a session flips an intensity-reduction flag for 48 hours; engagement multipliers get scaled by 0.5 on subsequent sessions.

Most of these are config-tunable — the model is in YAML, not buried in code.

How it's built

A standard Python package under src/, no heroics.

• src/tiktok_warmup/cli/ — Typer-based CLI commands (tiktok-warmup device list, account status, …).
• src/tiktok_warmup/config/ — YAML loader + Pydantic schema for the behavior/safety params.
• src/tiktok_warmup/core/ — session lifecycle, the warmup phase machine, the per-session behavior model.
• src/tiktok_warmup/device/ — screen-state detection, ban monitor, device guard (lock-screen, app-foreground checks).
• src/tiktok_warmup/adb/ — AdbTransport abstraction over adbutils.
• src/tiktok_warmup/input/ — gesture composition + the TouchServer client (injector.py).
• src/tiktok_warmup/server/ — InputBridge.java and its compiled DEX.
• src/tiktok_warmup/persistence/ — SQLAlchemy 2.0 models, session/account state, daily-limit counters.
• src/tiktok_warmup/scheduling/ — time-window scheduler with break enforcement.
• src/tiktok_warmup/mcp_server.py — the MCP surface.
• tests/ — 13 pytest modules covering the warmup state machine, behavior model, ban monitor, persistence, screen-state detector, injector, session, scheduler, plus three on-device integration scripts for end-to-end verification.

Stack and reasoning

• Python 3.11+ — most of the ADB / Android surface is Python-native via adbutils. Matching the ecosystem.
• FastMCP for the MCP server — async-first, decorators for tool registration. The stdio transport is what Claude Code expects.
• Pydantic v2 for config validation — Zod's Python cousin; nice error messages out of the box.
• SQLAlchemy 2.0 with the new typed ORM — relational state (accounts, sessions, daily counters) lives well in SQLite for a single-host tool.
• Typer for the CLI — built on Click but with type-hint-driven parsing; the CLI is small and Typer is overkill but ergonomic.
• Standard library logging with a per-module hierarchy and a SanitizingFormatter that redacts device IPs from logs. Nothing fancy.
• Java + d8 for the input bridge — there's no Python alternative for InputManager.injectInputEvent(). Single .java file, single .dex artifact.
• pytest with pytest-cov, ruff for lint, mypy for types.

Getting started

There are two ways to drive this:

1. As an MCP server for Claude Code — Claude is the decision-maker; the Python package is the executor. This is the primary path and the reason the repo exists.
2. As a standalone Python script — examples/run_session.py runs the loop headless against a profile YAML, with niche-classification rules baked into Python instead of delegated to a model. Useful when you want the loop to run without a Claude session in the picture.

Prerequisites

• Python 3.11+
• An Android device (real or emulator) with USB debugging enabled
• adb on PATH — brew install android-platform-tools on macOS
• Optional: Android SDK with d8 if you want to rebuild the DEX from InputBridge.java. The pre-built DEX is checked in, so this is only needed if you change the Java.

Setup (common to both paths)

git clone https://github.com/mantotan/mcp-mobile-agent.git
cd mcp-mobile-agent

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Connect a device
adb devices                          # USB
adb connect <device-ip>:<port>       # Wi-Fi

# Verify the setup
tiktok-warmup device list            # should list your connected device

If tiktok-warmup device list doesn't show your phone, no MCP / Python session is going to work — fix adb first.

Path A — Drive it from Claude Code (recommended)

This is the headline path. The MCP integration runs on a Claude subscription (Claude Code wraps your Pro / Max plan) — there's no per-call Anthropic API cost for the reasoning. The agent loop's model turns all go through your subscription, and every "action" call (take_screenshot, scroll_to_next_video, etc.) is a local Python call against the device. The only metered cost is the proxy bandwidth or mobile data you'd be paying for anyway.

A 30-minute warmup session is on the order of one Claude turn per video for several hundred videos, so a Pro subscription is fine for occasional one-shot sessions. If you're going to run sessions on a recurring schedule (/loop, see below), Max or higher gives you the headroom.

Register the MCP server. Edit your Claude Code settings.json:

{
  "mcpServers": {
    "tiktok-warmup": {
      "command": "tiktok-warmup-mcp",
      "args": []
    }
  }
}

Restart Claude Code. The 11 tools and the workflow://niche-engagement resource should appear under the tiktok-warmup server.

Run a one-shot session. Type something like:

Run a 30-minute warmup session on device 192.168.1.5:5555 for
account my_account. Niche: cooking, recipe, bake, foodie.
Language filter: English. Stop on captcha, login prompt, or
daily-limit signals.

Claude reads the workflow resource, calls start_session, then loops through check_screen_state → get_current_video_info → decide-and-act → scroll_to_next_video until the deadline or a stop signal. See docs/MCP_USAGE.md for the full tool surface and what each call returns.

Run it on a recurring schedule. Account warmup wants short, spaced sessions throughout the day, not one long burst. Claude Code's built-in /loop skill makes that one line:

/loop 8h Run a 30-minute warmup session on device 192.168.1.5:5555
for account my_account. Niche: cooking. Stop on captcha or daily-limit.

That fires the same prompt every 8 hours until you stop the loop. Claude Code keeps the loop running locally — there's no remote infrastructure involved.

There's also a /schedule skill for Claude Cowork-style remote scheduled agents (cron-based, runs on Anthropic's infrastructure). For this project, prefer /loop — Cowork-scheduled runs execute remotely and can't reach a device sitting on your home LAN unless you expose the MCP server publicly, which you almost certainly do not want to do. /schedule is the right tool for cloud-only agents (Gmail, Calendar, etc.); /loop is the right tool for "Claude Code on my laptop with reach to my devices."

Path B — Run it headless (no Claude needed)

The same primitives are exposed as a standalone Python script that takes a profile YAML. No model in the loop — niche classification is the dumb-but-fast substring matcher in examples/run_session.py.

# Copy a profile template and edit it
cp examples/profiles/example_english.yaml examples/profiles/my_niche.yaml
# Edit my_niche.yaml — set device_id, account_id, niche_keywords

# Run a single session
python examples/run_session.py --profile examples/profiles/my_niche.yaml

See examples/profiles/README.md for the profile schema.

What's not in here

This is a single-developer research tool, not a hosted product, so a few things are intentionally absent:

• iOS support. None. The InputBridge is Android-specific.
• Multi-platform abstraction. The screen-state detector, the TikTok-specific gestures, the gift-event schema — all TikTok. A "BrowserAgent" / "GenericAndroidAgent" abstraction would be the right way to generalize, but I haven't done it.
• GUI dashboard. Session state is in SQLite. Queries are run directly or via the Typer CLI.
• Cloud / managed deployment. Designed for a single host with USB/Wi-Fi reach to physical devices. The _sessions dict in the MCP server is process-local; running two MCP servers against the same device would race.
• Multi-finger gestures. The protocol carries one pointer at a time. Pinch-zoom would require a small protocol revision.

A note on how this was built

Most of the typing in this repo was driven through Claude Code, Anthropic's CLI coding agent. The architectural calls — building a custom Java input server to escape adb shell input latency, treating the engagement loop as a state machine with screen-state recovery rather than a happy-path script, paying the per-PID-file dance to set oom_score_adj correctly, making Claude itself the decision-maker over screenshots rather than encoding niche-matching as Python rules — are mine, and so is the debugging when the Phantom Process Killer made the server vanish on every Android 12+ device. The agent made the implementation faster; the system design and the bugs-to-fix loop are the same engineering work as without it.

What I'd do differently

The biggest item is the in-process connection manager: it's straightforward to reason about for a single device, but once you have 3+ phones I'd push the session-orchestration layer out of the MCP server into a dedicated process (or a job queue like Redis-backed RQ) so a crash in the MCP layer doesn't take all the active sessions with it. The MCP tools would then talk to the session service over a local socket, and the MCP server itself would be stateless and trivially restartable.

Second item: the niche keyword matcher in examples/run_session.py is intentionally dumb (lowercase substring match). That's fine when you're driving the loop yourself, but when Claude is the decision-maker the keyword list ends up unused — Claude classifies directly from the screenshot + on-screen text. The right move is probably to drop the keyword matcher from the example and lean fully into Claude's judgment, with the keyword list staying around only as a fallback when running headless via run_session.py.

License

MIT — see LICENSE.