mcp-fault-injection-harness

MCP Fault-Injection Harness

Chaos engineering for AI agents that use the Model Context Protocol.

mcpfault is a transparent stdio proxy that sits between any MCP client and any MCP server and injects the failures real integrations hit: timeouts, errors, malformed JSON, schema drift and stale results. Around it sits a harness that measures how agents cope. It includes a mock server, a task suite with ground truth, scripted agents with different reliability policies, a hook for a real LLM agent, and a benchmark that reports success rate, retries and recovery time per fault type.

Why

Agents act through tools, and tools fail. Demos rarely show the failures that matter in production:

• a write succeeds on the server but its response is lost, so a retry repeats it
• a server is redeployed mid-session and renames a field
• a cache or replica returns an earlier answer after a write
• a proxy or crash puts half a JSON object on the wire

The worst of these are not crashes. They are silent failures: the agent reports a wrong answer with full confidence. This project makes each failure reproducible with a seed and measurable with a number.

Results

All numbers come from docs/results/report.md. The run covers 8 fault profiles, 3 agents and 100 tasks per session, 2,400 tasks in total. You can regenerate them with mcpfault bench.

Fault profile	naive	retry-only	resilient
baseline (no faults)	100%	100%	100%
timeout	55%	94% (5% silent)	99%
error	60%	99%	99%
malformed	68%	92% (7% silent)	100%
stale	86% (14% silent)	86% (14% silent)	100%
drift_input	52%	52%	100%
drift_output	52% (48% silent)	52% (48% silent)	100%
mixed	53% (11% silent)	69% (31% silent)	99%

What the run shows:

• Retries alone trade loud failures for silent ones. Under mixed, the retry-only agent finishes more tasks than the naive one (69% vs 53%). Its silent-failure rate also climbs from 11% to 31%. Blind retries of a non-idempotent write created 19 duplicate incidents across profiles. Retries also cannot notice stale or drifted data.
• The resilient policy finishes 99–100% of tasks in every profile with zero silent failures. It is the same task code with a different call boundary. Its few remaining failures come from exhausting the retry budget (four faulted attempts in a row), and it reports them as errors.
• Recovery is cheap except for timeouts. The median time to recover is about 20 ms for errors, malformed responses and stale reads. For timeouts it is about 520 ms, because the client has to wait out its 0.5 s deadline before it can retry.
• The proxy adds about 0.1 ms to the median tool call (0.07 ms direct vs 0.16 ms through the proxy).

How it works

flowchart LR
    A["Agent<br/>naive · retry-only · resilient · LLM"] -- "JSON-RPC over stdio" --> P["mcpfault proxy<br/>fault engine"]
    P -- "forwarded requests" --> S["MCP server<br/>bundled mock or any stdio server"]
    S -- "responses" --> P
    P -- "faulted / clean responses" --> A
    P -.-> E[("fault events<br/>(by request id)")]
    A -.-> T[("attempt trace<br/>(by request id)")]
    S -.-> ST[("server state")]
    E & T & ST --> M["metrics + report"]

• The proxy relays newline-delimited JSON-RPC (the MCP stdio transport). Every client request passes through the fault engine. The engine decides, from a per-rule seeded RNG, whether to forward the request, answer it itself, or corrupt, delay or drop the response. Notifications, server-initiated requests and anything it does not touch are relayed byte for byte.
• The client (mcpfault.client) maps each failure mode to its own exception and records every request as an attempt.
• Metrics join the agent's attempts to the proxy's fault events on the JSON-RPC request id. They judge each answer against ground truth: the mock world's definition plus the server's final state. Each task then gets one outcome:

Outcome	Meaning
success	correct, and no fault touched the task
recovered	correct despite at least one injected fault
failed_detected	the agent gave up or crashed, so the failure is visible
failed_silent	the agent returned a wrong answer or left wrong state on the server

Fault catalog

Fault	Where it acts	What the client sees	Real-world analogue
timeout	Server acts; response dropped or delivered late	no reply, then maybe an orphaned late reply	network partition, overloaded server
latency	Server acts; response delayed	a slow but valid reply	congestion, cold start
error	Proxy answers; server never contacted	JSON-RPC error, or MCP tool result with isError	upstream outage, rate limiting
malformed	Server acts; response corrupted	truncate, not_json, missing_result, wrong_id, wrong_type	buggy server, proxy error page, encoding bugs
schema_drift	Persistent from call N	renamed arguments (-32602) or renamed result fields; tools/list updated; optional list_changed	server redeployed mid-session
stale	Proxy answers from a cache of earlier responses	a valid but outdated reply	replica lag, caching layer

Faults target tools/call. Schema drift also rewrites tools/list (and initialize capabilities when notify = true). Everything else passes through untouched.

Quickstart

git clone https://github.com/GauthamPrabhuM/mcp-fault-injection-harness
cd mcp-fault-injection-harness
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pytest                                                        # 77 tests, ~30 s
mcpfault run --profile profiles/stale.toml --agent naive      # one session
mcpfault run --profile profiles/stale.toml --agent resilient
mcpfault bench --iterations 25 --out results                  # full matrix + report

mcpfault run prints a summary and writes the full traces (attempts.jsonl, proxy_events.jsonl, server_state.json, records.jsonl) under runs/.

Put it in front of any MCP server

The proxy needs nothing from the server. It wraps any stdio MCP server command:

mcpfault proxy --config profiles/mixed.toml --events faults.jsonl \
  -- npx -y @modelcontextprotocol/server-filesystem ~/sandbox

Claude Code:

claude mcp add flaky-fs -- mcpfault proxy --config /abs/path/profiles/mixed.toml \
  -- npx -y @modelcontextprotocol/server-filesystem ~/sandbox

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "flaky-fs": {
      "command": "mcpfault",
      "args": ["proxy", "--config", "/abs/path/profiles/mixed.toml",
               "--events", "/tmp/mcpfault-events.jsonl",
               "--", "npx", "-y", "@modelcontextprotocol/server-filesystem", "/Users/me/sandbox"]
    }
  }
}

Every injected fault lands in the events file with its request id. On exit the proxy prints a summary to stderr.

Writing a fault profile

name = "flaky-writes"
seed = 7                         # same seed + same request sequence = same faults

[[faults]]
type = "timeout"                 # forward the request, drop the response
tools = ["open_*", "set_*"]      # glob patterns over tool names (default: all)
probability = 0.2
after = 5                        # let the first 5 matching calls through
limit = 10                       # at most 10 injections from this rule

[[faults]]
type = "schema_drift"
tools = ["get_device_status"]
after = 40                       # the "deploy" happens on the 41st call
rename_args = { device_id = "device" }
rename_fields = { status = "state" }
notify = true                    # send notifications/tools/list_changed

Type	Options (defaults)
timeout	delay_s (unset: drop the response)
latency	delay_s = 0.25, jitter_s = 0.0
error	mode = "protocol" or "tool", code = -32603, message
malformed	modes = [...] any of truncate, not_json, missing_result, wrong_id, wrong_type
schema_drift	rename_args, rename_fields, notify = false
stale	pick = "previous" or "oldest", history = 8

Rules are checked in order and the first one that fires wins. Schema drift is a state change rather than a one-off fault, so it applies in addition. Validate profiles with mcpfault validate profiles/*.toml.

The agents

All three scripted agents run the same task code (agents/tasks.py). That code is written the way most integration code is written: it trusts field names and types. Only the policy at the call boundary differs, so the comparison works as an ablation.

Mechanism	naive	retry-only	resilient
Per-call timeout	✓	✓	✓
Retries with exponential backoff and full jitter, on retryable failures only		✓	✓
Output validation against the tool's outputSchema			✓
Freshness check: reject results whose as_of revision is not newer than one already seen			✓
Schema reconciliation: re-read tools/list, map unambiguous renames, fail loudly on ambiguous ones			✓
Idempotency keys on non-idempotent writes			✓

The task suite has four workflows. Each one stresses a different weakness:

Task	What it exposes
site_triage	fan-out reads; a renamed field silently changes the answer
interface_check	numeric thresholds; wrong-typed values crash naive code
maintenance_roundtrip	read-after-write; a stale read gives the wrong verdict
open_incident	non-idempotent write; a blind retry after a lost response creates a duplicate

LLM agent (bring your own API key)

agents/llm.py solves the same tasks from natural-language prompts. It uses the tools advertised by tools/list, and every failure comes back to the model as an is_error tool result. Recovery is left entirely to the model.

pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
mcpfault run --profile profiles/mixed.toml --agent llm --iterations 2
mcpfault run --profile profiles/mixed.toml --agent llm --iterations 2 --reliability-hints

• Model. It defaults to claude-opus-5-5 at --effort medium, with server-side refusal fallback enabled (--no-fallbacks turns it off).
• Prompting. --reliability-hints tells the model the connection is unreliable and explains as_of. Comparing runs with and without it is a ready-made experiment.
• Cost. Each task takes several model turns, so start with a few iterations.
• Swapping the model. The backend is a small interface (LLMBackend), so other models can be plugged in.

The committed results cover the scripted agents only, because the LLM agent needs an API key to run.

Project layout

src/mcpfault/
  proxy.py          stdio proxy: relays bytes, routes requests through the engine
  faults.py         fault engine: seeded decisions, transforms, drift, stale cache
  config.py         TOML profiles and validation
  client.py         minimal async MCP client with typed failures and attempt tracing
  mock/             deterministic network-ops MCP server and its world (ground truth)
  agents/           task suite, scripted policies, LLM agent
  harness.py        sessions and the benchmark matrix
  metrics.py        outcome classification and aggregation
  report.py, chart.py   Markdown report and SVG chart
profiles/           ready-made fault profiles
tests/              unit, end-to-end (real subprocesses) and official-SDK interop tests

Design notes

• No runtime dependencies. The proxy, client, mock server and harness use only the standard library. anthropic is an optional extra for the LLM agent, and the official mcp SDK is used only in interop tests.
• Server-agnostic, checked against the official SDK. The tests drive the proxy with the official MCP Python SDK on both sides: an SDK client through the proxy to the mock server, and the harness client through the proxy to an SDK-built server.
• Deterministic. Each rule has its own RNG seeded from the profile seed, and decisions are made at request time. A given profile, seed and request sequence always injects the same faults, so failures can be replayed.
• Faults act at the right layer. A timeout lets the server act and loses only the response, which is what produces duplicate writes. An error short-circuits before the server. Stale replays an earlier response without contacting the server.
• Freshness via revisions. The mock server stamps every result with as_of, a monotonic revision, much like etcd revisions or Kubernetes resourceVersion. That is what lets a client detect stale reads at all.
• Fail loudly on ambiguity. Schema reconciliation maps a renamed property only when exactly one new property has the same shape and nothing else claims it. Otherwise the agent raises ContractChanged instead of guessing. A test covers this.
• Robust stdio. The proxy reads and writes its own stdin and stdout on threads, with raw os.read/os.write. This works the same for pipes, files and TTYs, and avoids the interpreter-shutdown crash that daemon threads holding buffered-IO locks can cause.

Limitations and roadmap

• stdio only. A Streamable HTTP transport is the next step. The engine is already transport-agnostic.
• tools/call only. Faults do not yet target resources, prompts or server-initiated requests such as sampling and elicitation.
• Simple scripted reliability. The resilient policy has no circuit breaker or cross-call retry budget. Per-flag ablation runs would quantify what each mechanism buys.
• No LLM benchmark yet. Benchmarking the LLM agent across models, with and without reliability hints, is still to be done.

License

MIT