AgentRigor

AgentRigor

Property-based reliability testing for tool-using AI agents under partial failures.

AgentRigor finds a class of production bugs that happy-path agent evaluations miss: a tool changes the world, its response is lost, and the agent retries the side effect.

tool commits an effect -> response times out -> agent retries -> duplicate side effect

Instead of judging prose, AgentRigor checks deterministic properties over tool calls and final state. It records the failure and is being built to reduce flaky, long agent trajectories into small regression tests.

Status: early research preview. The Python runtime, contract-guided campaign generator, and MCP v2 adapter/runtime run today across in-memory, stdio, and Streamable HTTP transports. Offline and live replay, a five-environment local benchmark, an operational web console, and a self-contained evidence dashboard are also executable. Do not use this as a safety boundary for irreversible actions.

简体中文 · Architecture · Benchmark · Model experiments · LangGraph adapter · Scenario generation · Interactive showcase · Research plan · Roadmap

Current evidence at a glance

Artifact fact	Current deterministic fixture
Resettable application environments	5
Generated fault scenarios	41
Blind-retry counterexamples	23
State-aware-recovery counterexamples	10
Counterexamples prevented by recovery	13
Automated tests / branch-aware coverage	97 / 87.36%

These counts validate the harness and make changes reviewable. They are not presented as live-LLM results or proof that one agent strategy is generally superior. The versioned interpretation policy lives in paper/benchmark-protocol.md.

Run experiments from the web console

This starts an operational local console rather than the read-only evidence showcase:

python -m pip install -e .
agentrigor web

Open http://127.0.0.1:8765/ to choose an environment, agent strategy, and fault occurrence bound; start a real local run; inspect violated properties, injected faults, timelines, and final business state; or download the JSON report. Reports persist under reports/web-runs/ and remain available in run history after a restart.

To run a live tool-calling model from the same page, configure dependencies and credentials in the PowerShell process that starts the server:

python -m pip install -e ".[openai,langgraph]"
$env:OPENAI_API_KEY="your-key"
$env:AGENTRIGOR_MODEL="your-model-id"
agentrigor web

Select Live model agent to configure the native loop or LangGraph, prompt profile, repeated trials, reasoning effort, and an optional compatible base URL. The API key remains server-side: the Web API rejects key fields and never persists a credential in a report.

Local live model (no API key)

A CUDA host can run a Transformers tool-calling model directly. This workspace keeps the downloaded Qwen3-0.6B files under local-model/models/Qwen3-0.6B. Start the local-model console with:

powershell -ExecutionPolicy Bypass -File scripts/start-local-model-web.ps1

Open http://127.0.0.1:8767/, select Live model agent, then Local Transformers (CUDA). Inference stays on the local GPU while all business tools remain inside resettable sandboxes. The report records the provider, model, CUDA device, dtype, decisions, tokens, and latency.

Use agentrigor showcase for the read-only paper/README evidence view and agentrigor web for interactive work.

The bug in 30 seconds

Run the vulnerable agent:

python -m pip install -e .
agentrigor doctor
agentrigor demo --agent vulnerable

AgentRigor injects a timeout after create_event commits. The agent retries blindly:

AgentRigor result: FAIL
[FAIL] no_duplicate_effects: found 1 duplicated logical side effect(s)

Injected fault:
  tool=create_event phase=after_effect committed=True

Final state:
  evt-1  Design review  2026-09-01T10:00:00+08:00
  evt-2  Design review  2026-09-01T10:00:00+08:00

Now run verify-before-retry behavior:

agentrigor demo --agent verified

The agent observes calendar state before retrying and the same property passes.

Now expand the two tool contracts into a bounded fault campaign:

agentrigor campaign --agent verified

The generator finds that verify-before-retry still fails under duplicate delivery and under a compound post-commit failure plus stale verification read.

Export those failures as integrity-checked research artifacts and replay them offline or live:

agentrigor campaign --agent verified --failure-dir failures
agentrigor replay failures/create_event.duplicate_delivery.1.yaml
agentrigor replay failures/create_event.duplicate_delivery.1.yaml --live

Run the 41-scenario benchmark across calendar, mail, helpdesk, commerce, and object-storage environments:

agentrigor benchmark --agent verified --report reports/benchmark.json

The blind-retry and state-aware reference strategies expose 23 and 10 deterministic counterexamples, respectively. These are harness-regression fixtures, not live-model results. See the benchmark guide for interpretation and complete trace/failure export.

Interactive showcase

Generate a self-contained dashboard plus both machine-readable benchmark reports:

agentrigor showcase --output reports/showcase
agentrigor showcase --output reports/showcase --serve

Open http://127.0.0.1:8765/ while the second command is running. The dashboard compares identical fault schedules, filters all 41 scenarios, and expands stable counterexample timelines for partial checkout, delayed object visibility, and ambiguous calendar commits. It deliberately labels the scripted baselines as harness evidence rather than LLM evidence.

Freeze the complete deterministic release evidence with hashes:

agentrigor release-bundle --output release/agentrigor-0.1.0-evidence

Run the same campaign through a live tool-calling model with repeated-trial statistics:

python -m pip install -e ".[openai,artifacts]"
agentrigor model-benchmark --model gpt-5.6-luna --trials 1 \
  --report reports/model-experiment.json

Every model decision is recorded inside the trace, so a YAML counterexample can replay the model's tool decisions and rerun the environment without another API call. See Model-driven experiments.

Run the same tasks, tools, fault schedules, properties, and report schema through LangGraph's current create_agent harness:

python -m pip install -e ".[langgraph,artifacts]"
agentrigor model-benchmark --framework langgraph --model gpt-5.6-luna --trials 1 \
  --report reports/langgraph-experiment.json

This is a framework-controlled agent loop, not a LangGraph-shaped wrapper around AgentRigor's native loop. Its messages are normalized into the same portable decision recording for API-free replay. To validate the complete framework path without a credential or billable request, run:

python -m examples.langgraph_scripted_smoke

The scripted smoke test checks a fixed harness fixture; it is not presented as an LLM result.

For a clean-machine reproduction checklist, expected outputs, and artifact inventory, see docs/reproducibility.md.

Why another agent testing project?

Most agent evaluations answer, "Did the task succeed?" Protocol conformance tests answer, "Was the message valid?" Observability tools answer, "What happened?"

AgentRigor asks a different question:

When tools partially fail, which business properties stop being true?

It focuses on failures with ambiguous outcomes:

• the operation committed but the response timed out;
• only part of a multi-resource update committed;
• a write succeeded but is not immediately visible;
• authorization expires halfway through a workflow;
• a schema changes between planning and execution;
• rate limiting or retries duplicate an irreversible effect.

Working core

• Effect-aware tool contracts (read, create, update, delete, send, execute)
• Logical side-effect identities derived from tool arguments
• Fault injection before a call, at named partial-effect checkpoints, or after its effect commits
• Duplicate-delivery and stale-response perturbations
• Deterministic contract-guided single and compound fault generation
• Campaign execution with one exportable JSON trace per scenario
• Versioned YAML counterexamples with trace and full-artifact SHA-256 checks
• Offline property replay and live fault-schedule reproduction
• Detached state snapshots and JSON execution traces
• Deterministic no_duplicate_effects property
• Effect-cardinality, state-cardinality, and bounded-recovery properties
• Portable final-state invariants with JSON-compatible paths
• Calendar, mail, multi-step helpdesk, non-atomic commerce, and delayed-visibility storage environments
• Versioned JSON benchmark reports with effect/fault and commit-outcome coverage
• Self-contained interactive comparison dashboard with no external frontend dependencies
• Provider-neutral model tool loop with strict schemas and bounded turns/calls
• Optional OpenAI Responses API adapter
• Optional LangGraph v1 / LangChain create_agent adapter using the framework-owned tool loop
• Model-decision recording, API-free replay, token/latency accounting, and Wilson intervals
• Framework-neutral Python tool runtime
• Wilson-interval, probability-aware fault-schedule shrinker
• Vulnerable and verified calendar agents
• Optional MCP v2 discovery/result adapter with an in-memory server example
• Async MCP fault interception with the same traces and properties as Python tools
• Official MCP v2 integration tests over in-memory, stdio, and Streamable HTTP
• Python 3.11-3.13 test matrix

See the MCP adapter guide for the official SDK integration. See Failure artifacts for the portable regression format.

Real MCP integration

python -m pip install -e ".[mcp]"
python -m examples.mcp_fault_campaign
python -m examples.mcp_stdio_smoke
python -m examples.mcp_http_smoke

The generated MCP campaign reproduces the same two counterexamples as the plain Python runtime, showing that the fault model is transport-independent rather than tied to the demo implementation.

Minimal API

from agentrigor import (
    EffectKind,
    FaultPhase,
    FaultRule,
    NoDuplicateEffects,
    ToolContract,
)
from agentrigor.demo import build_calendar_case
from agentrigor.runner import run_case

case = build_calendar_case(verified=False)
faults = (
    FaultRule("create_event", FaultPhase.AFTER_EFFECT),
)
result = run_case(case, faults)

assert not result.passed
assert result.violated[0].name == "no_duplicate_effects"

Research thesis

AgentRigor is designed as a research artifact, not a paper-shaped wrapper around a CLI. The central hypothesis is that effect-aware, stateful partial-failure testing discovers agent reliability bugs that protocol tests and fixed end-to-end benchmarks systematically miss.

The planned paper contributes:

1. an effect-aware fault model for tool-using agents;
2. stateful generation across pre-commit, post-commit, visibility, authorization, and schema faults;
3. deterministic state properties that avoid LLM-as-a-judge where possible;
4. probabilistic shrinking for non-deterministic failure trajectories;
5. an empirical study across agent frameworks, models, and MCP/OpenAPI tools;
6. externally validated bugs, fixes, and reusable regression tests.

No paper results are claimed yet. Hypotheses, baselines, metrics, and threats to validity are tracked in paper/README.md before the large evaluation campaign begins.

Development

python -m pip install -e ".[dev]"
ruff check .
pytest --cov=agentrigor --cov-report=term-missing

The core deliberately has no runtime dependencies. Framework and protocol adapters will remain optional extras.

Project principles

• State before prose: prefer business-state assertions over model judges.
• Ambiguous failures first: inject faults where the agent cannot infer whether an effect committed.
• Small reproductions: every discovered failure should become a readable regression test.
• Framework neutral: test agents without requiring a rewrite into a new orchestration framework.
• Evidence over benchmark theater: confirmed bugs and accepted fixes matter more than a single score.
• Local and safe by default: examples use simulated services and never trigger real external effects.

Contributing

AgentRigor is looking for tool contracts, realistic partial-failure scenarios, framework adapters, and reproducible bugs. See CONTRIBUTING.md. Only test services you own or have explicit permission to test; see SECURITY.md.

See CHANGELOG.md for the implemented preview surface and known research gaps. Maintainers can use RELEASE.md for the complete 0.1.0 release gate.

License

Apache-2.0. See LICENSE.