AgentTwin

Pre-production assurance and runtime containment for production AI agents.

AgentTwin helps teams answer one question:

Can I safely ship this agent change — and what evidence supports that decision?

---

Why AgentTwin

AI agents can technically succeed while being wrong.

A request may return 200 OK, the agent may produce a confident final answer, and the run may still have:

• called the wrong tool;
• passed dangerous arguments;
• retried an irreversible action;
• skipped required approval;
• followed a malicious instruction hidden in retrieved content;
• looped and burned budget;
• claimed success although external state never changed;
• changed behavior after a prompt, model, tool or permission update.

Traditional observability tells you what happened. AgentTwin is designed to turn that evidence into a decision about what is safe to ship next.

Tracing is infrastructure. The product is the closed loop from real behavior to regression protection, simulation, release evidence and runtime containment.

---

The assurance loop

The key workflow is deliberately circular:

production behavior
      ↓
trace + verified outcome evidence
      ↓
failure / regression mining
      ↓
permanent regression case
      ↓
change detection + blast radius
      ↓
production-shaped twin simulation
      ↓
deterministic + semantic evaluation
      ↓
explainable release gate
      ↓
canary / production
      ↓
runtime containment
      └──────────────→ back to evidence

A bad production trace should not end as an incident note. It should become a durable test that protects every future release.

---

What makes it different

🧬 Production failures become regression protection

AgentTwin can turn failed outcomes, policy violations, loops, unsafe retries and human-flagged traces into reviewed regression cases that automatically join future release checks.

🪞 Production-shaped digital twins

Scenarios are more than prompts. They include initial state, tool twins, permissions, expected outcomes, fault rules and final-state assertions so agents are tested against stateful reality, not toy inputs.

🕸️ Change-aware blast radius

A prompt, model, tool schema, policy or permission change is mapped to affected components, dependency paths, historical failures and relevant scenarios — with evidence for why each test was selected.

🧭 Trajectory-aware evaluation

AgentTwin evaluates final outcome and the path taken: tool choice, arguments, ordering, retries, approvals, loops, side effects, latency, cost, policy decisions and verified final state.

🚦 Explainable release gates

Release decisions are PASS / WARN / BLOCK with machine-readable evidence. One average score can never hide a new critical irreversible failure.

🛡️ Runtime containment

The optional runtime gateway can allow, constrain, require approval or deny tool actions using explicit policy, idempotency and audit — especially for irreversible operations.

---

The questions AgentTwin asks before you ship

WHAT CHANGED?
      ↓
WHAT CAN THAT CHANGE AFFECT?
      ↓
WHICH REAL BEHAVIORS SHOULD WE REPLAY?
      ↓
WHAT FAILS IN A SAFE TWIN?
      ↓
IS THIS CHANGE SAFE TO SHIP?
      ↓
WHAT SHOULD BE WATCHED OR CONTAINED AFTER RELEASE?

This is the product center of gravity — not token charts, prompt playgrounds or another generic trace viewer.

---

Architecture

AgentTwin uses microservices where there is a real difference in workload, language/runtime, scaling, failure mode or security boundary — not for microservice theatre.

Service	Runtime	Responsibility
control-plane	Go	auth, tenancy, projects, agents, releases, gate decisions, audit, API edge
trace-service	Go	OTLP ingestion, normalization, traces, spans, outcomes
graph-service	Go	components, dependency evidence, bounded blast-radius traversal
evaluation-service	Python / FastAPI	datasets, evaluators, comparisons, regression mining, embeddings, review
simulation-service	Python / FastAPI	scenarios, stateful tool twins, replay, fault injection, evidence
runtime-gateway	Go	policy decisions, approvals, idempotency, outbound safety, audit

The V1 infrastructure stays intentionally lean:

PostgreSQL + pgvector · RabbitMQ · OpenTelemetry Collector · S3-compatible object storage · Docker Compose

Kafka, Qdrant, Neo4j, ClickHouse, Temporal, Kubernetes and Redis are not default dependencies. They only enter when measured scaling needs justify them.

Full architecture: docs/architecture/system.md

---

Release gate

The gate is evidence-first. A secondary risk index may help sorting, but it is never the authority.

BLOCK examples

Critical deterministic scenario failure · cross-tenant leakage · missing approval · duplicate irreversible side effect · authorization bypass · prompt-injection critical failure · a claimed success disproven by state verification.

WARN examples

Meaningful latency or cost regression · new tool permission · new affected dependency without test coverage · evaluator uncertainty · insufficient simulation coverage.

PASS

Critical gates are green and configured policy thresholds are satisfied.

---

Core product surfaces

Trace Explorer

Inspect trajectories, tool calls, model calls, policy decisions, retries, outcome verification and linked regressions.

Regression Miner

Redact, match, embed, group, classify and promote production failures into permanent tests.

Scenario Library

Versioned scenarios with state, tool twins, fault injection, evaluators, severity and reproducible seeds.

Blast Radius

Map changed components to impacted dependencies, scenarios, policies and historical failures.

Baseline vs Candidate

Replay identical initial states and surface improved, unchanged, regressed and new-critical-failure slices.

Runtime Gateway

Enforce action policy with allow, allow_with_limits, require_approval and deny.

---

Security by design

AgentTwin treats agent reliability and agent security as the same production problem.

• Content capture is off by default.
• Client-side redaction supports drop, mask and hash modes.
• Tenant identity comes from verified credentials — not agent-supplied span attributes.
• Irreversible runtime actions fail closed when policy evaluation fails.
• Approval tokens are single-use, short-lived and bound to the exact action hash.
• Simulation workers are designed for isolation and restricted egress.
• Remote tool descriptions and results are treated as untrusted input.
• Multi-tenant isolation, SSRF controls and prompt-injection suites are release-blocking concerns.
• Gate decisions and evidence are designed to be immutable and auditable.

Threat model: docs/security/threat-model.md

---

Agent Assurance Coverage

Traditional line coverage does not describe agent safety.

AgentTwin is designed to measure coverage across:

Coverage surface	Example question
Tool coverage	Did we exercise every tool the agent can call?
Critical tool coverage	Did we test every irreversible action?
Policy coverage	Did scenarios exercise active policies?
Failure-mode coverage	Which known failure classes were replayed?
Regression coverage	Did every promoted regression run again?
Dependency coverage	Are affected components represented in the suite?
Fault coverage	Did we test timeouts, 429/500/503, stale and duplicate responses?
Outcome verification	Did we verify business state instead of trusting agent self-report?

AgentTwin does not turn these into a misleading “95% safe” claim.

---

Development

AgentTwin is under active phased development. Phases 0–3 are done; the blast radius, release gate, regression miner and runtime gateway follow.

Track live build status and the evidence of each phase: docs/plan/implementation-board.md

What works today (Phases 1–3)

• Trace ingestion from any OpenTelemetry-instrumented agent through the OTel Collector into the trace service: GenAI semantic conventions, per-project content policy, idempotent ingest, deterministic trace summaries, claimed vs verified outcomes with contradiction detection.
• Python SDK (packages/sdk-python) with client-side redaction, content capture off by default and measured overhead (benchmarks); it also starts simulations, builds datasets and runs evaluations.
• Control plane: dev and OIDC authentication, organizations, projects, RBAC, API keys, immutable agent manifests and versions, hash-chained audit log.
• Simulation: versioned scenarios run against declarative, stateful tool twins with injected faults (timeout after mutation, a tool that lies about success, rate limits…); every case records its trajectory, the twin's state diff and a verdict per expectation; the same seed reproduces the verdicts.
• Evaluation: deterministic expectations (state, trajectory, policy, side effects) and semantic ones graded by a judge — a deterministic fake by default, Anthropic or any OpenAI-compatible model behind configuration (JUDGE_PROVIDER), with a spending budget and a verdict cache. An evaluation runs a baseline and a candidate version on one pinned suite and seed, and classifies every case: new critical failure, regressed, incomplete, improved or unchanged, with the first step where the two diverged. Versioned datasets, human reviews that replace a verdict, and judge calibration against labels people gave (agreement and Cohen's kappa).
• Web UI: trace explorer and trace detail with waterfall; scenarios and simulations with the evidence down to the tool call and the state diff; evaluations, compared cases side by side, datasets, the review queue and judge calibration.
• Demo: Demo Co's support-refund agent with production-like tools, driven by a deterministic scripted planner (or Anthropic Claude when DEMO_AGENT_MODEL=anthropic and a key are set). Version 1.2.4 is good, 1.3.0 regresses (it refunds before checking the policy and trusts a tool's false success), 1.3.1 fixes it. make dev seeds 9 scenarios, simulations of 1.2.4 and 1.3.0, the refund-regression-suite dataset and an evaluation of 1.3.0 against 1.2.4 that finds the regression: two new critical failures and one regressed case.

Quick start (local)

Requirements: Docker with Compose 2.20+, make, curl.

git clone https://github.com/Ozgurisikdamar/AgentTwin.git
cd AgentTwin

cp .env.example .env
make dev        # builds and starts the stack, waits until healthy, seeds the demo workspace

Then open http://localhost:3000 and sign in with a demo account (development authentication):

Account	Role
owner@demo.agenttwin.dev	owner
admin@demo.agenttwin.dev	admin
engineer@demo.agenttwin.dev	engineer
reviewer@demo.agenttwin.dev	reviewer
viewer@demo.agenttwin.dev	viewer

Useful developer entrypoints:

make help
make demo       # one more refund conversation through the demo agent; prints the trace link
make doctor     # checks Docker, env, ports, PostgreSQL, RabbitMQ, OTel, migrations, services
make e2e        # Playwright tests against the running stack
make test       # unit + integration (real PostgreSQL + RabbitMQ) + frontend tests
make lint       # formatters, linters and type checkers for Go, Python and TypeScript
make down       # stop (keeps data) · make reset: wipe data and start again

Instrument your own agent with the Python SDK: see packages/sdk-python/README.md. Point AGENTTWIN_OTLP_ENDPOINT at http://localhost:4318 and use the demo project key from .env (AGENTTWIN_DEMO_API_KEY).

---

Repository map

apps/
  web/                Next.js UI and its BFF (/api/v1 through the control plane)

services/
  control-plane/      Go · auth, tenancy, projects, agents, audit, API edge
  trace-service/      Go · OTLP ingestion, traces, outcomes
  graph-service/      Go · dependency graph, bounded blast radius (Phase 4, in progress)
  simulation-service/ Python · scenarios, tool twins, faults, simulation runs
  evaluation-service/ Python · datasets, evaluation runs, judges, reviews

packages/
  contracts/          shared versioned event schemas and OpenAPI documents
  gokit/              shared Go infrastructure primitives
  core-py/            shared Python kit (service runtime, jobs, evaluators, contract checks)
  sdk-python/         Python SDK
  scenario-schema/    versioned scenario, twin, manifest and policy schemas

demo/
  support-refund-agent/  Demo Co's refund agent, its tools, manifests and scenarios

docs/
  architecture/       system architecture and domain references
  adr/                architectural decision records
  plan/               phased implementation board
  security/           threat model and security design

infra/
  otel/               OpenTelemetry collector configuration
  prometheus/         metrics collection
  grafana/            local observability provisioning

The runtime gateway, the CLI and the TypeScript SDK join as their phases land.

---

Design principles

1. Deterministic where possible. Use an LLM only when semantic reasoning genuinely adds value.
2. Evaluate trajectories, not only final text.
3. Outcome evidence beats agent self-report.
4. Production failures become future tests.
5. Simulation should resemble deployment.
6. Blast radius is probability × possible damage.
7. Human approval is selective, not universal.
8. Every release decision must be explainable.
9. No bespoke ML model until reviewed data justifies it.
10. No infrastructure component without a measured reason to exist.

---

Documentation

Document	Purpose
System Architecture	C4-style architecture, request paths, data ownership and events
Implementation Board	Build phases and acceptance criteria
Architecture Decisions	Why the system is built this way
Threat Model	Security boundaries and abuse cases
Shared Contracts	Versioned internal event and contract definitions

---

License

Apache License 2.0 — see LICENSE.

Ship agents with evidence, not hope.

AgentTwin · Simulation · Regression Mining · Blast Radius · Release Assurance · Runtime Containment