traceplay

traceplay

Record, replay, and test AI agent trajectories. VCR-style replay and assertions for AI agents — capture a real run once, replay it offline with zero tokens, and assert behavior in CI.

Why

AI agents fail silently. A prompt edit, a model upgrade, or a tool schema change can make an agent "succeed" while doing the wrong thing. Existing tools either watch production traces (Langfuse, Phoenix) or lint skill markdown (skillkit, skilllint). traceplay gives you a local, deterministic, language-agnostic regression suite for multi-step agent trajectories — the same way VCR/nock gave web apps reliable HTTP tests.

• Record once — point your agent's BASE_URL at the local proxy; every LLM call is persisted to a JSONL cassette with redacted secrets.
• Replay offline — subsequent runs return recorded responses by request hash. No API keys, no tokens, no flakiness.
• Assert trajectories — declare tool call order, arguments, final answer, token budgets, and forbidden tools in YAML.
• CI-ready — one binary, exit-code gating, console/JSON/Markdown reporters, GitHub Actions included.

30-second demo

# 1. Record a real run (after building from source; see Installation)
node dist/cli.js record --port 8123 --out ./cassettes/weather.jsonl
#    in another terminal: BASE_URL=http://localhost:8123 node my-agent.js

# 2. Write a test suite
cat > suite.yaml <<'EOF'
suite: weather-agent
cases:
  - name: fetches weather and answers
    cassette: ./cassettes/weather.jsonl
    assertions:
      - { kind: tool.called, name: get_weather }
      - { kind: tool.args, name: get_weather, jsonPath: $.city, equals: Xiamen }
      - { kind: answer.contains, text: sunny }
      - { kind: budget.maxTokens, value: 2000 }
      - { kind: forbid.tool, name: execute_shell }
EOF

# 3. Run offline, in CI
node dist/cli.js test suite.yaml

● fetches weather and answers
  [PASS] tool.called — tool "get_weather" called 1 time(s)
  [PASS] tool.args — tool "get_weather" args at $.city equals "Xiamen"
  [PASS] answer.contains — answer contains "sunny"
  [PASS] budget.maxTokens — used 144 tokens, budget 2000
  [PASS] forbid.tool — forbidden tool "execute_shell" not called

5 passed, 0 failed, 0 scaffolded (TODO)

More examples

The examples/ directory ships with ready-to-run suites covering common scenarios. Each has a cassette + suite YAML — run them with traceplay test examples/<name>/suite.*.yaml.

Example	What it demonstrates
demo/	Single tool call + answer + budget assertions (the 30-second demo)
anthropic/	Anthropic-format cassette (content blocks, system field) — proves provider-agnostic replay
multi-tool/	Agent calling two tools in order, with tool.args JSONPath checks and forbid.tool guards

Installation

The npm package is not published yet. Install the current source version:

git clone https://github.com/DorianChn/traceplay
cd traceplay
npm ci
npm run build
node dist/cli.js --help

Requires Node ≥ 20.

Commands

traceplay record

Start a recording proxy. Point your agent's BASE_URL at http://localhost:<port>.

traceplay record [--port 8123] [--upstream https://api.openai.com/v1] [--out cassette.jsonl] [--project name] [--no-redact]

• --port: local proxy port (default 8123)
• --upstream: real LLM provider URL (default OpenAI)
• --out: cassette output path (default cassette.jsonl)
• --project: project name stored in cassette metadata
• --no-redact: disable secret redaction (not recommended)

Supports OpenAI-compatible /chat/completions and Anthropic /v1/messages endpoints. Authorization headers and secret body fields are redacted before persistence.

traceplay replay

Start an offline replay server from a cassette. Incoming requests are matched by hash; on a hit, the recorded response is returned exactly.

traceplay replay --cassette cassette.jsonl [--port 8124]

• --cassette: path to a recorded cassette (required)
• --port: local server port (default 8124)

Unmatched requests return 404 with a hint to re-record. No network calls are ever made.

traceplay test

Run a test suite against one or more cassettes.

traceplay test suite.yaml [--format console|json|markdown] [--output report.md]

• suite.yaml|suite.json: test suite file (required)
• --format: output format (default console)
• --output: write report to file instead of stdout

Exit code 0 = all pass, 1 = any failure, 2 = usage error.

traceplay init

Scaffold a new traceplay project in a directory.

traceplay init [dir]

Creates suite.yaml, cassettes/, and appends traceplay entries to .gitignore.

Assertions reference

Kind	Checks	Example
tool.called	tool invoked (optionally exact times)	{ kind: tool.called, name: get_weather, times: 1 }
tool.order	tools invoked in given subsequence	{ kind: tool.order, names: [search, summarize] }
tool.args	JSONPath match on tool arguments	{ kind: tool.args, name: get_weather, jsonPath: $.city, equals: Xiamen }
forbid.tool	tool never invoked	{ kind: forbid.tool, name: execute_shell }
answer.contains	final answer contains text	{ kind: answer.contains, text: "sunny" }
answer.matches	final answer matches regex	{ kind: answer.matches, regex: "\\d+C" }
answer.judge	LLM-as-judge with rubric (cached)	{ kind: answer.judge, rubric: "mentions temperature" }
budget.maxTokens	total token usage ≤ value	{ kind: budget.maxTokens, value: 2000 }
budget.maxSteps	number of LLM requests ≤ value	{ kind: budget.maxSteps, value: 5 }

tool.args supports equals (exact JSON match) or matches (regex). answer.judge requires TRACEPLAY_JUDGE_API_KEY and caches verdicts to .traceplay/judge-cache/ for deterministic reruns; without a key it is marked todo.

Full field-by-field reference with troubleshooting: docs/assertions.md

Reporters

• console (default): human-readable pass/fail output
• json: structured TestReport for programmatic consumption
• markdown: PR-comment-ready table with pass/fail icons

traceplay test suite.yaml --format markdown --output pr-report.md

Testing Agent Skills

traceplay can generate test cassettes for Agent Skills (SKILL.md) using a mock agent runtime:

import { runSkill } from 'traceplay/src/skills/runner.js';
import { generateSkillSuite } from 'traceplay/src/skills/adapter.js';

// Run a skill once and produce a cassette
await runSkill({
  skillPath: './skills/code-review/SKILL.md',
  userMessage: 'review src/index.ts',
  outPath: './cassettes/code-review.jsonl',
});

// Generate a full test suite from multiple test inputs
await generateSkillSuite({
  skillPath: './skills/code-review/SKILL.md',
  outDir: './skill-tests',
  inputs: [
    { name: 'finds-bugs', userMessage: 'review this buggy file', assertions: [{ kind: 'answer.contains', text: 'bug' }] },
    { name: 'stays-in-budget', userMessage: 'review large file', assertions: [{ kind: 'budget.maxTokens', value: 500 }] },
  ],
});

For real skill testing against your actual agent runtime, record with traceplay record instead.

How it works

your agent ──BASE_URL──► traceplay record ──► LLM provider
                              │
                              ▼
                    cassette.jsonl (header + events)
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
     traceplay replay (offline)      traceplay test (assert)
     match by requestHash             YAML assertions → exit code

A cassette is JSONL: line 0 is metadata, every subsequent line is one TraceEvent (user.message, llm.request, llm.response, tool.call, tool.result, agent.error). The replayer matches incoming requests by sha256(canonicalized request body) and returns the corresponding recorded llm.response (raw body + status + headers).

Full cassette schema reference: docs/cassette-format.md

Project structure

src/
├── cli.ts                  # entry point, command routing
├── types.ts                # core data model (frozen)
├── core/
│   ├── hash.ts             # canonicalization + sha256
│   ├── redact.ts           # secret redaction
│   └── jsonpath.ts         # minimal JSONPath evaluator
├── cassette/
│   ├── store.ts            # JSONL read/write
│   └── normalize.ts        # provider request/response normalization
├── recorder/
│   ├── proxy.ts            # recording HTTP proxy
│   └── forward.ts          # upstream request forwarding
├── replayer/
│   ├── server.ts           # offline replay server
│   └── matcher.ts          # request hash matching
├── assert/
│   ├── engine.ts           # assertion dispatcher
│   ├── judge.ts            # LLM-as-judge with disk cache
│   └── matchers/
│       ├── tool.ts         # tool.called/order/args/forbid
│       ├── answer.ts       # answer.contains/matches/judge
│       └── budget.ts       # budget.maxTokens/maxSteps
├── report/
│   ├── console.ts          # console reporter
│   ├── json.ts             # JSON reporter
│   └── markdown.ts         # Markdown reporter
├── commands/
│   ├── record.ts           # `traceplay record`
│   ├── replay.ts           # `traceplay replay`
│   ├── test.ts             # `traceplay test`
│   └── init.ts             # `traceplay init`
└── skills/
    ├── runner.ts           # mock agent skill runner
    └── adapter.ts          # generate test suites from skills

Comparison

	traceplay	skillkit / skilllint	Langfuse / Phoenix
What it tests	Full multi-step trajectories	Skill markdown structure	Production traces
Replay	Offline, deterministic, zero tokens	No	No
Language	Any (HTTP-boundary proxy)	TS/Python	SDK-specific
Where it runs	Local + CI	Local + CI	Hosted/self-hosted platform
Assertions	9 types (tools, answer, budget)	Lint rules	Metrics & dashboards
Best for	Regression-gating agent behavior	Linting skill packages	Observability & debugging

Development

npm install
npm run build
npm test          # 59 tests, including record→replay integration
npm run dev -- test examples/demo/suite.example.yaml

---

More in this suite

AI agent security toolkit by DorianChn:

• agent-canary — zero-false-positive honeypot tripwires: decoy MCP tools + canary tokens
• agent-gate — tool-call policy gateway: least privilege, short-lived credentials, audit
• reposieve — privacy-first repository context packs for coding agents

License

MIT — see LICENSE.