probity

Probity

Open tools that check claims about AI agents against evidence you pinned: an offline verifier, an observer that records agent runs, and the tools around them. The conformance corpus, vocabulary and admission rules live in https://github.com/probityai/agent-evidence-vectors.

It's for auditors, incident reviewers and relying parties who need a decision they can replay, and for tool authors who want their records checked the same way.

Install the released verifier: uv add agent-evidence-verify==0.5.1, or pip install agent-evidence-verify==0.5.1.

Verified by Probity (github.com/probityai/probity). The probityai command line is released: uv add probityai==0.1.1, or pip install probityai==0.1.1.

Quick start

Check a sample claim with the released package. You need Python 3.12 or newer and uv; there is nothing to clone:

samples="$(uvx --from agent-evidence-verify==0.5.1 python -c 'import importlib.resources as r; print(r.files("agent_evidence_verify") / "samples")')"
uvx agent-evidence-verify==0.5.1 "$samples/case.json" --policy "$samples/policy-june.json"

It prints a decision packet that opens:

Probity decision packet v1
Case: "source-coverage-demo"
Claim: source_text_coverage/v1
Decision: contradicted
Reason: selected_span_missing_from_record

A selected passage is in the source capture but missing from the stored report. Swap in policy-april.json for not_established, or case-complete.json with policy-complete.json for supported.

Status

Two name families, on purpose. agent-evidence-* is the neutral evidence layer: formats and checks any tool can pin without adopting ours. probity* is the product built on it. Each package keeps its own version and release tag, <name>-v<version>.

Evidence layer

package	registry	version	downloads
agent‑evidence‑mcp	PyPI	0.2.5
agent‑evidence‑observer	PyPI	0.2.0

• agent-evidence-mcp: offline, read-only MCP server: checks a claim against evidence bytes and replays the conformance vectors
• agent-evidence-observer: records what an agent run did as pinned evidence

Product

package	registry	version	downloads
probityai	PyPI	0.1.1
agent‑evidence‑verify	PyPI	0.5.1
probity‑ratchet	PyPI	0.4.0
probity‑ratchet‑cases	PyPI	0.2.0
probity‑effects	PyPI	0.2.0	-
agent‑evidence‑replay	PyPI	0.1.0	-

• probityai: the probityai command; probityai verify today
• agent-evidence-verify: offline verifier: supported, contradicted or not established
• probity-ratchet: signed, replayable receipts for improving an agent against held-out cases
• probity-ratchet-cases: agent task cases graded on final state: a simulated company, tool faults, exact graders
• probity-effects: keeps one effect per agent tool call when its reply is lost, records the trail, tests it against lost-reply scenarios
• agent-evidence-replay: replays a lab's agent action record and flags a changed or missing step

Install any of them pinned to the version shown: pip install <package>==<version>.

The Claude Code and Cursor plugins ship from harnesses/agent-evidence-plugins.

Work on the repository

uv sync --all-packages --all-extras
uv run python -m pytest

Documentation

page	read it for
agent‑evidence‑verify	the verifier, its adapters and its recipes
Repository layout	the workspace, the probityai command seam, CI and releases
Agent instructions	commands to run before a change
Contributing	reporting a problem and opening a pull request

License

Apache-2.0, per directory: each member carries its own LICENSE.