mcp server

agentsec-agentdrill

Attack packs run in CI against your deployed agent policies

agentic-suite/agentsec-agentdrillcommunitylistedunreviewed

Use tool

Set up

Runs on your computer. Pick your agent and operating system.

Before you start

uv (`curl -LsSf https://astral.sh/uv/install.sh | sh`)

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "agentsec-agentdrill": {
      "command": "uvx",
      "args": [
        "agentsec-agentdrill"
      ]
    }
  }
}

same format Cursor, GitHub Copilot CLI, Junie, JetBrains AI Assistant, Augment (Auggie CLI), Factory Droid, Kiro, Devin CLI +12

Values in angle brackets are placeholders. Restart Claude Desktop after editing its config.Claude Desktop docs

README

synced Oct 29, 2026

agentdrill

Prove — Reproducible attack packs run in CI against your deployed agent-defense stack, with honest, published results — including the attacks we did not stop.

Part of the agentic-suite — open-source, harness-agnostic security for AI agents.

Status: v0.1.0 — released (2026-10-25). Install and run:

pip install agentsec-agentdrill        # package: agentsec-agentdrill · CLI + import: agentdrill
agentdrill quickstart                  # first honest drill result in ~15 minutes (bundled demo)

Field-tested end to end (73/73 units, Track B 19/19 agents across 8 frameworks) — see the field test report · release notes. The PRD is the design of record, published early on purpose so we can be held to it.

Install & Quick Start

# Install (PyPI) — latest: v0.1.0
pip install agentsec-agentdrill          # package: agentsec-agentdrill · CLI + import: agentdrill
# pin the first release for reproducibility:
# pip install agentsec-agentdrill==0.1.0
agentdrill --help                        # list commands

# Verify what you downloaded (optional, offline): hash check + SBOM + build attestation
pip download agentsec-agentdrill==0.1.0 -d ./dl --no-deps
shasum -a 256 ./dl/*.whl                 # compare against the release SHA256SUMS
#   PyPI: https://pypi.org/project/agentsec-agentdrill/
#   SBOM (CycloneDX) + SHA256SUMS + signed provenance: docs/release/v0.1.0/release-evidence.md

# First run in ~15 minutes, against a bundled demo stack (no real stack required)
agentdrill quickstart

# Drill your own deployed stack
agentdrill run --pack replit-destructive-command --target ./my-stack.yaml --sandbox

# Gate a policy change in CI — a rule that un-blocks an attack is rejected
agentdrill run --pack incidents-core --target ./stack.yaml --ci

# Verify a published result with nothing installed
agentdrill verify ./benchmark-release.json

The proof loop: pick a pack → point at your deployed stack → run locally or in CI → read an honest result (blocked / missed / degraded) + kill-time. See PRD 04.

See it before installing

Run agentdrill quickstart to boot a static, bundled demo stack and produce a first drill report — one pack's honest outcome split, a kill-chain timeline, and an exportable evidence bundle. It runs offline, opens from file://, needs no account, and is pure synthetic data watermarked demo — never evidence.

Why

Every agent-security vendor oversells detection, and almost nobody publishes reproducible evidence of what their controls actually stop. The benchmark record is brutal: simple prompt injection retains ~46% ASR against some of the best-known defenses, and one hacking prompt nearly doubles ASR against GPT-4 ReAct agents. Meanwhile red-team tooling (PyRIT, garak, promptfoo, AgentDojo) is strong but offline and point-in-time — it red-teams a snapshot, not a live policy stack.

Nothing treats agent policies as continuously regression-tested code. agentdrill is that layer: the ecosystem's proof-of-work engine. See PRD 01.

What it does

Capability	Description
Attack packs	Curated, versioned incident-replay packs (Replit destructive-command, MCP tool poisoning, Plugin4Shell supply-chain, Slack-AI exfil, credential evasion)
Runs the deployed stack	Drives the real enforcement path (MCP proxy, credential broker, confinement, compiled policy, halt) — never a simulation
Honest results	Every published table includes the misses; no scoring theater
Kill-time verification	Times revocation to actual cessation (probed, not inferred) — a full kill-chain timeline, not one latency
CI gate	A policy PR that un-blocks a known attack is rejected before merge
Public benchmark	Reproducible per-release results — a third party re-runs and gets the same numbers
Efficacy control	A `blocked` number only counts if the attack demonstrably succeeds unguarded (the control group no one else runs)
Evasion pack	Attacks on the ecosystem itself (tamper, hook-disable, policy-edit, telemetry-injection) run every release
Interop	Import promptfoo / garak / AgentDojo / PyRIT suites — don't reinvent
Standards-tagged	Every pack mapped to MITRE ATLAS + OWASP (LLM + Agentic)
Harness-agnostic	Works across harnesses by construction, or ships named per-harness packs

Honest by design

This is the differentiator, so it is enforced, not aspirational (PRD 07, PRD 20):

Misses are first-class — a run cannot publish without them.
A single run is never a claim — published numbers are mean ± std over ≥3 pinned seeds; kill-time reports p50/p95/max.
Contamination is disclosed, not hidden — cases overlapping a judge/target model's training data are flagged in the table.
Flag, never silently fix — expected outcomes are preregistered; a change is flagged with a re-pin note.
Disputes get answered — a public adjudication process with a 14-day SLA on every published finding.

Compatibility

A harness is "compatible" only when its pack passes in CI; release notes carry the compatibility table (PRD 05 §Compatibility).

Tier	Harness	Support
1 — Coding agents	Claude Code · Cursor · GitHub Copilot · Codex CLI · Gemini CLI	≥2 at v0.1.0 (per-harness packs)
2 — Frameworks	LangGraph · CrewAI · OpenAI Agents SDK · AutoGen/AG2 · LlamaIndex · PydanticAI · smolagents	via promptfoo/garak interop; adapter packs in scope
3 — Protocol/enterprise	Generic MCP clients · enterprise platforms	MCP-layer coverage via agentgate; enterprise via standards

Security & supply chain

Signed packs + provenance on every release (PRD 17).
Trust levels (built-in / local / external) — external packs are sandbox-restricted, never used for benchmark results.
Target attestation (live / fixture / mock) — results are labeled at the weakest link; mock never publishes.
Standalone verifier — a zero-install, offline check of signatures, hashes, manifests, and labels; a bundle is verified on a machine that never ran the drill.
Evidence privacy — target-side secrets redacted before bundling, with verifiable redaction receipts.
Self-audit — the evasion pack is the sanctioned way we attack ourselves; disclosure per SECURITY.md.

Verification & quality gates

make setup            # create the venv + install dev extras
make lint             # ruff, zero violations
make typecheck        # mypy --strict
make test             # functional test suite (excludes docker/omlx infra tests)
make cov              # functional suite with the coverage gate (>=95%)
make test-all         # full suite incl. docker/omlx (needs a daemon + oMLX server)

Self-attack, benchmark, and release-verification targets (drill-self / benchmark / verify-release) land with their milestones (M14 / M10 / M32).

Published benchmark: honest outcome split per release, misses included (PRD 07).
Every published number is efficacy-backed and multi-seed; every row links its traces (PRD 19).

Development

Python core (pack format, runner, driver/target adapters, oracle engine, reporter, registry), CLI wrapper, and a pytest-style API — one engine, every surface (PRD 02).

make setup        # editable install with dev extras
agentdrill list packs        # discover packs / oracles / targets / drivers
agentdrill run --dry-run --pack incidents-core --target ./stack.yaml
agentdrill quickstart        # end-to-end against the bundled demo stack