llm-verify

llm-verify

Verify the LLM endpoint you are actually using: model authenticity, billing inflation, relay provenance, performance and silent downgrades.

English · 简体中文

A single binary with no runtime dependencies. You point it at a base URL, a key and a model name; it probes the endpoint from the outside and writes an HTML report you open in a browser.

Contents

• Install
• Use
• Reading the report
• Use it from an AI coding tool
• What it checks
• Use it as a library
• Limits
• Licence

Install

From crates.io, with a Rust toolchain (1.82+):

cargo install llm-verify

Without a toolchain, the install script fetches a prebuilt binary:

curl -fsSL https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.sh | sh

On Windows: irm https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.ps1 | iex

Use

llm-verify --base-url https://api.anthropic.com \
           --api-key sk-ant-... \
           --model claude-opus-4-5

It writes an HTML report and opens it.

Credentials can live in .env or the environment instead, leaving only the model on the command line:

# .env
LLM_VERIFY_BASE_URL=https://api.anthropic.com
LLM_VERIFY_API_KEY=sk-ant-...
LLM_VERIFY_MODEL=claude-opus-4-5

llm-verify

See the suite without spending anything

probes prints every step, what it asks, and which group it belongs to. It contacts nothing and needs no key, so it is also the way to find the ids that --only and --skip accept:

llm-verify probes                              # all 28 steps, 40 results
llm-verify probes --only identity,consistency  # just those groups
llm-verify probes --lang zh

Example run

Probing gpt-5.6-sol through OpenRouter at the default balanced depth — 40 probes, 47 requests, 131.2s:

The summary that follows carries the verdict, the two axes, and the per-group scores:

Relayed / Relay: a real model, reached through one hop. Channel provenance and performance are what drag the groups down, not identity or billing.

Full HTML reports from two runs against the same model:

Report	Verdict	Origin	Score
openrouter.ai (preview)	Relayed	Relay	92 / 100
gw.asale.ai (preview)	Relayed	Undetermined	93 / 100

Same model, same verdict, two points apart — and the reports still differ. OpenRouter names itself in the response headers, so the origin is pinned to a single relay hop; the second path carries no channel markers at all, which leaves the origin undetermined rather than proven clean. Origin is a separate reading from both score and verdict.

Options

Flag	Meaning
--protocol anthropic|openai	Inferred from the URL and model name if omitted
--depth fast|balanced|forensic	Default balanced. forensic samples more — slower and costlier, but firmer
--turbo	The quickest run that still reaches a verdict: 9 requests instead of 21, several at a time. Gives up the corroborating identity probes and the contract checks that catch a reconstructed endpoint
--concurrency N	Requests to keep in flight. Default 1. Only raise it against an endpoint you know answers that many at once — latency probes always run alone
--claimed-model <ID>	Use when the vendor's advertised name differs from the ID you request. This is how you check for a downgrade
--lang en|zh	Report language. Follows the system locale, then falls back to English
-o <path>	HTML report path; pass a directory to auto-name the file
--json <path>	Also emit machine-readable JSON
--no-open	Do not open a browser

Exit codes

Code	Meaning
0	Clean
1	Failing score, or a suspicious / counterfeit / inconclusive verdict
2	A hard gate tripped

Usable as a CI gate as-is.

Reading the report

The verdict has two independent axes:

• Authenticity — genuine / genuine-with-defects / relayed / suspicious / counterfeit / inconclusive
• Origin — direct from vendor / cloud platform / subscription-derived / relay / reconstructed channel / undetermined

A real model behind a relay is relayed, not counterfeit. Longer path, same model.

Hard gates are facts no weighted score can excuse. Any one of them forces a suspicious verdict and exit code 2:

silent fallback · shared-pool forwarding · tier downgrade · third-party wrapper injection · cache replay · hidden prompt injection · response replay

Use it from an AI coding tool

The skill lives in this repository at skills/llm-verify/SKILL.md. Install it with skills, which supports Claude Code, Codex, Cursor, OpenCode and 70-odd other agents:

npx skills add asale-ai/llm-verify

npx skills add asale-ai/llm-verify -g              # user-level, every project
npx skills add asale-ai/llm-verify -a claude-code  # one agent only
npx skills add asale-ai/llm-verify --list          # look before installing

It is also published on ClawHub:

clawhub install @asale-ai/llm-verify

Then just ask: "is this API actually giving me what I paid for?"

The skill is only the usage guide — the llm-verify binary does the work, so you need both.

What it checks

40 probes across seven groups:

Group	Question it answers
Protocol contract	Is this a genuine API channel?
Streaming	Does streaming follow the protocol, or arrive empty?
Metering & billing	Are the token counts honest, or are you overcharged?
Channel provenance	What relays sit on this path?
Performance	First-token latency, throughput and jitter
Model identity	Is the model behind this the one that was sold?
Cross-request consistency	Does the endpoint behave the same way every time?

Use it as a library

The binary is a thin CLI over the crate, so an embedder gets the same probes, the same order and the same verdict — which is what lets it claim its own results and the published tool agree.

[dependencies]
llm-verify = { version = "0.4", default-features = false }

default-features = false drops the CLI half: clap, the terminal writer and the HTML renderer are dead weight in a service.

use llm_verify::{engine, probes::Cancel, Endpoint};

let cfg = engine::RunConfig::new(Endpoint {
    base_url: "https://api.anthropic.com".into(),
    api_key: std::env::var("ANTHROPIC_API_KEY")?,
    model: "claude-opus-4-5".into(),
    ..Default::default()
});
let report = engine::run(cfg, &Cancel::new(), &mut |_| {}).await?;
println!("{:?} {}", report.verdict.authenticity, report.verdict.score);

Probing through a relay

Half the suite asks about the endpoint — its error envelopes, its response headers, the token counts it reports. Behind a relay those describe the relay, not the model, and several of them deliberately send malformed or unauthenticated requests, which is not traffic you want attributed to somebody else's account.

Selection::model_only() keeps only the steps whose evidence is the generated text itself, which survives any number of hops:

let cfg = engine::RunConfig::new(endpoint)
    .model_only()
    .depth(llm_verify::Depth::Fast)
    // Chosen by the caller and recorded in the report, so a contested verdict
    // can be replayed probe for probe.
    .seed(0x5EED)
    // Spread the run out instead of issuing it as one recognisable burst.
    .pace(Duration::from_secs(20), Duration::from_secs(180));

When somebody is waiting for the answer

Selection::turbo() is model_only cut down to the probes a conclusion actually rests on, run several at a time:

let cfg = engine::RunConfig::new(endpoint).turbo(3);

The argument is how many requests the endpoint will answer at once — a statement about the far end, not about this crate. Set it too high and the run measures the endpoint's saturation instead of its behaviour, and on anything with a per-account concurrency budget it spends that budget on being examined. When in doubt, three.

Two things make overlapping safe rather than merely fast. Steps that measure a clock run alone, so a latency figure is never taken while this run has three other requests in the air. And each step draws its random payloads from its own generator, seeded from the run's seed and the step's id, so the same seed reproduces the same questions whatever order the scheduler happened to run them in.

What turbo gives up is documented probe by probe on Selection::turbo. The short version: the corroborating and merely descriptive identity probes, the dated-trivia knowledge check, and the three contract checks that catch an endpoint reconstructed from a web session. Put the last group back if you have not otherwise established what is on the far end:

Selection::turbo().plus(["max_tokens_truncation", "stop_sequence", "system_adherence"])

Progress, cancellation, and your own probes

engine::run reports each step as it starts and finishes, and takes a Cancel that is honoured between steps. Selection::with appends your own [Probe] implementations, which share the run's client, seed and observation buffers — so a private probe is indistinguishable from a built-in one both in the report and on the wire. That matters if you are policing a marketplace: everything in this repository is readable by the endpoint you are probing.

To put your own graded questions in place of the published battery — worth doing for exactly that reason — use replacing, which drops the built-in step and lets your probe answer to its id:

Selection::turbo().replacing("capability", Arc::new(my_bank))

Not minus(["capability"]).with(bank). skip applies to caller-supplied probes too, deliberately, so that spelling removes both and reports nothing: the run comes back with every other probe passing and the questions never asked.

What it costs

Measured, not estimated — tests/request_budget.rs pins these so that adding a probe cannot quietly multiply anyone's bill:

Selection	Depth	Requests
turbo	fast	9
model_only	fast	21
model_only	forensic	46
everything	balanced	49

Limits

One false accusation against an honest provider costs far more than one miss. This tool abstains when the evidence is thin rather than guessing. Please know the following:

• Resolution stops at tier granularity (flagship / mid / light). Adjacent versions inside one tier — say 4.5 and 4.6 of the same line — cannot be separated.
• The tier call depends on sampling. The default depth asks 9 capability questions, and adjacent tiers can still swing on a single one. The tool abstains when the margin is narrow, which costs it some genuine downgrades. Use --depth forensic (15 questions) when the answer has to hold up. The report states how many questions the call rests on and by how much the winner beat the runner-up.
• Delivering above the claimed tier is not fraud and carries no risk weight. Only measuring below the claim counts.
• Middle layers contaminate identity fingerprints, which is why the contract layer runs first; where injection is found, identity confidence is reduced automatically.
• Server-side weights cannot be proven — only whether behaviour matches expectations.
• One run describes one moment. Gradual degradation needs periodic re-runs and comparison.
• The probes are public. That is the point of an auditable tool, and it is a ceiling: an endpoint can read this repository too, and answer these questions honestly while serving everything else from somewhere cheaper. Timing can be disguised (RunConfig::pace); the questions cannot be. Anyone using this to police supply they do not control needs probes of their own on top — see Selection::with — and should treat the result as a probability rather than a proof.

Licence

Apache-2.0

Polished with repolish

This card is generated by repolish and is a plain file in this repository — no external fonts, no scripts, nothing hosted by a third party. To score your own: npx @asale/repolish.