QAIA

QAIA — Agentic QA platform, open source

🇫🇷 Lire ce README en français

A user story goes in. A traceable Gherkin test book and runnable Playwright tests come out — as Claude Code plugins that run inside your session. No API key, no backend, no data leaving your session beyond what you already send to Claude.

▶ Try it live — paste a Gherkin test book and watch QAIA judge it. A high score can still fail; the demo shows why.

The whole product in one screen

One acceptance criterion from eval/gold-set/US-004-expense-approval.md:

A report under €500 total needs one approval (the employee's direct manager). €500–€5000 needs manager then finance.

Under €500 and €500–€5000 do not say what happens at exactly €500.00. Here is what came out (approval-chain.feature, verbatim):

  @QAIA-US-004-009 @AC2 @P1 @boundary @low-confidence
  # condition: AC2-C2 — priority P1 — open: Q1 (exact-€500 boundary — read as inclusive
  # in band B: manager then finance)
  Scenario: A report of exactly €500.00 needs manager then finance
    Given a submitted report "R" by "employee@demo" totalling exactly 500.00 EUR
    When "manager@demo" approves report "R"
    Then report "R" still awaits approval from "finance"

The tool did not silently pick a reading of the boundary. It picked one, tagged it @low-confidence, numbered the open question, and wrote its assumption in the file — so a human can overturn it in one line instead of finding it in production. A stable ID that survives regeneration, the criterion it came from, the technique that produced it, and the ambiguity declared rather than guessed away.

38 scenarios came from that one story, 11 flagged low-confidence. Disclosed, because a demo should be: that ticket is our own gold-set fixture with ambiguities planted on purpose, the run was non-interactive (every human decision recorded as simulated), and the acting model had read the file's sequestered judge section — all three stated in the run's own journey file. It shows the shape of the output, not that it works on your ticket.

Two real defects, in software we did not write

Everything above is measured on code this project produced itself, which is the weakest kind of evidence. So the pipeline was pointed at typicode/json-server — 75,694 stars — and allowed to read only its README. Never the code, never the issues, never the fix commits.

• A one-character contract break. The docs promised _dependent; the code read dependent. The endpoint answered success, deleted the post, and left every dependent record in place. A suite written by reading the code cannot find this — it copies the mistake. Filed by a real user as issue #1551; fixed in 1b7c0fb.
• Two filters that overwrote each other. views_gt=100&views_lt=300 returned everything. Fixed in e6055e6.
• A third finding, refused. _start alone returns an empty list — a fact — but the README only ever shows it paired. Counted as contested. Two, not three.

And the part that goes against us. Against the current version four tests fail and three are our fault: those features left the documentation and the suite kept demanding retired promises. Now detected by check_requirement_drift.py rather than quietly closed.

This is not a pilot. No human has used QAIA in their own work. One target, one API, no UI, 32 scenarios. Full campaign, protocol and limits →

Install

/plugin marketplace add QAIA-Project/QAIA
/plugin install qaia-core@qaia

Then describing what you want in plain language — "write the tests for this user story" — is enough: generate carries the journey step by step and stops wherever a human has to decide.

/plugin install qaia-playwright@qaia      # runnable Playwright tests, contract probing, defect reports
/plugin install qaia-score@qaia           # judge: scoring and release gate

plugins/qaia-core/CATALOGUE.md maps "I want to X → use Y" across all 7 skills. Worked examples with their real output are in examples/.

What separates it, and what does not

Applying ISTQB techniques is no longer a differentiator — QASkills.sh alone publishes ~380 competently written MIT skills, and Claude-Code-native competitors (Agentic QE Fleet, QA Orchestra) overlap directly. Three things do separate QAIA, all verifiable by a stranger in five minutes:

• No producer scores itself. The structural score lives in a separate read-only plugin (qaia-score), apart from the semantic LLM judge, and since 2026-08-09 ships as pinned Python you can read, diff or refuse — not an algorithm the model rebuilds from prose each run. A score that is not reproducible is not a score (ADR 0002).
• Zero API key, nothing that auto-executes. Skills are Markdown, invoked on demand. Installing QAIA registers no hook, no agent, no MCP server. The scorers run only when you invoke the scoring skill, with your permissions.
• The failures are published too. A generated suite runs on a GitHub Actions runner with no Claude session and no skill loaded (run 30702503888); every number cited as measured points at the raw file it came from — including a benchmark that concludes QAIA costs ~2.9× a direct prompt and does not find more.

The honest counterweight: QAIA is younger and far less used than any of them, no human pilot has ever run it end to end, and what it produces for a real user is unmeasured. Which tool should you install? We recommend others for 3 of 4 cases →

Status and limits

Pre-alpha, in active development. qaia-core 0.5.0 (2 skills), qaia-playwright 0.2.0 (4 skills), qaia-score 0.3.1 (1 skill, judge) — 7 skills — all validating --strict, proven end-to-end on two independent domains: healthcare (examples/medibook/, 26 tests / 32 executions, all green) and finance/HR (examples/expense-demo/, 56 green tests, real bugs found during automation), plus a 24-case multi-model robustness corpus.

• It runs on your Claude quota. Per-command cost is published per plugin, measured, and higher than this project's own prior estimates on 13 of 14 commands.
• No learning loop. generate reads a git-versioned knowledge base in your repo, which your team keeps by hand. No model training, no central server.
• Web-first. Mobile coverage means browser emulation, not native iOS/Android.
• Not a regulatory claim. The original "medical software / regulated environments" framing was retired (D114): QAIA maps no actual framework — not IEC 62304, not 21 CFR Part 11, not ISO 13485. examples/medibook/ is a healthcare-shaped demo, not a certified artifact.
• What it deliberately does not do (ADR 0004): unit and component tests, internal integration, coverage-driven white-box testing. QAIA starts from a promise observable from the outside — a test written against a function is written against the implementation, which is the oracle it exists to avoid.

Willing to be the first pilot? docs/PILOT-KIT.md is a 15-minute guided run on a ready-made story, and the only thing asked in return is where it went wrong. Honest state: docs/STATUS-en.md (English) · docs/STATUS.md (full French record).

Agents — an opt-in tier

agents-tier/ ships six named agents. It is not installed by any plugin and is never a prerequisite — the 7 skills work without it. Only two earn their own context window (camille-judge, elian-refuter), because a producer never grades its own output; the other four group a phase behind one name — real ergonomics, no new capability. Two caveats the tier's own README documents rather than hides: tools: is a request to the harness, not a capability boundary, and a delegated agent runs the validation checkpoints with nobody in the room.

Repository map

Contributing

Read CONTRIBUTING.md first. Every PR requires a DCO sign-off; PRs touching skills additionally require a traced adversarial agent review — skills are prompts, and a malicious instruction is invisible to linters. Security reports: SECURITY.md.

License: MIT.