decisionAgent

DecisionAgent

A decision-support agent that checks its evidence—and knows when not to answer.

Both responses are real outputs from the v1.0 eval run (Q-055 and Q-036), replayed for the demo.

Ask it something about your internal docs and you get one of two things back: an answer with its sources, or a flat "not enough evidence." Never anything in between.

Finding the right document isn't the same as finding the answer. A document can mention your topic without containing the answer. When the evidence appears insufficient, the agent searches again. When it still doesn't cover the question asked, an answerability gate stops the run before an answer gets written.

Two calls the model doesn't get to make:

• Whether you're allowed to read a document. A SQL filter settles that before retrieval runs.
• Which sources it can cite. Only documents that turned up in the search. It can't make one up.

Underneath is hybrid search in Postgres, plus a router that sends easy questions to a cheap model and hard ones to a strong one. The whole agent is tested against an eval that's part of the project: 70 questions, 25 of them traps.

The gate took correct abstentions from 5 to 9 of the 10 unanswerable questions. The router cut model spend by 64% compared with running everything on the strong model.

Stack: Python 3.13 · Pydantic AI · FastAPI · PostgreSQL + pgvector · Sentence Transformers · OpenRouter · MCP · Langfuse

One contract, two valid outcomes

The demo above shows two runs. Here they are in detail — every response has the same shape, only the content changes.

When the evidence answers the question:

Do we still have logs from 80 days ago?

An older overview says logs are kept for 90 days. A newer operations note cuts that to 30. The agent follows the current policy, and flags the stale document as a risk:

{
  "answer": "No. Log retention is 30 days. Older logs may be recoverable from cold storage, but that is not guaranteed.",
  "confidence": "high",
  "sources": [
    { "source_id": "NOTE-003#body", "doc_id": "NOTE-003" },
    { "source_id": "OVW-005#operational-notes", "doc_id": "OVW-005" }
  ],
  "risks": ["OVW-005 still contains the superseded 90-day policy."],
  "abstained": false
}

When it doesn't:

What specific latency does the Gateway SLA guarantee?

The knowledge base mentions the SLA but never states the number. A plausible guess would be easy here. The agent returns nothing instead:

{
  "answer": null,
  "confidence": "insufficient",
  "sources": [],
  "risks": [],
  "abstained": true
}

Both are real outputs from the v1.0 evaluation run — Q-055 and Q-036, trimmed of citation snippets and eval-only fields.

How one request works

flowchart TD
    Q["Question"] --> M["Choose answer model<br/>(from question complexity)"]
    M --> H["Search in-scope evidence<br/>(vector + keyword)"]
    C["Caller scope<br/>(application state)"] --> H

    H --> S{"Is more evidence<br/>needed?"}
    S -- "Yes · max 3" --> F["Reformulate query"]
    F --> H
    S -- "No" --> A{"Does the evidence answer<br/>the exact question?"}

    A -- "No" --> X["Return structured abstention"]
    A -- "Yes" --> G["Generate typed recommendation"]
    G --> V["Validate citations against<br/>retrieved evidence"]
    V --> P["Audit claims against evidence<br/>(reports, doesn't block)"]
    P --> O["Return cited recommendation"]

    classDef outcome stroke-width:3px;
    class X,O outcome;

Each control in the table is one box in the diagram. The last column is what it measurably changed:

Control	What it does	Measured
Complexity router	Sends simple questions to a cheap model, hard ones to a strong one	64% less model spend than strong-only, at three fewer correct answers out of 45
Hybrid retrieval	Runs vector and keyword search, merges the two rankings	Questions with every needed source in the top ten: 34 → 37 of 45
Sufficiency judge	Decides whether to reformulate and search again	Re-queried 38 of 70 questions for two more correct answers, 33 → 35 of 45. Honestly marginal
Answerability gate	Checks the evidence against the exact question before an answer is written	Correct abstentions 5 → 9 of 10, no measurable correctness cost
Typed contract	One response shape for answers and abstentions alike	All 70 eval responses parsed into it
Citation validator	Accepts only source IDs from evidence retrieved during that run	0 of 50 answered questions shipped with an empty source list
Grounding audit	Re-checks each claim in the finished answer against the evidence	168 of 169 claims supported. It reports, it doesn't block

Every number is from a logged run in the eval ledger.

One choice here has no number behind it: Pydantic AI rather than LangChain or LangGraph. The pipeline above is a straight line with one bounded loop, so a graph runtime would sit unused. The part that had to be reliable is the response contract, and a validated output schema is what Pydantic AI is built around. Everything between model calls is plain Python.

How it was evaluated

Every change ran against the same benchmark: 64 synthetic documents and 70 questions with known answers. The documents were written for this project, so no model has seen them in training. Twenty-five of the questions are traps. Ten have no answer in the corpus, ten ask about documents outside the caller's scope, and five try prompt injection through the documents themselves.

An LLM judge scores correctness. The judge is from a different model family than the answerers, its prompt is versioned in the repo, and a human spot-checked its verdicts. The point is comparing versions on a controlled set, not estimating accuracy on anyone else's documents. Repeat runs are not identical: the cheap model ignores sampling seeds, and the strong one is a mixture-of-experts model, so answers vary even at temperature 0.

Two results shaped the design more than the rest. A stronger answer model broke the multi-hop ceiling, 7 to 13 of 15 correct, at 7.8 times the cost. Latency tells the same story: P50 is 3.6 seconds, but P95 is 33.5 seconds, dominated by the multi-hop questions that go to the strong model. Managing that tradeoff is why the router exists. And following links between documents improved nothing, so the graph layer stayed out of the answer path.

The traps mostly measure one thing: whether the agent knows when not to answer. That is the answerability gate's job, and it is what took correct abstentions from 5 to 9 of 10. The eval also caught it failing. On one out-of-scope question the gate let the run through, and the model invented a detail the evidence never stated. The grounding audit flagged the claim, but the audit reports rather than blocks, so the answer still went out.

CI runs a deterministic subset on every push: retrieval hit-rate on frozen embeddings, a check that nothing out of scope comes back, and the abstention logic on hand-written model replies. The full eval calls live models and runs manually before each release — that is where every number here comes from.

Full history: eval ledger · v1.0 run

API and MCP

POST /recommend runs the full pipeline: one question in, one typed response out — the same contract shown above, answer or abstention. The request is the question plus the caller's scopes: {"question": "...", "caller_scope": ["platform"]}. The scope arrives as application state, never as model input.

The MCP server exposes three read-only tools for other agents and clients: vector_search, get_document, and graph_lookup. Scope is bound server-side, and a forbidden document looks exactly like a missing one to the client. uv run decisionagent mcp starts it standalone; the checked-in .mcp.json registers it for MCP clients, with the caller pinned to the platform scope.

One boundary to be clear about: an MCP client gets the retrieval layer only. The routing, the answerability gate, the grounding audit, and the evaluated numbers above belong to /recommend.

Run locally

You need Docker, uv, and an OpenRouter API key.

git clone https://github.com/krmaschek/decisionAgent.git
cd decisionAgent
cp .env.example .env    # set OPENROUTER_API_KEY; everything else can stay as is
docker compose up -d    # Postgres + pgvector
uv sync
uv run decisionagent ingest    # build the index: chunk + embed the corpus
uv run decisionagent api

Then ask it the question from the demo:

curl -X POST http://127.0.0.1:8000/recommend \
  -H "Content-Type: application/json" \
  -d '{"question": "Do we still have logs from 80 days ago?", "caller_scope": ["platform"]}'

The models are called through OpenRouter. Every call uses the OpenAI-compatible chat protocol, so a local server such as Ollama or vLLM can take its place if your documents can't leave your infrastructure.

For deployment, Dockerfile packages the API as an image. The embedding model is built in, so a new container starts immediately instead of fetching 129 MB first. The commands are at the top of the file, and Postgres still comes from compose.

Use your own knowledge base

After completing local setup, you can replace the example corpus with your own Markdown documents. Every document needs YAML front matter:

---
doc_id: HANDBOOK-001
team_scope: [all]
---

doc_id must be unique across the corpus; a file without one is skipped at ingestion. team_scope controls access: [all] for a document every caller may read, or a list of allowed teams, such as [platform, payments]. A document with no team_scope is indexed but visible to no one. Subdirectories are scanned recursively.

Documents are split into chunks at ## headings; one without headings is split on paragraph boundaries at around 350 words instead. A title in front matter (optional) goes in front of every chunk at indexing time.

Point CORPUS_DIR in .env at your directory and rebuild the index with uv run decisionagent ingest. The rebuild replaces the current index; restart a running API or MCP server afterward.

Known limits

• There is no authentication. The caller's scope arrives with the request, so a real deployment has to derive it from an authenticated identity.
• The safety behaviour is measured, not guaranteed. The answerability gate is itself a model call. In the v1.0 run the agent still answered 2 of 10 questions it should have declined, built from tangential in-scope evidence — the scope filter held, and no restricted document was retrieved.
• Runs aren't reproducible: the cheap model ignores sampling seeds and the strong one is a mixture-of-experts model, so the same configuration could score differently each time.
• The eval measures this corpus: 64 documents written for this project. Nothing here predicts accuracy on your own documents.
• Questions and retrieved evidence are sent to OpenRouter, and to Langfuse when tracing is on. The models can be pointed at a local server instead, but every measured result here used the hosted ones.

Further reading

• Evaluation ledger — measured runs and the decisions they informed.
• Threat model — trust boundaries and residual risks.
• Eval questions and answer key — the 70 questions with the documents each one needs, and the traps: which documents are noise, which facts were left out on purpose, and where the injections are hidden.

License

Apache-2.0. See LICENSE.