nocap

---

🧢 Why

Most RAG pipelines pass the top-k chunks straight to the generator. When those chunks only mention the topic, leave out the key detail, or contradict each other, the model fills the gap anyway, and does it confidently.

"No cap" is slang for "no lie". NoCap adds one step between retrieval and generation that asks: do these excerpts support an answer, partly support it, miss it, or conflict? Then your own policy decides what happens next.

Q: How many days do I have for a refund?
A: You have 90 days. ✨        ← invented

Q: How many days do I have for a refund?
→ review_conflict  (conflicting 0.88)
  refund-policy.md: "within 30 days"
  refund-draft.md:  "within 14 days"
  LLM not called. Ask which source wins.

⚡ System One judges, System Two speaks

Kahneman's System 1 is fast, intuitive judgment; System 2 is slow, deliberate reasoning. A new class of System One models, such as TypeSafe's hosted Jev and the open-weight Laya, don't write text at all. You send them state plus typed questions, and they return a probability distribution your code can act on.

That is exactly the question a RAG pipeline should ask before it generates:

state:    { question, excerpts[] }
question: choice → supported | partial | missing | conflicting
answer:   { supported: 0.86, partial: 0.10, missing: 0.02, conflicting: 0.02 }

NoCap sends that typed question to any /v1/systemone endpoint, turns the distribution into a route with your thresholds, and only then lets your System Two (the generator LLM: slower, pricier, eloquent) speak. For Laya it averages four label rotations by default, to cancel one known source of option-order bias. No System One model at hand? Ollama or any OpenAI-compatible LLM can play the judge with the same schema, traces and replay.

gate = Gate(provider="laya")  # open weights, local · or "jev" (hosted), "ollama", "openai"

✨ What you get

	Feature	Why it matters
🛡️	Evidence gate	Four routes: answer · retrieve_more · abstain · review_conflict. Your generator runs only on answer.
🐍	3-line Python SDK	@gate.guard wraps any generator. Accepts strings, dicts, LangChain Documents and LlamaIndex nodes.
🤖	MCP server	Give Claude Code, Cursor, Codex or any MCP agent a check_evidence tool.
⚡	System One judges	Native /v1/systemone support for open-weight Laya and hosted Jev, or use Ollama / any OpenAI-compatible LLM (OpenAI, DeepSeek, Qwen, vLLM, LM Studio…).
🧪	CI for hallucinations	nocap eval runs labelled route tests and fails the build on leaks. Also available as a GitHub Action.
⏪	Zero-cost policy replay	Change thresholds on a saved decision and re-route it with 0 model calls.
🔍	Inspectable traces	Exact excerpts sent, the distribution, timings and citations, as JSON with a permalink.
🏠	Local-first	FastAPI + SQLite + a no-build web UI. The demo needs no keys, models, embeddings or vector DB.

🚀 60-second quickstart

pip install nocap-ai
nocap demo     # load fictional sample docs
nocap serve    # open http://127.0.0.1:8787

Or from source with uv: git clone https://github.com/xi029/nocap && cd nocap && uv sync && uv run nocap demo && uv run nocap serve.

The default demo judge is a clearly labelled lexical simulation, so you can explore everything offline. Switch to a real model in the UI or with NOCAP_PROVIDER (see models).

Actual workbench screenshot in demo mode. Demo scores are a lexical simulation, not AI probabilities.

Try this: ask "How do I request a refund?" → drag Minimum support to 95% and watch the route change with no model call → click Add conflicting policy and ask again.

🔌 Three ways to plug it in

1. Python SDK: wrap your generator

from nocap import Gate

gate = Gate(provider="ollama")  # or "openai", "laya", "jev", "demo"


@gate.guard(fallback="I couldn't find that in our docs.")
def answer(question, evidence):
    return llm(question, evidence)  # runs ONLY when the evidence supports an answer


answer("How do I request a refund?", retriever.invoke("How do I request a refund?"))

Need the details? verdict = gate.check(question, docs) gives you verdict.action, verdict.probabilities, verdict.evidence and verdict.ok. Async functions work too. Runnable: examples/quickstart.py.

2. MCP: make your agent show its receipts

claude mcp add nocap -e NOCAP_PROVIDER=ollama -- uvx --from "nocap-ai[mcp]" nocap mcp

Your agent gets check_evidence, ask_knowledge_base, add_document and replay_decision. Each verdict carries a next_step, such as "Tell the user the sources do not contain the answer. Do not guess." Configs for Cursor, Claude Desktop and Codex are in the MCP guide.

3. HTTP: any language, any stack

curl -s http://127.0.0.1:8787/api/decide -H "Content-Type: application/json" -d '{
  "question": "How do I request a refund?",
  "evidence": [{"id": "r1", "source": "refund-policy.md",
                "text": "To request a refund, email support with the order number."}]
}'
# → {"action": "answer", "reason": "...", "decision": {"probabilities": {...}}, ...}

Route	What your app should do
answer	Generate from the returned (judged) excerpts
retrieve_more	Search again, ask a clarifying question, or stop. Bound the retry loop.
abstain	Say "I don't know" and skip generation
review_conflict	Show both sources to a person or your source-of-truth rules

See the integration guide and API reference.

🧪 CI for hallucinations

Write the routes you expect, then fail the build when your RAG would answer without evidence.

{"id": "refund", "question": "How do I request a refund?", "expect": "answer"}
{"id": "pricing", "question": "What is the exact team plan pricing?", "expect": "retrieve_more"}
{"id": "off-topic", "question": "Who won the lunar chess championship?", "expect": "abstain"}

nocap eval cases.jsonl --docs ./knowledge --provider ollama --max-leaks 0

A leak means the gate allowed an answer it should have blocked. An over-refusal means it blocked an answer the sources supported. Both are counted; you choose which one fails CI. As a GitHub Action:

- uses: xi029/nocap@v0.2.0
  with:
    cases: tests/rag-cases.jsonl
    docs: knowledge/
    provider: openai          # set NOCAP_OPENAI_* in env
    max-leaks: 0

🧠 Bring your own judge

Provider	Setup	Score meaning
⚡ laya	System One, open weights: self-hosted Laya /v1/systemone (uv sync --extra laya)	Choice-head distribution, averaged over 4 label rotations
⚡ jev	System One, hosted: TypeSafe Jev API key (NOCAP_JEV_API_KEY)	Hosted decision distribution
ollama	ollama pull qwen3.5:4b and choose Ollama · local	Self-reported, normalized, uncalibrated
openai	NOCAP_OPENAI_URL, NOCAP_OPENAI_MODEL, NOCAP_OPENAI_API_KEY. Works with OpenAI, DeepSeek, Qwen/DashScope, OpenRouter, vLLM, LM Studio	Self-reported, normalized, uncalibrated
demo	Nothing	Lexical simulation for exploring the UI

# .env — for example, DeepSeek as the judge
NOCAP_PROVIDER=openai
NOCAP_OPENAI_URL=https://api.deepseek.com/v1
NOCAP_OPENAI_MODEL=deepseek-chat
NOCAP_OPENAI_API_KEY=sk-...

Every judge answers the same typed question; the policy, traces and replay are identical. Provider setup and troubleshooting →

A real request on a CPU-bound Windows machine, not the demo simulation. It took about 85 seconds end to end there; speed depends entirely on your hardware and model.

⏪ Policy replay

Every decision is saved with its full distribution. Move the thresholds and NoCap recomputes the route from the saved judgment with zero new model calls, while the original response stays attached to its original policy. Use it to see how many questions a stricter policy would block before you ship that policy.

uv run python examples/replay.py    # replay a real Qwen trace, no model running

🤔 How is this different?

• Evaluation frameworks (RAGAS, DeepEval, promptfoo…) score answers after generation, usually offline. NoCap makes a runtime routing decision before generation and can skip the generator. Use both: NoCap's nocap eval tests routes, not answer quality.
• Guardrail libraries usually validate outputs (format, toxicity, PII). NoCap judges whether the inputs support an answer at all, and tells you whether to answer, dig deeper, abstain or escalate.
• "Answer only from context" prompts leave the decision inside the same model call that writes the answer. NoCap makes it a separate, inspectable, replayable step with your thresholds.

📏 Honest limits

• answer means "meets your threshold", not "guaranteed true". LLM judges are uncalibrated; validate thresholds on your own labelled questions.
• Generated citations are checked for valid IDs, not semantic entailment.
• The built-in retriever is BM25 (no embeddings). For semantic search, bring your own retriever via the SDK, MCP or /api/decide.
• The bundled 8-case fixture is a smoke test, not a benchmark. We publish no invented accuracy, speed or hallucination-reduction numbers. Methodology →
• Local, single-user workbench without auth. See SECURITY.md.

🗺️ Roadmap

• OpenAI-compatible judges · MCP server · Python SDK · nocap eval + GitHub Action
• Claim-level entailment check of generated answers
• Risk / coverage curves from labelled sets
• Native LangChain Runnable and LlamaIndex postprocessor wrappers
• Shareable, redacted trace cards
• OpenAI-compatible proxy mode: put NoCap in front of any chat app

Ideas and PRs welcome: see CONTRIBUTING.md or open an issue.

⭐ Support

If NoCap stopped your bot from making something up, a star helps other builders find it.

License

Apache-2.0. NoCap is an independent project. Thanks to Laya, TypeSafe, Ollama and the Model Context Protocol. Model weights are not included; upstream models keep their own licenses. See NOTICE.