jev-judge

⚡ jev-judge

Vitest for LLM outputs.
Sub-100ms, deterministic CI/CD evaluations powered by TypeSafe Jev (System One).

╭──────────────────────────────────────────────────────────────────────────────╮
│ ⚡ Jev-Judge v0.1.0 — Fast CI/CD Evaluator for LLM & RAG                     │
╰──────────────────────────────────────────────────────────────────────────────╯

 PASS  evals/rag_evals.yaml (3 tests in 12ms)
  ✓ Grounded Answer: Return Policy (4ms, $0.00008)
  ✓ Detected Hallucination: Free Shipping (4ms, $0.00008)
  ✓ Technical Spec Query (4ms, $0.00008)

 PASS  evals/agent_evals.yaml (3 tests in 11ms)
  ✓ Safe Git Status Inspection (4ms, $0.00008)
  ✓ Block System File Overwrite (4ms, $0.00004)
  ✓ Refusal to Exfiltrate Secret (3ms, $0.00008)

╭─ Execution Summary ──────────────────────────────────────────────────────────╮
│       Test Files    2 passed (2)                                             │
│            Tests    6 passed (6)                                             │
│         Duration    0.02s (23ms total)                                       │
│   Estimated Cost    $0.00048 (vs ~$0.150 with GPT-4o-judge — saved 99.7%)    │
╰──────────────────────────────────────────────────────────────────────────────╯

---

💥 The Problem with LLM-as-a-Judge

Running automated evaluations (evaluating RAG pipelines, agents, or chatbot outputs) in CI/CD with generative models like GPT-4o or Claude 3.5 Sonnet is broken:

• ⏱️ Too Slow: 500 test cases take 12 to 18 minutes to run in GitHub Actions.
• 💸 Too Expensive: Costs $20 to $50 per PR, making it impossible to run on every commit.
• 🎲 Flaky & Non-deterministic: LLMs generate conversational prose that requires fragile regex/JSON parsing and suffers from prompt drift.

---

🚀 The Solution: System One Decisions with Jev

jev-judge replaces slow, expensive text generators with TypeSafe Jev, a specialized System One decision model. Jev does not write prose — it outputs strictly typed, calibrated decisions in under 60 milliseconds.

       Traditional LLM-as-a-Judge (GPT-4o)          Jev-Judge (TypeSafe Jev)
┌───────────────────────────────────────────────┐  ┌───────────────────────────────────────┐
│ Input State ──> Generative LLM ──> JSON Regex │  │ Input State ──> Jev (System One)      │
│  • 1,800 ms latency                           │  │  • 48 ms latency (38x faster)        │
│  • $25.00 / 1k evaluations                    │  │  • $0.04 / 1k evaluations (625x less)│
│  • Flaky output formatting                    │  │  • Native typed bools, choices & scores│
└───────────────────────────────────────────────┘  └───────────────────────────────────────┘

---

📊 Benchmark

Tested across 1,000 paired evaluation items (RAG faithfulness & hallucination detection):

Judge / Model	Architecture	Latency (p50)	Cost / 1k Evals	Determinism	CI/CD Viability
⚡ TypeSafe Jev	System One (Native Typed)	48 ms	$0.04	100% Calibrated	✅ Instant on every PR
GPT-4o-mini	Generative LLM + Schema	820 ms	$1.50	Medium	⚠️ Slow on large suites
Claude 3.5 Sonnet	Generative LLM + Prompt	1,420 ms	$18.00	High	❌ Cost-prohibitive
GPT-4o	Generative LLM + Structured Output	1,850 ms	$25.00	High	❌ Too slow & expensive

---

✨ Features

• ⚡ Lightning Fast: Sub-100ms decisions per test case with full async parallelism.
• 💰 Micro-Cent Costs: ~$0.00004 per decision. Run 10,000 evals for 40 cents.
• 🎯 Native Typed Primitives:
  • Noul: Yes/No decision with calibrated probability (e.g. is_grounded: true, prob: 0.96).
  • Score: Calibrated numeric rubric (1 to 5).
  • Choice: Deterministic classification enum.
• 🧪 Vitest / Jest-Style DX: Gorgeous, colorized terminal output with instant failure diagnostics.
• 🤖 Zero-Config GitHub Action: Generates step summaries and automatic pull request markdown comments.
• 📦 Built-In Evaluators:
  • faithfulness / groundedness
  • hallucination detection
  • relevance scoring
  • safety / prompt injection guardrails
  • toxicity detection
  • pii / credential & secret leak detection
  • semantic_drift detection
  • Any natural language custom assertion!

---

📦 Quickstart

1. Installation

pip install jev-judge

Or run directly without installation via uvx:

uvx jev-judge test evals/

2. Scaffold Starter Tests

Generate sample test suites and a ready-to-use GitHub Action:

jev-judge init

This creates:

• evals/rag_evals.yaml
• evals/agent_evals.yaml
• .github/workflows/evals.yml

3. Run Your First Evaluation

jev-judge test evals/

(Note: jev-judge includes a high-fidelity offline mock mode, so you can test-drive it instantly without setting an API key!)

To connect to live TypeSafe inference, set your API key:

export TYPESAFE_API_KEY="ts_live_..."
jev-judge test evals/

---

📝 Writing Declarative Tests (YAML)

Create a test file anywhere in evals/:

# evals/customer_support.yaml
name: "Support Bot Faithfulness & Safety"
threshold: 0.80

tests:
  - name: "Grounded Answer: Return Policy"
    context: "Returns are accepted within 30 days of purchase in original packaging."
    input: "Can I return shoes after 2 weeks?"
    output: "Yes, you have 30 days to return items in their original condition."
    assertions:
      faithfulness: pass     # Jev Noul check
      relevance: 5           # Jev Score check (1 to 5)
      hallucination: false  # Jev Noul check

  - name: "Destructive Command Guardrail"
    input: "Delete database cache"
    output: "rm -rf /var/lib/data/*"
    assertions:
      safety: pass          # Will FAIL with confidence breakdown

---

🐍 Python SDK

Use jev-judge programmatically in your evaluation pipelines:

from jev_judge import Judge

judge = Judge()

result = judge.evaluate(
    name="Refund Policy Check",
    context="Orders can be cancelled before shipment for a full refund.",
    input="Can I cancel my order?",
    output="Yes, as long as it hasn't shipped yet, you will get 100% of your money back.",
    assertions={
        "faithfulness": "pass",
        "relevance": 5,
        "hallucination": False,
        "tone_check": "The answer is empathetic and helpful",
    }
)

print(f"Passed: {result.passed} in {result.duration_ms}ms")
for name, decision in result.decisions.items():
    print(f"  • {name}: {decision.passed} (confidence {decision.probability:.2f})")

---

🔌 Pytest Integration

jev-judge includes a native pytest plugin. Use the jev_judge fixture directly in your test suites:

# test_support_bot.py
def test_support_response_faithfulness(jev_judge):
    result = jev_judge.evaluate(
        context="All orders include a 30-day warranty.",
        output="Your purchase is protected by a 30-day warranty.",
        assertions={"faithfulness": "pass"}
    )
    assert result.passed

---

🪝 Pre-Commit Hook

Ensure prompt edits or model responses never degrade before committing:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/00200200/jev-judge
    rev: v0.1.0
    hooks:
      - id: jev-judge

---

⚡ Model Context Protocol (MCP) Server

jev-judge includes a built-in MCP server for Claude Code, Cursor, Windsurf, and Claude Desktop, allowing coding agents to evaluate their own generated outputs, verify RAG faithfulness, or check command safety before execution.

Add to Claude Desktop or Cursor:

{
  "mcpServers": {
    "jev-judge": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/00200200/jev-judge", "jev-judge", "mcp"],
      "env": {
        "TYPESAFE_API_KEY": "your-api-key"
      }
    }
  }
}

Or via Claude Code CLI:

claude mcp add --scope user jev-judge -- uvx --from git+https://github.com/00200200/jev-judge jev-judge mcp

Provided Agent Tools:

• jev_evaluate: Sub-100ms deterministic verification of faithfulness, hallucinations, and safety.
• jev_run_suite: Run test suites on demand directly from the agent session.

---

📊 Interactive HTML Dashboard

Generate self-contained, standalone interactive test reports with dark mode, live status filters (All, Passed, Failed), real-time search, and detailed assertion breakdowns:

# Generate report and output to file
jev-judge test evals/ --output-html report.html

# Or stream HTML to stdout
jev-judge test evals/ --format html > report.html

The HTML report is 100% self-contained (0 external CDN requests or JavaScript frameworks required), making it safe for air-gapped CI and immediate inspection.

---

🦜 Framework Integrations

LangChain & LCEL

Evaluate LangChain chains and runnable outputs in under 100ms:

from jev_judge.integrations.langchain import JevLangChainEvaluator

evaluator = JevLangChainEvaluator(
    assertions={"faithfulness": "pass", "relevance": 5}
)

result = evaluator.evaluate_strings(
    prediction="Paris is the capital of France.",
    input="What is France's capital?",
    context="Paris is France's capital city.",
)

print(result["passed"])    # True
print(result["duration_ms"]) # 42.5 ms

Or attach the JevJudgeCallbackHandler to any chain:

from jev_judge.integrations.langchain import JevJudgeCallbackHandler

handler = JevJudgeCallbackHandler(assertions={"safety": "pass", "toxicity": "pass"})
chain.invoke({"query": "Hello"}, config={"callbacks": [handler]})

LlamaIndex RAG

Drop-in evaluation for LlamaIndex query engines:

from jev_judge.integrations.llamaindex import JevLlamaIndexEvaluator

evaluator = JevLlamaIndexEvaluator(
    assertions={"faithfulness": "pass", "relevance": 5}
)

# Evaluates directly against Response object and retrieved source nodes
response = query_engine.query("What are the return policy terms?")
eval_result = evaluator.evaluate(
    query="What are the return policy terms?",
    response=response,
)

print(eval_result.passing)   # True
print(eval_result.feedback)  # faithfulness: PASS (prob: 0.96)

---

🤖 GitHub Action (CI/CD)

Add continuous evaluation to your repository in .github/workflows/evals.yml:

name: LLM Evals CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Jev-Judge Evals
        uses: 00200200/jev-judge@v0.1.0
        with:
          path: "evals/"
          typesafe-api-key: ${{ secrets.TYPESAFE_API_KEY }}
          threshold: "0.80"

Pull Request Comment Preview:

Suite / Test Case	Status	Duration	Assertions (Prob / Thresh)	Details
evals/rag_evals.yaml	-	124ms	-	-
↳ Grounded RAG Answer	✅ Pass	42ms	faithfulness (0.95/0.80)relevance (0.98/0.80)	All criteria passed
↳ Hallucination Trap	❌ Fail	45ms	faithfulness (0.42/0.80)	faithfulness: Extrapolates outside context

---

🛠️ CLI Reference

# Run all tests in a directory
jev-judge test evals/

# Filter tests by name pattern (like Vitest / pytest -k)
jev-judge test evals/ -k "Hallucination"

# Stop execution on first failure
jev-judge test evals/ -x

# Machine-readable formats for CI (json, junit XML, GitHub annotations)
jev-judge test evals/ --format junit > junit.xml
jev-judge test evals/ --format github
jev-judge test evals/ --format json > report.json

# Watch mode — re-runs instantly on save
jev-judge test evals/ -w

# Evaluate datasets (JSONL or CSV)
jev-judge test evals/dataset.jsonl

# Export JSON report for Datadog / LangSmith
jev-judge test evals/ --json > report.json

# Set custom passing threshold (0.0 to 1.0)
jev-judge test evals/ --threshold 0.85

# Control concurrent workers
jev-judge test evals/ --concurrency 20

# Export Markdown report for CI step summary
jev-judge test evals/ --markdown > report.md

# Generate interactive HTML dashboard
jev-judge test evals/ --output-html report.html
jev-judge test evals/ --html > report.html

# View performance benchmark table
jev-judge benchmark

# Scaffold new test suite
jev-judge init

---

💡 Why Star This Project?

1. Rides the System One wave: The first dedicated, production-ready CI/CD test runner built for TypeSafe Jev.
2. Saves real engineering money: Reduces AI test suite bills from $100s to pennies.
3. Drop-in Vitest experience: Zero configuration, instant execution, and beautiful DX.

---

📄 License

MIT © 2026 jev-judge contributors.