{"id":"913dd5bb-290c-47f3-b038-d746452334ed","slug":"agentops","kind":"mcp","name":"AgentOps","summary":"Self-hosted LLM operations gateway in one Go binary: OpenAI-compatible router, durable agent tracker, faithfulness evals + drift, traces, MCP server, console","owner":"yosrikhiari","repoUrl":"https://github.com/yosrikhiari/AgentOps","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-09-15T06:22:40.000Z","addedAt":"2026-10-21T09:41:09.016Z","logoUrl":"https://avatars.githubusercontent.com/u/148056014?u=3243730cca2f0f5325a4755ed25b425fbecdff6b&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["search","cloud-infra"],"tags":["evals","go","llm","llmops","ollama","pgvector","rag"],"runtime":"go","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T07:28:14.594Z","sample":false,"url":"https://conduits.dev/tools/agentops","language":"Go","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# AgentOps\n\n[](https://github.com/yosrikhiari/AgentOps/actions/workflows/ci.yml) [](https://github.com/yosrikhiari/AgentOps/releases) \n\nA self-hosted **LLM operations gateway** in one Go binary: an OpenAI-compatible **router** over local and cloud models, a **durable task tracker**, a **faithfulness eval + drift alert** loop, **OTel-shaped traces**, an **MCP server** for AI assistants, and the **Tower console** for humans.\n\nBuilt solo, on an RTX 4060 (8 GB VRAM), against local [Ollama](https://ollama.com) models — informed by how TensorZero, Bifrost, Temporal and Langfuse solve the same problems, but with every core piece written here so it can be explained line by line. Standard library + one dependency (`pgx`).\n\n```\n ┌──────────────────────────── agentops (one binary) ────────────────────────────┐\n OpenAI clients ──► │ router: auth (API keys) → Classify → Plan (fail-closed) → Backend │\n POST /v1/chat │ Ollama /api/chat (stream) · any OpenAI-style API · health prober │\n │ ├─ Prometheus /metrics (hand-written exposition) │\n │ └─ 3 spans per chat ──► async writer ──► Postgres spans │\n --run-tracker ───► │ tracker: Researcher → Drafter → Reviewer, row-status durability, kill -9 safe │\n --score ─────────► │ evals: pgvector top-k → split answer into claims → LLM judge → faithfulness │\n │ eval_runs + eval_pair_scores → drift report + alert │\n Claude Desktop ──► │ mcp (--mcp): list_models · get_stats · route_test_request · inspect_trace · │\n │ get_drift_report (JSON-RPC 2.0 over stdio) │\n browser / ─────► │ Tower console: overview · traffic · backend health · trace inspector · │\n │ evals & drift · workflows · cockpit (embedded, no build step) │\n └───────────────────────────────────────────────────────────────────────────────┘\n```\n\n\n\nTower, the console embedded in the binary (no build step): the same k6 run seen from the span store — 3,738 requests in the hour, p50 142 ms / p99 261 ms, every request routed with its reason and traceable. The [Evals & drift page](docs/img/tower-evals.png) shows the golden v3 history against the 0.70 alert threshold and the worst-scoring pairs of the latest run.\n\n\n\nThe provisioned Grafana board during a 5-minute k6 run on 2026-09-19 (2 VUs of short prompts routed to `qwen2.5:3b-instruct`, 1 VU of long prompts routed to `qwen3:8b`): 3,737 requests, 1 failure — the quality-tier request that hit the 180 s client timeout, which is the 100 % error step on the yellow line. Faithfulness 0.972 is the golden v3 suite (72 pairs) scored minutes earlier with `--score`.\n\n**Current release: [v1.0.0](https://github.com/yosrikhiari/AgentOps/releases/tag/v1.0.0)** — 68 tests, `go vet` + `-race` + `staticcheck` + `govulncheck` in CI, plus 11 **executable policies** that fail the build when a rule in [`docs/RULES.md`](docs/RULES.md) is broken. See [`CHANGELOG.md`](CHANGELOG.md).\n\n## What it does\n\n| Capability | Where | Proof |\n|---|---|---|\n| Route chat requests between a fast and a quality local model on a rule you can read, with a cloud fallback that is *never* used for sensitive data | `router/` | `reason` on every response; `TestSensitiveNeverLeavesBox` |\n| Speak the OpenAI chat shape, streaming included, to any client library | `router/server.go` | `TestOpenAIShape`, `TestStreamSSE`; [`docs/API.md`](docs/API.md) |\n| Virtual API keys with per-minute limits and token budgets | `router/keys.go`, `--create-key` | live 200/429/429/429 on a 2-rpm key |\n| Hand-written Prometheus exposition + Tower-native dashboard panels (per-model counts, p50/p99, tokens, error rate) | `router/metrics.go`, `console/` `#/overview` | k6: 0/2725 failed at 50 RPS, p99 70 ms |\n| Keep a multi-step agent alive through `kill -9` and resume without repeating work | `tracker/` | `TestKillResume` + a recorded live kill |\n| Score answers claim-by-claim against retrieved sources, store every run, alert on drift | `evals/` | golden v3 (72 pairs) **0.972** — 70/70 faithful where retrieval hit, 2 retrieval misses; v2 (48) 1.000 |\n| One trace per request: `trace_id / span_id / parent_id`, prompts never stored | `GET /v1/traces/{id}`, `--trace`, MCP | 3 chained spans per chat |\n| Let Claude (or any MCP client) ask \"which model handled the most traffic?\" | `mcp/` | 5 tools, live handshake |\n| Show it all to a human and let them run an eval or resume a workflow | `console/` → `http://localhost:8080` | verified page by page in the browser |\n\n## Quickstart\n\nRequirements: Go 1.22+, Docker, [Ollama](https://ollama.com) with `nomic-embed-text` and one chat model (`qwen2.5:3b-instruct` / `qwen2.5:7b-instruct-q4_K_M` by default, or set `FAST_MODEL`/`QUALITY_MODEL`), Python 3 for the corpus cleaner.\n\n```bash\ndocker compose up -d # Postgres 16 + pgvector on :5432\ngo run . --migrate # migrations/*.sql, tracked in schema_migrations\npython scripts/clean_docs.py # corpus/*.md → evals/corpus/clean/*.jsonl\ngo run . --ingest # embed chunks with nomic-embed-text → pgvector\ngo run . # gateway + Tower console on :8080 — open http://localhost:8080\n```\n\nOr containerised (21 MB distroless image, talks to Ollama on the host):\n\n```bash\ndocker compose --profile app up -d --build # gateway on :8080 + postgres\ndocker compose run --rm agentops --migrate # once\n```\n\nReleases ship `agentops-vX.Y.Z-{linux,windows,darwin}-amd64` binaries built by CI on every `v*` tag.\n\nReleasing (Track T checklist — all local, CI has no GPU/Postgres):\n1. `bash scripts/verify.sh` green.\n2. `go run . --release-gate` (default: last 3 runs of v3 clear `EVAL_THRESHOLD` on one judge).\n3. Tag `vX.Y.Z` and push; CI builds + attaches binaries.\n\nThen:\n\n```bash\ncurl -s localhost:8080/v1/chat/completions -d '{\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n# OpenAI shape: {\"id\":\"chatcmpl-…\",\"object\":\"chat.completion\",\"model\":\"qwen2.5:3b-instruct\",\"choices\":[…],\"usage\":{…},\n# \"reason\":\"short-simple-prompt\",\"backend\":\"ollama\",\"trace_id\":\"598d…\"} ← AgentOps extensions\ncurl -s -N localhost:8080/v1/chat/completions -d '{\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"stream\":true}' # SSE\ncurl -s localhost:8080/v1/models # served models with tier, backend, health\ncurl -s localhost:8080/v1/traces/598d… # 3 spans: route.decide → model.generate → router.respond\ncurl -s localhost:8080/metrics # router_*, router_backend_up, router_key_*, eval_faithfulness\n```\n\nTo require keys: `go run . --create-key alice --key-rpm 60` (prints `ak_…` once), then run with `REQUIRE_API_KEY=true` and send `Authorization: Bearer ak_…`.\n\nEvery mode is a flag on the same binary (`go run . --help`):\n\n| Flag | What it does |\n|---|---|\n| *(none)* | HTTP gateway + Tower console on `ADDR` (default `:8080`) |\n| `--mcp` | MCP stdio server for Claude Desktop (config in [`mcp/README.md`](mcp/README.md)) |\n| `--migrate` / `--ingest` / `--search \"q\"` | apply migrations · embed the clean corpus into pgvector · cosine-search it |\n| `--create-key NAME --key-rpm N --key-budget T` | mint a virtual API key (secret printed once, SHA-256 stored) |\n| `--run-tracker [--tracker-input \"q\"]` / `--resume-tracker ` | run / resume the 3-step toy agent; kill it mid-step and resume |\n| `--trace ` | print redacted spans for a chat or workflow |\n| `--advise` | print the static model advisor report (Track K: per-model size/VRAM fit, presence, co-residency verdict) and exit |\n| `--draft-golden` → review → `--freeze-golden` | build a golden set (`--golden-version vN`): LLM drafts, human reviews, validator rejects unscorable answers, hash frozen |\n| `--score` / `--drift` / `--schedule-evals 24h` | run the faithfulness suite, print the drift report, or loop it (`--golden-version vN`, `--golden path`; `--drift-golden vN` picks the version the report and console show) |\n| `--score-model NAME` | with `--score`: generate each answer with MODEL (temperature 0, 256 tokens) instead of scoring frozen answers; tags the run for Track L model comparison |\n| `--release-gate` (`--golden-version vN`, `--gate-k 3`) | pre-tag check (Track T): exit 0 only if the last k golden-answer runs clear today's `EVAL_THRESHOLD` on one judge; run locally before tagging, CI has no GPU/Postgres |\n| `--version` | print the version stamped at release |\n\nEnvironment variables are listed in [`docs/API.md`](docs/API.md#backends-environment).\n\n## Layout\n\n```\nmain.go wiring: flags → mode; pgx pool + adapters; graceful shutdown\nconfig.go OLLAMA_URL, ADDR, FAST_MODEL, QUALITY_MODEL (all with defaults)\nrouter/ gateway: API keys → classify → plan (fail-closed) → backends (Ollama, OpenAI-style) → metrics → spans\ntracker/ workflows/steps row-status durability, spans (Store iface: SQLStore + MemStore)\nevals/ corpus ingest, pgvector search, golden validation, judge, retry/backoff, drift, migrations\nmcp/ JSON-RPC 2.0 over stdio, 5 tools\nconsole/ Tower web UI (embedded static/) + its JSON endpoints\nmigrations/ 0001_init.sql, 0002_drift.sql, 0003_api_keys.sql (applied once each, recorded in schema_migrations)\ncorpus/ 15 short docs describing this system — the RAG target; golden sets in evals/golden/\nload-tests/ router.js (k6: 50 RPS on /health, p99 gate)\nlessons/ 33 plain-language HTML lessons on this repo — open lessons/index.html\ndocs/ MARKET.md (market study + roadmap) · RULES.md (engineering rules) · API.md · VRAM.md · adr/ (8 decision records) · tower-design-system.html (the console design system, live against console/static/tower.css) · tower-enhancements.html (20 worked UI/UX enhancements with demos)\npolicy/ executable policies: tests that fail the build when a rule in docs/RULES.md is broken\nAGENTS.md what a coding agent must read first (rules, the Tower design system, the traps). OpenCode/Codex read it natively;\n per-tool pointers to it (CLAUDE.md, GEMINI.md, …) are local and git-ignored; .github/copilot-instructions.md is tracked\n```\n\n## Rules and decisions\n\nThe engineering rules — architecture style, sync vs async, retries, latency and scalability budgets, security requirements, guardrails, testing/experiment rules, static analysis, and the executable policies that enforce them — are in [`docs/RULES.md`](docs/RULES.md). Full decision one-pagers in [`docs/adr/`](docs/adr/README.md).\n\nBefore every commit, run the gate (Track O — the same script CI runs):\n\n```bash\nbash scripts/verify.sh # gofmt, vet, tests, build, policy, diff stat\n```\n\nWindows without bash: run the same five commands by hand (`gofmt -l .`, `go vet ./...`, `go test -count=1 ./...`, `go build ./...`, `go test -count=1 ./policy/`). CI adds `-race` via `VERIFY_RACE=1` (needs cgo).\n\n- **One module, one binary** — every mode is a flag; packages reach Postgres through tiny interfaces and never import the driver. (ADR-0001)\n- **pgvector, not a vector DB** — one Postgres for workflows, spans, evals *and* vectors; HNSW `m=16, ef_construction=128`. (ADR-0004)\n- **Postgres-native durability, not DBOS/Temporal** — `(workflow_id, seq)` idempotency keys + `pending→running→done` rows; resume = re-run the same id and skip `done`. (ADR-0003)\n- **Scheduler is a flag, not a service** — `--schedule-evals 24h` loops the same `scoreOnce` that `--score` and the console call. (ADR-0006)\n- **Per-question eval rows** — `eval_pair_scores` lets the drift report name the worst cases without re-running judges. (ADR-0005)\n- **Heuristic router first** — `len > 200` or a complexity keyword → quality tier. One `if` you can explain beats a classifier you can't demo.\n- **Fail-closed on sensitive data** — the routing plan is built once; a sensitive request has no cloud candidate, ever. (ADR-0007)\n- **Judge bias guards** — justification before verdict, negative verdicts parsed first, temperature 0, judge model + prompt version logged, `judge_changed` flag in the drift report.\n- **Golden answers are standalone propositions** — the judge never sees the question, so one-word, list-fragment and \"Because…\" answers score 0 regardless of truth; `--freeze-golden` rejects them. Learned from v1 (0.969) → v2 (1.000).\n- **Telemetry never on the request path** — spans go through a 1024-deep channel to one writer goroutine; a dead DB costs a chat nothing (verified: chat stays 200 with Postgres stopped).\n- **The console is read-mostly and framework-free** — three embedded files on the Tower design tokens; abortable fetches, retry on 5xx, inline error + Retry, polling that never replaces a field you are typing in. (ADR-0008)\n\n## Known limits (honest list)\n\n- 8 GB VRAM: one chat model resident at a time; an eval run and live traffic contend for the GPU (`docs/VRAM.md`, `EVAL_TIMEOUT`).\n- The API-key rate limiter is per process; usage is charged after the response, so a budget can overshoot by one request.\n- Console endpoints are unauthenticated (localhost tool) — proxy them if exposed.\n- Tracker: no lease for two concurrent resumers, no `failed` terminal state; step spans use the workflow id as parent.\n- Retrieval is vector-only; hybrid BM25 + RRF is parked.\n\nRoadmap and the full defect log live in [`docs/PLAN.md`](docs/PLAN.md) (§7 backlog, §9 v1.0 tracks). Where this sits in the market and what to build next: [`docs/MARKET.md`](docs/MARKET.md).\n\n## Privacy & license\n\nThe gateway never stores prompts; sensitive requests never leave the machine — see [`PRIVACY.md`](PRIVACY.md). Licensed under [Apache-2.0](LICENSE).\n"}