penumbra

Penumbra — a cache-aware, tool-gating MCP gateway

Penumbra: the partial outer shadow. It exposes only part of the toolset to the agent; the rest stays in shadow until it's relevant.

Penumbra sits in front of many MCP servers and exposes to the agent only the task-relevant subset of their tools — without breaking prompt caching.

The problem (grounded in Datadog's 2026 State of AI Engineering)

• Fact 4: 69% of all input tokens were system prompts — internal instructions, policy, and tool guidance. The report says shorten this.
• Fact 5: Context quality, not volume, is the new limiting factor; teams need systems that "select, compress, and structure the most decision-relevant information."
• Caching: only 28% of LLM calls used cached reads; the named culprit is unstable prompt prefixes — "blocks that are supposed to be stable getting reordered between requests."

When an agent connects to N MCP servers, every request stuffs the full schema of every tool from every server into the prompt. Connect 10 servers × 20 tools and you ship 200 tool definitions per call — most irrelevant to the task. That's a direct hit to the 69%.

What Penumbra does

                  agent (one MCP connection)
                          │  tools/list  { _meta: { "penumbra/task": "<user turn>" } }
                          ▼
        ┌─────────────────────────────────────────┐
        │                PENUMBRA                  │
        │                                          │
        │  1. fan out to all upstreams (goroutines)│
        │     → aggregate every tool definition    │
        │  2. rank tools vs the task (embeddings)  │
        │  3. assemble CACHE-AWARE manifest:       │
        │       core  = pinned, deterministic      │  ← stable, cacheable prefix
        │       tail  = top-K relevant, bounded    │  ← varies per request
        │  4. proxy tools/call to the owning server│
        └───────┬──────────────┬──────────────┬────┘
                ▼              ▼              ▼
            MCP srv A      MCP srv B      MCP srv C

Beyond tools: cache-aware context gating

The same core/tail split now applies to context — conversation-history fragments and retrieved documents — via internal/contextgate and the POST /context/select endpoint. Datadog 2026 names context quality (not volume) the limiting factor: a stable, cacheable core (e.g. a rolling summary) anchors the prefix while a bounded, task-relevant tail of docs/history varies per request. Pinned items stay byte-stable and hashed exactly like the tool core.

Cross-service token budget (runaway-loop termination)

internal/budget charges Penumbra's tool-call traffic against the same per-session budget Sentinel tracks (/v1/budget/{session}), so a runaway agent loop — not just a runaway model loop — is throttled fleet-wide. The session id rides the shared trace as W3C baggage (session.id); when the budget is exhausted, tools/call short-circuits with an error result instead of doing more upstream work. Leave budget.base_url empty to run standalone.

The differentiator: cache-aware gating

Naive "top-K tools per request" destroys prompt caching, because the tool list is part of the model's stable prefix. Penumbra splits the manifest into:

• Core — pinned, always-present tools in canonical order. Byte-stable across requests, so it anchors a cacheable prefix. We hash it and emit the hash as a span attribute; if it flaps, cache hits collapse and you'll see it.
• Tail — a small, bounded, task-relevant set that adapts per request.

You get the token reduction (Fact 4/5) and keep the cache prefix stable (the page-10 caching point). The 8-star RAG-MCP routers that exist don't do this.

Prior art (honest)

Semantic tool routing exists in small form: smart-mcp (Python/FAISS), ElBruno.ModelContextProtocol (C#), rag-mcp-router (TS), and MetaMCP lists it "coming soon." None in Go, none cache-aware, none production-grade. Penumbra is a known pattern executed well in Go — not a unicorn.

Integration with Sentinel & Twinshot

Penumbra is the tool layer of a three-project AI-reliability stack:

Project	Layer	Pattern	Lang
Sentinel	LLM responses	quality circuit breaker	Python
Twinshot	LLM model upgrades	shadow mode + canary	Python
Penumbra	MCP tool exposure	cache-aware gating	Go

They stitch together through one shared trace:

Sentinel: llm.complete            (model decides to call a tool)
   └─ agent
        └─ penumbra.tools/list     (gate.select — 200 in, 9 out)
        └─ penumbra.tools/call     (routed to owning upstream)
             └─ upstream MCP server

• Shared OpenTelemetry — all three propagate the same W3C traceparent and export to the same Jaeger (otel.endpoint). One trace, end to end.
• Embeddings through Sentinel — set embedding.base_url to http://sentinel:8000/v1 and Penumbra's tool-ranking embeddings inherit Sentinel's circuit breakers, cache accounting, and tracing.

Protocol surface (spec 2025-11-25, streamable HTTP)

Toward upstreams (as an MCP client): lazy initialize handshake with MCP-Session-Id capture and MCP-Protocol-Version on every subsequent request, transparent re-initialize on session expiry (HTTP 404), paginated tools/list (follows nextCursor), POST responses accepted as either application/json or text/event-stream, and a best-effort GET SSE stream watching notifications/tools/list_changed. Pre-lifecycle upstreams that answer initialize with method-not-found are served bare.

Toward the agent (as an MCP server): initialize (version negotiation + instructions teaching the _meta["penumbra/task"] hint), ping, notifications answered 202, gated tools/list, proxied tools/call, and spec-required Origin validation (localhost by default, allowed_origins to extend).

Aggregation is cached (manifest_ttl, default 30s) and invalidated the moment a watched upstream announces a tool-list change.

Metrics

/metrics (Prometheus text) and /metrics.json (same numbers plus the current hash strings) expose the two headline signals with zero client-library deps:

• penumbra_tokens_saved_total — estimated prompt tokens the gate withheld (tool name+description+schema of every dropped definition).
• penumbra_core_hash_changes_total — how often the cacheable core flapped. This should stay at 0 between config/schema changes; growth means the stable prefix is breaking and cache hits are collapsing.

Plus request counts, exposed/dropped gauges, budget-blocked calls, upstream warnings (degraded upstreams, name collisions), and listChanged invalidations.

Run

cp config.example.yaml config.yaml      # set upstreams + embedding endpoint
cp .env.example .env                    # set API keys
go mod tidy
go test ./...
go run ./cmd/penumbra -config config.yaml
# or: docker compose up --build
open http://localhost:16686             # Jaeger — watch tools dropped per request
curl -s localhost:7000/metrics.json     # tokens saved + core-hash stability

Point an agent at http://localhost:7000/mcp as its single MCP server. To get task-aware gating, the client sets the latest user turn on tools/list:

{"method":"tools/list","params":{"_meta":{"penumbra/task":"what's the weather in tokyo"}}}

Without the hint, Penumbra returns just the pinned core — small and cacheable either way.

Layout

cmd/penumbra/main.go          gateway wiring + mcp.Handler impl (telemetry + metrics)
internal/mcp/
  types.go                    JSON-RPC + MCP shapes (initialize, tools/list, tools/call)
  client.go                   upstream client: handshake, sessions, pagination, SSE, watch
  server.go                   agent-facing endpoint: lifecycle, gating entry, origin check
  sse.go                      minimal SSE reader (POST streams + GET notifications)
internal/upstream/manager.go  concurrent fan-out, manifest cache + invalidation, routing
internal/gating/
  ranker.go                   embedding-based relevance ranking (tool embeds cached)
  cache_aware.go              core/tail split + canonical order + core hash  ← the idea
  gate.go                     request-time entrypoint
internal/contextgate/         the same core/tail split applied to context items
internal/budget/              shared per-session token budget (Sentinel's ledger)
internal/embed/               embedder interface + OpenAI/Sentinel-compatible client
internal/metrics/             stdlib-only registry: Prometheus text + JSON views
internal/observability/       OTel setup, W3C propagation, traced HTTP transport

Status

Complete. The full request path is implemented and tested end to end: MCP lifecycle on both sides, paginated aggregation with cache + listChanged invalidation, cache-aware tool/context gating, cross-service budget, tracing, and metrics. go test ./... covers the gate's cache guarantees, the client's handshake/pagination/SSE/session-recovery, the server lifecycle, the manager's cache and failure isolation, and the metrics exposition.