snappcloud-bot

snappcloud-bot

A ChatOps SRE assistant for Kubernetes and OpenShift that cannot exceed the asking user's own access.

Ask it a question in chat — "why are the pods in team-a crashing on prod-1?" — and it investigates across your clusters with a reasoning model driving MCP servers: pods, logs, events, rollouts, quotas, network flows, packet drops, ingress, GitOps state and Prometheus metrics. It answers with the evidence it found, the root cause, and what to do.

It can also investigate Alertmanager alerts on its own: mark a channel, and alerts posted there are batched, investigated, and answered in the thread before anyone opens it.

User documentation: docs.snappcloud.io · Related: mcp-authz (per-cluster authorization) · openshift-mcp (read-only Kubernetes tools)

Why this exists

An LLM with cluster credentials is a liability: prompts are not a security boundary, and "only show the user their own namespaces" is a request, not a control. This bot is built the other way around.

• It holds no cluster credentials. All reads happen in MCP servers, each with its own read-only role.
• Authorization is resolved per request against the same RBAC oc uses, via mcp-authz — one per cluster.
• Results are filtered before the model sees them. A record naming a namespace the caller cannot access is dropped; a bare IP is resolved and gated; if scope cannot be verified the result is withheld. The model never receives data the user is not entitled to, so it cannot leak it.
• Metrics queries are rewritten, not filtered. A PromQL result carries no namespace to filter on, so every selector is pinned to the caller's namespaces before the query runs — unless the server is deliberately marked unscoped.
• Strictly read-only. It tells you the fix; you apply it.

What you need to run it

• A chat front end — Mattermost today (the agent loop is independent of it)
• An Anthropic-style /v1/messages LLM endpoint
• One mcp-authz per cluster
• One or more MCP servers, e.g. openshift-mcp for Kubernetes, plus anything else that speaks MCP

make build
./bin/snappcloud-bot -config config.example.yaml

config.example.yaml is a full, commented configuration. Deployment manifests live in the SnappCloud helm repository; the binary needs only a config file, a Mattermost token, an LLM key, and network reach to its MCP servers.

How it works

Mattermost user ── message (WebSocket)
        ▼
snappcloud-bot ── resolve SSO email
        │  scope = mcp-authz(every region): cluster -> {namespaces, clusterWide}
        │          (groups-aware SAR, admin fast-path, cached — authz.cacheTTL)
        ▼  if authorized somewhere
   agent loop (streaming reasoning model, all authorized clusters at once):
        │  model proposes cluster-tagged tool calls (in parallel)
        │  ── infra tool (nodes/BGP/agent status)?
        │        cluster-admin → run, return unfiltered
        │        otherwise    → denied ("requires cluster-admin")
        │  ── tenant tool → MCP call → FILTER result vs the cluster's namespaces
        │        drop records in unauthorized namespaces; resolve bare IPs via
        │        mcp-authz /v1/resolve (fail-closed); cap oversized output
        │  ── feed only authorized data back
        ▼
   answer ── Mattermost (threaded in channels, split if long)

Enforcement (why the model can't leak)

MCP tools take pods/IPs/services, not namespaces — the namespace lives in the result data. So the bot filters every tenant-data tool result before the model sees it: a record referencing a namespace the user can't access is dropped; a bare IP is resolved to its namespace via mcp-authz and gated; if resolution is unavailable the result is withheld (fail-closed).

Two shapes need more than per-record namespace matching:

• Peer-shaped records (a Hubble flow names a source and a destination) are kept when the caller owns either side. A tenant's own egress always names a peer they do not own — CoreDNS, another team's service, the world — so the all-sides rule hid their own traffic from them, including DNS failures and cross-namespace policy denials. They are a party to that traffic and can already see it with tcpdump inside their own pod. A flow they own neither side of is still dropped, and the relaxation applies only to peer-shaped records: an ordinary resource never becomes visible for mentioning an authorized namespace.
• Aggregates (flow_summary) hide the namespace in the map key ({"top_source_pods": {"team-b/api-0": 42}}), where a namespace-field scan cannot see it. Those entries are scoped by key before the result is returned. The model only ever receives authorized data — authorization is not the model's job, and the prompt requires withheld data to be reported as an access limitation, never as "does not exist".

Three exemption classes:

• Cluster-infrastructure tools (toolRules.<tool>.clusterAdminOnly): the Cilium datapath (BGP, BPF maps, identities, monitor, statedb), node tools, get_resource, and Grafana's label-enumeration tools. Denied outright for non-admins; returned unfiltered for callers whose cluster-wide SAR passed (clusterWide in the mcp-authz scope response). Unfiltered because there is no namespace in the output to filter to — not because the caller is trusted with more.

  clusterWide is a second SubjectAccessReview with an empty namespace ("may this subject get pods at cluster scope?"), resolved per cluster. It gates exactly three things: these tools, global servers marked clusterAdminOnly, and the diagnostics command. It is not a filter bypass — an ordinary tool called by a cluster-admin runs the same namespace enforcement and the same result filter as anyone else's. An admin sees every namespace because their namespace list contains every namespace, not because a flag turned the filter off.

• Self-authorized servers (servers[].selfAuthorized): trusted, identity-aware MCP servers (e.g. argocd-mcp). The caller's SSO identity is forwarded as the X-Remote-User header and the server authorizes the caller itself, so the bot skips namespace filtering and returns results unfiltered — trusting it the same way it trusts mcp-authz. The identity comes from the authenticated Mattermost user (never a tool argument, so the model can't spoof it); a request with no identity is refused, never sent unscoped (fail-closed).

• Unscoped servers (servers[].unscoped): results every user authorized on the cluster may see in full. The bot skips namespace enforcement, PromQL pinning and result filtering for such a server's tools; allowTools and clusterAdminOnly rules still apply. An operator's deliberate widening — e.g. Prometheus metrics, so an investigation can see the whole cluster — and logged as a warning at startup so it is never forgotten. Note these are two independent axes: metric data is open to every authorized user because the server is unscoped, while label enumeration on that same server stays clusterAdminOnly, since listing every label value would hand one tenant the namespace and pod names of all the others.

• Global servers (the general docs): namespace-agnostic, available to any authorized user, not scope-filtered. A global server marked clusterAdminOnly is served only to callers holding cluster-wide access on some cluster, and its tools are not listed at all for anyone else — the model cannot propose them or name them. Where the URL is itself a credential (a capability token in the path), set urlEnv and keep it in the Secret.

Behavior

• Identity. Sender's SSO email via the Mattermost API; identityMap can override email → username.
• Authorization. A user with no namespaces on any cluster never reaches the MCP servers (hard gate). Group-aware SARs; per-region fail-closed.
• Multi-cluster. Tools are tagged [cluster X]; the agent calls the right cluster and combines across clusters. A question that names its clusters is given only those — under any of the names a cluster goes by, including the region label an alert carries, resolved in code rather than inferred — so a question about one cluster carries one cluster's tools. A question naming none reaches every cluster the user can access. See clusters[].names.
• Thorough tool use. The system prompt pushes the model to investigate with every relevant tool (pods + logs + events + flows + policy + ingress) and reconcile them. Extend with your own MCP "skills" via agent.toolGuidance.
• Unattended runs. Scheduled reports and alert investigations are told that nobody will reply: no questions, no options, no commentary on effort. What does not fit the budget is marked n/a with a reason and listed in one closing line, because a question posted to a channel nobody is reading is a report that never arrives.
• Commands. help lists everything the bot understands, and only the features that are switched on. Commands are matched on the whole message after normalising case, punctuation and politeness (Alerts On, please works), but a message that merely mentions a command word is treated as a question — "which alerts are firing on prod-1" goes to the agent, not the command handler.
• Access refresh. Scope is cached per user (authz.cacheTTL). A user whose authorization just changed can say "refresh" to flush their own cache and get their live cluster/namespace list immediately — no wait, no restart. Lower cacheTTL for faster automatic propagation (more mcp-authz load).
• Schedules. A user can save a recurring query — schedule every day at 09:00 are any pods failing in my-ns? — plus schedules to list, run <id> to run one immediately and unschedule <id> to remove. run goes through the scheduler, not around it: same unattended framing, same empty history, same timeout, same worker pool, one run of an id at a time, and it does not consume the next scheduled run. Pasting the question into the channel instead is a different request — it carries the thread's history and the chat timeout — so it is not a way to reproduce what a schedule did. An interval schedule can name its first run (every 4h starting at 16:10 ...). Times are read and displayed in schedules.timezone, not the pod's zone. Bounded by schedules.perUser (5) / total (500) / minInterval (4h) — total ÷ minInterval is the worst-case hourly load — run by a small worker pool (concurrency) so recurring work never crowds out interactive users, and disabled automatically after repeated failures. Authorization is resolved at run time, never stored: a schedule cannot outlive the access it was created with.
• Alert channels. A user says alerts on in a channel where Alertmanager posts; from then on every post produced by an integration is treated as an alert, investigated, and answered in the channel. Humans talking there are unaffected — the discriminator is Mattermost's own marking of the post (from_webhook, from_bot, an overridden username), not the message shape, so a team's alert template can be anything. Note this is a property of the POST, not the author: an incoming webhook posts under the account of whoever created it, so an Alertmanager integration set up by an engineer carries that engineer's identity and would otherwise look like them talking. Investigations run with the marking user's authorization, resolved fresh each batch and never stored, and the bot says so publicly when the channel is marked: everyone in the channel sees what that person can see. Each cluster in a window is investigated as one unit: alerts that fire together on a cluster are usually one incident, so they go into a single investigation asked whether they share a cause, and produce one message naming the cluster. Alerts from several clusters in the same minute become one investigation each, with that cluster's full tool set and its own maxPerWindow, run in parallel up to alerts.concurrency. Alerts with no cluster label share one investigation. Alerts still inside their cooldown ride along as context — what else is firing is often what identifies the incident — without being re-diagnosed. Noise is bounded by the window, the per-alert cooldown (a window with nothing new produces no message at all), and maxPerWindow, which bounds the prompt with anything over it listed rather than dropped silently. Watchdog, resolved notifications and below-minSeverity alerts are skipped; an alert with no recognisable severity is investigated rather than dropped.
• Best-practice report. Every Saturday at 00:00 (bestPractice.schedule, in schedules.timezone) the bot scans every cluster for single-instance RabbitMQ and Redis, PDBs that block node drains (maxUnavailable: 0, minAvailable ≥ the pod count, overlapping PDBs, drains held up by unhealthy pods), and quorum systems whose PDB lets a drain take a majority, or that have no PDB. Downtime and quorum risks are reported for production namespaces only; drain blockers everywhere. The rules are code, not the model, so the same cluster state always gives the same report. Findings are grouped by the namespace's snappcloud.io/team label and the Team CR's admins, and posted as a preview in bestPractice.adminChannel. Nothing is sent until a cluster-admin says report approve <id> there: the bot then scans again, sends only what is still true (what was fixed meanwhile is dropped, what is new waits for next week), and emails it through mail (SMTP), with the findings and their fixes in a PDF. bestPractice.recipients routes it: each group of teams gets one email, to its teams' admins; the teams in no group share one email to others; cc is on every email. report reject|exclude|preview|run|status do what they say. Teams mark a finding intentional with the snappcloud.io/best-practice-ignore: <rule> annotation; the preview counts ignores so they stay visible. PDBs that block node drains (pdb-max-unavailable-zero, pdb-min-available-all, pdb-overlap, pdb-blocked-unhealthy) cannot be ignored: they must be fixed. This is the one feature that reads without a caller's scope: the bot audits every cluster on its own authority through openshift-mcp's disruption_audit and list_teams (keep both clusterAdminOnly in toolRules, so chat users cannot reach them). What it reads leaves only after a cloud admin approves the preview, which names every email and who receives it. Recipients are not checked against mcp-authz: a group's admins see every team in that group, and others and cc see whatever they get.
• Memory. Per Mattermost thread (and each DM), a transcript is kept and replayed for context; persisted to a file (memory.memoryPath, a PVC) so it survives restarts.
• Replies. Channels: in-thread, only when @-mentioned. DMs: always. Typing indicator while working; long answers split transparently.
• Mentions from other bots. The mention is detected from the event's mention list or from the message text, because Mattermost does not populate mention metadata for posts authored by other bots, webhooks, and integrations. An ignored channel message is logged at debug and counted as messages_total{outcome="ignored"}. A calling bot still needs an identity the platform can authorize: bot accounts usually have no email, so map theirs with mattermost.identityMap ("" -> username is not possible; give the bot account an email or map it) — otherwise the call is refused as unauthorized and logged as no identity for sender.
• Singleton. One WebSocket listener — a single replica on a single cluster.

Reliability

• Streaming LLM (SSE) with retries: every text/tool-use delta is accumulated, so no part of a long answer is lost; if the stream ends before completion it is retried (never returns partial). Transient failures (network, 429, 5xx) retry with backoff+jitter; 4xx do not. Falls back to a non-streaming JSON body if the endpoint ignores stream:true. HTTP/2 keep-alive transport.
• Model failover. An optional backup model (agent.fallbackLLM) behind a circuit breaker: after failureThreshold consecutive primary failures traffic moves to the backup for cooldownPeriod, then the primary is probed once and reinstated if healthy. The failing request itself is retried on the backup, so users still get an answer. A single primary success resets the counter. Watch snappcloud_bot_llm_model_requests_total{role} and snappcloud_bot_llm_failover_total{state}.
• Bounded prompts. A single tool result is capped (~100k chars, applied after filtering) so a verbose dump cannot blow the model's request budget; empty content blocks are normalized (the API rejects them).
• MCP mux skips a dead server (best-effort tool listing); a cluster with no reachable servers is dropped, not fatal. Which servers failed is recorded and told to the model, because a cluster that answered with five of its six servers looks exactly like a cluster that never had the sixth one's tools. SSE responses up to 32 MiB per line.
• Answers are finished, not posted half-written. A response that stopped at the model's output limit is continued from where it broke off and joined on (up to three times). A turn that makes no tool call but whose tail announces work it did not do is sent back — twice at most — rather than posted as the answer. Running out of tool calls produces an answer, not a plan: the final request carries no tools and says so, and the reply ends by saying it stopped.
• A tool never runs on arguments that failed to parse. Streamed tool-call arguments that do not decode are a retryable error naming the tool, never an empty argument map — a metrics query with no expression answers, and the model reasons from whatever comes back.
• Crash isolation. Each message is handled in its own goroutine with a panic recover, so one bad message can never take the singleton process down.
• Abuse guards (limits): per-user token-bucket rate limit (ratePerMin/rateBurst) and a max query length (maxQueryRunes). Both protect the LLM budget and the downstream MCP servers from a single client.

HTTP query API

The same enforced agent loop, callable outside Mattermost (api.enabled, port 8081 by default). Callers authenticate with their own OpenShift token — a user token or a ServiceAccount token:

TOKEN=$(oc whoami -t)                     # or: oc create token my-sa -n my-ns
BOT=https://snappcloud-bot.prod-1.example.com

curl -sS -H "Authorization: Bearer $TOKEN" "$BOT/v1/whoami"

curl -sS -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"query":"why are pods in my-ns crashing on prod-1?"}' "$BOT/v1/query"

Endpoint	Purpose
POST /v1/query	{"query", "history"?, "review"?} → {"answer", "user", "requestId"}
GET /v1/whoami	verify a token and see the scope it grants (no agent run)
POST /mcp	the same, as an MCP server for other agents (below)

review: true re-checks the answer against its tool results before returning — worth it when the caller acts on the answer without a person reading it. Errors are {"error", "requestId"}; quote the requestId to find the run in the logs.

As an MCP server

/mcp serves the bot to any MCP client (streamable HTTP, stateless): two read-only tools, ask (question, history?, review?) and whoami. The caller's bearer token is its identity, with the same scoping, filtering, rate limit and answer cleaning as REST. A refused caller gets a tool error the agent can read, not an HTTP 401.

claude mcp add --transport http snappcloud \
  https://snappcloud-bot.prod-1.example.com/mcp \
  --header "Authorization: Bearer $(oc whoami -t)"

An ask can run for minutes. When the client accepts text/event-stream (all streamable-HTTP clients do), the answer is streamed: headers at once, a progress notification every 15s, then the result — so HTTP clients that abort a silent response (Node's default is 5 minutes) keep waiting. The client's own tool timeout must still allow the run (in Claude Code, MCP_TOOL_TIMEOUT in ms).

The bot holds no cluster credentials, so the token is verified by mcp-authz via TokenReview (POST /v1/authenticate, requires create on authentication.k8s.io/tokenreviews). Every region is probed; the cluster that issued the token authenticates it. The resulting identity — including a ServiceAccount's system:serviceaccounts:* groups — is scoped and filtered exactly like a Mattermost user's, so an API caller can never see more than they can with oc. No token is ever logged or stored. Rate limiting is shared with Mattermost, so one identity has a single budget across both entrypoints.

Metrics

Prometheus metrics on /metrics (same port as the health probes), scraped via the chart's ServiceMonitor. Labels are deliberately low-cardinality — bounded enums only, never a user identity, namespace, or free text.

Metric	Labels	Use
snappcloud_bot_api_requests_total	outcome	HTTP query API traffic
snappcloud_bot_messages_total	outcome	answered / denied / rate_limited / agent_error / …
snappcloud_bot_message_duration_seconds	—	end-to-end answer latency
snappcloud_bot_tool_calls_total	cluster, tool, outcome	per-tool call rate
snappcloud_bot_tool_errors_total	cluster, tool, reason	which MCP tool is broken, and why (timeout, unreachable, auth, not_found, bad_args, server_error)
snappcloud_bot_tool_call_duration_seconds	cluster	slow MCP servers
snappcloud_bot_llm_requests_total / _duration_seconds	outcome	model health
snappcloud_bot_authz_requests_total / _duration_seconds	region, outcome	per-region mcp-authz health
snappcloud_bot_schedules, _schedule_owners, _schedule_limit	—	stored recurring queries, distinct owners, configured ceiling
snappcloud_bot_schedule_runs_total	outcome	scheduled runs: ok / error / skipped (owner lost access)
snappcloud_bot_schedule_run_duration_seconds, _schedule_runs_in_flight	—	scheduled run latency and worker-pool saturation
snappcloud_bot_schedules_disabled_total	—	schedules dropped after repeated failures
snappcloud_bot_alert_channels, _alerts_pending	—	watched channels; alerts buffered in open windows
snappcloud_bot_alerts_received_total	disposition	queued / duplicate / cooldown / ignored / resolved / low_severity — shows whether noise suppression is working
snappcloud_bot_alert_investigations_total	outcome	ok / error / empty / unauthorized
snappcloud_bot_alert_investigation_duration_seconds	—	investigation latency
snappcloud_bot_report_runs_total	outcome	best-practice report builds: ok / partial (a cluster could not be read) / error
snappcloud_bot_report_findings	rule, severity	findings in the latest report
snappcloud_bot_report_deliveries_total	outcome	report emails: sent / rejected (every address refused) / unrouted (nobody to send to) / send_error
snappcloud_bot_report_collect_failures_total	cluster	clusters a build could not read
snappcloud_bot_mcp_list_failures_total	server	an MCP server that failed to list its tools — its tools are missing from every answer until it recovers
snappcloud_bot_tools_offered, _tools_per_cluster	—, cluster	tools put in front of the model last turn, and what each cluster advertised
snappcloud_bot_tools_dropped_total	cluster	tools omitted to fit maxTools — a cluster the question named appearing here is the case worth acting on
snappcloud_bot_turn_outcomes_total	outcome	answered / max-iters — the second means the model ran out of tool calls mid-investigation
snappcloud_bot_answer_continuations_total	—	answers that hit the model's output limit and had to be continued; climbing means raise maxTokens
snappcloud_bot_answer_nudges_total	—	turns sent back because the model announced work it did not do
snappcloud_bot_conversation_trims_total	—	turns where the oldest tool output was dropped to fit the model's window
snappcloud_bot_active_conversations, _messages_in_flight, _handler_panics_total	—	live state

Dashboard: core/dashboards/Network/SnappCloudBot. Alerts: core/helm/apps/monitoring-int/templates/rules/snappcloud-bot.yaml.

Configuration

See config.example.yaml. Secrets are read from the environment (never YAML):

Env	Purpose
MATTERMOST_TOKEN	bot account token
LLM_API_KEY	x-api-key for the Anthropic-style endpoint
MCP_AUTHZ_TOKEN	bearer to every mcp-authz
MAIL_PASSWORD	SMTP password for mail.username (best-practice report)
<per-server>	Authorization header for an authed MCP server (e.g. CILIUM_TEH1_AUTH)

Key config sections: agent.llm (endpoint/model), agent.clusters[].servers[] (MCP servers per cluster), agent.globalServers[] (namespace-agnostic servers like docs), agent.toolGuidance (tool-usage skills), agent.toolRules (per-tool namespace-arg overrides + clusterAdminOnly infra gating), authz.regions[] (mcp-authz endpoints). A cluster's name must match an authz.regions[].name.

A question that names its clusters is given only those clusters' tools. Every tool definition is re-sent each round, and an LLM endpoint that cannot carry them all drops the tail silently, so set agent.budgets.maxTools to what your endpoint carries. The bot trims to it, one tool per server per round with named clusters first, and tells the model what was trimmed. Watch snappcloud_bot_tools_offered and snappcloud_bot_tools_dropped_total.

A cluster's names lists what else it is called (alert labels, datasource names, shorthand); agent.clusterNamePrefixes (e.g. ["okd4-"]) gives every cluster its unprefixed name too. Matching is exact, then prefix, then substring, longest name winning, ignoring case and punctuation.

Environment knowledge belongs in config, not code: identity in agent.persona, platform facts and tool usage in agent.toolGuidance. The built-in system prompt holds only the investigation method and output rules.

Review. Scheduled reports and alert investigations are re-checked against their own tool results before posting — figures that disagree between summary and table, numbers no result contains, causes stated without evidence. It is one extra model call per unattended answer, never applied to chat, and the draft is kept whenever the review would drop content. agent.reviewUnattended: false turns it off; agent.reviewPrompt replaces the instruction. Watch snappcloud_bot_answer_reviews_total{outcome}: a rising share of shrank or lost-structure means the review is not helping.

Adding a new MCP server

Append one entry under the cluster — no code change:

agent:
  clusters:
    - name: prod-1
      servers:
        - url: https://hubble-mcp.prod-1.example.com/mcp
          authHeaderEnv: HUBBLE_TEH1_AUTH   # only if it needs auth (per region)

If authed, add the key to the snappcloud_bot.mcpAuth secret (the full Authorization header). MCP Basic auth is per region — one key each. Its tools appear automatically, cluster-tagged and enforced. A namespace-agnostic server (docs) goes under agent.globalServers; cluster-infrastructure tools it exposes should be listed in agent.toolRules with clusterAdminOnly: true.

For a trusted, identity-aware server that authorizes the caller itself (e.g. argocd-mcp), set selfAuthorized: true on the server entry. The caller's SSO identity is then forwarded as X-Remote-User and all of that server's tools skip the bot's namespace enforcement, returning results unfiltered:

agent:
  clusters:
    - name: prod-1
      servers:
        - url: https://argocd-mcp.prod-1.example.com/mcp
          selfAuthorized: true   # forwards X-Remote-User; server scopes the result

Enable it only for servers you trust to enforce the caller's access from the forwarded identity — it bypasses the bot's own filtering.

To share a server's results with every tenant of the cluster instead — metrics, typically, so an investigation can correlate across namespaces — set unscoped: true. Nothing is scoped or filtered; allowTools still confines the server to its read-only tools and clusterAdminOnly rules still hold:

        - url: https://grafana-mcp.prod-1.example.com/mcp
          unscoped: true         # every tenant sees every metric
          allowTools: [query_prometheus, list_prometheus_metric_names]

The bot logs a warning for each unscoped server at startup.

Diagnosing a thin answer

diagnostics (cluster-admins, in chat) lists every MCP server the bot has for your clusters, the number of tools each advertises, and the error for any that is not answering — a server that is down, one refusing the bot's credentials, and one that genuinely lacks a tool are indistinguishable from an answer and obvious here. It also reports the model, the per-question tool-call limit and the per-request tool limit, which are the other two reasons an answer stops early, plus the running build — the image tag is republished in place, so a pod that has not been restarted since a fix answers exactly like one that has. An answer that ran out of tool calls says so in its own last line.

what you see	where to look	usual cause
"this cluster has no such tool"	diagnostics	that server did not answer — token, route, or pod
"that is not documented / not supported"	tool_errors_total{reason="backend_error"}	a doc or search tool failed before it looked: a failed search is not an empty one, and the answer should say the tool is broken
diagnostics shows 403 on servers that are up	the status cell names the cause	"no credential sent" is a key missing from the bot's Secret, not a sick server; "credential refused" is a token to rotate
every metric cell empty	check_datasources_health	the metrics backend is refusing the credentials it was given
right cluster, wrong numbers	list_datasources	several datasources exist and the default is not always the complete one
a tool missing for one cluster	tools_dropped_total	the list was trimmed; named=true in the warning is the case worth acting on
"this cluster has no metrics tool" while the log shows the server offering them	tools offered … tools= ceiling=	the endpoint carries 128 definitions and drops the rest of the array silently; the bot caps itself at that, so a count at the ceiling means something was trimmed
a tool whose result is always cut	results_truncated_total	called too broadly: the model is spending a call on a fraction of an answer
answer stops mid-sentence	answer_continuations_total	hit maxTokens; it is continued, but a climbing count means raise it
answer describes work it did not do	answer_nudges_total	the model announced calls and made none
"no dashboard measures it" with no search in the log	tool call rejected before it was sent	the model passed a uid it invented; it is refused before it is spent
investigation stops early	turn_outcomes_total{outcome="max-iters"}	ran out of tool calls
HTTP 400 asking for a shorter prompt	agent.budgets.conversationRunes	the transcript exceeded the model's window; it fails identically on every retry

Worth alerting on: rate(snappcloud_bot_mcp_list_failures_total[10m]) > 0. A server that stops answering is otherwise invisible until someone reads an answer and doubts it.

Develop

make build   # binary -> bin/snappcloud-bot
make test
make run      # config.example.yaml
make docker   # multi-arch via build/package/docker-bake.json

Deploy

Helm chart: core/helm/apps/snappcloud-bot. Singleton (replicas: 1, Recreate), no inbound Service, no cluster RBAC. Ships Deployment, ConfigMap (the config values → config.yaml), ServiceAccount, a Secret (all keys exposed as env via envFrom — including per-region mcpAuth entries), and a PVC for conversation memory. Secrets are grouped under the snappcloud_bot sops key; the shared mcp_authz.authToken sops key is read by both this chart and mcp-authz so the bearer can never drift.

License

Apache 2.0 — see LICENSE.