{"id":"8f0326da-2c28-4b09-89a8-10478d0991f2","slug":"snappcloud-bot","kind":"mcp","name":"snappcloud-bot","summary":"ChatOps SRE assistant for Kubernetes/OpenShift. Answers questions and investigates Alertmanager alerts over MCP servers — strictly inside each user's own RBAC, read-only, with no cluster credentials o","owner":"snapp-incubator","repoUrl":"https://github.com/snapp-incubator/snappcloud-bot","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":2,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-06-22T18:38:08.000Z","addedAt":"2026-10-21T06:17:02.839Z","logoUrl":"https://avatars.githubusercontent.com/u/89763510?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["cloud-infra","communication","memory-knowledge"],"tags":["ai-agent","chatops","cilium","devops-tools","golang","incident-response","kubernetes","llm"],"runtime":"go","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-27T00:32:37.860Z","sample":false,"url":"https://conduits.dev/tools/snappcloud-bot","language":"Go","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# snappcloud-bot\n\n**A ChatOps SRE assistant for Kubernetes and OpenShift that cannot exceed the\nasking user's own access.**\n\nAsk it a question in chat — *\"why are the pods in team-a crashing on prod-1?\"* —\nand it investigates across your clusters with a reasoning model driving\n[MCP](https://modelcontextprotocol.io) servers: pods, logs, events, rollouts,\nquotas, network flows, packet drops, ingress, GitOps state and Prometheus\nmetrics. It answers with the evidence it found, the root cause, and what to do.\n\nIt can also **investigate Alertmanager alerts on its own**: mark a channel, and\nalerts posted there are batched, investigated, and answered in the thread before\nanyone opens it.\n\n**User documentation:**\n[docs.snappcloud.io](https://docs.snappcloud.io/docs/observability/snappcloud-bot)\n· **Related:** [mcp-authz](https://github.com/snapp-incubator/mcp-authz) (per-cluster\nauthorization) · [openshift-mcp](https://github.com/snapp-incubator/openshift-mcp)\n(read-only Kubernetes tools)\n\n## Why this exists\n\nAn LLM with cluster credentials is a liability: prompts are not a security\nboundary, and \"only show the user their own namespaces\" is a request, not a\ncontrol. This bot is built the other way around.\n\n- **It holds no cluster credentials.** All reads happen in MCP servers, each\n  with its own read-only role.\n- **Authorization is resolved per request** against the same RBAC `oc` uses, via\n  [mcp-authz](https://github.com/snapp-incubator/mcp-authz) — one per cluster.\n- **Results are filtered before the model sees them.** A record naming a\n  namespace the caller cannot access is dropped; a bare IP is resolved and\n  gated; if scope cannot be verified the result is withheld. The model never\n  receives data the user is not entitled to, so it cannot leak it.\n- **Metrics queries are rewritten, not filtered.** A PromQL result carries no\n  namespace to filter on, so every selector is pinned to the caller's namespaces\n  *before* the query runs — unless the server is deliberately marked `unscoped`.\n- **Strictly read-only.** It tells you the fix; you apply it.\n\n## What you need to run it\n\n- A chat front end — Mattermost today (the agent loop is independent of it)\n- An Anthropic-style `/v1/messages` LLM endpoint\n- One [mcp-authz](https://github.com/snapp-incubator/mcp-authz) per cluster\n- One or more MCP servers, e.g.\n  [openshift-mcp](https://github.com/snapp-incubator/openshift-mcp) for\n  Kubernetes, plus anything else that speaks MCP\n\n```sh\nmake build\n./bin/snappcloud-bot -config config.example.yaml\n```\n\n`config.example.yaml` is a full, commented configuration. Deployment manifests\nlive in the SnappCloud helm repository; the binary needs only a config file, a\nMattermost token, an LLM key, and network reach to its MCP servers.\n\n## How it works\n\n```\nMattermost user ── message (WebSocket)\n        ▼\nsnappcloud-bot ── resolve SSO email\n        │  scope = mcp-authz(every region): cluster -> {namespaces, clusterWide}\n        │          (groups-aware SAR, admin fast-path, cached — authz.cacheTTL)\n        ▼  if authorized somewhere\n   agent loop (streaming reasoning model, all authorized clusters at once):\n        │  model proposes cluster-tagged tool calls (in parallel)\n        │  ── infra tool (nodes/BGP/agent status)?\n        │        cluster-admin → run, return unfiltered\n        │        otherwise    → denied (\"requires cluster-admin\")\n        │  ── tenant tool → MCP call → FILTER result vs the cluster's namespaces\n        │        drop records in unauthorized namespaces; resolve bare IPs via\n        │        mcp-authz /v1/resolve (fail-closed); cap oversized output\n        │  ── feed only authorized data back\n        ▼\n   answer ── Mattermost (threaded in channels, split if long)\n```\n\n## Enforcement (why the model can't leak)\n\nMCP tools take pods/IPs/services, not namespaces — the namespace lives in the\n**result data**. So the bot filters every tenant-data tool result before the\nmodel sees it: a record referencing a namespace the user can't access is\ndropped; a bare IP is resolved to its namespace via mcp-authz and gated; if\nresolution is unavailable the result is withheld (**fail-closed**).\n\nTwo shapes need more than per-record namespace matching:\n- **Peer-shaped records** (a Hubble flow names a source *and* a destination) are\n  kept when the caller owns **either** side. A tenant's own egress always names\n  a peer they do not own — CoreDNS, another team's service, the world — so the\n  all-sides rule hid their own traffic from them, including DNS failures and\n  cross-namespace policy denials. They are a party to that traffic and can\n  already see it with `tcpdump` inside their own pod. A flow they own **neither**\n  side of is still dropped, and the relaxation applies only to peer-shaped\n  records: an ordinary resource never becomes visible for mentioning an\n  authorized namespace.\n- **Aggregates** (`flow_summary`) hide the namespace in the map *key*\n  (`{\"top_source_pods\": {\"team-b/api-0\": 42}}`), where a namespace-field scan\n  cannot see it. Those entries are scoped by key before the result is returned. The model\nonly ever receives authorized data — authorization is not the model's job, and\nthe prompt requires withheld data to be reported as an access limitation, never\nas \"does not exist\".\n\nThree exemption classes:\n- **Cluster-infrastructure tools** (`toolRules.<tool>.clusterAdminOnly`): the\n  Cilium datapath (BGP, BPF maps, identities, monitor, statedb), node tools,\n  `get_resource`, and Grafana's label-enumeration tools. Denied outright for\n  non-admins; returned unfiltered for callers whose cluster-wide SAR passed\n  (`clusterWide` in the mcp-authz scope response). Unfiltered because there is\n  no namespace in the output to filter *to* — not because the caller is\n  trusted with more.\n\n  `clusterWide` is a second SubjectAccessReview with an empty namespace (\"may\n  this subject `get pods` at cluster scope?\"), resolved per cluster. It gates\n  exactly three things: these tools, global servers marked `clusterAdminOnly`,\n  and the `diagnostics` command. **It is not a filter bypass** — an ordinary\n  tool called by a cluster-admin runs the same namespace enforcement and the\n  same result filter as anyone else's. An admin sees every namespace because\n  their namespace list *contains* every namespace, not because a flag turned\n  the filter off.\n- **Self-authorized servers** (`servers[].selfAuthorized`): trusted,\n  identity-aware MCP servers (e.g. argocd-mcp). The caller's SSO identity is\n  forwarded as the `X-Remote-User` header and the server authorizes the caller\n  itself, so the bot skips namespace filtering and returns results unfiltered —\n  trusting it the same way it trusts mcp-authz. The identity comes from the\n  authenticated Mattermost user (never a tool argument, so the model can't spoof\n  it); a request with no identity is refused, never sent unscoped (fail-closed).\n- **Unscoped servers** (`servers[].unscoped`): results every user authorized on\n  the cluster may see in full. The bot skips namespace enforcement, PromQL\n  pinning and result filtering for such a server's tools; `allowTools` and\n  `clusterAdminOnly` rules still apply. An operator's deliberate widening —\n  e.g. Prometheus metrics, so an investigation can see the whole cluster — and\n  logged as a warning at startup so it is never forgotten. Note these are two\n  independent axes: metric *data* is open to every authorized user because the\n  server is `unscoped`, while *label enumeration* on that same server stays\n  `clusterAdminOnly`, since listing every label value would hand one tenant the\n  namespace and pod names of all the others.\n- **Global servers** (the general docs): namespace-agnostic, available to any\n  authorized user, not scope-filtered. A global server marked\n  `clusterAdminOnly` is served only to callers holding cluster-wide access on\n  some cluster, and its tools are **not listed at all** for anyone else — the\n  model cannot propose them or name them. Where the URL is itself a credential\n  (a capability token in the path), set `urlEnv` and keep it in the Secret.\n\n## Behavior\n\n- **Identity.** Sender's SSO **email** via the Mattermost API; `identityMap` can\n  override email → username.\n- **Authorization.** A user with no namespaces on any cluster never reaches the\n  MCP servers (hard gate). Group-aware SARs; per-region fail-closed.\n- **Multi-cluster.** Tools are tagged `[cluster X]`; the agent calls the right\n  cluster and combines across clusters. A question that **names** its clusters\n  is given only those — under any of the names a cluster goes by, including the\n  region label an alert carries, resolved in code rather than inferred — so a\n  question about one cluster carries one cluster's tools. A question naming none\n  reaches every cluster the user can access. See `clusters[].names`.\n- **Thorough tool use.** The system prompt pushes the model to investigate with\n  every relevant tool (pods + logs + events + flows + policy + ingress) and\n  reconcile them. Extend with your own MCP \"skills\" via `agent.toolGuidance`.\n- **Unattended runs.** Scheduled reports and alert investigations are told that\n  nobody will reply: no questions, no options, no commentary on effort. What\n  does not fit the budget is marked `n/a` with a reason and listed in one\n  closing line, because a question posted to a channel nobody is reading is a\n  report that never arrives.\n- **Commands.** `help` lists everything the bot understands, and only the\n  features that are switched on. Commands are matched on the whole message\n  after normalising case, punctuation and politeness (`Alerts On, please` works),\n  but a message that merely *mentions* a command word is treated as a question —\n  \"which alerts are firing on prod-1\" goes to the agent, not the command handler.\n- **Access refresh.** Scope is cached per user (`authz.cacheTTL`). A\n  user whose authorization just changed can say **\"refresh\"** to flush their own\n  cache and get their live cluster/namespace list immediately — no wait, no\n  restart. Lower `cacheTTL` for faster automatic propagation (more mcp-authz load).\n- **Schedules.** A user can save a recurring query — `schedule every day at\n  09:00 are any pods failing in my-ns?` — plus `schedules` to list,\n  `run <id>` to run one immediately and `unschedule <id>` to remove.\n  `run` goes through the scheduler, not around it: same unattended framing, same\n  empty history, same timeout, same worker pool, one run of an id at a time, and\n  it does not consume the next scheduled run. Pasting the question into the\n  channel instead is a *different* request — it carries the thread's history and\n  the chat timeout — so it is not a way to reproduce what a schedule did.\n  An interval schedule can name its first run\n  (`every 4h starting at 16:10 ...`). Times are read and displayed in\n  `schedules.timezone`, not the pod's zone. Bounded by `schedules.perUser` (5) / `total`\n  (500) / `minInterval` (4h) — `total ÷ minInterval` is the worst-case hourly\n  load — run by a small worker pool (`concurrency`) so recurring work never\n  crowds out interactive users, and disabled automatically after repeated\n  failures. **Authorization is resolved at run time, never stored**: a schedule\n  cannot outlive the access it was created with.\n- **Alert channels.** A user says `alerts on` in a channel where Alertmanager\n  posts; from then on every post **produced by an integration** is treated as an\n  alert, investigated, and answered in the channel. Humans talking there are\n  unaffected — the discriminator is Mattermost's own marking of the post\n  (`from_webhook`, `from_bot`, an overridden username), not the message shape, so\n  a team's alert template can be anything. Note this is a property of the POST,\n  not the author: an incoming webhook posts under the account of whoever created\n  it, so an Alertmanager integration set up by an engineer carries that\n  engineer's identity and would otherwise look like them talking. Investigations run with the **marking\n  user's** authorization, resolved fresh each batch and never stored, and the\n  bot says so publicly when the channel is marked: everyone in the channel sees\n  what that person can see. **Each cluster in a window is investigated as one unit**:\n  alerts that fire together on a cluster are usually one incident, so they go\n  into a single investigation asked whether they share a cause, and produce one\n  message naming the cluster. Alerts from several clusters in the same minute\n  become one investigation each, with that cluster's full tool set and its own\n  `maxPerWindow`, run in parallel up to `alerts.concurrency`. Alerts with no\n  cluster label share one investigation. Alerts\n  still inside their `cooldown` ride along as context — what else is firing is\n  often what identifies the incident — without being re-diagnosed. Noise is\n  bounded by the `window`, the per-alert `cooldown` (a window with nothing new\n  produces no message at all), and `maxPerWindow`, which bounds the prompt with\n  anything over it listed rather than dropped silently. `Watchdog`, resolved notifications and\n  below-`minSeverity` alerts are skipped; an alert with *no* recognisable\n  severity is investigated rather than dropped.\n- **Best-practice report.** Every Saturday at 00:00 (`bestPractice.schedule`,\n  in `schedules.timezone`) the bot scans every cluster for single-instance\n  RabbitMQ and Redis, PDBs that block node drains (`maxUnavailable: 0`,\n  `minAvailable` ≥ the pod count, overlapping PDBs, drains held up by\n  unhealthy pods), and quorum systems whose PDB lets a drain take a majority, or\n  that have no PDB. Downtime and quorum risks are reported for production\n  namespaces only; drain blockers everywhere. The rules are code, not the\n  model, so the same cluster state always gives the same report. Findings are\n  grouped by the namespace's `snappcloud.io/team` label and the `Team` CR's\n  admins, and posted as a **preview** in `bestPractice.adminChannel`. Nothing is\n  sent until a cluster-admin says `report approve <id>` there: the bot then\n  scans again, sends only what is still true (what was fixed meanwhile is\n  dropped, what is new waits for next week), and emails it through `mail`\n  (SMTP), with the findings and their fixes in a PDF. `bestPractice.recipients`\n  routes it: each group of teams gets one email, to its teams' admins; the\n  teams in no group share one email to `others`; `cc` is on every email.\n  `report reject|exclude|preview|run|status` do what they say. Teams mark a\n  finding intentional with the `snappcloud.io/best-practice-ignore: <rule>`\n  annotation; the preview counts ignores so they stay visible. PDBs that block\n  node drains (`pdb-max-unavailable-zero`, `pdb-min-available-all`,\n  `pdb-overlap`, `pdb-blocked-unhealthy`) cannot be ignored: they must be fixed.\n  **This is the one feature that reads without a caller's scope**: the bot\n  audits every cluster on its own authority through openshift-mcp's\n  `disruption_audit` and `list_teams` (keep both `clusterAdminOnly` in\n  `toolRules`, so chat users cannot reach them). What it reads leaves only\n  after a cloud admin approves the preview, which names every email and who\n  receives it. Recipients are not checked against mcp-authz: a group's admins\n  see every team in that group, and `others` and `cc` see whatever they get.\n- **Memory.** Per Mattermost thread (and each DM), a transcript is kept and\n  replayed for context; persisted to a file (`memory.memoryPath`, a PVC) so it\n  survives restarts.\n- **Replies.** Channels: in-thread, only when @-mentioned. DMs: always. Typing\n  indicator while working; long answers split transparently.\n- **Mentions from other bots.** The mention is detected from the event's mention\n  list *or* from the message text, because Mattermost does not populate mention\n  metadata for posts authored by other bots, webhooks, and integrations. An\n  ignored channel message is logged at debug and counted as\n  `messages_total{outcome=\"ignored\"}`. A calling bot still needs an identity the\n  platform can authorize: bot accounts usually have no email, so map theirs with\n  `mattermost.identityMap` (`\"\" -> username` is not possible; give the bot\n  account an email or map it) — otherwise the call is refused as unauthorized\n  and logged as `no identity for sender`.\n- **Singleton.** One WebSocket listener — a single replica on a single cluster.\n\n## Reliability\n\n- **Streaming LLM (SSE)** with retries: every text/tool-use delta is accumulated,\n  so no part of a long answer is lost; if the stream ends before completion it is\n  **retried** (never returns partial). Transient failures (network, `429`, `5xx`)\n  retry with backoff+jitter; `4xx` do not. Falls back to a non-streaming JSON\n  body if the endpoint ignores `stream:true`. HTTP/2 keep-alive transport.\n- **Model failover.** An optional backup model (`agent.fallbackLLM`) behind a\n  circuit breaker: after `failureThreshold` consecutive primary failures traffic\n  moves to the backup for `cooldownPeriod`, then the primary is probed once and\n  reinstated if healthy. The failing request itself is retried on the backup, so\n  users still get an answer. A single primary success resets the counter.\n  Watch `snappcloud_bot_llm_model_requests_total{role}` and\n  `snappcloud_bot_llm_failover_total{state}`.\n- **Bounded prompts.** A single tool result is capped (~100k chars, applied after\n  filtering) so a verbose dump cannot blow the model's request budget; empty\n  content blocks are normalized (the API rejects them).\n- **MCP mux** skips a dead server (best-effort tool listing); a cluster with no\n  reachable servers is dropped, not fatal. Which servers failed is recorded and\n  told to the model, because a cluster that answered with five of its six\n  servers looks exactly like a cluster that never had the sixth one's tools.\n  SSE responses up to 32 MiB per line.\n- **Answers are finished, not posted half-written.** A response that stopped at\n  the model's output limit is continued from where it broke off and joined on\n  (up to three times). A turn that makes no tool call but whose tail announces\n  work it did not do is sent back — twice at most — rather than posted as the\n  answer. Running out of tool calls produces an answer, not a plan: the final\n  request carries no tools and says so, and the reply ends by saying it stopped.\n- **A tool never runs on arguments that failed to parse.** Streamed tool-call\n  arguments that do not decode are a retryable error naming the tool, never an\n  empty argument map — a metrics query with no expression answers, and the model\n  reasons from whatever comes back.\n- **Crash isolation.** Each message is handled in its own goroutine with a panic\n  recover, so one bad message can never take the singleton process down.\n- **Abuse guards** (`limits`): per-user token-bucket rate limit\n  (`ratePerMin`/`rateBurst`) and a max query length (`maxQueryRunes`). Both\n  protect the LLM budget and the downstream MCP servers from a single client.\n\n## HTTP query API\n\nThe same enforced agent loop, callable outside Mattermost (`api.enabled`, port\n`8081` by default). Callers authenticate with **their own OpenShift token** — a\nuser token or a **ServiceAccount** token:\n\n```bash\nTOKEN=$(oc whoami -t)                     # or: oc create token my-sa -n my-ns\nBOT=https://snappcloud-bot.prod-1.example.com\n\ncurl -sS -H \"Authorization: Bearer $TOKEN\" \"$BOT/v1/whoami\"\n\ncurl -sS -H \"Authorization: Bearer $TOKEN\" -H 'Content-Type: application/json' \\\n  -d '{\"query\":\"why are pods in my-ns crashing on prod-1?\"}' \"$BOT/v1/query\"\n```\n\n| Endpoint | Purpose |\n|---|---|\n| `POST /v1/query` | `{\"query\", \"history\"?, \"review\"?}` → `{\"answer\", \"user\", \"requestId\"}` |\n| `GET /v1/whoami` | verify a token and see the scope it grants (no agent run) |\n| `POST /mcp` | the same, as an MCP server for other agents (below) |\n\n`review: true` re-checks the answer against its tool results before returning —\nworth it when the caller acts on the answer without a person reading it. Errors\nare `{\"error\", \"requestId\"}`; quote the `requestId` to find the run in the logs.\n\n### As an MCP server\n\n`/mcp` serves the bot to any MCP client (streamable HTTP, stateless): two\nread-only tools, `ask` (`question`, `history`?, `review`?) and `whoami`. The\ncaller's bearer token is its identity, with the same scoping, filtering, rate\nlimit and answer cleaning as REST. A refused caller gets a tool error the agent\ncan read, not an HTTP 401.\n\n```bash\nclaude mcp add --transport http snappcloud \\\n  https://snappcloud-bot.prod-1.example.com/mcp \\\n  --header \"Authorization: Bearer $(oc whoami -t)\"\n```\n\nAn `ask` can run for minutes. When the client accepts `text/event-stream` (all\nstreamable-HTTP clients do), the answer is streamed: headers at once, a progress\nnotification every 15s, then the result — so HTTP clients that abort a silent\nresponse (Node's default is 5 minutes) keep waiting. The client's own tool\ntimeout must still allow the run (in Claude Code, `MCP_TOOL_TIMEOUT` in ms).\n\nThe bot holds no cluster credentials, so the token is verified by **mcp-authz\nvia TokenReview** (`POST /v1/authenticate`, requires `create` on\n`authentication.k8s.io/tokenreviews`). Every region is probed; the cluster that\nissued the token authenticates it. The resulting identity — including a\nServiceAccount's `system:serviceaccounts:*` groups — is scoped and filtered\nexactly like a Mattermost user's, so an API caller can never see more than they\ncan with `oc`. No token is ever logged or stored. Rate limiting is shared with\nMattermost, so one identity has a single budget across both entrypoints.\n\n## Metrics\n\nPrometheus metrics on `/metrics` (same port as the health probes), scraped via\nthe chart's ServiceMonitor. Labels are deliberately low-cardinality — bounded\nenums only, never a user identity, namespace, or free text.\n\n| Metric | Labels | Use |\n|---|---|---|\n| `snappcloud_bot_api_requests_total` | `outcome` | HTTP query API traffic |\n| `snappcloud_bot_messages_total` | `outcome` | answered / denied / rate_limited / agent_error / … |\n| `snappcloud_bot_message_duration_seconds` | — | end-to-end answer latency |\n| `snappcloud_bot_tool_calls_total` | `cluster`, `tool`, `outcome` | per-tool call rate |\n| `snappcloud_bot_tool_errors_total` | `cluster`, `tool`, `reason` | which MCP tool is broken, and why (`timeout`, `unreachable`, `auth`, `not_found`, `bad_args`, `server_error`) |\n| `snappcloud_bot_tool_call_duration_seconds` | `cluster` | slow MCP servers |\n| `snappcloud_bot_llm_requests_total` / `_duration_seconds` | `outcome` | model health |\n| `snappcloud_bot_authz_requests_total` / `_duration_seconds` | `region`, `outcome` | per-region mcp-authz health |\n| `snappcloud_bot_schedules`, `_schedule_owners`, `_schedule_limit` | — | stored recurring queries, distinct owners, configured ceiling |\n| `snappcloud_bot_schedule_runs_total` | `outcome` | scheduled runs: `ok` / `error` / `skipped` (owner lost access) |\n| `snappcloud_bot_schedule_run_duration_seconds`, `_schedule_runs_in_flight` | — | scheduled run latency and worker-pool saturation |\n| `snappcloud_bot_schedules_disabled_total` | — | schedules dropped after repeated failures |\n| `snappcloud_bot_alert_channels`, `_alerts_pending` | — | watched channels; alerts buffered in open windows |\n| `snappcloud_bot_alerts_received_total` | `disposition` | `queued` / `duplicate` / `cooldown` / `ignored` / `resolved` / `low_severity` — shows whether noise suppression is working |\n| `snappcloud_bot_alert_investigations_total` | `outcome` | `ok` / `error` / `empty` / `unauthorized` |\n| `snappcloud_bot_alert_investigation_duration_seconds` | — | investigation latency |\n| `snappcloud_bot_report_runs_total` | `outcome` | best-practice report builds: `ok` / `partial` (a cluster could not be read) / `error` |\n| `snappcloud_bot_report_findings` | `rule`, `severity` | findings in the latest report |\n| `snappcloud_bot_report_deliveries_total` | `outcome` | report emails: `sent` / `rejected` (every address refused) / `unrouted` (nobody to send to) / `send_error` |\n| `snappcloud_bot_report_collect_failures_total` | `cluster` | clusters a build could not read |\n| `snappcloud_bot_mcp_list_failures_total` | `server` | an MCP server that failed to list its tools — its tools are missing from every answer until it recovers |\n| `snappcloud_bot_tools_offered`, `_tools_per_cluster` | —, `cluster` | tools put in front of the model last turn, and what each cluster advertised |\n| `snappcloud_bot_tools_dropped_total` | `cluster` | tools omitted to fit `maxTools` — a cluster the question named appearing here is the case worth acting on |\n| `snappcloud_bot_turn_outcomes_total` | `outcome` | `answered` / `max-iters` — the second means the model ran out of tool calls mid-investigation |\n| `snappcloud_bot_answer_continuations_total` | — | answers that hit the model's output limit and had to be continued; climbing means raise `maxTokens` |\n| `snappcloud_bot_answer_nudges_total` | — | turns sent back because the model announced work it did not do |\n| `snappcloud_bot_conversation_trims_total` | — | turns where the oldest tool output was dropped to fit the model's window |\n| `snappcloud_bot_active_conversations`, `_messages_in_flight`, `_handler_panics_total` | — | live state |\n\nDashboard: `core/dashboards/Network/SnappCloudBot`. Alerts:\n`core/helm/apps/monitoring-int/templates/rules/snappcloud-bot.yaml`.\n\n## Configuration\n\nSee [`config.example.yaml`](config.example.yaml). Secrets are read from the\nenvironment (never YAML):\n\n| Env | Purpose |\n|-----|---------|\n| `MATTERMOST_TOKEN` | bot account token |\n| `LLM_API_KEY`      | `x-api-key` for the Anthropic-style endpoint |\n| `MCP_AUTHZ_TOKEN`  | bearer to every mcp-authz |\n| `MAIL_PASSWORD`    | SMTP password for `mail.username` (best-practice report) |\n| `<per-server>`     | Authorization header for an authed MCP server (e.g. `CILIUM_TEH1_AUTH`) |\n\nKey config sections: `agent.llm` (endpoint/model), `agent.clusters[].servers[]`\n(MCP servers per cluster), `agent.globalServers[]` (namespace-agnostic servers\nlike docs), `agent.toolGuidance` (tool-usage skills), `agent.toolRules`\n(per-tool namespace-arg overrides + `clusterAdminOnly` infra gating),\n`authz.regions[]` (mcp-authz endpoints). A cluster's `name` must match an\n`authz.regions[].name`.\n\nA question that **names its clusters** is given only those clusters' tools.\nEvery tool definition is re-sent each round, and an LLM endpoint that cannot\ncarry them all drops the tail silently, so set `agent.budgets.maxTools` to what\nyour endpoint carries. The bot trims to it, one tool per server per round with\nnamed clusters first, and tells the model what was trimmed. Watch\n`snappcloud_bot_tools_offered` and `snappcloud_bot_tools_dropped_total`.\n\nA cluster's `names` lists what else it is called (alert labels, datasource\nnames, shorthand); `agent.clusterNamePrefixes` (e.g. `[\"okd4-\"]`) gives every\ncluster its unprefixed name too. Matching is exact, then prefix, then substring,\nlongest name winning, ignoring case and punctuation.\n\nEnvironment knowledge belongs in config, not code: identity in `agent.persona`,\nplatform facts and tool usage in `agent.toolGuidance`. The built-in system\nprompt holds only the investigation method and output rules.\n\n**Review.** Scheduled reports and alert investigations are re-checked against\ntheir own tool results before posting — figures that disagree between summary\nand table, numbers no result contains, causes stated without evidence. It is one\nextra model call per unattended answer, never applied to chat, and the draft is\nkept whenever the review would drop content. `agent.reviewUnattended: false`\nturns it off; `agent.reviewPrompt` replaces the instruction. Watch\n`snappcloud_bot_answer_reviews_total{outcome}`: a rising share of `shrank` or\n`lost-structure` means the review is not helping.\n\n### Adding a new MCP server\n\nAppend one entry under the cluster — no code change:\n\n```yaml\nagent:\n  clusters:\n    - name: prod-1\n      servers:\n        - url: https://hubble-mcp.prod-1.example.com/mcp\n          authHeaderEnv: HUBBLE_TEH1_AUTH   # only if it needs auth (per region)\n```\n\nIf authed, add the key to the `snappcloud_bot.mcpAuth` secret (the full\n`Authorization` header). MCP Basic auth is **per region** — one key each.\nIts tools appear automatically, cluster-tagged and enforced. A namespace-agnostic\nserver (docs) goes under `agent.globalServers`; cluster-infrastructure tools it\nexposes should be listed in `agent.toolRules` with `clusterAdminOnly: true`.\n\nFor a trusted, identity-aware server that authorizes the caller itself (e.g.\nargocd-mcp), set `selfAuthorized: true` on the server entry. The caller's SSO\nidentity is then forwarded as `X-Remote-User` and all of that server's tools skip\nthe bot's namespace enforcement, returning results unfiltered:\n\n```yaml\nagent:\n  clusters:\n    - name: prod-1\n      servers:\n        - url: https://argocd-mcp.prod-1.example.com/mcp\n          selfAuthorized: true   # forwards X-Remote-User; server scopes the result\n```\n\nEnable it only for servers you trust to enforce the caller's access from the\nforwarded identity — it bypasses the bot's own filtering.\n\nTo share a server's results with every tenant of the cluster instead — metrics,\ntypically, so an investigation can correlate across namespaces — set\n`unscoped: true`. Nothing is scoped or filtered; `allowTools` still confines the\nserver to its read-only tools and `clusterAdminOnly` rules still hold:\n\n```yaml\n        - url: https://grafana-mcp.prod-1.example.com/mcp\n          unscoped: true         # every tenant sees every metric\n          allowTools: [query_prometheus, list_prometheus_metric_names]\n```\n\nThe bot logs a warning for each unscoped server at startup.\n\n### Diagnosing a thin answer\n\n`diagnostics` (cluster-admins, in chat) lists every MCP server the bot has for\nyour clusters, the number of tools each advertises, and the error for any that\nis not answering — a server that is down, one refusing the bot's credentials,\nand one that genuinely lacks a tool are indistinguishable from an answer and\nobvious here. It also reports the model, the per-question tool-call limit and\nthe per-request tool limit, which are the other two reasons an answer stops\nearly, plus the **running build** — the image tag is republished in place, so a\npod that has not been restarted since a fix answers exactly like one that has.\nAn answer that ran out of tool calls says so in its own last line.\n\n| what you see | where to look | usual cause |\n|---|---|---|\n| \"this cluster has no such tool\" | `diagnostics` | that server did not answer — token, route, or pod |\n| \"that is not documented / not supported\" | `tool_errors_total{reason=\"backend_error\"}` | a doc or search tool failed before it looked: a failed search is not an empty one, and the answer should say the tool is broken |\n| `diagnostics` shows 403 on servers that are up | the status cell names the cause | \"no credential sent\" is a key missing from the bot's Secret, not a sick server; \"credential refused\" is a token to rotate |\n| every metric cell empty | `check_datasources_health` | the metrics backend is refusing the credentials it was given |\n| right cluster, wrong numbers | `list_datasources` | several datasources exist and the default is not always the complete one |\n| a tool missing for one cluster | `tools_dropped_total` | the list was trimmed; `named=true` in the warning is the case worth acting on |\n| \"this cluster has no metrics tool\" while the log shows the server offering them | `tools offered … tools= ceiling=` | the endpoint carries 128 definitions and drops the rest of the array silently; the bot caps itself at that, so a count at the ceiling means something was trimmed |\n| a tool whose result is always cut | `results_truncated_total` | called too broadly: the model is spending a call on a fraction of an answer |\n| answer stops mid-sentence | `answer_continuations_total` | hit `maxTokens`; it is continued, but a climbing count means raise it |\n| answer describes work it did not do | `answer_nudges_total` | the model announced calls and made none |\n| \"no dashboard measures it\" with no search in the log | `tool call rejected before it was sent` | the model passed a uid it invented; it is refused before it is spent |\n| investigation stops early | `turn_outcomes_total{outcome=\"max-iters\"}` | ran out of tool calls |\n| HTTP 400 asking for a shorter prompt | `agent.budgets.conversationRunes` | the transcript exceeded the model's window; it fails identically on every retry |\n\nWorth alerting on: `rate(snappcloud_bot_mcp_list_failures_total[10m]) > 0`. A\nserver that stops answering is otherwise invisible until someone reads an answer\nand doubts it.\n\n## Develop\n\n```bash\nmake build   # binary -> bin/snappcloud-bot\nmake test\nmake run      # config.example.yaml\nmake docker   # multi-arch via build/package/docker-bake.json\n```\n\n## Deploy\n\nHelm chart: `core/helm/apps/snappcloud-bot`. Singleton (`replicas: 1`,\n`Recreate`), no inbound Service, no cluster RBAC. Ships Deployment, ConfigMap\n(the `config` values → `config.yaml`), ServiceAccount, a Secret (all keys exposed\nas env via `envFrom` — including per-region `mcpAuth` entries), and a **PVC** for\nconversation memory. Secrets are grouped under the `snappcloud_bot` sops key; the\nshared `mcp_authz.authToken` sops key is read by both this chart and mcp-authz so\nthe bearer can never drift.\n\n## License\n\nApache 2.0 — see [LICENSE](LICENSE).\n"}