k8s-rca-mcp-server

k8s-rca-mcp-server

An MCP server for Kubernetes root-cause analysis. It exposes compact, model-friendly tools for resource topology, bounded Service health checks, metric time series, and ranked log retrieval.

Alpha software. The read-only tools and 30-scenario evaluation suite run against a disposable local kind cluster; the APIs may still change before 1.0.

Why a separate MCP server?

The boundary decouples tools from orchestration and model providers:

• Model-independent integration. MCP is the protocol between a host and the tool service. The model sees the rendered tool schemas, not the transport.
• Reusable across hosts. The same server powers k8s-rca-agent and can be attached to other MCP hosts for interactive debugging.
• Independent tool iteration. Descriptions, schemas, and response shape have a large effect on agent behavior and can be tested without changing the graph.

Architecture

flowchart LR
    Host[MCP host / RCA agent] -->|stdio or HTTP| Server[k8s-rca-mcp-server]
    Loopback[Host loopback NodePorts] -->|9090| Prom
    Loopback -->|9200| ES
    Server --> Topology[Topology tool]
    Server --> Health[Service health probe]
    Server --> Metrics[Metrics tool]
    Server --> Logs[Ranked log tool]
    Server --> Knowledge[Historical knowledge tool]
    Topology --> Kube[Kubernetes API]
    Health --> Kube
    Metrics --> Prom[(Prometheus)]
    Logs --> ES[(Elasticsearch incident indices)]
    Knowledge --> ES2[(Elasticsearch knowledge index)]
    Pods[Container stdout] --> FB[Fluent Bit] --> ES
    Manifests[30 baseline/fault/recovery cases] --> Lifecycle[Lifecycle CLIs]
    Lifecycle --> Kube

The server is read-only. Scenario lifecycle CLIs are separate test utilities with an exact kind-context and namespace contract; they are never exposed as model tools.

Tools

Tool	Purpose	Status
get_resource_topology	Native ownership, mounted dependencies, peer replicas, Services, and Endpoints	Implemented
probe_service_health	Read-only API Server Service proxy check without arbitrary Pod execution or response bodies	Implemented
get_container_metrics	Prometheus container memory/CPU and joint JVM direct-buffer used/capacity/count trends	Implemented
search_relevant_logs	Elasticsearch BM25 retrieval, template grouping, and explainable reranking	Implemented
search_historical_knowledge	Versioned retrieval from a separately labelled local knowledge corpus	Implemented
ping	Connectivity and server-version check	Implemented

The two-stage log pipeline addresses log volume that exceeds an LLM context window. A stateless model request resends prior tool results on later rounds, so oversized output is repeatedly amplified. Logs must therefore be reduced at the tool boundary, before they enter conversation state.

Structure

src/k8s_rca_mcp_server/
├── server.py       MCP entry point and tool registration
├── tools/          Model-facing descriptions, schemas, and compact summaries
├── clients/        Kubernetes, Prometheus, and Elasticsearch data access
└── resources/      Packaged scenarios, images, and knowledge evaluation data
deploy/             Local cluster and observability-stack configuration
tests/              Unit, contract, and installed-wheel smoke tests

Clients retrieve and normalize data. Tools decide what a model should see. This separation keeps raw infrastructure payloads out of the prompt and lets tests mock data sources while validating model-facing output.

Quick start

Requirements: Python 3.12+, uv, and a reachable Kubernetes API. Prometheus and Elasticsearch are optional unless their corresponding tools are called.

uv sync

# stdio transport
uv run python -m k8s_rca_mcp_server.server

# Streamable HTTP transport
uv run python -m k8s_rca_mcp_server.server --http

For the complete local stack, use the sibling Agent repository's canonical lifecycle. It creates only the fixed kind-rca-chaos cluster and maps the three data services to host loopback:

cd /path/to/k8s-rca-agent
export RCA_MCP_SERVER_DIR=/absolute/path/to/k8s-rca-mcp-server
uv run k8s-rca-demo bootstrap
uv run k8s-rca-doctor --target demo

Dependency	Kubernetes endpoint	Host endpoint
Redis	NodePort 30379	127.0.0.1:6379
Prometheus	NodePort 30090	127.0.0.1:9090
Elasticsearch	NodePort 30200	127.0.0.1:9200

The loopback binding is part of the safety contract, not a production exposure pattern. See troubleshooting before changing ports or reusing an older kind cluster.

The verified local lifecycle is: bootstrap, seven-probe Doctor READY, one isolated fault, investigation, mechanism-specific recovery, scenario cleanup, then k8s-rca-demo cleanup. In the network-partition path, connectivity returned without replacing either Pod; both UIDs remained stable and restart counts stayed at zero.

To attach the stdio server to an MCP host during development:

claude mcp add k8s-rca -- uv --directory /path/to/k8s-rca-mcp-server \
  run python -m k8s_rca_mcp_server.server

The server reads the current Kubernetes client configuration. Use a dedicated, least-privilege service account or kubeconfig; do not give an RCA process write access merely to run these tools.

probe_service_health needs read access to the namespaced services/proxy subresource. It performs one fixed HTTP GET through the API Server, discards the response body, and exposes no command or container argument. Success proves that the backend responds from outside an affected client path; it does not identify a CNI or kube-proxy defect by itself. Combine it with client logs, stable Pod UIDs, ready Endpoints, and resource metrics.

For network scenarios, the synthetic checkout client logs every health-state transition but rate-limits unchanged heartbeat lines. This preserves the decisive healthy-to-failing transition in bounded Pod-log tails during a longer RCA run.

The jvm_direct_memory preset combines BufferPoolMXBean used bytes, total capacity, and reported buffer count. Joint growth is stronger retention evidence than used bytes alone; it still cannot identify the allocation code path, so the tool keeps that link explicitly unknown and treats a larger limit as containment.

Reproducible scenarios

src/k8s_rca_mcp_server/resources/scenarios/ contains 30 standard RCA cases plus a separate Pod-kill smoke test. Standard cases have a healthy baseline, a deterministic fault, recovery, idempotent cleanup, and unit-tested manifest contracts. Lifecycle CLIs refuse protected namespaces and require the expected kind context.

For example:

uv run k8s-rca-controller-scenario \
  missing-service-account-reference smoke

uv run k8s-rca-state-scenario \
  configmap-script-not-executable smoke

The state runner observes Pods that are Pending, not Ready, or restarting. The controller runner covers admission failures where the ReplicaSet reports FailedCreate and no Pod exists. Service, network, and JVM cases use their specialized runners so every lifecycle proves mechanism-specific evidence rather than treating all outages as a generic rollout failure.

Run scenarios only on the disposable kind-rca-chaos cluster and only one at a time. See deploy/chaos/README.md and each scenario's README for setup and expected evidence.

Test

uv run python -m unittest discover -s tests -v
uv build --no-sources

See CONTRIBUTING.md for change and scenario requirements and SECURITY.md for private vulnerability reporting. Participation is governed by the Code of Conduct.

Technology

Python 3.12, uv, MCP SDK 2.x (MCPServer), Kubernetes Python client, Prometheus HTTP API, Elasticsearch, Fluent Bit, and httpx.

License

Apache License 2.0. See LICENSE.