{"id":"111ee640-611a-4025-8600-a2cbc4da1612","slug":"gke-observability","kind":"skill","name":"gke-observability","summary":"Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collecti","owner":"google","repoUrl":"https://github.com/google/skills/tree/HEAD/skills/cloud/gke-observability","license":"Apache-2.0","licenseTier":"permissive","trust":"official","stars":21260,"match":null,"fewStars":false,"format":"skill","createdAt":null,"addedAt":"2026-10-14T05:35:44.128Z","logoUrl":"https://avatars.githubusercontent.com/u/1342004?v=4","linkOnly":false,"collection":{"name":"skills","repo":"google/skills"},"skillPath":"skills/cloud/gke-observability","categories":["cloud-infra","data-analytics","browser-web"],"tags":["google","googlecloud"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-14T05:35:44.126Z","sample":false,"url":"https://conduits.dev/tools/gke-observability","language":"Python","transports":[],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"> Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection, and to troubleshoot Managed Service for Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets, PodMonitoring misconfiguration, rule/alert evaluation failures, and monitoring permission errors. Don't use to configure local application logging frameworks or external APMs outside GKE.\n\n# GKE Observability\n\nThis reference covers monitoring, logging, and metrics configuration for GKE.\nThe golden path enables comprehensive observability including control-plane\nmetrics.\n\n> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`,\n> `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud\n> container clusters update --monitoring=...`, `gcloud logging read`\n\n## Golden Path Observability Defaults\n\nSetting | Golden Path Value | Notes\n--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----\n`loggingConfig` components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging\n`monitoringConfig` components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane\n`managedPrometheusConfig.enabled` | `true` | Google-managed Prometheus\n`advancedDatapathObservabilityConfig.enableMetrics` | `true` | Dataplane V2 flow metrics\n`loggingService` | `logging.googleapis.com/kubernetes` | Cloud Logging\n`monitoringService` | `monitoring.googleapis.com/kubernetes` | Cloud Monitoring\n\n### Control-Plane Metrics (Golden Path Addition)\n\nThe golden path adds three control-plane monitoring components not present in\ndefault clusters:\n\n| Component | What It Monitors |\n| -------------------- | ---------------------------------------------------------------------- |\n| `APISERVER` | API server request latency, error rates, admission webhook performance |\n| `SCHEDULER` | Scheduling latency, pending pods, scheduling failures |\n| `CONTROLLER_MANAGER` | Controller work queue depth, reconciliation latency |\n\nThese are critical for diagnosing cluster-level issues (slow API responses,\nscheduling delays, stuck controllers).\n\n## Enabling Full Monitoring\n\n**Say this whenever you hand over a `--monitoring` command:**\n\n1. **Control-plane metrics are NOT enabled by default.** State this outright in\n your answer — do not leave it implied by the fact that you are supplying an\n enable command. `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off\n on every new cluster and collect nothing until explicitly turned on, and the\n same is true of `DCGM`, `CADVISOR`, `KUBELET`, and kube-state (`POD`,\n `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`, `JOBSET`).\n `SYSTEM` is the only package on by default. A user asking \"why are there no\n API server metrics\" has almost always simply never enabled them.\n2. **The flag replaces, it does not append.** The set supplied to `--monitoring`\n overrides the previous setting entirely, so omitting a component silently\n turns it off. Always pass the full desired list, and always include `SYSTEM`\n — it cannot be disabled while monitoring is on, and never on Autopilot.\n3. **These metrics bill per sample ingested** via Managed Service for\n Prometheus. Enabling the full suite on a large cluster is a real cost\n increase; mention it rather than presenting the list as free.\n\n> **The gcloud flag and the API field use different spellings for the same\n> components.** Do not copy names between them:\n>\n> Component | `gcloud --monitoring=` | `monitoringConfig` API enum\n> ---------------- | ---------------------- | ---------------------------\n> System | `SYSTEM` | `SYSTEM_COMPONENTS`\n> API server | `API_SERVER` | `APISERVER`\n> Controller mgr | `CONTROLLER_MANAGER` | `CONTROLLER_MANAGER`\n>\n> The remaining components share a spelling. Using an API enum in the CLI flag\n> (or the reverse) fails the command — this is a common and confusing error.\n\n```bash\n# Enable golden path monitoring suite\ngcloud container clusters update --region \\\n --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \\\n --quiet\n\n# Enable Managed Prometheus\ngcloud container clusters update --region \\\n --enable-managed-prometheus \\\n --quiet\n\n# Enable Dataplane V2 observability metrics\ngcloud container clusters update --region \\\n --enable-dataplane-v2-flow-observability \\\n --quiet\n```\n\n## Managed Prometheus\n\nGolden path enables Google Managed Prometheus for metrics collection and\nquerying.\n\n**Querying metrics:**\n\n- Use Cloud Monitoring Metrics Explorer in the console\n- Use PromQL via the Prometheus UI or API\n- Grafana dashboards via Managed Grafana\n\n**Key GKE metrics:**\n\n| Metric | Source | Use |\n| -------------------------------------------------- | ------------------ | ---------------------- |\n| `container_cpu_usage_seconds_total` | cAdvisor | Pod CPU usage |\n| `container_memory_working_set_bytes` | cAdvisor | Pod memory usage |\n| `kube_pod_status_phase` | kube-state-metrics | Pod lifecycle |\n| `apiserver_request_duration_seconds` | API Server | Control plane latency |\n| `scheduler_scheduling_attempt_duration_seconds` | Scheduler | Scheduling performance |\n| `kubernetes.io/node/cpu/core_usage_time` | Cloud Monitoring | Node CPU |\n| `DCGM_FI_DEV_GPU_UTIL` | DCGM | GPU utilization |\n\n## Live Resource Usage (kubectl-only)\n\nNo MCP or gcloud equivalent exists for live resource usage. Use `kubectl top`:\n\n```bash\nkubectl top pods --all-namespaces --sort-by=cpu\nkubectl top nodes\nkubectl top pods --containers -n # per-container breakdown\n```\n\n## Cloud Logging (gcloud-only)\n\n**Querying cluster logs** (no MCP equivalent — use `gcloud logging read`):\n\n```bash\n# System component logs\ngcloud logging read \\\n 'resource.type=\"k8s_cluster\" AND resource.labels.cluster_name=\"\"' \\\n --project --limit 50 \\\n --quiet\n\n# Workload logs for a specific namespace\ngcloud logging read \\\n 'resource.type=\"k8s_container\" AND resource.labels.cluster_name=\"\" AND resource.labels.namespace_name=\"\"' \\\n --project --limit 50 \\\n --quiet\n\n# Audit logs (who did what)\ngcloud logging read \\\n 'resource.type=\"k8s_cluster\" AND logName:\"cloudaudit.googleapis.com\"' \\\n --project --limit 50 \\\n --quiet\n```\n\n## Diagnostic Settings\n\nFor security monitoring and troubleshooting, enable control-plane audit logs:\n\n```bash\n# View current logging config\ngcloud container clusters describe --region \\\n --format=\"yaml(loggingConfig)\" \\\n --quiet\n```\n\n## Alerting\n\nSet up alerts for critical conditions:\n\nCondition | Metric | Threshold\n----------------------- | --------------------------------------------------- | ---------\nHigh API server latency | `apiserver_request_duration_seconds` | P99 > 5s\nPod crash loops | `kube_pod_container_status_restarts_total` | > 5 in 10min\nNode not ready | `kube_node_status_condition` | condition=Ready, status!=True\nHigh GPU utilization | `DCGM_FI_DEV_GPU_UTIL` | > 95% sustained\nPVC near capacity | `kubelet_volume_stats_used_bytes / capacity` | > 85%\nScheduling failures | `scheduler_schedule_attempts_total{result=\"error\"}` | > 0\n\n> **Prerequisite:** The `kube_*` series above (e.g., `kube_pod_status_phase`,\n> `kube_pod_container_status_restarts_total`, `kube_node_status_condition`)\n> come from **kube-state-metrics**, which GKE does not collect by default.\n> Deploy the Managed Prometheus kube-state-metrics package first.\n\n### Proposing Dashboards & Alerts (Production Rules)\n\nWhen designing or proposing alerting and dashboard strategies for GKE:\n\n1. **Always explicitly name Google Cloud Monitoring** as the platform to\n implement these alerts and dashboards.\n2. **Always include API server latency** (via\n `apiserver_request_duration_seconds` metric) on the dashboard as a critical\n indicator of control plane health, alongside node CPU/Memory and pod crash\n loops.\n\n### Node Health (Production Rules)\n\nA comprehensive assessment of node health relies on analyzing these two metrics together:\n\n1. **`kubernetes.io/node/status_condition`** (filtered by `status_condition=\"Ready\"`): Use this to track healthy nodes. Note that it will only report values for nodes that have successfully bootstrapped.\n2. **`compute.googleapis.com/instance_group/size`** (filtered by `instance_group_name=\"gke--.*\"`): Use this to track the total number of nodes in a specific cluster. Note that it does not differentiate between healthy and unhealthy nodes.\n\n## Cost Considerations\n\nMonitoring and logging have associated costs:\n\n- **Cloud Logging**: Charged per GiB ingested beyond free tier (50\n GiB/project/month)\n- **Cloud Monitoring**: Free for GKE system metrics; custom metrics charged\n per time series\n- **Managed Prometheus**: Charged per samples ingested\n\nTo reduce costs in non-production:\n\n```bash\n# Reduce to system-only monitoring\ngcloud container clusters update --region \\\n --monitoring=SYSTEM \\\n --quiet\n```\n\n## Distributed Tracing & Continuous Profiling (Recommended)\n\n**Not golden path defaults** — recommended for production microservice\narchitectures and performance-sensitive workloads.\n\n- **Cloud Trace**: Add OpenTelemetry SDK to your app with the\n `opentelemetry-operations-go` (or equivalent) exporter. Traces appear in\n Cloud Trace console. Identifies cross-service latency bottlenecks.\n- **Cloud Profiler**: Add the Cloud Profiler agent to your app. Profiles CPU\n and memory usage in production with low overhead. Identifies hotspots and\n compares across versions.\n\n**Recent additions:**\n\n- **Managed OpenTelemetry for GKE (Preview)**: Managed in-cluster OTLP\n endpoint plus auto-instrumentation for traces, metrics, and logs. Requires\n GKE 1.34.1-gke.2178000+; enable with `gcloud beta container clusters\n update ... --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS`.\n- **PSI (Pressure Stall Information) metrics**: cAdvisor\n `container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total` series\n (beta in Kubernetes 1.34) can be collected via a Managed Prometheus\n `ClusterNodeMonitoring` resource; GKE's documented collection path requires\n GKE 1.35+.\n\n## LQL Query Examples\n\nCommon Logging Query Language patterns for GKE troubleshooting:\n\n```\n# Error logs for a specific container\nresource.type=\"k8s_container\" AND resource.labels.container_name=\"my-app\" AND severity>=ERROR\n\n# OOMKilled events\nresource.type=\"k8s_event\" AND jsonPayload.reason=\"OOMKilling\"\n\n# Pod scheduling failures\nresource.type=\"k8s_event\" AND jsonPayload.reason=\"FailedScheduling\"\n\n# Audit logs (who did what)\nresource.type=\"k8s_cluster\" AND logName:\"cloudaudit.googleapis.com\"\n```\n\n## Troubleshooting Managed Prometheus (GMP)\n\nDiagnose GMP ingestion, rule, and query problems. Stay read-only (`kubectl get`\n/ `describe` / `logs`) and propose config changes; do not mutate live resources\ndirectly.\n\n### First: split ingestion-side vs query-side\n\nBefore anything else, query the `up` metric in the **Metrics Explorer PromQL\ntab** in Cloud Monitoring. If `up` returns data, ingestion works and the problem\nis query-side (Grafana / PromQL / permissions). If `up` is empty, the problem is\ningestion-side (collectors, scrape config, or write permission).\n\n### Ingestion-side\n\n1. **Check GMP system pods.** They run in `gmp-system` on Standard clusters and\n `gke-gmp-system` on Autopilot. Look for `gmp-operator`, `collector`\n (DaemonSet), and `rule-evaluator` not `Running` or with high restarts:\n\n ```bash\n kubectl get pods -n gmp-system # gke-gmp-system on Autopilot\n kubectl logs -n gmp-system -l app.kubernetes.io/name=collector -c prometheus\n ```\n\n A collector in `CrashLoopBackOff` with `OOMKilled` usually means high metric\n cardinality - drop unneeded series/labels (see cost section below) or apply a\n VPA to the collector.\n\n2. **Check PodMonitoring / ClusterPodMonitoring.** The three classic mistakes:\n - `spec.selector.matchLabels` does not match the target Pod labels.\n - A `PodMonitoring` only discovers targets **in its own namespace** - use\n `ClusterPodMonitoring` for cluster-wide scope.\n - `spec.endpoints.port` must reference the **named** container port (e.g.\n `port: web`), not the port number.\n\n3. **Enable target status for scrape errors.** Propose patching\n `OperatorConfig` in `gmp-public` with `features.targetStatus.enabled: true`;\n once applied, `kubectl describe podmonitoring ` and read `Active Targets`,\n `Unhealthy Targets`, and `Last Error` (for example `connection refused`, HTTP 404,\n `context deadline exceeded`). Disable it again when done - it can OOM the\n operator on large clusters.\n\n### Permissions (403 / no data written)\n\nGMP components inherit the **node service account**. Ingestion needs\n`roles/monitoring.metricWriter` (error `Permission monitoring.timeSeries.create\ndenied` in collector logs); the `rule-evaluator` and query paths need\n`roles/monitoring.viewer` (403 / `PermissionDenied`). If a query app (like\nGrafana) uses Workload Identity, the bound Google service account also needs\n`roles/monitoring.viewer`.\n\n### Rule and alert evaluation\n\nRule scope is decided by the resource kind: `Rules` (single namespace),\n`ClusterRules` (whole cluster), and `GlobalRules` (all data in the metrics\nscope). You **must** use `GlobalRules` to write rules against Cloud Monitoring\nmetrics - a `Rules`/`ClusterRules` resource silently returns no data for them.\nCheck `rule-evaluator` logs (`-c evaluator`) for parse/permission errors.\n\n### Query-side (Grafana / PromQL)\n\n- **Data source** must point at the GMP frontend query proxy, not\n `localhost:9090`, and the HTTP **Method must be GET** - `POST` fails with\n `no match[] parameter provided`.\n- **Grafana template variables:** use the two-argument form\n `label_values(,