cloud-trace-querying

Query Cloud Trace spans, filter by latency thresholds or error status, correlate distributed traces with Cloud Logging, and diagnose latency bottlenecks across Google Cloud services. Use when investigating slow requests, analyzing trace hierarchies, or resolving latency regressions. Do NOT use for querying non-GCP telemetry or database query optimization outside Cloud Trace.

Cloud Trace Querying Skill (cloud-trace-querying)

This skill equips the agent to interact with Cloud Trace by providing standard query patterns, diagnostic workflows for latency and errors, and tools to fetch and inspect trace hierarchies.

[!IMPORTANT] AUTONOMOUS TOOL EXECUTION MANDATE: You MUST execute the tool scripts directly in bash to query telemetry. Do NOT just describe query strings or write Python scripts. Run the provided bash tools directly and report the exact results.

Prerequisites & Setup

1. Google Cloud SDK: Verify gcloud is installed (Installation Guide).
2. Authentication: Authenticate CLI and application default credentials:

   gcloud auth login && gcloud auth application-default login
   

3. Project & Billing: Identify the target <PROJECT_ID> (from the user's prompt or via bash scripts/get_trace_project/run.sh), ensure an active billing account is attached, and configure the project if not already set:

   gcloud config set project <PROJECT_ID>
   

4. API Enablement: Verify Cloud Trace and Cloud Logging APIs are enabled:

   gcloud services enable cloudtrace.googleapis.com logging.googleapis.com
   

Quickstart Workflows

1. Identify Google Cloud Project ID

If the user specified a Google Cloud project ID in their prompt, use it.

If they did not, ALWAYS determine the Google Cloud project ID by running:

bash scripts/get_trace_project/run.sh

2. Search Traces (Latency & Errors)

To find traces matching specific criteria, such as a root span of /v1/booking/book with a duration of at least 1.5s:

bash scripts/search_traces/run.sh --filter 'root:/v1/booking/book latency:1.5s' --projects '<PROJECT_ID>'

Note: --filter is required. To list traces without specific filtering, pass --filter '' or --filter 'root:'. Filter syntax: The Cloud Trace API uses latency:<duration> (such as latency:1.5s for duration >= 1.5s), which corresponds to "Span duration" in Trace Explorer. The script outputs JSON where each cluster's examples array contains sample trace objects with trace_id, duration_seconds, span count, and error status.

3. Fetch Entire Trace Hierarchy

To view all spans and status codes/labels within a trace:

bash scripts/fetch_entire_trace/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>'

4. Fetch Correlated Application Logs

To retrieve application log entries correlated with a trace ID:

bash scripts/fetch_related_logs/run.sh --trace-id '<TRACE_ID>' --trace-projects '<PROJECT_ID>'

Application log entries are useful when inspecting stack traces, exception messages, and log events emitted during span execution. When diagnosing errors or service crashes, inspect the returned log entries for severity: "ERROR" to confirm root causes or uncover unhandled exceptions. Note: fetch_related_logs queries the default log scope (_Default), searching across configured log views in the target project. An empty response can occur if: (1) no log entries match the criteria, (2) the caller lacks IAM permissions to view the matching entries, or (3) matching entries reside in log views outside the searched log scope.

5. Fetch Specific Trace Span

To inspect attributes and metadata of a single span within a trace:

bash scripts/fetch_trace_span/run.sh --trace-id '<TRACE_ID>' --span-id '<SPAN_ID>' --project '<PROJECT_ID>'

6. Generate Cloud Console Deep Links

To generate direct Cloud Console URLs for browser inspection of traces or correlated logs:

bash scripts/generate_links/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>' [--span-id '<SPAN_ID>']

7. Format Output

• Terminal / CLI: Render span trees as indented ASCII timelines using |--.
• UI Presentation: Render as A2UI JSON graph structures when requested for UI.

8. Upstream Service Error Handling

If the Cloud Trace API or Cloud Logging API returns an upstream HTTP service error (such as 500 Internal Server Error, 503 Service Unavailable, or GoogleAPICallError), report the upstream service outage directly to the user. Do not attempt to inspect virtual environments, modify internal scripts, or debug local libraries.

Reference Documentation

• Master Index: Reference Documentation Index

Operational Workflows

• Troubleshoot Latency Workflow
• Correlated Logs Workflow
• Trace Querying Decision Tree
• Console Deep Links
• Workflows Overview

Query Syntax & Tools

• Filter Syntax (root:, latency:, +span:, error:true)
• API Limits and Constraints
• Observability Analytics (BigQuery SQL)
• Cloud Trace MCP Tools

Semantic Conventions

• Conventions Overview
• OpenTelemetry Core
• OpenTelemetry HTTP
• OpenTelemetry gRPC & RPC
• OpenTelemetry GenAI
• GCP App Hub & Service Conventions
• GCP Legacy Labels

Concepts & Architecture

• Observability Scopes
• Distributed Traces Concept
• Spans Concept
• Logs-to-Trace Correlation
• Knowledge Overview

Templates & Assets

• ASCII Formatting Templates
• A2UI Formatting Templates
• Query Scenarios

Official Documentation Links

• Cloud Trace Documentation
• Finding and Viewing Traces
• OpenTelemetry Semantic Conventions