{"id":"69402ef4-6420-4152-93d3-eda01c25127e","slug":"vss-summarize-video","kind":"skill","name":"vss-summarize-video","summary":"Use to summarize a recorded video via the LVS summarization microservice (HITL-gated) with a VLM fallback. Not for report generation or live RTSP captioning.","owner":"NVIDIA","repoUrl":"https://github.com/NVIDIA/skills/tree/HEAD/skills/vss-summarize-video","license":"Apache-2.0","licenseTier":"permissive","trust":"official","stars":3586,"match":null,"fewStars":false,"format":"skill","createdAt":null,"addedAt":"2026-10-14T05:52:40.001Z","logoUrl":"https://avatars.githubusercontent.com/u/1728152?v=4","linkOnly":false,"collection":{"name":"skills","repo":"nvidia/skills"},"skillPath":"skills/vss-summarize-video","categories":["design-media","writing","developer-tools"],"tags":["agentic-ai","ai-agents","claude-code","codex","coding-agents","cuda","developer-tools","llm"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-14T05:52:39.998Z","sample":false,"url":"https://conduits.dev/tools/vss-summarize-video","language":"Python","transports":[],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"## Instructions\n\nFollow the routing tables and step-by-step workflows below. Each section that ends in *workflow*, *quick start*, or *flow* is intended to be executed top-to-bottom. Detailed reference material lives in `references/`.\n\n## Examples\n\nWorked end-to-end examples are kept under `evals/` (each `*.json` manifest contains a runnable scenario) and inline in the per-workflow `curl` blocks below. Run a Tier-3 evaluation with `nv-base validate <this-skill-dir> --agent-eval` to replay them.\n\nCall the VLM NIM or the video summarization microservice **directly**.\nAlways run `curl` commands yourself; never instruct the user to run them.\n\nPrimary video workflow query type: **\"Summarize this video.\"** Direct video summarization API\nand service-ops requests are handled by the reference-routed sections below.\n\n## Purpose\n\nProduce a single, polished narrative summary of one recorded video clip, with\ntimestamped events when the LVS microservice path is reachable.\n\n**Do NOT use this skill for:**\n- Live RTSP captioning — use `vss-deploy-dense-captioning`.\n- Report generation, including incident or alert-window reports — use `vss-generate-video-report` Mode B.\n- Semantic search across the archive — use `vss-search-archive`.\n\n## Prerequisites\n\n- VSS `lvs` profile running on `$HOST_IP` (port 38111) OR a reachable\n  VLM/RT-VLM endpoint as a fallback. The `vss-deploy-profile` skill brings\n  these up.\n- Network reachability from the agent host to both endpoints; clip URLs from\n  VIOS must be fetchable by the chosen backend.\n- `jq` and `curl` available on the agent host.\n\n## Limitations\n\n- Direct VLM fallback uses a single fixed prompt and cannot target\n  scenario/events — output quality is lower than the LVS path.\n- Remote VLM endpoints generally cannot reach `localhost`/private clip URLs.\n- One backend call per request; no parallel hedging or multi-pass summaries.\n\n## Troubleshooting\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `/v1/ready` returns 503 repeatedly | LVS service still warming up | Retry up to ~30 s as shown in *Setup*; if it never returns 200 the service may not be deployed |\n| Empty `video_summary` and `events` | Clip does not contain the requested events | Re-run with broader `scenario` or different `events` |\n| VLM returns `<think>` block | Cosmos reasoning mode | Strip everything up to `</think>` before rendering |\n| Empty stdout from `curl /v1/ready` | Service legitimately returns 200 with empty body | Always check HTTP status with `-o /dev/null -w '%{http_code}'`, never inspect the body |\n\nSee [`references/video-summarization-debugging.md`](references/video-summarization-debugging.md) for deeper diagnostics.\n\n## Reference Map\n\nUse these references only when the user asks for the relevant detail, or when\nthe core workflow below needs deeper video summarization information:\n\n- **video summarization API details**: [`references/video-summarization-api.md`](references/video-summarization-api.md) for\n  `/v1/summarize`, `/summarize`, `/v1/generate_captions`,\n  `/v1/stream_summarize`, health probes, `/models`, `/recommended_config`,\n  `/metrics`, request fields, response shapes, and API gotchas.\n- **video summarization service configuration and ops**:\n  [`references/video-summarization-deployment.md`](references/video-summarization-deployment.md) for\n  the VSS `lvs` profile, ports, required env vars, logs, status, dry-runs,\n  teardown, model/backend swaps, Elasticsearch/Neo4j/ArangoDB backend\n  selection, and service-level troubleshooting.\n- **Extended video summarization ops references**:\n  [`references/video-summarization-environment-variables.md`](references/video-summarization-environment-variables.md),\n  [`references/video-summarization-debugging.md`](references/video-summarization-debugging.md), and\n  `assets/video-summarization.env.example`.\n\nLoad `video-summarization-api.md` only when you need a request field, response shape, or\nendpoint that is not already covered by the Step 2 LVS or fallback VLM\nexample below, or when handling a direct video summarization API\nrequest. Load `video-summarization-deployment.md` only for deployment,\nconfiguration, or service operations.\n\n## Video Summarization API And Service Ops Requests\n\nIf the user asks to call or debug video summarization endpoints directly, answer from\n[`references/video-summarization-api.md`](references/video-summarization-api.md) instead of running the\nend-to-end video summarization workflow. Examples: list video summarization models, check\nreadiness, get recommended chunking config, inspect metrics, explain a 422\nresponse, or build a `/v1/summarize` request body.\n\nIf the user asks to configure, deploy, restart, tear down, or troubleshoot the\nvideo summarization service, prefer the `vss-deploy-profile` skill for full VSS profile\ndeployment and use [`references/video-summarization-deployment.md`](references/video-summarization-deployment.md)\nfor video summarization-specific service details.\n\n## Routing\n\nDecide purely from video summarization service availability (probed in\n*Setup → Availability checks* below). **Duration does not drive routing.**\n\n| `/v1/ready` | Backend | Endpoint |\n|---|---|---|\n| HTTP 200 | LVS microservice with HITL | `POST ${LVS_BACKEND_URL}/v1/summarize` |\n| Anything else | VLM / RT-VLM with the default prompt + fallback note | `POST ${VLM_BASE_URL}/v1/chat/completions` |\n\nFallback message when the LVS service is unreachable — copy verbatim above the summary:\n\n> ⚠ **Note:** Input video `<name>` is `<N>`s long.\n> The video summarization service is not deployed, so this summary was\n> produced by the VLM alone with a generic default prompt. Deploy the\n> `lvs` profile for higher-quality summaries with scenario/events\n> targeting.\n\n## Deployment prerequisite\n\nThe VSS **lvs** profile on `$HOST_IP` is the primary backend. If the\n`/v1/ready` probe (see *Setup → Availability checks*) returns anything\nother than 200 after the warmup retries, ask the user:\n\n> *\"The VSS `lvs` profile isn't running on `$HOST_IP`. Shall I deploy it now using the `/vss-deploy-profile` skill with `-p lvs`? Reply `no` to summarize with the VLM-only fallback instead (lower quality, no scenario/events targeting).\"*\n\n- **Yes** → hand off to `/vss-deploy-profile`, then re-probe and continue with Step 2 (LVS + HITL).\n- **No** → go straight to **Step 2 fallback (VLM with default prompt)** and prepend the Routing fallback note. Do not ask again, and do not run scenario/events HITL.\n- **Pre-authorized to deploy autonomously** (caller said so explicitly) → skip the confirmation and invoke `/vss-deploy-profile` directly.\n- **Pre-authorized to use VLM fallback** (\"skip lvs, just use the VLM\") → go straight to Step 2 fallback without prompting.\n\n---\n\n## Setup\n\n**Endpoints (defaults for a local VSS `lvs` deployment):**\n\n- VLM / RT-VLM: `${VLM_BASE_URL}` — default `${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}`\n- LVS service: `${LVS_BACKEND_URL}` — default `http://${HOST_IP:-localhost}:38111`\n- VIOS: owned by `vss-manage-video-io-storage`; refer there.\n\nUse env vars when set (strip trailing `/v1` from the VLM base — the skill appends it). Otherwise use the defaults. If neither works, ask the user — do not scan ports or read config files to guess.\n\n**Model name:** read `${VLM_NAME}` (default\n`nim_nvidia_cosmos3-nano-reasoner_bf16-final`). It must match the id RT-VLM\n`/v1/models` advertises; do not substitute the friendly\n`nvidia/cosmos3-nano-reasoner`.\n\nFor endpoint schemas, optional fields, response envelopes, and error handling, see [`references/video-summarization-api.md`](references/video-summarization-api.md).\n\n**Availability checks** (run both before routing).\n**Readiness is determined by the HTTP status code only** — the LVS\n`/v1/ready` may legitimately return `200` with an empty body, so do not\ninspect the body.\n\n```bash\nVLM=\"${VLM_BASE_URL:-${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}}\"\nVLM=\"${VLM%/v1}\"\n\n# VLM / RT-VLM: 200 on /v1/models\nvlm_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 10 \\\n  \"$VLM/v1/models\")\n[ \"$vlm_code\" = \"200\" ] && echo \"VLM OK\" || echo \"VLM not reachable (HTTP $vlm_code)\"\n\n# Video summarization service: 200 on /v1/ready, with retry on 503 (warmup) for up to ~30s\nVIDEO_SUMMARIZATION_URL=${LVS_BACKEND_URL:-http://${HOST_IP:-localhost}:38111}\nvideo_sum_code=000\nfor i in $(seq 1 10); do\n  video_sum_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 10 \"$VIDEO_SUMMARIZATION_URL/v1/ready\")\n  case \"$video_sum_code\" in\n    200) echo \"video summarization OK\"; break ;;\n    503) sleep 3 ;;                 # warming up; keep polling\n    *)   break ;;                   # any other code = not reachable, stop retrying\n  esac\ndone\n[ \"$video_sum_code\" = \"200\" ] || echo \"video summarization service not reachable (HTTP $video_sum_code)\"\n```\n\n**How to interpret the results:**\n\n- `video_sum_code = 200` → **Step 2 (LVS + HITL)** for every video.\n- `video_sum_code != 200`, `vlm_code = 200` → **Step 2 fallback (VLM)**; prepend the Routing fallback note.\n- `vlm_code != 200` → fail; at least one backend must be reachable.\n- A non-200 LVS code after the retry loop is the ONLY signal of unavailability. Empty stdout or missing JSON fields are NOT \"unavailable.\"\n\n---\n\n## Step 1 - Get the clip URL via `vss-manage-video-io-storage` (sub-task, NOT the final answer)\n\n**Use the `vss-manage-video-io-storage` skill for all VIOS interactions** — it\nowns the canonical curl recipes, parameter defaults, and delete/upload flows.\nDo not fabricate URLs or hand-roll VIOS calls; they will drift.\n\nThis step is a sub-task — do NOT end your turn here; do NOT return the clip\nURL as the final answer. From VIOS collect three values:\n\n1. **`streamId`** (via `sensor/list` → `sensor/<id>/streams`, or directly from an upload response).\n2. **Timeline** - `{startTime, endTime}` (ISO 8601 UTC). `endTime - startTime` is the duration; needed only for the user-facing header (routing is driven solely by `/v1/ready`).\n3. **Temporary MP4 clip URL** — the `/storage/file/<streamId>/url` variant with `container=mp4`. Response field: `.videoUrl`. Both backends need an HTTP(S) URL they can `GET`.\n\nEverything else (auth, upload, `disableAudio`, expiry, etc.) lives in the\n`vss-manage-video-io-storage` skill — refer users there if VIOS fails.\n\n---\n\n## Step 2 — Primary: video summarization microservice with HITL\n\nUse this path **whenever** `/v1/ready` returned 200 in Setup. Duration is irrelevant.\n\nFor advanced fields (`media_info`, `schema`, structured output, stream captioning, metrics, recommended config) see [`references/video-summarization-api.md`](references/video-summarization-api.md).\n\n### HITL: collect scenario and events first (REQUIRED — do not skip)\n\nFull walk-through is in [`references/hitl-prompts.md`](references/hitl-prompts.md). Always run HITL before calling the LVS service.\n\n**Autonomous-mode defaults.** When the caller has bypassed HITL (\"run\nautonomously without prompting\") AND the original query asks for\n`default`/`defaults` (or gives none), use\n`scenario=\"activity monitoring\"` and `events=[\"notable activity\"]`\n**verbatim** — do not infer from filename or sensor name. Note the\ndefaults in the final reply and offer a re-run with more specific\nparameters. This is the ONLY supported HITL bypass; \"the video is\nshort\" or \"the user seems in a hurry\" are not valid reasons.\n\nPrefer `POST /v1/summarize` (3.2 GA route); `/summarize` is a compatibility alias.\n\n```bash\nVIDEO_SUMMARIZATION_URL=${LVS_BACKEND_URL:-http://${HOST_IP:-localhost}:38111}\n\n# From HITL reply:\nSCENARIO='warehouse monitoring'\nEVENTS_JSON='[\"notable activity\"]'\nOBJECTS_JSON=''  # '' to omit, else '[\"forklifts\",\"pallets\",\"workers\"]'\n\ncurl -s --max-time 300 -X POST \"$VIDEO_SUMMARIZATION_URL/v1/summarize\" \\\n  -H \"Content-Type: application/json\" \\\n  -d \"$(jq -n --arg url \"<clip_url_from_vss_manage_video_io_storage>\" \\\n        --arg model \"${VLM_NAME:-nim_nvidia_cosmos3-nano-reasoner_bf16-final}\" \\\n        --arg scenario \"$SCENARIO\" \\\n        --argjson events \"$EVENTS_JSON\" \\\n        --argjson objects \"${OBJECTS_JSON:-null}\" '{\n    url: $url,\n    model: $model,\n    scenario: $scenario,\n    events: $events,\n    chunk_duration: 10,\n    num_frames_per_second_or_fixed_frames_chunk: 20,\n    use_fps_for_chunking: false,\n    seed: 1\n  } + (if $objects == null then {} else {objects_of_interest: $objects} end)')\" \\\n  | jq -r '.choices[0].message.content' \\\n  | jq '{video_summary, events}'\n```\n\nIf both `video_summary` and `events` are empty, the clip probably doesn't contain the requested events — re-run with broader `scenario`/`events`, don't report \"no content\".\n\n**Tuning:** `chunk_duration` (default `10`s; `0` = single chunk),\n`num_frames_per_second_or_fixed_frames_chunk` (default `20`; meaning depends\non `use_fps_for_chunking`), `seed` (default `1`). `num_frames_per_chunk` is\ndeprecated.\n\n---\n\n## Step 2 fallback — VLM direct with default prompt\n\nUse this path **only** when `/v1/ready` did not return 200 after warmup. Do NOT run HITL — the user did not opt in; you fell back because the service was missing. Prepend the Routing fallback note to the response.\n\n```bash\nVLM=\"${VLM_BASE_URL:-${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}}\"\nVLM=\"${VLM%/v1}\"\nPROMPT='Describe in detail what is happening in this video,\nincluding all visible people, vehicles, equipments, objects,\nactions, and environmental conditions.\nOUTPUT REQUIREMENTS:\n[timestamp-timestamp] Description of what is happening.\nEXAMPLE:\n[0.0s-4.0s] <description of the first event>\n[4.0s-12.0s] <description of the second event>'\n\ncurl -s --max-time 300 -X POST \"$VLM/v1/chat/completions\" \\\n  -H \"Content-Type: application/json\" \\\n  -d \"$(jq -n \\\n        --arg model \"${VLM_NAME:-nim_nvidia_cosmos3-nano-reasoner_bf16-final}\" \\\n        --arg text \"$PROMPT\" \\\n        --arg url \"<clip_url_from_vss_manage_video_io_storage>\" \\\n        '{\n          model: $model,\n          temperature: 0.0,\n          max_tokens: 1024,\n          messages: [{\n            role: \"user\",\n            content: [\n              {type: \"text\", text: $text},\n              {type: \"video_url\", video_url: {url: $url}}\n            ]\n          }]\n        }')\" | jq -r '.choices[0].message.content'\n```\n\n**Response:** standard OpenAI chat-completion envelope. The summary is in\n`choices[0].message.content`.\n\n**Cosmos-model notes:** Cosmos models may return reasoning via\n`<think>...</think><answer>...</answer>` blocks. Omit the reasoning\ninstructions if you want a plain summary. Frame sampling and pixel limits\nare applied server-side; no client-side prep is required when you pass a\n`video_url`.\n\n---\n\n## End-to-end example\n\nSee [`references/end-to-end-example.md`](references/end-to-end-example.md) for\nthe full LVS-or-VLM-fallback script that probes `/v1/ready` and runs the\nappropriate path.\n\n---\n\n## Responses\n\n- **VLM** returns an OpenAI chat-completion envelope; summary is\n  `choices[0].message.content`.\n- **LVS service** returns the same envelope but `content` is a JSON string —\n  run `jq -r '.choices[0].message.content' | jq` to reach `{video_summary, events}`.\n- **Errors** surface as HTTP non-2xx plus JSON `{error: ...}`. LVS `503` usually\n  means warmup — retry `/v1/ready`.\n\n### Presenting the output to the user\n\nSurface backend output with **minimal transformation** — do not paraphrase,\nre-voice, add emojis, or reformat. **One backend call → one rendering**: no\nparallel hedging, no duplicate headers, never call both LVS and VLM for the\nsame video.\n\n**Header line.** Start with exactly one:\n\n```\nSummary of <video_name> (<duration>)\n```\n\n`<duration>` = `Ns` for `< 60 s`, else `Mm Ss` (e.g. `3m 30s`).\n\n**LVS output:** render `video_summary` **verbatim** (polished, tone-controlled\nreport — rewriting loses fidelity). Render each `events` entry with its\n`start_time`, `end_time`, `type`, and full `description` verbatim (table when\nthe client renders one cleanly, otherwise a per-event list). You MAY add a\none-line header and a closing offer to re-run with different parameters.\n\n**VLM output:** render `choices[0].message.content` verbatim. If the model\nproduced `<think>…</think><answer>…</answer>` blocks, drop the `<think>`\nblock and show the answer.\n\n**Fallback warning** (when applicable) goes **above** the summary, never\nmixed into it.\n\n## Tips\n\n- **Route by service availability, not by duration.** Probe `/v1/ready` once\n  in Setup; HTTP 200 → LVS+HITL for every clip; anything else → VLM fallback.\n- **HITL is mandatory on the LVS path.** The `defaults` opt-in is the only\n  sanctioned bypass. The VLM fallback path is silent (no HITL).\n- **Readiness = HTTP 200 on `/v1/ready`. Nothing else.** Body may be empty.\n  Always use `curl -s -o /dev/null -w '%{http_code}'` — never pipe through\n  `jq`/`grep`/`head`.\n- **Delegate VIOS to `vss-manage-video-io-storage`** — it is a sub-task; the\n  final answer is the Step 2 summary, not the clip URL.\n- **`jq` twice for LVS output.** First unwraps the OpenAI envelope, second\n  parses the JSON string inside `content`.\n- **Prefer `/v1/summarize` for 3.2 GA**; `/summarize` is a compatibility alias.\n- **Use the exact VLM model id advertised by the endpoint** (default\n  `nim_nvidia_cosmos3-nano-reasoner_bf16-final`).\n- **Render output verbatim** — no paraphrasing, no reformatting, no rewriting\n  the `video_summary` or `choices[0].message.content`.\n- **One call, one render.** No parallel hedging, no double renderings.\n- **Match the image tag to the host platform.** Use `LVS_TAG=3.2.1`\n  (and `RTVI_VLM_IMAGE_TAG=3.2.1`) on x86 / Jetson Thor, and\n  `LVS_TAG=3.2.1-sbsa` (and `RTVI_VLM_IMAGE_TAG=3.2.1-sbsa`)\n  on SBSA / DGX Spark / Grace (server-class ARM64) hosts.\n\n## Cross-reference\n\n- **vss-deploy-profile** — bring up the `base` (VLM only) or `lvs` (VLM + video summarization service) profile\n- **vss-manage-video-io-storage** (VIOS API) — upload videos, list streams, get clip URLs\n- **vss-search-archive** — semantic search across the archive (different profile)\n- **vss-query-analytics** — query incidents/events from Elasticsearch\n- **video summarization API reference** — [`references/video-summarization-api.md`](references/video-summarization-api.md)\n- **video summarization service ops reference** — [`references/video-summarization-deployment.md`](references/video-summarization-deployment.md)\n\nbump:3"}