{"id":"c91c1481-1b97-442f-87d7-bf679a3a6d0b","slug":"tao-generate-video-reasoning-annotations","kind":"skill","name":"tao-generate-video-reasoning-annotations","summary":"Multi-step video annotation pipeline that turns raw videos into Chain-of-Thought training data — multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning tr","owner":"NVIDIA","repoUrl":"https://github.com/NVIDIA/skills/tree/HEAD/skills/tao-generate-video-reasoning-annotations","license":"Apache-2.0","licenseTier":"permissive","trust":"official","stars":3586,"match":null,"fewStars":false,"format":"skill","createdAt":null,"addedAt":"2026-10-14T05:50:48.740Z","logoUrl":"https://avatars.githubusercontent.com/u/1728152?v=4","linkOnly":false,"collection":{"name":"skills","repo":"nvidia/skills"},"skillPath":"skills/tao-generate-video-reasoning-annotations","categories":["design-media","data-analytics","developer-tools"],"tags":["agentic-ai","ai-agents","claude-code","codex","coding-agents","cuda","developer-tools","llm"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-14T05:50:48.738Z","sample":false,"url":"https://conduits.dev/tools/tao-generate-video-reasoning-annotations","language":"Python","transports":[],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"> Multi-step video annotation pipeline that turns raw videos into Chain-of-Thought training data — multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning traces, via VLM/LLM distillation. Use when the user wants to \"create video training data\", \"generate video QA datasets\", \"build CoT reasoning traces from videos\", \"auto-label videos\", or run the video_reasoning_annotation pipeline. Triggers include \"video annotation\", \"video CoT\", \"video QA\", \"chain-of-thought\", \"video captioning pipeline\", \"video distillation\".\n\n# Video Reasoning Annotation Pipeline\n\n> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).\n\nGenerate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain.\n\n## Purpose\n\nTransform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as \"teacher\" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls).\n\n## Pipeline architecture\n\n```\nStep 0:  [Optional] Filter & classify videos  → Keep domain-relevant, classify anomaly vs normal\nStep 1a: Global + dense captions               → VLM: narrative summary + timestamped events\nStep 1b: Chunk captions                         → VLM: fixed-duration segment micro-captions\nStep 1c: [Optional, anomaly only] Highlight     → LLM extracts anomaly timestamp, VLM captions clip\nStep 2:  Description synthesis                  → LLM: synthesize captions into structured narrative\nStep 3:  QA generation                          → LLM: MCQ, binary, open-ended with reasoning\nStep 4:  Parse outputs                          → Per-task `tao-vl-reason-v1.0` JSON files\n```\n\nSteps are individually selectable via `workflow.steps`. The pipeline has built-in resume — each step skips already-processed videos, so re-running after a prompt tweak is safe.\n\n## Initial consultation\n\nWhen the user invokes this skill, walk through these questions in order. Don't skip — getting domain and VLM access right up front prevents wasted runs.\n\n### 1. Videos\n\n- Path to the video directory and/or a JSONL with `{\"video_path\": \"...\"}` per line.\n- Confirm format (`.mp4` preferred; `.avi`, `.mov`, `.mkv` also walked).\n\n### 2. Domain — drives prompt selection\n\nAsk the user: *\"What domain are these videos from?\"* Choose one of the following branches:\n\n| Domain | What to do |\n|---|---|\n| **general** | Use the default prompts. Set `prompts_module: \"\"` (or omit). The built-in `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts` covers domain-agnostic content. |\n| **traffic** (CCTV intersections, highways; dashcam excluded) | Use the reference module. Set `prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_traffic\"`, **or** copy `references/prompts_traffic.py` into the user's project and tune for their specific camera angles, then point `prompts_module` at the copy. |\n| **warehouse** (industrial site CCTV — safety, operations, security) | Same pattern. Set `prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_warehouse\"`, or copy `references/prompts_warehouse.py` and tune. |\n| **custom** (any other domain) | **Run the workshop in [references/domain_adaptation.md](references/domain_adaptation.md)**. It walks through: Phase 1 — question types the user wants the model to answer; Phase 2 — caption-requirements checklist; Phase 3 — fill the `[PLACEHOLDER]` markers in `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template`. The two reference modules above are working examples to model after. Do this **before** any pipeline runs. |\n\n### 3. Anomaly / normal / mixed\n\n- Mixed dataset → `workflow.mode: \"auto\"` (Step 0 classifies each video).\n- Pre-split anomaly only → `workflow.mode: \"anomaly\"`, drop Step 0.\n- Pre-split normal only → `workflow.mode: \"normal\"`, drop Steps 0 and 1c.\n\n### 4. VLM / LLM endpoint — confirm access **before** running\n\n- **Gemini** (default for both `vlm.backend` and `llm.backend`): user needs `GOOGLE_API_KEY` set, or to put the key in the YAML.\n- **OpenAI-compatible** (Qwen via vLLM, NIM endpoint, etc.): user provides `base_url`, `model_name`, and `api_key`.\n- Steps 2–3 are text-only — a smaller/cheaper LLM is fine for `llm.backend` even when `vlm.backend` is a frontier video model.\n\nIf the user has **no endpoint at all** and wants to self-host, point them at the `skills/applications/tao-run-inference-service` skill — a workflow that stands up a network-specific TAO inference microservice locally and exposes an OpenAI-compatible endpoint. Should support Cosmos, Qwen, and Gemma. Check `skills/applications/tao-run-inference-service/references/service.yaml` for the current `valid_network_arch_config_basenames` list before relying on a specific model.\n\nIf the user doesn't have endpoint access ready and isn't ready to set one up, stop here and help them figure it out first.\n\n### 5. Pilot vs full run\n\n- **Recommend a 5–10 video pilot** when domain is `custom`, when any prompt was edited, or when this is the user's first run.\n- **Full-run is fine** for `general` / `traffic` / `warehouse` once the user has previously verified output quality on the same data type.\n- The pipeline has built-in resume, so a pilot followed by a full run does not re-process the pilot videos.\n\n## Quick start\n\nThe pipeline runs inside the TAO Toolkit container via the `auto_label` CLI:\n\n```bash\nauto_label generate -e /path/to/spec.yaml \\\n    results_dir=/results \\\n    video_reasoning_annotation.data.video_root=/videos \\\n    video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY \\\n    video_reasoning_annotation.workflow.mode=auto\n```\n\nGenerate a default spec to start from:\n\n```bash\nauto_label default_specs results_dir=/results module_name=auto_label\n# then set:  autolabel_type: \"video_reasoning_annotation\"\n```\n\nAll fields support Hydra dot-notation overrides on the command line. For the full YAML reference (every field, model/endpoint setup, error patterns), see [references/configuration.md](references/configuration.md).\n\n## Pilot workflow\n\nUse this when running a 5–10 video pilot:\n\n1. Run the pipeline on the pilot subset with the chosen `prompts_module` and `workflow.mode`.\n2. Inspect `results_dir/step_1a_caption/captions.jsonl` — captions accurate, capturing the right level of detail?\n3. Inspect `results_dir/step_3_qa/qa_output.jsonl` — questions meaningful, answers correct, reasoning logical?\n4. If quality is insufficient: adjust the prompts (in `prompts_module` if domain-customized, or fall back to `general` if a domain module is over-tuned), and re-run. The pipeline auto-skips already-processed videos.\n5. Once satisfied, scale to the full dataset by pointing `data.video_root` (or `data.input_jsonl_files`) at the full set and re-running with the same `results_dir` (resume) or a fresh one (full re-run).\n\nQuality compounds downstream — bad captions produce bad descriptions which produce bad QA. Focus iteration on Step 1a/1b output first; descriptions and QA usually improve once captions are right.\n\n## Configuration summary\n\nKey fields (full reference in [references/configuration.md](references/configuration.md)):\n\n| Field | Default | Description |\n|---|---|---|\n| `workflow.steps` | `[\"0\",\"1a\",\"1b\",\"1c\",\"2\",\"3\",\"4\"]` | Which pipeline steps to execute |\n| `workflow.mode` | `\"auto\"` | `\"auto\"`, `\"anomaly\"`, or `\"normal\"` |\n| `vlm.backend` | `\"gemini\"` | `\"gemini\"` or `\"openai\"` (OpenAI-compatible) |\n| `llm.backend` | `\"gemini\"` | Same options; text-only, cheaper model works |\n| `workflow.max_workers` | `4` | Parallel threads per step (watch API rate limits) |\n| `license` | `\"\"` | Optional: written to `metadata.license` in step 4 outputs (e.g. `\"CC-BY-4.0\"`) |\n| `description_extra` | `\"\"` | Optional: extra text appended to per-task descriptions in step 4 metadata |\n| `prompts_module` | `\"\"` | Dotted import path to custom prompts module |\n\n## Prompts\n\n- **Built-in (general)**: `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts` — domain-agnostic, used by default.\n- **Template**: `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template` — same 26 keys with `[PLACEHOLDER]` markers for domain customization.\n- **Reference modules** (working examples for the consultation's `traffic` / `warehouse` branches): [references/prompts_traffic.py](references/prompts_traffic.py), [references/prompts_warehouse.py](references/prompts_warehouse.py).\n- **Custom domains**: see [references/domain_adaptation.md](references/domain_adaptation.md) for the full workshop and placeholder reference.\n\n## Inputs\n\n- **`video_root`**: Directory of videos (walked recursively for `.mp4`, `.avi`, `.mov`, `.mkv`).\n- **`input_jsonl_files`**: List of JSONL files with `{\"video_path\": \"...\"}` per line. The `video` key is also accepted; extra fields are allowed.\n- **`filter_field`**: Optional boolean field to filter JSONL entries.\n\nProvide `video_root`, `input_jsonl_files`, or both (lists merge).\n\n## Outputs\n\nAll outputs go to `results_dir/` with per-step subdirectories (`step_0_filter/`, `step_1a_caption/`, …, `step_4_output/`):\n\n- **Steps 0–3**: JSONL — one JSON object per video per line.\n- **Step 4**: One `<task>.json` per non-empty task type, in the **`tao-vl-reason-v1.0`** envelope. Up to 10 files: `mcq.json`, `mcq_openended.json`, `bcq.json`, `bcq_openended.json`, `open_qa.json`, `causal_linkage.json`, `temporal_localization.json`, `temporal_description.json`, `scene_description.json`, `video_summarization.json`.\n\nEach step 4 file looks like:\n\n```json\n{\n  \"format\": \"tao-vl-reason-v1.0\",\n  \"metadata\": {\"type\": \"annotation\", \"task\": \"<task>\", \"date\": \"YYYY-MM-DD\",\n               \"description\": \"<per-task + description_extra>\", \"license\": \"<from config>\"},\n  \"media_root\": \"<data.video_root>\" | null,\n  \"items\": [{\"video_id\": \"...\", \"question\": \"...\", \"answer\": \"...\", \"reasoning\": \"...\"}, ...]\n}\n```\n\n`media_root` mirrors `data.video_root` (or `null` when unset); each item's `video_id` is the entry's video path with the `video_root` prefix stripped. Set `license` and `description_extra` in the spec to populate the metadata.\n\n## Prerequisites\n\n- **Container**: `nvcr.io/nvidia/tao/tao-toolkit:7.2.0-pyt`. <!-- versions-key: images.tao_toolkit.pyt -->\n- **ffmpeg / ffprobe**: required for chunk captioning (Step 1b) and highlight extraction (Step 1c).\n- **VLM endpoint**: at least one — Gemini API key or OpenAI-compatible endpoint."}