programmatic-video-skill

Programmatic Video Skill

Deterministic 60fps MP4 videos from a single HTML5 Canvas/SVG file, Playwright frame-stepping, multimodal contact-sheet QA, Python audio synthesis, and FFmpeg.

Built for Claude Code, Cursor, OpenCode, Codex, Grok, and any AI coding agent with a shell and image viewer.

Above: Real 9-keyframe contact-sheet.png generated in ~3 seconds from examples/release-wrapped/scene.html so a multimodal coding agent can visually inspect every scene before encoding the full 60fps MP4.

All it does, in one pipeline

scene.html (window.__setFrame) ──► render.mjs (--timestamps) ──► contact-sheet.py (Agent Visual QA)
                                                                        │
final-60fps.mp4 ◄── ffmpeg (image2pipe + AAC) ◄── render.mjs (60fps) ◄──┴── synth-audio.py (WAV + SFX cues)

When you ask an AI coding agent to "make an animated video," naive approaches either set up a heavy React/Remotion bundler, record a browser window in real time (dropping frames whenever CPU spikes), or render thousands of frames blindly with overlapping text.

programmatic-video-skill gives your coding agent a repeatable, zero-framework engineering workflow:

1. Deterministic frame clock (window.__setFrame(f, t)): Every pixel at frame f (t = f / 60) is a pure mathematical function of time. Zero dropped frames, even on heavy 1080p Canvas/SVG scenes.
2. Multimodal Contact-Sheet Self-Review: In ~3 seconds, the agent captures key timestamps across all scenes, stitches a labeled contact-sheet.png, looks at the image with its multimodal Read/View tool, fixes any text collisions or layout bugs, and only then runs the full 60fps encode.
3. Procedural Audio + Sample-Accurate WAV SFX Cues: Synthesizes a multi-segment stereo 44.1 kHz WAV soundtrack (bassline, stereo arpeggiator, electronic drums, transition risers, sub-drops) in Python/NumPy and mixes WAV sound effects (like Warcraft III "Job's done!" clips from claude-code-sound-packs) at exact sub-second timestamps with automatic music ducking.
4. Zero-Disk-Bloat FFmpeg Pipe: Streams raw PNG frame buffers from headless Playwright straight into ffmpeg -f image2pipe (H.264 yuv420p, CRF 18, +faststart) — no 10 GB folder of temporary PNG files.

Why This Approach Beats Alternatives for Coding Agents

Install

Option 1: Install via skills CLI (Cursor, Claude Code, Codex, OpenCode, Windsurf, Gemini CLI)

npx skills add jblxo/programmatic-video-skill --skill programmatic-video
npx skills add jblxo/programmatic-video-skill --skill programmatic-video -g   # install globally

Option 2: Claude Code Plugin Marketplace

Inside Claude Code:

/plugin marketplace add jblxo/programmatic-video-skill
/plugin install programmatic-video@programmatic-video-skill

Option 3: Zero-Install Prompt (Point Any Agent at This Repo)

Paste this directly into Claude Code, Cursor, Grok, Codex, or OpenCode:

Clone or read https://github.com/jblxo/programmatic-video-skill (especially skills/programmatic-video/SKILL.md and examples/release-wrapped/).
Using its deterministic window.__setFrame(f, t) + contact-sheet.py multimodal QA loop + synth-audio.py + render.mjs pipeline, build a 60fps 1080p MP4 animated video for:
[describe your release recap, team farewell, product launch, or engineering Wrapped video here]

Quick Start: Run the Included 60fps Showcase

The examples/release-wrapped/ folder contains a complete, self-contained 30-second 1080p showcase ("Project Hyperion v2.0 — Ship It Wrapped") featuring 4 scenes:

• Scene 01 (0.0s – 7.0s): Priority Omega CRT Terminal Alert → Interactive Button Click → Live Ingestion Waveform
• Scene 02 (7.0s – 15.0s): 3D Perspective Synthwave Grid + Spring-Physics Hero Counters + Quarterly Throughput Bars
• Scene 03 (15.0s – 23.0s): Split-Screen Codebase Telemetry + Animated Semi-Circular Cluster Health Gauge
• Scene 04 (23.0s – 30.0s): 8-Card Core Squad Bento Grid + "WORK, WORK... JOB'S DONE!" Finale Banner & Seeded Confetti Burst

git clone https://github.com/jblxo/programmatic-video-skill.git
cd programmatic-video-skill/examples/release-wrapped

# 1. Capture 9 keyframe snapshots and build contact-sheet.png (~3 seconds)
npm run qa

# 2. Synthesize the 30-second 124 BPM stereo synthwave soundtrack
npm run audio

# 3. Render the full 30-second 60fps 1080p H.264 MP4 video (1,800 frames)
npm run render

Bundled Tools

All scripts live in skills/programmatic-video/scripts/ and run headlessly from any agent shell:

Standalone CLI Examples

# Step 1: Fast Visual QA (captures 9 timestamps + builds contact-sheet.png)
node skills/programmatic-video/scripts/render.mjs \
  --html ./scene.html \
  --timestamps 1.8,3.0,5.2,8.5,12.5,16.5,20.5,24.5,28.0 \
  --out-dir ./qa-frames

python3 skills/programmatic-video/scripts/contact-sheet.py \
  --dir ./qa-frames \
  --cols 3 \
  --thumb-width 560 \
  --out ./contact-sheet.png

# Step 2: Procedural Audio + Custom WAV Sound Effect Cues
python3 skills/programmatic-video/scripts/synth-audio.py \
  --duration 30.0 \
  --bpm 124 \
  --style synthwave \
  --transitions 3.4,7.0,15.0,23.0,25.2 \
  --cue "25.2:~/.claude/hooks/sounds/warcraft-peasant/stop.wav:+2.0" \
  --out ./soundtrack.wav

# Step 3: Full 60fps 1080p H.264 MP4 Encode
node skills/programmatic-video/scripts/render.mjs \
  --html ./scene.html \
  --duration 30 \
  --fps 60 \
  --audio ./soundtrack.wav \
  --crf 18 \
  --out ./release-video.mp4

How the Deterministic Frame Contract Works

Instead of relying on real-time CSS animations or setInterval, your scene.html exposes a single function:

window.__setFrame = function(frameIndex, timeSeconds, fps = 60) {
  if (window.__rafId) {
    cancelAnimationFrame(window.__rafId);
    window.__rafId = null;
  }
  const t = typeof timeSeconds === 'number' ? timeSeconds : frameIndex / fps;
  renderSceneAtTime(frameIndex, t);
};

1. Interactive Browser Preview: When you open scene.html in a normal browser tab, a standard requestAnimationFrame loop calls renderSceneAtTime(frame, elapsed) so you can preview animations live.
2. Headless Deterministic Capture: The moment render.mjs calls window.__setFrame(f, f / 60), the live loop is cancelled and the canvas renders the exact mathematical state for frame f. Even if a single frame takes 250ms to draw complex SVG charts and 200 particles, the resulting MP4 plays back at a locked, butter-smooth 60.00 fps.

Repository Structure

programmatic-video-skill/
├── .claude-plugin/
│   ├── plugin.json                        # Claude Code plugin manifest
│   └── marketplace.json                   # Plugin marketplace catalog
├── skills/
│   └── programmatic-video/
│       ├── SKILL.md                       # Complete 6-step workflow & rules for coding agents
│       ├── references/
│       │   ├── visual-patterns.md         # Easing curves, mulberry32 PRNG, 3D synthwave grid, confetti
│       │   ├── visual-qa-loop.md          # Multimodal 6-point contact-sheet inspection checklist
│       │   └── audio-synthesis.md         # Procedural NumPy synth + WAV SFX cue mixing & ducking
│       └── scripts/
│           ├── render.mjs                 # Playwright __setFrame stepper + FFmpeg image2pipe encoder
│           ├── contact-sheet.py           # Labeled keyframe contact-sheet generator
│           └── synth-audio.py             # Multi-segment soundtrack synth + WAV SFX cue mixer
├── examples/
│   └── release-wrapped/
│       ├── scene.html                     # 1920x1080 60fps 4-scene HTML5 Canvas showcase
│       ├── generate-audio.py              # Soundtrack generator for the showcase
│       ├── package.json                   # npm run qa | audio | render
│       └── contact-sheet.png              # Rendered 9-keyframe visual QA preview
├── llms.txt                               # Single-file digest for LLMs and answer engines (llmstxt.org)
├── LICENSE                                # MIT License
└── README.md

Requirements

• Node.js 18+ and Playwright (npm i -D playwright or global playwright-core + Chrome/Chromium)
• FFmpeg (brew install ffmpeg on macOS, apt install ffmpeg on Linux)
• Python 3.8+ (pip install numpy Pillow recommended for accelerated audio synthesis and labeled contact sheets; automatic stdlib/FFmpeg fallbacks included)

FAQ

How do I generate an MP4 video with Claude Code, Cursor, or an AI coding agent?

Install this skill (npx skills add jblxo/programmatic-video-skill) or point your agent to https://github.com/jblxo/programmatic-video-skill. The agent will design a storyboard, write a single scene.html implementing window.__setFrame(f, t), verify the layout via contact-sheet.png, synthesize a WAV soundtrack with synth-audio.py, and pipe 60fps frames into FFmpeg via render.mjs.

How do I render HTML5 Canvas and SVG animations at 60fps with Playwright and FFmpeg?

Expose window.__setFrame(frameIndex, timeSeconds) in your HTML file so every visual element is computed from t = frameIndex / 60. Then run node skills/programmatic-video/scripts/render.mjs --html scene.html --duration 30 --fps 60 --out video.mp4. render.mjs steps frame-by-frame in headless Chromium and pipes PNG buffers to ffmpeg -f image2pipe -framerate 60 -c:v libx264 -pix_fmt yuv420p.

How can an AI agent visually verify video frames before rendering the full MP4?

Instead of encoding thousands of frames blindly, run render.mjs --timestamps 1.8,5.2,12.5,20.5,28.0 --out-dir ./qa-frames followed by contact-sheet.py --dir ./qa-frames --out ./contact-sheet.png. In ~3 seconds, the agent gets a single grid image stamped with frame numbers and timestamps, inspects it with its multimodal Read/View tool, and fixes any overlapping text or broken layout math before full rendering.

How do I sync sound effects (WAV SFX) and procedural music to programmatic video frames?

Pass your scene cut timestamps (--transitions 3.4,7.0,15.0,23.0) and WAV SFX cues (--cue "25.2:./sfx/jobs-done.wav:+2.0") to synth-audio.py. Because t = frame / 60 is deterministic in scene.html and sample = int(t * 44100) is deterministic in synth-audio.py, visual events and audio cues stay locked with 0 ms drift. Works out of the box with sound packs from jblxo/claude-code-sound-packs.

License

MIT © Ondřej Hliba