{"id":"7d402b9b-6e56-4865-b418-bbf55fa3a826","slug":"clipwise","kind":"skill","name":"clipwise","summary":"Agent skill: give your coding agent (Claude Code, Codex, Cursor, Hermes, OpenClaw) a video or link, get viral 9:16 Shorts back. Face-tracked crop, word-by-word captions, SFX. Local, free, no API keys.","owner":"RajkumarDake","repoUrl":"https://github.com/RajkumarDake/clipwise","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"skill","createdAt":"2026-10-26T19:18:07.000Z","addedAt":"2026-11-01T02:17:37.216Z","logoUrl":"https://avatars.githubusercontent.com/u/108937933?u=f2e68d57b16d4a9cfaa7159439831874c04871da&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["design-media","developer-tools","files-documents"],"tags":["ai-agents","captions","claude-code","codex","face-tracking","ffmpeg","opus-clip-alternative","reels"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-11-01T02:39:42.486Z","sample":false,"url":"https://conduits.dev/tools/clipwise","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"

\n \"clipwise\"\n

\n\n

Give your coding agent a video link. Get viral Shorts back.

\n\n

\n \"MIT\n \"Python\n \"ffmpeg\n \"agents\n

\n\nclipwise is an agent skill: one folder with a `SKILL.md`, a scoring rubric and a few Python\nscripts. Install it into your coding agent, paste a YouTube link or a file path, and the agent\ntranscribes the video, picks the moments worth posting, checks them with you, and renders\nfinished 9:16 clips with face-tracked framing, word-by-word captions, sound effects and\nbroadcast-level loudness. Everything runs on your machine. No API keys, no uploads, no\nper-minute fees.\n\n## Demo\n\n

\"A

\n\n

\"Finished

\n\nA real contact sheet from the test render (40.6 s clip from a CC BY interview, default\nquality, measured on a 2-core CPU). Ten evenly spaced frames; the red line marks where the\nplatform UI starts.\n\n

\"Contact

\n\n## What it does\n\n- **Takes any source**: local file, YouTube and other yt-dlp sites, public Google Drive or\n Dropbox links, direct `.mp4/.mov/.webm` URLs. iPhone HDR / variable-frame-rate footage is\n tone-mapped and conformed automatically.\n- **Transcribes locally** with faster-whisper (large-v3-turbo by default), word-level timings,\n on CPU or CUDA.\n- **Your agent picks the moments**, not a keyword heuristic: it reads the transcript and\n scores candidates with a rubric (hook, emotion, curiosity, story, value, relatability), then\n shows you the list before anything is rendered.\n- **Hook first**: a strong line from mid-clip can be moved to the first 2-3 seconds.\n- **Dead air removed**: pauses over 0.35 s are cut with a threshold tuned to each clip.\n- **Face-tracked 9:16 crop** from 16:9 (OpenCV YuNet), with a sticky tracker and a smooth\n camera that only snaps on cuts. Blurred-background layout for wide multi-person shots.\n- **Punch-in zooms** every 2-4 s, aligned to cuts and sentence ends.\n- **Word-by-word captions** (active word highlighted), placed below the chin and above the\n bottom 300 px that Shorts/Reels/TikTok cover with buttons.\n- **Sound**: voice cleanup, sound effects synthesised at render time (whoosh, thud, pop), an\n optional music bed of your own that ducks under the voice, two-pass loudness to -14 LUFS.\n- **Ready to post**: title, description, pinned-comment idea and hashtags per clip, plus a\n contact sheet the agent inspects before handing the clip over.\n\n## How it works\n\n

\"Pipeline:

\n\n1. `fetch.py` downloads and conforms the source to a clean H.264/AAC file.\n2. `transcribe.py` writes `transcript.txt` with timestamps and word indices.\n3. The agent reads it with `references/viral-rubric.md` and writes `picks.json`.\n4. You approve (or edit) the list.\n5. `render.py` cuts, tracks, captions, mixes and encodes each pick, then `look.py` makes a\n contact sheet the agent checks with its vision tool.\n\n## Install\n\n**1. Copy the `clipwise/` folder into your agent's skills directory.**\n\n```bash\ngit clone https://github.com/RajkumarDake/clipwise.git\n```\n\n| Agent | Where the folder goes |\n| --- | --- |\n| Claude Code | `~/.claude/skills/clipwise` (all projects) or `.claude/skills/clipwise` (one project) |\n| Codex CLI / IDE | `~/.agents/skills/clipwise` (or the older `~/.codex/skills/clipwise`, still scanned); per repo: `.agents/skills/clipwise` |\n| Hermes Agent | `~/.hermes/skills/clipwise` |\n| OpenClaw | `openclaw skills install ./clipwise` (workspace) or add `--global`; or copy into `/skills/clipwise` |\n| Cursor / Windsurf / Cline | anywhere in the repo (e.g. `tools/clipwise`), then add a project rule (`.cursor/rules`, `.windsurfrules`, `.clinerules`): \"To turn a video into Shorts, read and follow `tools/clipwise/SKILL.md`\" |\n| Any other agent | put the folder anywhere and tell it: \"Read `path/to/clipwise/SKILL.md` and follow it\" |\n\nExample for Claude Code:\n\n```bash\nmkdir -p ~/.claude/skills\ncp -r clipwise/clipwise ~/.claude/skills/clipwise\n```\n\nIf your Codex setup does not pick the skill up, add one line to `AGENTS.md`:\n`To make Shorts from a video, follow ~/.agents/skills/clipwise/SKILL.md.`\n\n**2. Run the setup once.** Read [`clipwise/setup.sh`](clipwise/setup.sh) first; it only\nwrites to `~/.clipwise` (override with `CLIPWISE_HOME`).\n\n```bash\nbash ~/.claude/skills/clipwise/setup.sh # path of wherever you copied the folder\n```\n\nIt checks Python 3.10+ and ffmpeg (with libass) and prints the install command if one is\nmissing, creates a venv, installs `requirements.txt`, downloads the YuNet face model and the\nLato Black caption font (both pinned and sha256-verified), and pre-downloads the whisper\nmodel (about 1.6 GB; skip with `--no-model`). Re-running it is safe.\nVerify with `python3 /scripts/run.py check`.\n\n## Usage\n\nTalk to your agent:\n\n```text\nMake 5 Shorts from https://www.youtube.com/watch?v=XXXXXXXXXXX\n```\n\n```text\nCut Reels from ~/Videos/podcast_ep12.mp4, only minutes 10-25, captions in Spanish.\n```\n\n```text\nMake 2 Shorts from interview.mov with ~/Music/bed.mp3 as a quiet music bed, green highlight.\n```\n\nThe agent shows you the picks (timestamps, hook, title, why) and waits for your OK before\nrendering. More prompts in [`examples/prompts.md`](examples/prompts.md).\n\n**How many Shorts?** If you name a number, you get exactly that many, unless the video does\nnot have that many strong moments; then the agent says so and proposes the honest number.\nWithout a number it scales with length: under 5 min -> 1-2, 5-15 min -> 3-5, 15-45 min -> 5-8,\n45 min+ podcast -> 8-15. Only moments that pass the rubric checklist are kept; weak clips are\nnever added to reach a count.\n\nYou can also run the scripts yourself:\n\n```bash\npython3 clipwise/scripts/run.py prep \"https://...\" --job talk\n# write ~/clipwise-work/talk/picks.json (see examples/picks.json)\npython3 clipwise/scripts/run.py render --job talk # add --fast for a preview\n```\n\n## Output\n\n```text\n~/clipwise-work/talk/\n├── source.mp4 conformed source (delete with: run.py clean --job talk)\n├── transcript.txt what the agent reads\n├── transcript.json segments + word timings\n├── words.tsv word index, start, end, probability\n├── picks.json the approved moments\n└── out/\n ├── clip_1.mp4 1080x1920, 30 fps, H.264 + AAC, -14 LUFS\n ├── clip_1.txt title, description, pinned comment, hashtags\n ├── clip_1.json probe numbers, layout, face coverage, SFX cues, timings\n ├── clip_1_cutmap.json\n └── clip_1_sheet.png contact sheet\n```\n\n`clip_1.txt` from the test render:\n\n```text\nTITLE: Is every flood caused by climate change?\n\nDESCRIPTION: A flood scientist on what climate change does and does not cause.\n\nPINNED COMMENT: Did you think every flood was climate change?\n\nHASHTAGS: #shorts #climate #science #floods\n\nWHY: Opens on a claim people don't expect from a climate scientist, resolves it with 'more likely, not always'.\nSOURCE: 1:05.50-1:50.35, hook 1:14.85-1:20.10\n```\n\n## Requirements\n\n- Python 3.10+ and ffmpeg built with libass (Ubuntu/Debian `apt install ffmpeg`, macOS\n `brew install ffmpeg`, Windows via WSL2).\n- About 2.5 GB disk for the venv and the whisper model, plus room for your videos.\n- 4 GB+ RAM. A GPU is optional.\n- A coding agent that can run shell commands and look at images.\n\n## Performance\n\nMeasured on a 2-core cloud VM, CPU only, while writing this README:\n\n| step | input | time |\n| --- | --- | --- |\n| fetch (download) | 85 MB 1080p WebM from Wikimedia | about 5 min (server-limited) |\n| fetch (conform) | 3 min 1080p VP9 window to H.264 | about 2.5 min |\n| fetch (conform) | 20 s 10-bit HDR iPhone clip, tone-map + CFR | 45 s |\n| transcribe | 3 min English, large-v3-turbo int8 | 110 s (0.61x realtime) + 19 s model load |\n| render, default quality | 40.6 s clip, face-tracked crop | 85 s |\n| render, `--fast` | 26.4 s clip, crop | 32 s |\n| render, `--fast --layout pad` | 17.1 s clip | 30 s |\n\nProbe of the default-quality clip: video 40.63 s / audio 40.57 s, 1080x1920, 30 fps,\n-14.0 LUFS integrated, -1.5 dBTP. On a modern laptop expect roughly 2-4x faster; with CUDA,\ntranscription is a few seconds per minute.\n\n## Limitations\n\n- The tracker follows the largest face, not the active speaker. In a wide two-person shot\n it holds on one person (no flicker), so the other speaker may be off-screen; use\n `--layout pad` there.\n- Whisper returns English for many non-Latin-script languages and code-mixed speech instead\n of transcribing them. The agent then has to write the captions in the speaker's language\n itself (supported via `caption_overrides`), and you should check those lines.\n- YouTube sometimes blocks downloads from servers and some home IPs (\"confirm you're not a\n bot\"). Pass `--cookies cookies.txt` or give the agent the file.\n- 720p sources look soft after the 9:16 crop (2.7x upscale).\n- Picking quality depends on your agent's model; the rubric keeps it honest but it is still\n judgement.\n- Tested on Linux. macOS should work (Homebrew ffmpeg); Windows only through WSL2.\n\n## FAQ\n\n**Does it need a GPU?** No. Everything was developed and measured on a 2-core CPU. If CUDA\nis available, transcription uses it automatically.\n\n**Which languages?** Any language whisper detects is transcribed with word timings. Captions\nare written in the speaker's language; for languages whisper translates instead (common with\nIndic and other non-Latin scripts, and mixed-language speech), the agent writes the caption\nlines itself and shows them to you before rendering.\n\n**YouTube is blocked. What now?** Export `cookies.txt` from a browser where you are logged\nin and pass `--cookies cookies.txt`, or download the video yourself and give the agent the\nfile path.\n\n**How many Shorts will I get?** The number you ask for, or a count based on video length\n(1-2 under 5 minutes up to 8-15 for a long podcast). The agent will tell you if fewer\nmoments are strong enough.\n\n**What does it cost?** Nothing beyond your agent subscription. All models run locally; no\npaid APIs.\n\n**Can I use my own sounds or music?** Yes: `--sfx-dir` with your own `whoosh_fast`,\n`whoosh_soft`, `thud_low`, `pop`, `tick` files, and `--music bed.mp3` for a ducked music bed.\nUse audio you have the rights to. No audio files ship with clipwise; the default SFX are\ngenerated by `scripts/sfx.py`.\n\n## Credits\n\n- [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (MIT) and the\n [large-v3-turbo CTranslate2 conversion](https://huggingface.co/mobiuslabsgmbh/faster-whisper-large-v3-turbo) (MIT) of OpenAI Whisper.\n- [OpenCV](https://opencv.org) and the [YuNet face detector](https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet) (MIT).\n- [FFmpeg](https://ffmpeg.org) (LGPL/GPL) with libass.\n- [yt-dlp](https://github.com/yt-dlp/yt-dlp) (Unlicense).\n- [Lato](https://fonts.google.com/specimen/Lato) by Łukasz Dziedzic (SIL Open Font License), downloaded by setup.sh.\n- Test footage in the demo contact sheet and `examples/picks.json`:\n \"[Interview on extreme weather with physical geographer Hannah Cloke](https://commons.wikimedia.org/wiki/File:Interview_on_extreme_weather_with_physical_geographer_Hannah_Cloke_%E2%80%93_The_Royal_Society.webm)\"\n by The Royal Society, [CC BY 3.0](https://creativecommons.org/licenses/by/3.0/), via Wikimedia Commons. Frames were cropped and captioned.\n\n## License\n\n[MIT](LICENSE) © 2026 Rajkumar Dake\n"}