nybls

---

$ nybls probe "https://www.youtube.com/watch?v=F2FmTdLtb_4"

id: F2FmTdLtb_4
title: System Design Concepts Course and Interview Prep
duration: 53.6 min · 1280x720 · 64 scenes
transcript: captions:video.en.vtt → ~/.nybls/store/F2FmTdLtb_4/transcript.txt
budget: 200 image units
cost so far: 0 images. Read the transcript first; then request sheets.

$ nybls study F2FmTdLtb_4 --adaptive

probing every 5s at low resolution (no vision cost)...
probed 644 frames free → kept the 72 biggest changes (median change score 1.56)
study pass: 72 tiles every 5s → 12 sheets (12 of 200 budget units)
  sheet_000.png  [00:07-01:52]
  sheet_002.png  [10:30-19:32]
  sheet_007.png  [34:12-37:30]
  ...

$ nybls ledger F2FmTdLtb_4

watched 54 min · examined 12 images (~12,480 visual tokens, ≈$0.02 at sonnet-5 input rate) of ~77,226 total frames · budget 12/200 units

A 54-minute course. 12 images. Two cents. Output from a real run, with the paths elided. The cost is an estimate at one model's input price.

It pulled 644 frames for free to work out which 72 were worth looking at, then packed them into 12 sheets. Pulling frames is nearly free, but looking at them is what costs money, so it only looks at what changed.

The frames it picked were the right ones. A single sheet from the middle of the course covers a tradeoffs diagram, the throughput versus latency slide, TCP against UDP, a TCP header breakdown, the networking infrastructure slide, and live browser devtools showing a real HTTP request. Six distinct teaching moments, and not one redundant shot of a talking head.

You do not have to ask it anything to get that. Pasting a URL on its own means "digest this", and that is the common case.

Why this exists

Almost every tool that claims to watch YouTube is really just reading the captions. That is not a cheap shot, it is what the code does: LangChain's YoutubeLoader, for one, wraps youtube-transcript-api, and Google's own documentation says a YouTube source in NotebookLM imports "only the text transcript."

The consequence is that a video with no speech returns nothing at all, a chart might as well not be there, and a demo that shows rather than tells is invisible. Worse, you cannot tell a tool that looked from one that guessed, because neither of them shows you any evidence.

The other half of the field takes the opposite approach and dumps a hundred evenly spaced frames on the model in the hope that something useful lands. That is expensive, mostly redundant, and still blind to anything that happened between two samples.

Install

Any agent, one line — installs the watch skill into the agents the skills.sh ecosystem supports (Claude Code, Codex, Cursor, Gemini CLI, and dozens more); installs count on the public leaderboard:

npx skills add nKOxxx/nybls

Or let Claude do it. Paste this repository's URL into Claude Code and say "install this". INSTALL.md is written to be followed by an agent: it detects the platform, installs what is needed, verifies the result with nybls doctor, and is told to stop and say so rather than improvise if something cannot be installed.

By hand, two lines:

brew install ffmpeg pipx

pipx install "nybls[download]"

Use pipx, not pip. Homebrew's Python, and most current Linux distributions, refuse pip install into the system environment under PEP 668, so pip install nybls fails outright on a normal modern Mac. pipx is built for command line tools: it puts nybls in its own environment and the nybls command on your PATH. If you would rather not add pipx, a virtual environment works too: python3 -m venv ~/.venvs/nybls && ~/.venvs/nybls/bin/pip install "nybls[download]".

ffmpeg is the only thing that needs a system package manager, because nothing else can decode a video. On Linux that line is sudo apt install ffmpeg pipx or your distribution's equivalent; INSTALL.md has the table. The [download] extra pulls in yt-dlp for URLs and adds about 25 MB. Plain pipx install nybls works fine on local files.

That is the whole setup. There are no API keys, no account and no telemetry, because your agent brings its own model and everything else runs on your machine. The base install comes to about 50 MB, which is numpy and Pillow and nothing else. It was 293 MB before 0.8.0, and the difference was two libraries we were using one function from each.

Videos without captions need speech recognition, which is one more line (brew install whisper-cpp). The model fetches itself the first time you need it, and the default is 141 MB rather than 1.5 GB.

nybls doctor

tells you what works, and what any missing piece would unlock.

What it does

probe	Ingest a URL or a local file, pull captions or transcribe locally, map the scenes. Costs nothing.
sheet	Six timestamped thumbnails in one image, so the model can see where to look before it spends.
frames	Full resolution stills at the moments that turned out to matter.
zoom	Crop into one frame to read small text, a chart axis, a log line.
study	Cover a whole video properly, for when you want to learn it rather than query it.
speakers	Work out who is on screen across a recorded call, from free probes.
digest	Index the on-screen text of every frame for free: a grep-able OCR index, a browsable contact sheet, a scene timeline.
verify	Check every cited timestamp against the transcript, mechanically.
contract	Shape an extraction for a purpose: teach, rebuild, procedure, brief.
corpus	Put several videos from one source on a timeline and see what changed.
extract-check	Validate an extraction against the shape it claims, so a malformed one fails loudly.
ledger	What you spent, in images, tokens and cents.
serve, inbox, approve, reject	Receive videos shared from your phone, held pending until you approve each one.

Everything lands in ~/.nybls/store/<id>/ as ordinary PNGs and text files. There is no API to integrate against, because your agent simply reads the files.

With Claude Code

The plugin drives the CLI, it does not replace it, so install the CLI first or the skill will have nothing to run:

brew install ffmpeg pipx
pipx install "nybls[download]"
/plugin marketplace add nKOxxx/nybls
/plugin install nybls@nybls

Then restart Claude Code, or run /reload-plugins. A newly installed plugin does not load into a session that is already running, and neither command says so.

Then just ask it about a video. The plugin ships the protocol itself, including a mandatory confidence check on every round, a rule that the model has to name what it is missing before it may ask for more, and a receipts contract. The discipline travels with the tool instead of depending on how carefully you prompted.

With any other agent

nybls never calls a model, so it does not care which one you use. Codex CLI, Cursor, Aider and the rest read AGENTS.md, which carries the same protocol the plugin ships. Any agent that can run a shell command and open an image can use it. Chat apps that cannot run local commands cannot.

The ledger prices images at Claude Sonnet 5 rates by default. Pass the model you are actually billed on and it recomputes the token count with that vendor's own formula, not just the price:

nybls ledger <id> --model gpt-5.5     # or export NYBLS_MODEL=gpt-5.5
nybls ledger --models                  # every model it knows, with sources

Each vendor counts differently, and each one also shrinks large images before counting. Anthropic bills 28 pixel cells with a cap per model tier, 1568 tokens on models before Claude 4.7 and 4784 after. OpenAI's newer models bill 32 pixel patches times a multiplier, up to a per-model patch budget; its older ones bill 512 pixel tiles plus a base. All of that is modelled, and the tests reproduce every row of Anthropic's own published resolution table exactly. The figures were read from the vendors' documentation on 2026-09-15 and each entry names its source.

Two honest limits. Grok 4.6 is listed at its published price with the token count marked UNVERIFIED, because xAI publishes no image token formula, and the tool will not pretend otherwise. And ledger entries written before 0.10.0 have no stored image sizes, so for those only the price changes between models; the summary says so when that happens.

Reviewing a recorded call

When a call is recorded in speaker view, the picture cuts to whoever is talking, which means a shot change is a turn boundary. nybls already detects visual change for free, so it can work out the shape of a conversation without spending anything on vision.

$ nybls speakers my-client-call.mp4

probed 600 frames every 3s over 30 min, free (no vision cost)
6 distinct shots · picture changes every 9s

  shot 0   48.5%   14.6 min  ###################
  shot 1   37.8%   11.3 min  ###############
  shot 2    5.7%    1.7 min  ##

You tell it once which shot is you and who the others are, and from that point every line of the transcript has a speaker attached. That is enough to answer the questions people actually want answered about their own calls: how much of it did you talk, did you ask something and then fill the silence yourself four seconds later, and which buying signal went past unremarked.

Naming solves identity and the video solves segmentation, so both halves are needed. If the camera was off there is nothing to attach a name to.

Reading the frames nobody looks at

probe answers what was said, for free, because captions and whisper are free. What was shown used to be a different story: the only way to know a slide, a dashboard or a leaderboard was on screen was to spend vision tokens looking. The digest pass closes that gap without spending anything:

$ nybls digest <id>

digest: 957 frames at 30s intervals (extracted now), OCR via tesseract...
ocr:      792/957 frames with text (83%)  -> ~/.nybls/store/<id>/digest/ocr-index.jsonl
scenes:   142 screen changes  -> ~/.nybls/store/<id>/digest/scenes.txt
browse:   ~/.nybls/store/<id>/digest/contact_sheet.html

One uniform frame pass (ffmpeg, a JPEG every 30 seconds, or whatever --every you ask for), one OCR pass over all of it — Apple Vision if the macos extra is installed, tesseract otherwise, never a model — and three artifacts land under digest/. ocr-index.jsonl turns "somewhere in nine and a half hours of stream there was a pricing table" into a one-line grep. contact_sheet.html is for human eyes: every frame in order, timestamped, with its OCR text under the tile so the page itself is find-in-page searchable. scenes.txt flags scene-change hints by JPEG-size delta, which is how you find the moment the demo started without looking at anything.

The pass costs zero vision tokens — the ledger is never touched — so running it is never a budget decision. nybls corpus <name> --digest shows which videos in a collection are still missing theirs.

Two honest limits. OCR reads text, not pictures: a chart's shape or a hand-drawn diagram still needs eyes, yours or the model's. And a 30-second grid can miss a flash shorter than that, which is what frames --scene is for afterwards.

You do not need to ask it anything

Paste a URL and stop there. That is a complete instruction, and it is the common case.

https://www.youtube.com/watch?v=...

With no question attached, nybls reads it as "digest this" and runs study mode: it covers the whole video, end to end, and hands back what the video actually contains. Spending as little as possible is the wrong goal here. A 48-minute lesson answered from a single image describes the layout instead of the content, so study mode deliberately does not stop early.

Ask a question instead and it switches to answer mode, which stops the moment it can answer. A 35-minute video, asked only what happens in it, came back for three images and about a cent.

You never choose between them. The presence or absence of a question decides it.

Turning a video into something that lasts

A digest you read once and lose is not much better than watching it yourself. If you want the video to leave something behind, a set of notes, a decision record, a procedure your agent can follow later, ask for a contract:

nybls contract --purpose "teach me the technique in this video" --shape teach

Four shapes, each a different job. teach pulls out concepts, prerequisites, worked examples and the errors people make. rebuild pulls out decisions and the reasoning behind them, so you can build your own version of what you watched. procedure pulls out ordered steps with checkpoints. brief pulls out claims and what backs each one.

nybls extract-check <id> --file out.json then checks the result against the shape it claims to be, so a malformed extraction fails loudly instead of looking finished. Every field carries a timestamp, and nybls verify checks those citations against the transcript mechanically. So what you keep is grounded in the video rather than in the model's memory of it, which is the difference between notes you can trust and notes you have to re-check.

Why it is cheap without being lazy

Scene detection turns out to be the wrong signal for a fixed camera. A 48-minute chess lesson filmed as one continuous screen capture produces three scene cuts, while the board, which carries all of the information, changes every move. Slide decks, IDEs and dashboards all have that same shape.

So --adaptive samples on a clock instead, scores each probe by how much the picture actually changed, and spends the budget where the changes are biggest. On that lesson it caught an on-screen framework card that a uniform pass every 30 seconds sampled straight past.

The deeper pattern is that the value of looking runs inverse to whatever the transcript already carries. A podcast barely needs frames at all. A silent screen recording, whether that is a build-in-public demo, a dashboard or a tutorial set to music, has no transcript worth reading, and the frames are the only place its content exists. Four short reels with one-line captions turned out to hold an entire data pipeline architecture, complete with table names and retry logic firing, along with a product restructure that no caption mentioned anywhere.

Does it actually work

Four rounds, 12 videos, 56 audited questions scored two points each (112 points). A control arm that gets a uniform grid of frames scores 83 of 112, or 74 percent, for 764,400 visual tokens served. nybls scores 101 of 112, or 90 percent, for 189,052. The control is served 4.0 times the tokens for 82 percent of the score.

Read this as a pilot, not a verdict. It is 12 videos, the judges are language models, the control is a uniform grid we built ourselves and not another product, and there are no confidence intervals. Round 4, the best designed one, is four silent screen recordings, which is the case this tool is built for. Two of its registered predictions missed their thresholds and are reported as missed in the results, which also records the correction that raised the control's cost from 645,120 to 764,400 tokens.

Round 4 was run properly, which matters more than the numbers. The questions and the ground truth were written by an agent with no access to nybls, working from a dense uniform grid. The arms answered separately. A third agent scored them with the arms labelled X and Y and the mapping withheld, so the scoring was blind. Earlier rounds had the experimenter write the questions and mark the results, which is exactly the weakness you would expect it to be.

The mechanism held up too. Every question was labelled at design time as persistent or transient, and on silent screen recordings the two arms tied 18 to 18 on persistent items. Every single point that separated them came from transient ones, and the control's failures were all on things that were on screen for under two seconds. That is the argument for adaptive sampling, tested directly rather than asserted.

Four earlier questions were voided when an audit found the narration stated the answer, and that audit is published alongside the results.

Testing the premises, not just the claims

For eight releases the first command in this README, pip install nybls, did not work on Homebrew Python or most current Linux, because PEP 668 refuses it. The tests and the benchmark were aimed at the claims, not at whether a stranger's first command runs. So the install is now executed rather than described: scripts/check_documented_install.py runs the exact commands from INSTALL.md on a clean runner on every push, weekly against PyPI, and as a gate before release. The rules that came out of it are in TESTING.md.

Honest limits

• No live video. Download first, then analyse.
• Motion is lossy, because it works from stills. That suits talks, tutorials, demos, dashboards and anything with text on screen, and it does badly on sports and fast action.
• Speech only, meaning words rather than music or sound events.
• Call review is new and thinly tested. The shot clustering has been measured properly on one call. Speaker view works; gallery view has not been tested; and during a screen share nobody's face is on screen, so attribution stops exactly where a demo begins. Screen time is also a proxy for talk time rather than a measurement of it.
• Platforms fight downloaders, so keep yt-dlp current and expect the occasional breakage.
• Still early. Interfaces may change before 1.0. See CHANGELOG.md.

Video content is treated as data and never as instructions. A video can put text on screen aimed squarely at your agent, and the protocol requires that it be reported to you rather than obeyed.

Docs

Install · Testing standards · Protocol · Research and evidence · Security · Changelog

Benchmarks: round 1, with a superseded early tally · round 2 · round 3 · round 4, blind judged · question audit · where information lives · change detection signals

MIT.