framefinder

framefinder

Film editors scan a reel on contact sheets. framefinder does the same for AI: one image per minute of video, with numbered tiles, so a vision model can say "it's at 23:40" and only look closely where it matters.

framefinder index talk.mp4 --no-asr --no-timeline
framefinder ask talk "a slide about robotics" --mode visual
# → 23:40  p=0.99  verified   (frame saved to disk)

It finds things that are shown, not just said: a slide, a product, a person walking in, an error dialog, a black screen. You don't need a transcript, training, or a vector database.

---

The trick

A video is thousands of images. Showing an AI every frame is slow, so framefinder lays the video out the way a film editor would: on contact sheets, one per minute, with every tile labelled.

 1 hour of video
   └─ ffmpeg: 1 frame per second            → 3,600 frames
       └─ tile 60 frames onto one image     → 60 contact sheets (one per minute)
           └─ label each tile T01 … T60
               └─ ask an AI, for every sheet in parallel:
                    "Is <your question> on this sheet?"      yes/no + probability
                    "Which tile shows it most clearly?"      T01 … T60, or none
                       └─ take the best few sheets and re-check frame by frame,
                          every 0.5 s around the picked tile
                              └─ timestamp, confidence, and the full-size frame

Why it works

• Numbered tiles let the model point at a specific second, not just "somewhere in this minute."
• Screen first, then verify. The rough pass ranks the minutes well even when it can't see small details. The frame-by-frame pass confirms the moment or rejects it.
• One question per judgment. "Is it here?" and "which tile?" are asked separately and combined in code, which is more reliable than one compound question.
• Shuffled re-checks. Borderline sheets are asked again with tiles and options reordered, because models tend to favour the first option.

Real result

A 57-minute recorded conference talk, searching "a slide about robotics", with no transcript:

Sheets	60 (built locally in 12 s)
Hits	23:40, 24:08, 25:06, 27:02, 28:20, all verified at 0.98–0.99
Correct?	Yes: every verified hit was one of the talk's robotics slides
Cost	~$0.02–0.04 with gpt-5-mini
Time	1–3 min (limited by API rate limits, not compute)

When to use something else

• You only care about what was said. Search the transcript. It's free.
• You'll search the same large library over and over. Build an embeddings index once. Each search is then nearly free, while framefinder pays per question.
• You need a managed, supported service. Use a cloud video indexer (for example Azure AI Video Indexer).

framefinder is best for one-off, plain-English questions about what's on screen.

---

Setup

You need Python 3.11+, ffmpeg (with ffprobe) on your PATH, and your own access to a vision model. framefinder doesn't ship with keys or a hosted service.

git clone <this repo> && cd framefinder
uv venv && uv pip install -e .          # or: python -m venv .venv && pip install -e .
cp .env.example .env                    # fill in ONE option, then: set -a; source .env; set +a

Pick one model provider (details in .env.example):

Option	Set	Notes
OpenAI	OPENAI_API_KEY, optional FRAMEFINDER_MODEL	Simplest. Default model gpt-5-mini.
Azure OpenAI / Foundry	AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT, plus az login or AZURE_OPENAI_API_KEY	Keeps footage inside your own Azure tenant.
OpenAI-compatible	OPENAI_BASE_URL, OPENAI_API_KEY, FRAMEFINDER_MODEL	Local or hosted. The model must support images and JSON-schema output.

The model needs to read images and support structured (JSON-schema) output. Tested with gpt-5-mini on Azure.

Privacy: video frames are sent to whichever provider you configure. Check that you're allowed to send the footage there.

Commands

Command	What it does
index <file or URL>	Builds contact sheets (local, free). --no-timeline skips the optional per-minute summary pass.
ask <video> "<what>"	Finds moments. --mode visual ignores speech. --start/--end limit the range.
find <video> <photo>	Finds every moment that matches a reference photo.
locate <video> <time> --target "<what>"	Draws a box around something in one frame and saves the crop.
zoom <video> <start> <end>	Saves full-resolution frames.
whole <video> "<question>"	One answer about the whole video.
lens, query	Your own questions asked of every minute, then filtered for free.
mcp	Runs as an MCP server so coding agents (Claude Code, Cursor, …) can use it.

<video> is the ID from index or list, or the first word(s) of the title.

Tuning

Variable	Default
FRAMEFINDER_REASONING_EFFORT	low	For gpt-5-family models. minimal missed captions in our tests.
FRAMEFINDER_CONCURRENCY	3	Raise it if your rate limit allows. Lower it if you see HTTP 429 errors.
FRAMEFINDER_VERIFY_TOP	6	How many top windows get the frame-by-frame check.
FRAMEFINDER_TILES_PER_SHEET	60	Fewer tiles means bigger thumbnails: more accurate, but more requests.
FRAMEFINDER_USD_PER_M_INPUT	per model	Sets the price used in the cost estimate.

Limits

• Confidence scores are the model's own estimates, not calibrated. Tune --threshold on your own footage.
• Unverified hits can be related rather than exact (for example, "physical objects" for "robots").
• Events shorter than about a second can be missed if their minute isn't ranked in the top few windows.
• It doesn't hear sound. Speech is only available through captions or local transcription (pip install -e .[asr]).

Credits

framefinder is a fork of jevreel by Madhav Gupta (MIT). The contact-sheet method, screen-then-verify search, lenses, caching, and most of the code are his. jevreel was built for TypeSafe's Jev decision model. framefinder removes that dependency and runs on any OpenAI-compatible vision model.

MIT licensed. See LICENSE.