Framegaze — Give your model eyes Skip to content

Video perception for language models · MCP

Give your model eyes.

Claude and GPT don't take video. Frame samplers drop the flash, the cut, the glimpsed object, and the sound. Framegaze lets a language model actually watch.

Request early access See the benchmark
clip.mp4 · 72 frames · 24 fps 1 fps sample caught by Framegaze

01 — The problem

Models are watching a slideshow.

Video doesn't go in

Claude and GPT accept no video at all. Tools cut it into frames, usually one per second.

Sound is lost

The audio usually disappears entirely: no speech, no laughter, no bang.

Fast things vanish

Even models that hear audio look at only 1–2 frames per second. A flash, a 2-frame insert, or an object crossing between samples simply doesn't exist.

02 — How it works

Look at everything. Show what matters.

  1. 01

    Attention on every frame

    Each frame gets a cheap check. Expect the next frame to be the last one shifted by motion; whatever motion can't explain is surprise. It is measured per cell and normalized to that cell's habit, so a speck in a corner stands out even on busy footage.

  2. 02

    Sound on the same clock

    Speech with words and timestamps (Whisper). What is making noise — laughter, applause, sirens, music (CLAP, zero-shot). Loudness. And attention per frequency band: a steady beat doesn't surprise, a new sound does.

  3. 03

    Spend pixels where it matters

    The model receives a protocol of what is seen, heard and said, and when; one storyboard image of the whole clip; and contact sheets before, at, and after each event.

  4. 04

    Rewind

    When something needs a closer look, the model asks for it — dense frames of a span, the sound and words of a stretch, or a single frame up close.

03 — Benchmark

Every modality of an event, caught.

83%
strict · 57 of 69 events
94%
loose · 65 of 69 events
Framegaze · strict
83% 57/69
Framegaze · loose
94% 65/69
Uniform · 2 fps
9% 6/69
Uniform · 1 fps
3% 2/69
Framegaze by event type · loose scoring
{{ d.name }}
{{ d.hit }} / {{ d.total }}
Frame budget · frames shown to the model
Framegaze
493
Uniform · 1 fps
367

04 — Works where you work

Local-first. MCP-native.

Runs on CPU on your own machine. Nothing is uploaded. Ships as an MCP server for Claude Code, Claude Desktop and any MCP client.

Six tools
  • watch
  • show
  • look
  • listen
  • frame
  • history
mcp · framegaze
# the model rewinds while it answers
› look("1:23", "1:25")
↳ dense frames of the span
› listen("1:20", "1:30")
↳ sound and words, timestamped
› frame(83.4)
↳ one frame, up close

05 — Where it helps

Anywhere a frame matters.

A

Checking edits

Find stray flashes, flash frames and glitches before the cut ships.

B

QA of screen recordings

Catch the toast that appears for a moment and the error dialog that blinks past.

C

Short-form video

Analyze fast cuts, on-screen text and sound cues the way a viewer meets them.

D

Meetings and calls

Review what was said, what was shared on screen, and when.

Let your model watch.

Early builds go to a small group first.

Your mail app should open with a ready message. If it doesn't, write to [email protected].