Over the past year, we’ve been deploying video AI agents into workflows at some of the world’s largest media enterprises.

A recurring challenge in these deployments is answering several questions about the same footage: Is this live play or a replay? Which person is visible? Does this segment need a closer review? The footage may be complex, but each question often needs just one answer from a small set of choices.

Vision-language models (VLMs) typically generate those answers token by token. For workflows that ask many such questions, we wanted a faster way to return the required labels—and a way to reuse the processed footage when new questions follow.

Work on Jev, SemIf and CLM offered a starting point: models can make fixed-answer decisions from text by scoring the allowed answers directly.

We adapted the finite-answer approach to open VLMs. Our implementation reads frames or video clips as evidence and returns probabilities without retraining the model or adding a classification head. It also reuses cached visual context, so follow-up questions about the same footage can build on work already done.* ‡

How we adapted the VLM

We found that a VLM can handle these decisions with two changes at inference time.

First, following SemIf’s work with open language models, we score the token paths for the allowed answers and turn those scores into probabilities. This lets us answer several questions from the same visual context without generating each answer as text.

Second, we reuse the VLM’s processed input. System instructions, a task hint, and video frames form a shared prefix; the questions come last. As the figure below shows, Calls A and B can ask different questions about the same cached clip. Call C can add the selected answers from A and B as text before asking a question that depends on them.

We implemented answer scoring in our llama.cpp fork, using its prompt cache to reuse the context. The repository also contains the scripts for the experiments below.

But does it work? VLM vs. VLM in System1 mode

Putting a VLM in System1 mode should make classification faster without sacrificing accuracy. We started with MNIST handwritten digits so we could measure both on a simple image task before moving to video.

In normal structured-output mode, Gemma 4 writes a schema-constrained JSON digit for each image, token by token.

In System1 mode, the same VLM scores the allowed labels 0–9 directly, sharing one input across 16 questions instead of writing 16 JSON fields. The benchmark covers all 10,000 MNIST test images in 16-image batches on an A100 GPU.

System1 classified 9,501/10,000 digits correctly (95.01%) in both runs; normal structured output got 9,480/10,000 (94.80%). On this test set, System1 recorded 0.21 percentage points higher accuracy while making the answer step much faster.

Answer step per image: 34.3× faster

Full request per image: input processing + answers

From raw game frames to actions

After MNIST showed that System1 could speed up answer scoring without sacrificing accuracy, we wanted to test it on a multimodal decision task where the visual input changes after every action. Jev’s Doom demo and CLM’s T-Rex and Super Mario experiments pointed us toward games. Their agents acted on text descriptions of game state; we wanted to find out whether an open VLM could make decisions from the raw frames.

On an A100, we gave a quantized Qwen3.5 9B one unmodified Q*bert frame per decision. It made two sequential calls: first, it described the cube colors, Q*bert, enemies, score, and lives as a predicted text state; then, using that state and the frame, it chose the next move. We compared native structured JSON output with System1 scoring of the allowed answers.

Answer step means count JSON generation or System1 scoring after input processing. Full time includes both calls, image processing, input prefill and answer scoring.

Each column averages its own episode, so these are descriptive timings rather than paired measurements on identical frames. Video playback time is accelerated and should not be used to estimate inference latency.

Where this fits into media workflows

Many media workflows ask several small questions about the same footage. Together, the answers determine what gets tagged, reviewed or sent to another system. Fixed-answer scoring could make those repeated decisions faster, while caching could avoid processing the same visual input for each new set of questions.

The following are applications to evaluate beyond the experiments above. Each starts with a specific task and an output the team can use.

Help review teams find segments that need attention

A content-review workflow could begin with checks for visible material such as cigarettes or nudity in defined video segments. Several checks could share the same visual context. Flagged segments would then go to a reviewer or a model that can examine the surrounding context and explain the concern.

The output would be a review queue linked to the footage. The team would define the checks and assess missed material, false flags and review effort. A visual flag alone would not establish whether a programme meets a policy.

Turn training footage into useful catalogue metadata

A fitness team could use several questions to tag a training clip: which equipment is in use, whether the trainer is seated or standing, and whether they are pedalling. A later call could reuse the clip for another set of labels.

Those answers could populate search filters and catalogue fields, using the team’s definitions and review rules. Producing precise equipment-control instructions would require additional timing and validation work.

Make appearances searchable in broadcast archives

For a programme with a known presenter and guest roster, sampled frames could be checked against that roster, including an “unknown” or “none” option. Repeating the checks across predefined segments could produce an appearance index linked to the source footage.

The index would help archive users find candidate footage. Identifying who is speaking or what they are discussing would require additional evidence. The workflow would also need evaluation for missed appearances and incorrect matches.

Narrow sports footage before detailed analysis

Short sports segments could be checked for live play versus replay or classified against a defined set of events. Those results could narrow the footage passed to a more expensive model for detailed description or precise clip boundaries.

The output would be candidate segments for the next stage of the workflow. Classifying a segment as containing a goal would not locate the exact second it occurred or produce a finished highlight. Evaluation would need to account for relevant footage filtered out before deeper analysis.

Fixed-answer scoring applies when the output can be expressed as a predefined set of labels. Tasks that require descriptions or explanations still need generative output.

Evaluate classification accuracy and end-to-end latency on representative footage, including input processing and downstream steps. Prefix caching reduces repeated input processing only when calls share an unchanged prefix; new footage must still be processed.

What’s next

We are applying this approach to training and sports footage: tagging exercises and movement phases, marking plays and athlete states, and adding search labels to video libraries.

Our open-source implementation includes the inference changes and scripts to reproduce the MNIST and Q*bert experiments on your own hardware.

* At the time of writing, OpenAI had announced a limited-preview Decisions API using a fine-tuned version of Luna for finite-answer questions with text or image context.

‡ Post-publication update: Cloudflare later released Clef, a multimodal decision model post-trained from Qwen3.8-27B. Its release shows growing interest in multimodal decision systems.