Video benchmarks are meant to show whether a model can understand video. We attacked 115 of them with five kinds of shortcuts that do not need the capability a benchmark claims to test, and 35 of them break without the attacker seeing a single frame. After screening out the questions that shortcut attacks can solve, we select the 840 hardest to form Video-Index.
视频 benchmark 本该检验模型能不能看懂视频。我们用五类不需要所考察能力的捷径去攻击 115 个 benchmark，其中 35 个连一帧画面都不用看就被攻破。筛掉捷径攻击能解的题之后，我们选出最难的 840 道题，组成 Video-Index。
Hover a tile for its audit record, or click it to open it in the benchmark table further down.悬停一块查看它的审计记录，点击在后文的 benchmark 表格中打开。
The question问题
Video benchmarks decide which models the field trusts. A model that scores well on the right ones is trusted to follow a soccer match, an egocentric kitchen, or an hour-long lecture. In the past five years, 605 video benchmarks have appeared, 342 of them with public data. The 115 multiple-choice benchmarks we audit hold 528,172 questions over 123,281 videos.
视频 benchmark 决定这个领域信任哪些模型。一个模型只要在对应的 benchmark 上拿到高分，就会被认为能看懂一场足球赛、一段第一人称视角的厨房录像、或者一小时的课程录像。过去五年里出现了 605 个视频 benchmark，其中 342 个数据公开。我们审计的 115 个选择题 benchmark 共有 528,172 道题、123,281 段视频。
Each benchmark was built against the models of its year. Language priors are stronger now, one frame carries more of the answer, and a deployed system can remember every item it has been asked. A question that forced a 2023 model to watch the video may need no video in 2026. So we asked the same question of every benchmark, namely whether it can still tell if a model has the capability it was built to test.
每个 benchmark 都是对着它那一年的模型造出来的。如今模型的语言先验更强，一帧画面里能读出的答案更多，一个部署中的系统还能记住它被问过的每一道题。一道在 2023 年迫使模型看视频的题，到 2026 年可能根本不需要视频。所以我们对每一个 benchmark 都问同一个问题，现在来看，它还能不能测出模型是否具备它要考察的能力？
The attack pyramid攻击金字塔
An attacker is any strategy that raises the score without the claimed capability, and no attacker sees the correct answer of the item it answers. We arrange attackers in five levels of growing access to an item, shown below, and call the arrangement the attack pyramid, since fewer benchmarks survive each level than reach it.
攻击者指任何不靠所声称的能力却能抬高分数的策略，任何攻击者都看不到当前这道题的标准答案。我们把攻击者按对题目的访问权限由少到多排成五层，见下图，称之为攻击金字塔，因为每一层幸存的 benchmark 都比到达这一层的少。
Five levels五个层级
The answer options alone. A fixed answer position, and Claude Opus 5 reading the options.
只有答案选项。固定答案位置，以及只读选项的 Claude Opus 5。
Adds the question. Gemini 3.5 Flash and Claude Opus 5 reading text only.
加上问题。只读文字的 Gemini 3.5 Flash 和 Claude Opus 5。
Adds the other evaluated items. Copying the nearest earlier answer, Qwen3-8B with retrieved pairs, a classifier on question embeddings.
加上其他被评测的题目。照抄最近的已答题、带检索的 Qwen3-8B、问题向量上的分类器。
Adds one frame or captions. Gemini 3.5 Flash on one frame, Claude Opus 5 on AuroraCap captions of 32 frames.
加上一帧或帧描述。看单帧的 Gemini 3.5 Flash，读 32 帧 AuroraCap 描述的 Claude Opus 5。
Adds the video, shuffled or a contiguous tenth. Gemini 3.5 Flash and Qwen3-VL-8B.
加上视频，但帧序打乱或只取连续的十分之一。Gemini 3.5 Flash 和 Qwen3-VL-8B。
The break rule攻破规则
The exploitability of a benchmark at a level is the largest margin over chance that any attacker of that level reaches. A reference model, Gemini 3.5 Flash at 32 frames, sets the full-protocol accuracy. A benchmark breaks at the first level whose exploitability comes within 5 points of the reference's own margin over chance, about the sampling noise of a 300-item accuracy. A benchmark that no level reaches within that tolerance is unbroken. Every benchmark takes every level, since a score is read as a video score whatever the benchmark claims to test.
一个 benchmark 在某一层的可利用度，是该层任一攻击者超出随机水平的最大幅度。参照模型是看 32 帧的 Gemini 3.5 Flash，它给出完整协议下的准确率。当某一层的可利用度距参照模型自身超出随机水平的幅度不足 5 个点时，benchmark 就在这一层被攻破；5 个点大约是 300 道题的准确率的抽样噪声。没有任何一层在这个容差内追上参照模型的 benchmark 记为未攻破。每个 benchmark 都要过全部五层，因为不管它声称测什么，分数都会被当作视频理解的分数来读。
The protocol in detail协议的细节
- The 115 benchmarks are English multiple-choice video benchmarks that do not require audio. We sample 300 questions per benchmark, preserving subtask proportions and keeping every question when fewer are available.115 个 benchmark 都是不依赖音频的英文选择题视频 benchmark。每个 benchmark 抽 300 道题，保持子任务比例，不足 300 道时全部保留。
- Videos are stored at a 720-pixel short side, reduced to 480 for clips longer than 30 minutes without on-screen text. Frames reach the models with a long side of at most 768 pixels.视频按短边 720 像素存储，超过 30 分钟且没有画面文字的片段降到 480。送入模型的帧长边不超过 768 像素。
- Gemini 3.5 Flash at 32 frames supplies the reference accuracy. On long-video benchmarks where it stays within the tolerance of chance, its 128-frame probe supplies the reference instead.参照准确率由看 32 帧的 Gemini 3.5 Flash 给出。在它与随机水平的差距不超过容差的长视频 benchmark 上，改用它的 128 帧探针作为参照。
- Every model answers once per item, without tools, retries, or an agent scaffold. Answers are scored by option letter, then by an answer-line regular expression, then by option text.每个模型对每道题只答一次，不用工具、不重试、不套 agent 框架。答案先按选项字母判分，再按答案行的正则表达式，最后按选项文本。
- Within each benchmark we keep one item per near-duplicate group, which removes 1.1% of sampled items, refill to 300 where possible, and screen the replacements.每个 benchmark 内每组近似重复题只留一道，去掉 1.1% 的抽样题目，能补的补回 300 道，补进来的题同样筛查。
- Items carry 18 fine capability categories in four groups, namely perception, temporal, spatial and physical, and reasoning and knowledge. Each benchmark takes the group that most of its items require.题目标注 18 个细粒度能力类别，归为四组：感知、时序、空间与物理、推理与知识。每个 benchmark 归入多数题目所需的那一组。
What each level takes每一层拿走什么
Across the 115 benchmarks, 17 break on the options alone and 13 on the question text. Another 5 break on the other evaluated items, 27 on a single frame or captions, and 15 under temporal perturbation. 38 benchmarks survive every level. Each level below also shows one sampled item. Its box gives the question and the options with the correct option in bold, the text above it states what each attacker saw and chose, and frames appear only where an attacker saw visual input.
在 115 个 benchmark 上跑完五层，17 个只靠选项就被攻破，13 个靠问题文本，5 个靠同批的其他题目，27 个靠单帧或帧描述，15 个在时序扰动下被攻破。38 个 benchmark 全部幸存。下面每一层还各配一道抽样题：框里是问题和选项，正确选项加粗；框上方的文字说明每个攻击者看到了什么、选了什么；只有攻击者看到了画面时才附上帧。
Options and question text选项与问题文本
The first two levels never see a frame. At the option level, chance accuracy is the reciprocal of the option count. A fixed-position attacker picks the most frequent answer position, and Claude Opus 5 reads the options alone in a call that shares no context with its other roles. Options raise accuracy by up to 63 points over chance, lift it significantly on 78 benchmarks, and break 17. The option order matters as well. Across 70 complete runs, eight permutations of the option order give Qwen3-VL-8B a median accuracy range of 5 points, and the selected answer changes on a median 37% of items.
前两层看不到任何画面。在选项层，随机水平是选项数的倒数。固定位置攻击者总选最常见的答案位置，Claude Opus 5 则在一次与其他角色不共享上下文的调用里只读选项。选项最多把准确率抬高到随机水平之上 63 个点，在 78 个 benchmark 上显著高于随机水平，攻破 17 个。选项的顺序也有影响。在 70 次完整运行里，八种选项排列让 Qwen3-VL-8B 的准确率中位数波动 5 个点，中位数 37% 的题目会换一个答案。
At the text level, Gemini 3.5 Flash and Claude Opus 5 read the question and the options without the video. Question text raises accuracy by up to 71 points over chance. Among the 98 benchmarks that pass the option level, the blind reader exceeds the reference on 4 and breaks 13.
在文本层，Gemini 3.5 Flash 和 Claude Opus 5 不看视频，只读问题和选项。问题文本最多把准确率抬高到随机水平之上 71 个点。在通过选项层的 98 个 benchmark 里，盲读模型在 4 个上超过参照模型，攻破 13 个。
Options alone. ExAct breaks at the option level. The options-only attacker, Claude Opus 5, sees the five options of item 3175 without the question or the video and picks D, the correct option. D is the only option that says participant instead of player, and the only one without a reason clause. On 240 of the 300 sampled ExAct items, this attacker picks the correct option.
只看选项。ExAct 在选项层被攻破。只看选项的攻击者 Claude Opus 5 看不到问题和视频，只看到第 3175 题的五个选项，就选了正确的 D。D 是唯一说 participant 而不是 player 的选项，也是唯一不带原因从句的选项。在抽样的 300 道 ExAct 题目里，这个攻击者答对了 240 道。
Question问题 Which expert commentary correctly assesses the skill execution shown in the video?
- A. The player should focus on jumping higher before releasing the ball to generate more backspin, improving the shot's precision and reducing bounce-off potential.
- B. The player should focus on holding the elbow away from the body to create a wider arc, which will make the shot more difficult for opposing players to contest.
- C. The player should keep the fingers stiff and straight while shooting to ensure a stronger shot and prevent the ball from being underpowered or easily blocked.
- D. The participant should focus on snapping the wrist down and pointing the fingers down when shooting.
- E. The player should concentrate on flicking the wrist upwards and extending the fingers straight out when shooting to achieve more height and distance on the shot.
Question text. MMR-V breaks at the text level. Gemini 3.5 Flash and Claude Opus 5 each read the question and the eleven options of item 337 without the video, and both pick D. Only D names a specific work, while the other ten options are generic titles about ratings, technology, and the workplace. Both readers answer 124 of the 300 sampled MMR-V items correctly without the video.
问题文本。MMR-V 在文本层被攻破。Gemini 3.5 Flash 和 Claude Opus 5 不看视频，各自读第 337 题的问题和十一个选项，都选了 D。只有 D 点出了一部具体作品，其余十个选项都是关于评分、科技和职场的泛泛标题。在抽样的 300 道 MMR-V 题目里，两个模型不看视频都答对了 124 道。
Question问题 Please suggest a suitable title for this video.
- A. A Digital Society Perspective
- B. Social Ratings and Personal Perceptions
- C. Online Influence on Personal Life
- D. Black Mirror - Social Credit System
- E. Living Under the Social Lens
- F. Empathy in the Age of Technology
- G. Corporate Culture vs. Personal Image
- H. The Intersection of Ratings and Reality
- I. Navigating Social Standards
- J. The Power of Community Approval
- K. Workplace Dynamics in a Digital World
Other evaluation items同批的其他题目
The pool level asks whether the answers to other items help solve a question. A copy attacker copies the answer of the nearest earlier question when their BGE-large embeddings are similar enough. A retrieval attacker gives Qwen3-8B the eight nearest question-answer pairs. Following the predictive auditing of Brown et al.1, a logistic regression on question embeddings predicts answer letters out of sample. The pool is ordered at random, and each item is answered before its ground truth enters the pool. The gap between the strongest attacker and the reference narrows as more items enter the sequence, and 5 benchmarks break.
题库层问的是其他题目的答案能否帮助解题。照抄攻击者在 BGE-large 向量足够相似时，直接照抄最近的一道已答题的答案。检索攻击者把最近的八个问答对交给 Qwen3-8B。按照 Brown 等人1的预测式审计，一个基于问题向量的逻辑回归在样本外预测答案字母。题库按随机顺序排列，每道题都在它的标准答案进入题库之前作答。随着进入序列的题目越来越多，最强攻击者与参照模型的差距不断缩小，5 个 benchmark 被攻破。
Other evaluation items. TimeScope breaks at the pool level. Items 1104 and 1892 come from videos of 3,600 and 180 seconds, yet they ask the same question with the same six options and the same answer. Their question embeddings coincide, and the copy attacker, which copies the answer of the nearest earlier question when the BGE-large cosine similarity exceeds 0.9, answers whichever item comes second.
同批的其他题目。TimeScope 在题库层被攻破。第 1104 和 1892 题分别来自 3,600 秒和 180 秒的视频，问的却是同一个问题，六个选项和答案也都相同。两道题的问题向量重合，而照抄攻击者会在 BGE-large 余弦相似度超过 0.9 时照抄最近一道已答题的答案，所以两道题里后出现的那道总会被它答对。
Question问题 What does the girl do after she get a cup of coffee?
- A. reading a book
- B. working with her computer
- C. taking some notes
- D. cleaning the kitchen
- E. preparing a breakfast
- F. washing the dishes
In Tables 14 and 15, superscript w marks a store of earlier items from the same benchmark and superscript c a store preloaded with other benchmarks, and n is the benchmark's full item count. εpool is the largest final-stage margin among the attackers, in %, and Att. names that attacker: L0 learns answer-position priors, L1 copies near-duplicates, L2-rule and L2-LLM use sibling items of the same video through rules or a language model, and L3 retrieves eight examples. ALC is that attacker's normalized area under the learning curve, from 0 at chance to 100 at perfect accuracy throughout.
表 14 和表 15 里，上标 w 表示题库只存放同一 benchmark 里此前的题目，上标 c 表示题库预先装入了其他 benchmark 的题目，n 是该 benchmark 的全部题目数。εpool 是各攻击者在最后阶段的最大幅度，单位为 %；Att. 指出是哪个攻击者：L0 学习答案位置的先验，L1 照抄近似重复题，L2-rule 和 L2-LLM 分别用规则和语言模型利用同一视频的兄弟题目，L3 检索八个样例。ALC 是该攻击者学习曲线下的归一化面积，随机水平为 0，全程全对为 100。
Table 15.表 15 Within-pool and cross-pool exploitability across audited benchmarks (2/2).各被审计 benchmark 的池内与跨池可利用度（2/2）。
Single frames and captions单帧与帧描述
The frame level replaces the video with a single frame or a text description. Gemini 3.5 Flash reads one frame, and Claude Opus 5 reads captions of the 32 sampled frames, each written independently by AuroraCap-7B-IMG2 without the question or options. Captions recover more of the reference accuracy than a single frame. Among the 78 benchmarks with all three probes at this level, one frame comes within tolerance of the reference on 8. It beats the blind reader by a median 3.3 points. Captions come within tolerance on 23 and exceed the reference on 9, and 27 benchmarks break at this level.
帧层用单帧或文字描述替代视频。Gemini 3.5 Flash 看一帧，Claude Opus 5 读 32 个抽样帧的描述，每条描述由 AuroraCap-7B-IMG2 在看不到问题和选项的情况下独立写出。帧描述比单帧更能追回参照模型的准确率。在这一层三种探针齐全的 78 个 benchmark 里，单帧在 8 个上追到参照模型的容差之内，比盲读模型高出中位数 3.3 个点。帧描述在 23 个上追到容差之内，在 9 个上超过参照模型，27 个 benchmark 在这一层被攻破。
Captions. NExT-QA breaks at the frame level. For item 7426, Claude Opus 5 reads the captions of 32 frames, each written independently by AuroraCap-7B-IMG, and picks C, the correct option. The captions describe a man with a small drone in a garden and later a small object resting on a white platform, and none of the 32 captions mentions a baby, a lady, or a girl. The same model shown a single frame picks B.
帧描述。NExT-QA 在帧层被攻破。对第 7426 题，Claude Opus 5 读 32 帧的描述，每条都由 AuroraCap-7B-IMG 独立写出，它选了正确的 C。这些描述写的是花园里一个男人和一架小型无人机，后来一个小物体停在白色平台上，32 条描述里没有一条提到婴儿、女士或女孩。同一个模型只看一帧时选了 B。
Question问题 why did the blade stop moving at the end
- A. baby stopped it
- B. no battery
- C. the man stopped it
- D. lady pressed a button
- E. girl pulled the sockets
Captions帧描述 “A man in a green t-shirt and black shorts stands barefoot on a lawn, facing a small, black, circular object on the ground, which appears to be a remote-controlled toy.” “The image shows a man standing in a garden, facing a drone that is in flight.” “The image captures a tranquil garden scene with a small, unmanned aircraft on the ground, a white rectangular object on the grass, and a variety of greenery.”
Tables 7 and 8 give the accuracy under each input condition. There, video cap. is a whole-video description written by Qwen3-VL-8B, which enters no attack level, Frame cap. is the caption attack, and Cov. is the share of multiple-choice items that the 32-frame run scored.
表 7 和表 8 给出每种输入条件下的准确率。其中 video cap. 是 Qwen3-VL-8B 写的整段视频描述，不属于任何攻击层级；Frame cap. 是帧描述攻击；Cov. 是 32 帧运行中得到评分的选择题所占比例。
A broader diagnostic sweep with Qwen3-VL-8B helps interpret sensitivity to visual detail and temporal coverage, and not all of its conditions are frame-level attacks. In Table 17, blind supplies no video, 1f uses position 16 of the 32-frame grid, and 32f and 128f give frame counts. The 448px and 224px columns vary resolution, shuf permutes frame order, half1 and half2 restrict input to each half of the video, and off1 and off2 shift the sampling grid. Table 18 summarizes the capability profiles derived from these conditions.
用 Qwen3-VL-8B 做的更大范围诊断扫描有助于理解模型对视觉细节和时间覆盖的敏感度，其中并非所有条件都是帧层攻击。表 17 里，blind 不给视频，1f 取 32 帧网格中的第 16 帧，32f 和 128f 是帧数；448px 和 224px 两列改变分辨率，shuf 打乱帧序，half1 和 half2 只给视频的前一半或后一半，off1 和 off2 平移采样网格。表 18 汇总由这些条件得出的能力画像。
Order and coverage顺序与覆盖
The order level tests whether accuracy depends on frame order and on coverage of the full clip. Gemini 3.5 Flash and Qwen3-VL-8B read 32 shuffled frames and, separately, a contiguous tenth of the video. Much of the score survives both. Among the 51 benchmarks with both probes at this level, shuffling retains a median 96% of the same model's full-video accuracy, and one tenth retains 78%. Across all 96 profiled benchmarks, either half of the video retains a median 92%. Of 15 benchmarks whose papers claim temporal reasoning, 6 retain over 90% under shuffling, yet 6 of the 7 that reach this level lose at least a quarter. 15 benchmarks break here, and 38 remain unbroken.
顺序层检验准确率是否依赖帧的顺序和对完整片段的覆盖。Gemini 3.5 Flash 和 Qwen3-VL-8B 读 32 个打乱顺序的帧，另外单独读视频中连续的十分之一。两种扰动下分数大多保得住。在这一层两种探针齐全的 51 个 benchmark 里，打乱帧序后中位数保留同一模型完整视频准确率的 96%，十分之一的片段保留 78%。在全部 96 个有完整画像的 benchmark 上，视频的任意一半都保留中位数 92%。论文声称考察时序推理的 15 个 benchmark 里，6 个在打乱后仍保留 90% 以上，但到达这一层的 7 个里有 6 个至少损失四分之一。15 个 benchmark 在这里被攻破，38 个保持未攻破。
Shuffled frames. MVBench breaks at the order level. For item action_prediction_88, Gemini 3.5 Flash reads the 32 frames in random order and still picks the correct next action, as it does with ordered frames. The question asks for the next action, yet the answer survives the loss of frame order.
打乱的帧。MVBench 在顺序层被攻破。对 action_prediction_88 这道题，Gemini 3.5 Flash 读打乱顺序的 32 帧，依然选出了正确的下一个动作，和读有序帧时一样。问题问的是接下来的动作，答案却不受帧序丢失的影响。
Question问题 What will the person do next?
- A. Take the shoe.
- B. Lie on the floor.
- C. Close the box.
- D. Tidy up the blanket.
In Table 19, the frame-budget ladder reads 8, 32, and 128 frames. The window probe reads 32 frames inside one random contiguous tenth of the video, and retention divides probe accuracy by 32-frame accuracy on the same items.
表 19 里，帧预算阶梯读 8、32 和 128 帧。窗口探针在视频中随机一段连续的十分之一里读 32 帧，保留率是探针准确率除以同一批题目上 32 帧的准确率。
What the audit finds审计结果
Newer benchmarks break earlier越新的 benchmark 越早被攻破
Among releases through 2024, 19% break before any visual input; among releases in 2025 and 2026, 34% do. The rise comes from the text and pool levels, where the newer releases hold 17 of the 18 breaks. A blind reader breaks 12 of the 89 newer benchmarks and 1 of the 26 older ones, mostly by coming within tolerance of a reference that scores far above chance. The option level breaks a similar share in both periods, 4 older and 13 newer benchmarks, especially where reference accuracy approaches chance. Recent benchmarks use language models to generate or refine questions and answers, which can leave the answer recoverable from the text alone.
2024 年及以前发布的 benchmark 里，19% 在看到任何画面之前就被攻破；2025 和 2026 年发布的里，这个比例是 34%。增量来自文本层和题库层，这两层的 18 次攻破里有 17 次落在新发布的 benchmark 上。盲读模型攻破了 89 个新 benchmark 里的 12 个，26 个老 benchmark 里的 1 个，多数情况是参照模型远高于随机水平，而盲读模型追到了容差之内。选项层在两个时期攻破的比例相近，老的 4 个、新的 13 个，多发生在参照模型接近随机水平的 benchmark 上。近期的 benchmark 用语言模型生成或润色问题和答案，这可能让答案单靠文字就能推出来。
Hover a bar for its benchmarks, or click it to list them in the benchmark table.悬停一段查看其中的 benchmark，点击在后文的表格中列出。
Each claim breaks differently每类能力各有软肋
Reasoning and knowledge benchmarks break on the text or on captions, which already carry the facts the question asks for. Perception benchmarks break most often at the frame level. Spatial and physical benchmarks break on the options and under temporal perturbation. Temporal benchmarks split, with 6 of 18 breaking on the options and 10 surviving every level. An audit should match its checks to the claimed capability and still screen the options of a temporal claim.
推理和知识类 benchmark 在文本层或帧描述上失守，因为问题要的事实已经写在文字里。感知类最常在帧层失守。空间和物理类在选项层和时序扰动下失守。时序类则分成两半，18 个里 6 个在选项层被攻破，10 个全部幸存。审计应当按所声称的能力选检查项，但即使是时序类也要筛一遍选项。
Hover a bar for its benchmarks, or click it to list them in the benchmark table.悬停一段查看其中的 benchmark，点击在后文的表格中列出。
Frames, pixels, and long video帧数、像素与长视频
Additional visual input can supply finer spatial detail or denser temporal coverage, and we asked which helps more. On the benchmarks that pass the pool level, Qwen3-VL-8B reads 32 frames at short sides of 168, 224, 336, and 448 pixels and at the stored resolution. It then reads 224-pixel frames sampled at 0.25, 0.5, 1, and 2 frames per second within its context. 11 benchmarks favor pixels, 3 favor frames, and the two gains differ by less than the tolerance on the rest. Resolution recovers small details in slides, endoscopy, and synthetic artifacts, and denser sampling catches brief events that sparse sampling misses.
额外的视觉输入可以提供更细的空间细节，也可以提供更密的时间覆盖，我们想知道哪一种帮助更大。在通过题库层的 benchmark 上，Qwen3-VL-8B 先以 168、224、336、448 像素的短边以及存储分辨率各读 32 帧，再在上下文允许的范围内以每秒 0.25、0.5、1、2 帧读 224 像素的帧。11 个 benchmark 更受益于像素，3 个更受益于帧数，其余的两种增益相差不到容差。分辨率找回幻灯片、内窥镜和合成瑕疵里的小细节，更密的采样抓住稀疏采样漏掉的短暂事件。
Hover a line or a name to follow one benchmark across both panels, or a legend entry to pick out its group. Click to keep the highlight.悬停在线或名字上，可在两个面板里同时追踪一个 benchmark；悬停在图例上可挑出整组。点击可保持高亮。
Long videos want more frames. Among the 30 benchmarks whose videos average at least 300 seconds, 18 gain at least 5 points from larger frame budgets in the reference sweep. Nine still gain at its 600-frame limit. In a diagnostic sweep at 224 pixels, only LongTimeScope and TimeScope keep improving at 1024 frames. Since the observed gains also depend on the reference model's long-context ability, other benchmarks may still place high demands on processing events across long time spans.
长视频需要更多帧。视频平均时长不低于 300 秒的 30 个 benchmark 里，18 个在参照模型的扫描中随帧预算增加至少提高 5 个点，9 个到它 600 帧的上限时仍在提高。在 224 像素的诊断扫描里，只有 LongTimeScope 和 TimeScope 在 1024 帧时仍在提高。观察到的增益也取决于参照模型的长上下文能力，所以其他 benchmark 仍可能对跨越长时间跨度的事件处理提出很高的要求。
Table 20 gives the reference sweep of Claude Opus 5 on the long-video set, with the same items, prompt, and scorer and uniform frames over the whole video. Median frames is the median number of frames sent at the largest budget that ran, since a video holds at most 1,024 stored frames and short videos send every frame once. Saturation is the first budget within 5% of the maximum over the budgets that ran.
表 20 是 Claude Opus 5 在长视频集合上的参照扫描，题目、提示词和评分方式都相同，帧在整段视频上均匀采样。帧数中位数是实际跑过的最大预算下发送帧数的中位数，因为一段视频最多存储 1,024 帧，短视频每帧只发送一次。饱和点是第一个落在各预算最大值 5% 以内的预算。
Errors and duplicates错误与重复
Errors concentrate in few capabilities. We attribute the reference model's 32-frame errors to the 18 fine capability categories on the 76 pool-level survivors with at least 20 agreed attributions. Claude Opus 5 receives the frames, question, options, gold answer, and the incorrect answer with its reasoning, and only attributions confirmed by two independent judgments count. The two most frequent categories of a benchmark hold a median 71% of its attributions, and profiles overlap weakly, with a mean cosine similarity of 0.29. Fine-grained action leads on 27 benchmarks, spatial relation on 13, and temporal order and action counting on 7 each. Covering the observed capability gaps therefore takes several sources.
错误集中在少数几种能力上。我们把参照模型在 32 帧下的错误归因到 18 个细粒度能力类别，对象是通过题库层且至少有 20 条一致归因的 76 个 benchmark。Claude Opus 5 拿到帧、问题、选项、标准答案以及错误答案及其推理，只有两次独立判断一致的归因才计入。一个 benchmark 最常见的两个类别占其归因的中位数 71%，各 benchmark 的错误画像重叠很少，平均余弦相似度 0.29。细粒度动作在 27 个 benchmark 上居首，空间关系在 13 个上，时序顺序和动作计数各在 7 个上。因此要覆盖观察到的能力缺口，需要多个来源。
A benchmark also lists more items than it measures. We embed each item's 32 frames with SigLIP 2 and its question with BGE-large. The Vendi score3 of a 200-item draw in the joint space is the benchmark's effective size. A 200-item draw holds a median of 32 effective items, and 39 benchmarks hold 20 or fewer. Near-duplicate questions make up at least half the items of 63 benchmarks, and weighting items by inverse duplicate-cluster size moves accuracy by more than 1 point on 93 benchmarks. Across sources, a tenth of items use videos that appear in another benchmark, and PAI-Bench-U and PhysicalAI share 191 questions.
一个 benchmark 列出的题目数也多于它真正测到的。我们用 SigLIP 2 编码每道题的 32 帧，用 BGE-large 编码问题，把 200 道题在联合空间里的 Vendi 分数3作为 benchmark 的有效题目数。抽 200 道题，中位数只有 32 道有效题目，39 个 benchmark 不超过 20 道。63 个 benchmark 里至少一半的题目是近似重复，按重复簇大小的倒数给题目加权，会让 93 个 benchmark 的准确率变动超过 1 个点。跨来源看，十分之一的题目用的视频也出现在别的 benchmark 里，PAI-Bench-U 和 PhysicalAI 共享 191 道题。
Table 16 covers 114 of the 115 benchmarks, since CG-Bench has no item embeddings. EffN is the Vendi score of a 200-item draw and VS/n that score per item, and the 8 smaller benchmarks use all their items.
表 16 覆盖 115 个 benchmark 中的 114 个，因为 CG-Bench 没有题目向量。EffN 是抽 200 道题的 Vendi 分数，VS/n 是平均到每道题的分数；8 个较小的 benchmark 用全部题目。
A near-duplicate pair needs a question cosine similarity of at least 0.9 under BGE-large, confirmed with MPNet. Table 11 lists the largest flows between benchmarks, with example questions, and Table 12 gives the share of each benchmark's items that match items of another.
近似重复要求两道题的问题在 BGE-large 下的余弦相似度至少为 0.9，并经 MPNet 确认。表 11 列出 benchmark 之间最大的几股重复流向并附例题，表 12 给出每个 benchmark 的题目在另一个 benchmark 里有匹配的比例。
Explore the 115 benchmarks逐个查看 115 个 benchmark
Every count in this post traces to the per-benchmark records below. Filter by breaking level, capability group, or release period, sort any column, and click a row to see how far each attacker gets on that benchmark.
本文的每一个数字都能追溯到下面的逐 benchmark 记录。可以按被攻破层级、能力组或发布时期筛选，按任意列排序，点击一行查看每个攻击者在该 benchmark 上能走多远。
Which benchmarks to use for a capability按能力选 benchmark
A benchmark that survives every level holds a certificate that none of our attackers can forge, and 38 of the 115 hold one. The survivors of each capability group are these.
通过全部层级的 benchmark 拿到了一份我们的攻击者都无法伪造的证书，115 个里有 38 个。各能力组的幸存者如下。
- Perception, 13 of 45:感知，45 个中的 13 个： Colon-Bench, K9-Bench, KFS-Bench, MME-VideoOCR, NaturalBench-Video, Perception Test, Q-Bench-Video, ST-Evidence, ScaleLong, TimeBlind, VELOCITI, VNBench, ViMUL-Bench.
- Temporal understanding, 10 of 18:时序理解，18 个中的 10 个： AoTBench, E.T. Bench, MotionBench, PhysicalAI, S-EMBER, SLVMBench, TOMATO, TVBench, VCR-Bench, Vinoground.
- Spatial and physical understanding, 7 of 20:空间与物理理解，20 个中的 7 个： CMU, ContPhy, OSI-Bench, SIS-Bench, UrbanVideo-Bench, VGenST-Bench, VSI-Bench.
- Reasoning and knowledge, 8 of 32:推理与知识，32 个中的 8 个： GUI-World, MORSE-500, Neptune, PAI-Bench-U, SFI-Bench, SciVideoBench, SmartHome-Bench, Video-Holmes.
A survivor resists the attackers we tried and certifies nothing beyond them. Its report card names the level at which it came closest to breaking, and a user who needs the strongest guard for one claim can rank the survivors of a group by that margin. Tables 34 to 37 rank the ten strongest benchmarks of each group, where the margin is the breaking threshold s* − c − δ minus the largest exploitability over the measured levels, in %.
幸存只说明它挡住了我们试过的攻击者，并不证明更多。它的报告卡写明它离被攻破最近的是哪一层；如果某项能力需要最可靠的把关，可以按这个余量给同组的幸存者排序。表 34 到表 37 给出每组最强的十个 benchmark，余量是攻破门槛 s* − c − δ 减去各已测层级中最大的可利用度，单位为 %。
Report cards报告卡
Every audited benchmark has a report card, released with the pool. Each card gives the attack gaps, the breaking level, the reference and shortcut accuracies, the dominant error categories, the effective size, the near-duplicate share, the video dependence, the protocol-error share, and the contribution to the pool. The cards regenerate from the same exported tables as the paper.
每个被审计的 benchmark 都有一张报告卡，随题库一起发布。每张卡给出各层的攻击差距、被攻破的层级、参照模型与各捷径的准确率、主要错误类别、有效规模、近似重复占比、视频依赖度、协议错误占比以及对题库的贡献。报告卡和论文由同一批导出的表格生成。
Video-Index
Screening every item逐题筛查
The items that no attacker solves are worth keeping. We screen all 381,614 multiple-choice items of 112 audited benchmarks. Qwen3-VL-8B removes the items it answers from the options, from the text, or from one frame, and Qwen3-VL-2B removes the items it answers from 32 frames in order or shuffled. Near-duplicate questions across sources go as well, and repeated videos are restricted during selection. A labeling agent assigns task and video-content labels, and every item records its provenance, duration, attack margins, and question and middle-frame embeddings. The chain below counts the items after each stage. The attacks remove 76% of the distinct questions, and 62,142 items remain with complete screening and labeled videos.
没有任何攻击者能解的题目值得留下来。我们筛查了 112 个已审计 benchmark 的全部 381,614 道选择题。Qwen3-VL-8B 剔除它靠选项、靠文本或靠单帧就能答对的题，Qwen3-VL-2B 剔除它看 32 帧、无论顺序正常还是打乱都能答对的题。跨来源的近似重复问题也被去掉，选题时限制重复的视频。一个标注 agent 给题目打上任务和视频内容标签，每道题记录来源、时长、攻击幅度以及问题和中间帧的向量。下面的链条给出每个阶段之后剩余的题目数。这些攻击去掉了 76% 的不重复问题，最后剩下 62,142 道筛查完整、视频已标注的题目。
The screened pool serves two kinds of selection. A coverage composition answers a request that specifies task and event coverage, an item budget, and a cap per source. A specification agent turns the request into a query over the pool, and a selector draws items round-robin across duration groups, event types, and sources. A red-team gate then applies adversarial filtering4 to what was selected. Video-Index takes the other route, difficulty first. It holds the 840 hardest verified questions from 76 sources, 210 per capability group and one per video, ranked by their maximum percentile over the five attacker margins. Claude Opus 5 labels the 3,000 hardest candidates by fine capability and verifies their answers against the frames, and selection ignores reference-model accuracy.
筛查后的题库支持两种选题方式。覆盖式组合回应一个指定任务和事件覆盖、题目预算和每个来源上限的请求：一个规格 agent 把请求翻译成对题库的查询，选择器在时长组、事件类型和来源之间轮流抽题，再由红队闸门对选出的题目做对抗式过滤4。Video-Index 走的是另一条路，难度优先。它收录来自 76 个来源的 840 道最难且经核验的题，每个能力组 210 道，每段视频一道，按五种攻击者得分幅度的最大百分位排序选出。Claude Opus 5 给最难的 3,000 道候选题标注细粒度能力，并对照画面核验答案，选题不看参照模型的准确率。
How the red-team gate works红队闸门如何工作
- The selector's initial caps allow one item per video and visual scene, five per question template, and 5% of the budget per source; scarce coverage cells trigger recorded relaxations. Within a source it prefers less templated questions and then weaker worst-case attacker scores.选择器的初始上限是每段视频和每个视觉场景一道题、每个问题模板五道、每个来源占预算的 5%；覆盖稀缺的格子会触发有记录的放宽。同一来源内优先选模板化程度低的题，其次是最坏情况攻击者得分更低的题。
- The gate tests the selected items with Qwen3-VL-8B at 32 frames and the caption attacker, then reruns all five levels. It drops solved items and refills for a bounded number of rounds.闸门先用看 32 帧的 Qwen3-VL-8B 和帧描述攻击者测试选出的题目，再重跑全部五层。它丢掉被解出的题并补充新题，轮数有上限。
- A registrar randomizes the option order, reports the residual exploitability, and logs the pool snapshot, query, selector version, and seed in a lockfile whose hash names the release.登记器打乱选项顺序，报告剩余的可利用度，并把题库快照、查询、选择器版本和随机种子写进一个锁文件，锁文件的哈希就是这次发布的名字。
- At a matched item budget, the gate holds the strongest attacker to 13%, against 19% for selections without it.在相同的题目预算下，闸门把最强攻击者压到 13%，没有闸门的选法是 19%。
Results结果
Nine open video models, a fixed-input Claude Opus 5, and five agents with local tools answer Video-Index. The fixed-input systems receive at most 512 frames at one frame per second with a 224-pixel short side. The agents work on the same videos stored at two frames per second and choose what to look at. With the same fixed input, Claude Opus 5 outscores every open model by more than 37 points, and tools raise the same model by 19.8 points on matched items. GPT-6-Astra leads at 79.3%, ahead of Claude Fable 5.1 at 71.5% and Claude Opus 5 with tools at 70.6%. Human volunteers reach 55.0%, below the three leading agents and above every open model, whose best is Molmo2-8B at 19.0%.
九个开源视频模型、固定输入的 Claude Opus 5 和五个带本地工具的 agent 回答 Video-Index。固定输入的系统最多拿到每秒一帧、共 512 帧，短边 224 像素。agent 在同样的视频上工作，视频按每秒两帧存储，看什么由 agent 自己决定。在相同的固定输入下，Claude Opus 5 比每个开源模型高出 37 个点以上；在相同的题目上，工具让同一个模型再提高 19.8 个点。GPT-6-Astra 以 79.3% 领先，其后是 Claude Fable 5.1 的 71.5% 和带工具的 Claude Opus 5 的 70.6%。人类志愿者拿到 55.0%，低于前三名 agent，高于所有开源模型，开源模型里最好的 Molmo2-8B 是 19.0%。
Three checks say the set measures what it should. Blind baselines in the style of MMStar5 give a mean text-only accuracy of 6% and a mean video gain of 11% over the ten fixed-input models. Video exceeds blind for all ten. Item-bootstrap signal-to-noise analysis6 gives a noise floor of 2.0% and a signal-to-noise ratio of 10.8, against 3.7 for the median source benchmark. And at a matched item budget, the red-team gate holds the strongest attacker to 13%, against 19% for selections without it.
三项检查说明这套题测的是该测的东西。按 MMStar5 的方式做盲读基线，十个固定输入模型只看文字的平均准确率为 6%，看视频的平均增益为 11%，十个模型看视频都高于盲读。题目 bootstrap 的信噪比分析6给出 2.0% 的噪声下限和 10.8 的信噪比，来源 benchmark 的中位数是 3.7。在相同的题目预算下，红队闸门把最强攻击者压到 13%，没有闸门的选法是 19%。
A Video-Index item that agents solve. NaturalBench-Video item 1481 plays an emergence in reverse, from a butterfly beside its chrysalis to the chrysalis alone. GPT-6-Astra, Claude Fable 5.1, and Claude Opus 5 working as agents pick B, the correct option. All nine open models and Claude Opus 5 with fixed input pick A, the forward transformation.
一道 agent 能解的 Video-Index 题。NaturalBench-Video 第 1481 题把一段羽化过程倒着放，从蛹旁的蝴蝶变成只剩下蛹。作为 agent 的 GPT-6-Astra、Claude Fable 5.1 和 Claude Opus 5 都选了正确的 B。九个开源模型和固定输入的 Claude Opus 5 全都选了 A，也就是正向的变化。
Question问题 What transformation is occurring in the cocoon?
- A. The insect in the cocoon turns into a butterfly.
- B. The butterfly turns back into an insect in the cocoon.
A Video-Index item that no system solves. S-EMBER item 3823 follows the wearer of a camera through a furniture store for about five minutes and asks how many price tags the wearer touches after closing a nightstand drawer. The answer is four. The three agents pick one, Claude Opus 5 with fixed input picks three, and all nine open models also miss the answer.
一道所有系统都没解出的 Video-Index 题。S-EMBER 第 3823 题跟着一位佩戴摄像头的人在家具店里走了大约五分钟，问佩戴者关上床头柜抽屉之后摸了几个价签。答案是四个。三个 agent 都选了一个，固定输入的 Claude Opus 5 选了三个，九个开源模型也都没答对。
Question问题 Since I closed the drawer on the nightstand, how many price tags have I touched?
- A. 2 price tags.
- B. 3 price tags.
- C. 1 price tag.
- D. 0 price tags.
- E. 4 price tags.
How agents gather evidenceagent 如何找证据
The execution logs show how the agents look. GPT-6-Astra, Claude Fable 5.1, and Claude Opus 5 all tile frames into grids on nearly every answer and then diverge. Any two of them give the same answer on about four in five items but choose the same sampling strategy on fewer than half. GPT-6-Astra seeks timestamps more often, while Claude Fable 5.1 and Claude Opus 5 favor cropping and zooming. Claude Opus 5 extracts similar image counts across durations. GPT-6-Astra decodes all stored frames on short clips and combines methods on longer videos, whereas Claude Fable 5.1 decodes exhaustively and extracts far more images without reaching GPT-6-Astra's accuracy. All three score lower on videos past ten minutes, GPT-6-Astra least of all. Evaluation should separate the volume of extracted imagery from its relevance and reward knowing when to stop.
执行日志显示了 agent 是怎么看的。GPT-6-Astra、Claude Fable 5.1 和 Claude Opus 5 几乎每道题都先把帧拼成网格，然后各走各路。任意两个 agent 在大约五分之四的题目上给出相同答案，但选择相同采样策略的题目不到一半。GPT-6-Astra 更常按时间戳定位，Claude Fable 5.1 和 Claude Opus 5 更爱裁剪和放大。Claude Opus 5 在不同时长的视频上抽取的图像数量相近。GPT-6-Astra 在短片段上解码全部存储帧，在长视频上组合多种方法；Claude Fable 5.1 则把帧解码得最彻底，抽取的图像多得多，准确率却没有追上 GPT-6-Astra。三个 agent 在超过十分钟的视频上分数都更低，GPT-6-Astra 掉得最少。评测应当把抽取图像的数量和它们的相关性分开，并奖励知道何时停下的系统。
Takeaways核心结论
Video benchmarks are meant to show whether a model can understand video. We attacked 115 of them with five kinds of shortcuts that do not need the capability a benchmark claims to test, and 35 of them break without the attacker seeing a single frame. After screening out the questions that shortcut attacks can solve, we select the 840 hardest to form Video-Index.
视频 benchmark 本该检验模型能不能看懂视频。我们用五类不需要所考察能力的捷径去攻击 115 个 benchmark，其中 35 个连一帧画面都不用看就被攻破。筛掉捷径攻击能解的题之后，我们选出最难的 840 道题，组成 Video-Index。
The remedy is measurement rather than abandonment. A builder can run the pyramid before release and report the breaking level beside the score. A user can draw items from the screened pool, as Video-Index does. Resistance holds only against the attacks tested, so both the audit and the benchmark have to evolve with new models and new shortcuts.
解决办法是测量，而不是弃用。造 benchmark 的人可以在发布前跑一遍金字塔，把被攻破的层级写在分数旁边。用 benchmark 的人可以从筛查后的题库里选题，Video-Index 就是这样做的。抵抗力只对测试过的攻击成立，所以审计和 benchmark 都要跟着新模型和新捷径一起演进。
References参考文献
Citation
The paper is Video-Index: A Curated Meta-Benchmark for Video Understanding Evaluation, 2026; the preprint, the data, and the per-benchmark report cards will follow. To cite this post:
论文为 Video-Index: A Curated Meta-Benchmark for Video Understanding Evaluation，2026 年；预印本、数据和逐 benchmark 的报告卡随后发布。引用本文：
@misc{song2026videoindex,
title = {Video-Index: A Curated Meta-Benchmark for Video Understanding},
author = {Song, Enxin},
year = {2026},
howpublished = {Blog post},
url = {https://enxinsong.com/blog/video-index/}
}
Comments