VibeVideoBench: coding agents top out at 76% of video tasks

VibeVideoBench handed nine general coding agents the same job: build a finished video with sound from nothing but code. The best of them still missed a core requirement on nearly a quarter of its tasks.

The benchmark comes from a team at Beijing University of Posts and Telecommunications, Peking University, Jilin University, Zhongguancun Academy and the Kling Team, and the paper went up on arXiv on October 26, 2026. The setup is strict. No image or video generation model is available inside the sandbox. Each agent gets one hour, a written specification, sometimes a folder of input assets, and a text-to-speech tool. Every frame and every sound has to come from code the agent writes, and the run has to deliver an MP4 plus an editable project that can re-render it. The project page lists 239 tasks across 12 categories, 43 subcategories and two task languages, English and Chinese, with 12,985 binary rubric checks. The paper's own Table 2 carries the scores.

That table's strict-success column is the headline number. A run counts as a strict success when it fails no core rubric check and passes at least one; a missing video fails. Claude Opus 5.5 leads at 76.3%, GPT-6 Astra is second at 74.5%, and Gemini 3.8 Flash sits last at 26.6%.

Quality and delivery pull apart. On delivered videos alone, Astra scored 4.126 out of 5 and Opus 4.074, and a blind human preference study ranked Astra first on Elo at 1242 and Opus second at 1203. Count the missing videos as zeros and the order moves: Gemini 4 Argon delivered only 184 of 239 and fell from fourth to seventh. Opus was the only system to deliver all 239, and it finished a median 24 minutes before the deadline.

Delivery failures cluster on the clock. Of the 224 runs that ended without a playable video, the paper reports 169 where the final render started too late, ran too slowly or never started, 45 stalled renders, 7 where the agent destroyed its own output and 3 other causes. Both stalled-render counts sit with the Gemini systems: 33 renders deadlocked because the agent's script piped frames into ffmpeg without ever reading its error stream, and in 12 runs the agent ended its turn while a render kept running in the background.

These are the authors' own figures, produced by one team, and no outside group has run the benchmark yet. For a creator shopping for an agent, strict success is the number that matters, because a video that never arrives is worth nothing. The paper also reports that 95% of runs view their own rendered frames, and how much they look does not predict quality.

The full leaderboard, the per-category scores and the task list are on the project page, and the paper is open on arXiv.

Benchmark card

• Benchmark: VibeVideoBench
• Makers: Beijing University of Posts and Telecommunications, Peking University, Jilin University, Zhongguancun Academy, Kling Team
• Posted: October 26, 2026
• Tasks: 239 across 12 categories, English and Chinese
• Rubric checks: 12,985 binary checks, about 54 per task
• Systems tested: 9 agent systems, 2,151 rollouts
• Time budget: one hour per task
• Read it: https://arxiv.org/abs/2610.35864