Slop TVNewsLatest
Explainers

SciWorldBench finds video models fail paired science experiments

The benchmark varies a single condition between two otherwise identical experiments, and the best of ten models got both outcomes right in 24.8% of pairs.

Illustration: SciWorldBench finds video models fail paired science experiments
Illustration: SLOP TV News.

Key takeaways

  • SciWorldBench holds 222 pairs of scientific experiments, each pair varying one condition while the setup stays fixed and requiring two mutually exclusive outcomes.
  • Across ten text-and-image-to-video models, Paired Accuracy ran from 1.4% for Wan2.2-TI2V-5B (5B) to 24.8% for Seedance 2.5.
  • Among pairs where both videos passed the checks, the models showed the same outcome under both conditions in 55.7% to 91.7% of cases, the benchmark's Collapse Rate.
  • The authors say they will publicly release the benchmark dataset and open-source the evaluation code.

SciWorldBench, a new benchmark from a research team led by Xue Yang at Peking University, asks whether video world models can turn a change in one experimental condition into a different outcome. Across ten text-and-image-to-video models, they mostly cannot. The paper, submitted to arXiv on November 16, reports Paired Accuracy that runs from 1.4% to 24.8%, so even the best model produced both required outcomes in a pair about a quarter of the time.

The finding matters because video generators are increasingly pitched as world simulators, tools that understand how a scene should evolve. SciWorldBench argues that a clip which looks correct for one setup proves nothing about whether the model reasoned about the condition that produced it.

What Is SciWorldBench?

The benchmark holds 222 pairs of scientific experiments across eight categories: mechanics, fluids, thermodynamics, optics, electromagnetism, chemistry, materials and biology. In each pair, the setup stays fixed while one condition changes, and the two conditions require outcomes that cannot both be true. A prompt describes the setup and names the condition, but never states the expected result.

The paper's running example is an orange dropped into water. An orange with its peel intact floats, because the air inside the peel lowers the fruit's average density. A peeled orange sinks. Same tank, same start, opposite results. If a model understands why, it should show two different futures. In the example the authors give, Seedance 2.5 floats the orange in both videos, so the peeled orange never sinks.

The 222 pairs split evenly by input. In 111 of them the two sides share a first frame, so the changed condition lives only in the text prompt. In the other 111 the condition is visible in the starting image. The reference frames were made with Qwen-Image 2.1 at 2048x2048 and resized to each model's output resolution.

How the Experiments Are Judged

An independent vision-language model, doubao-seed-2-1-pro, scores every video at temperature 0. It receives 49 frames sampled evenly across the clip, their timestamps and a description of the shared setup. The paper withholds the condition text, the name of the generating model, the A/B side label, and the label that says which outcome is correct.

The judge answers five questions. Three check that the target's behavior is visible, that it stays the same object and that the experimental prerequisites are met. The other two ask whether a specific candidate outcome appears. A video counts as correct only when all three checks pass, the expected outcome appears and the other one does not. A pair counts as correct only when both of its videos are.

Four numbers come out of this. Prerequisite Pass Rate (PPR) is the share of videos that clear the checks. Scientific Outcome Accuracy (SOA) is the share of individual videos that are correct. Paired Accuracy (PA) is the share of pairs where both videos are correct, and it is the paper's main metric. Collapse Rate (CR) is the share of decided pairs, pairs where both videos pass the checks and each shows exactly one outcome, in which the two videos show the same outcome.

What the Ten Models Scored

All ten models generated one video per condition in separate requests, using the same random seed (42) and a target duration of eight seconds. Table 1 ranks them.

Model Type PPR SOA PA CR
Seedance 2.5 Commercial 94.6 53.8 24.8 55.7
Kling 3.0 Commercial 92.6 45.0 15.8 66.9
Vidu Q3 Pro Commercial 88.7 44.6 15.3 72.2
Wan 3.0 Commercial 92.1 43.2 12.6 74.0
Cosmos3-Super Open-weight, 64B 90.1 37.8 9.0 79.3
MiniMax-H3 Open-weight, 33B 91.7 39.9 7.2 84.3
LTX-2.3 Open-weight, 22B 78.4 32.0 7.2 75.6
Cosmos3-Nano Open-weight, 16B 77.0 30.4 5.4 82.1
Wan2.2-I2V-A14B Open-weight, 14B active 75.0 28.8 3.6 87.1
Wan2.2-TI2V-5B Open-weight, 5B 48.0 16.2 1.4 91.7

Seedance 2.5 leads with PA 24.8% and SOA 53.8%, and it has the lowest CR at 55.7%. Kling 3.0 follows at PA 15.8%, then Vidu Q3 Pro at 15.3% and Wan 3.0 at 12.6%. The four commercial models take the top four places. The six open-weight models run from PA 1.4% to 9.0%. Cosmos3-Super (64B) is the strongest open-weight entry at PA 9.0%, and the weakest, Wan2.2-TI2V-5B (5B), scores PA 1.4% with a CR of 91.7%.

The gap between the columns is the finding. A high prerequisite score does not carry into a high pair score. Six models pass their prerequisite checks in 88.7% to 94.6% of videos, yet their SOA never beats 53.8%. Seedance 2.5 gets the scientifically correct outcome in roughly half of individual videos and still does so under both conditions in only a quarter of all pairs.

The Repeated Outcome Is the Failure

When a model shows the same outcome under both conditions, it is right for one side and wrong for the other, and PA records a zero. The Collapse Rate puts a number on how often that happens. Every one of the ten models repeated an outcome more often than it produced two different ones, with CR from 55.7% to 91.7%.

The benchmark runs the same test two ways and the difference points at the cause. When the condition is visible in the first frame, PA reaches as high as 30.6% (Seedance 2.5). When the two sides share a frame and only the text differs, PA falls to a range of 0.0% to 18.9%, and every model's CR climbs. Seedance 2.5's CR goes from 43.8% with different first frames to 67.2% with a shared one.

The authors read this as a bias toward a familiar outcome. In their discussion, they write that models lean more on the visual association between a scene's first appearance and a familiar result than on the condition specified in the prompt. They call current models renderers rather than world simulators.

Knowing the Answer Is Not Showing It

Two further experiments separate understanding from generation. First, the authors appended the correct outcome to each prompt and regenerated videos with the six open-weight models. Every one improved and every one repeated the outcome less often. Averaged across the six, PA rose from 5.6% to 19.5%. MiniMax-H3 gained the most, from PA 7.2% to 33.3%, with its CR falling from 84.3% to 40.2%. That 33.3% is higher than the 24.8% Seedance 2.5 reached without such hints in the main table. Even so, five of the six models still repeated the same outcome in more than half of their decided pairs.

Second, the team asked two models that both reason and generate, Cosmos3-Super and Cosmos3-Nano, to answer the outcome question in text. Cosmos3-Super predicted both outcomes correctly for 63.1% of pairs in words, against a video PA of 9.0%. For Cosmos3-Nano the figures are 44.6% in text and 5.4% in video. The models know the answer more often than they can draw it.

What SciWorldBench Does Not Measure

The benchmark scores scientific condition sensitivity. It does not score cinematic quality. Nothing in it judges lighting, motion smoothness or how a clip would work in an edit, and a low Paired Accuracy is not a verdict on what a creator can shoot with a model that scores badly here.

The authors list two limits. Their explanation of the mechanism comes from behavioral outputs and text predictions rather than from probing the models' internal representations, so the deeper reason for the failures remains unknown. The scoring also depends on the VLM judge, whose agreement with human annotators is imperfect and can introduce errors. That caveat matters for close calls: Kling 3.0 at PA 15.8% and Vidu Q3 Pro at 15.3% sit within a rounding of each other. The outcome-stated test covers only the six open-weight models, not the four commercial ones.

What This Means for Creators

Video world models already do what most creators need: plausible motion, consistent subjects, a shot that holds together for eight seconds. SciWorldBench points at a narrower weakness, cases where the outcome depends on a physical or chemical condition the model has to reason about rather than recognize. Pooled across the ten models, optics was the easiest category at PA 18.1% and chemistry the hardest at 6.2%. Even in optics, the authors note, nearly two thirds of decided pairs still showed the same outcome under both conditions.

The practical reading is to treat scientific correctness as its own axis. A model can look convincing and still show the wrong result, so a clip meant to demonstrate a physical effect deserves a check against the science, not only a look at the frames. The paper compares its approach with earlier paired tests, including WorldBench, which isolates single physics concepts, plus What-If World and CRONOS, which vary conditions in driving, robotic manipulation and physical events. SciWorldBench extends the idea to expert-reviewed scientific experiments across eight categories.

SciWorldBench is not a released artifact yet. The authors say they will publicly release the benchmark dataset and open-source the evaluation code, so the numbers so far come from their own runs, and the dataset and code are a commitment rather than a download you can use today.

The full paper is at arXiv:2611.20899, and the authors say the dataset and evaluation code will be released publicly.

Sources

  1. arxiv.org
  2. arxiv.org
  3. alphaxiv.org
  4. thelatestinai.com
  5. cinecourier.com
  6. arxivsignals.io