GeoFlip: best video model scores 27.98% on physics tests

Video generation models can make a falling object look real. A new benchmark says most of them still get the fall wrong. GeoFlip, a preprint the authors titled GeoFlip Preview in its latest version and posted to arXiv on October 30, tests whether image-to-video models read the geometry of a scene's first frame and generate the physical outcome that geometry requires. On the authors' own numbers, they usually do not.

The benchmark puts the best result at 27.98% event correctness, averaged across tasks, on the 442 conditions that all 16 tested models attempted. That top score belongs to Alibaba's Wan2.2. The work is a preprint and has not been peer reviewed, so every figure here is the authors' own measurement rather than a settled result.

What GeoFlip actually tests

GeoFlip pairs two scenes that share the same prompt and the same physical settings but differ in one piece of initial geometry, and asks whether the model produces the different event the physics demands. The paper's opening example is a box near a table edge: moving the overhang from 0.39 m to 0.79 m flips the simulated reference outcome from staying put to falling. Wan2.2 makes the box fall in both, so a single convincing clip says nothing about whether the model used the geometry at all.

The authors built 482 such scene pairs, with 964 reference videos, across six categories of physical mechanism: support and static stability, torque-driven rotation, constrained gravitational dynamics, contact and impact, elastic interaction, and pulley-mediated coupling. They constructed the scenes programmatically, simulated each one with a physics engine, checked the outcome numerically, and rendered the first frame and the reference video in Blender. A pair enters the benchmark only when both conditions pass validation and their reference events differ.

Three scores come out of each test. Event correctness asks whether the model produced the right outcome. Structural integrity asks whether the apparatus survived intact. Reference similarity uses PSNR and SSIM to compare the generated video with the simulation.

What the 16 models scored

The comparison pool is 442 conditions across 34 tasks, the intersection of event judgments every model received. The overall ranking averages three dimensions equally, and Wan2.2, MiniMax-H3 and LingBot-World-Fast take the first, second and third slots, with mean ranks of 3.17, 3.67 and 4.33. Their strengths differ. Wan2.2 leads on event correctness at 27.98%. LingBot-World-Fast leads on structural integrity at 60.81%. MiniMax-H3 combines 48.31% structural integrity with strong reference similarity.

The dimensions come apart. DynamiCrafter scores 59.01% structural integrity but only 16.87% event correctness, while Wan2.2 leads on events and sits at 34.38% on structure. On raw image similarity, LTX-Video has the highest PSNR at 29.87 dB but event correctness of 18.41%; LingBot-World-Fast has a lower PSNR of 21.88 dB yet reaches 26.69% event correctness. Wan2.2 leads on SSIM at 0.9659. A model that makes a physically wrong video that looks like its reference can rank well on similarity alone.

Which physics the models miss

Averaged across the 16 models, event correctness is highest for pulley coupling (30.1%) and support stability (28.2%). It collapses to 8.6% for constrained gravitational dynamics and 9.9% for elastic interaction. Structural integrity follows a different pattern: support stability is highest at 37.9%, while pulley coupling is lowest at 23.9%.

Leaders change with the mechanism. Wan2.2 leads constrained gravitational dynamics at 30.6% and contact and impact at 40.6%. LingBot-World-Fast leads support stability at 64.5% and torque-driven rotation at 33.3%. Wan2.1 leads elastic interaction at 28.0%. The paper's paired example for support stability moves the top block of a stack by 0.182 m, which flips the reference from stable to collapsed; LingBot-World-Fast reproduces both outcomes, while Wan2.2 collapses the stack in both and still produces clearer images.

How the failures look

The authors group the failures into patterns a creator will recognize. Some models ignore the geometric change: when reducing a chair's leg spread flips the reference from stable to tipping, MiniMax-H3 keeps the chair upright in both conditions. Some produce the right outcome for the wrong reason: in one stack test only the top block should collapse, and Wan2.2 collapses it but overturns the middle block too, changing which objects move and how they interact. Others break the apparatus: SkyReels-V2 visibly bends a rigid object, and I2VGen-XL adds fragments that were not in the first frame.

How reliable the grading is

Two vision-language judges, GPT-6-Astra and Opus, label the events, and a deterministic scorer compares those labels with the simulation. On 346 reference videos with known labels, Opus recognizes the event 94.22% of the time and reaches 93.72% task-macro balanced accuracy. Across the 16 models and 5,536 generated videos, the two judges agree on 81.54% of event labels and 73.55% of structural labels. Their rank orderings of the models correlate strongly, at 0.906 for event correctness and 0.929 for structural integrity.

The disagreement concentrates in the structural score. The mean absolute difference between judges is 2.47 percentage points for event correctness but 15.14 points for structural integrity; on DynamiCrafter, structural integrity reads 57.64% under one judge and 21.03% under the other. The paper treats structural assessment as the noisier dimension, which means the event-correctness numbers are the more stable signal.

What a creator should change today

Do not ask a video model to resolve a geometry-critical outcome. A top score of 27.98% event correctness means the model is wrong more often than right whenever the physics depends on how the scene is arranged. Use a reference frame, a keyframe or a physics-aware pipeline for any shot where a fall, a tip or a contact has to happen. Watch the two weakest categories in particular: constrained gravitational dynamics at 8.6% and elastic interaction at 9.9% are where the models are least able to follow the geometry. And check the apparatus, not just the outcome, because the paper separates them for a reason: a model can produce the right event while bending a rigid body or adding parts, the failure modes the authors flag for SkyReels-V2 and I2VGen-XL.

The other benchmark: GlobeBench

A separate Patronus AI benchmark, GlobeBench, points the same way for a different kind of world model. Where GeoFlip tests video generation, GlobeBench tests whether language models can predict how a text-based software environment responds to an action. Its paper, submitted to arXiv on October 15 and accepted to a NeurIPS 2026 workshop, covers 1,632 tasks across five domains, scored by executing the action to get the true next state. The blog reports that the best model, GPT-6 Astra, succeeds on 49.57% of tasks, that no model solves even half, and that 631 tasks are solved by none of the 20 models tried. The paper adds that when a change has to propagate to dependent objects, average task success drops to 5.1%. GlobeBench releases its code and data; GeoFlip, so far, does not.

Both results land in the same place for anyone treating a generator as a simulator: plausible output is not the same as a correct response to the conditions, and the gap is large enough to plan around. The GeoFlip paper's own conclusion is that progress from visual realism to physical consistency is the open problem, and the scores it reports show how far that is from done.

The GeoFlip paper is on arXiv at arxiv.org/abs/2610.42002. The authors link no project page or code release, so there is nothing to run yet; GlobeBench's benchmark and harness are open-sourced at github.com/patronus-ai/globebench.