Video-Coder writes Blender code before the video model runs

Video-Coder makes an AI write and run Blender Python to build a rough 3D scene before a video model generates a single frame. The agent renders that scene into what the paper calls a Draft Video, and the draft carries the structure and the motion while the video generator supplies the appearance.

It is the work of ten researchers across six institutions, and the paper arrived on arXiv on November 6. The claim is narrow: routing a generator through executable code improves how faithfully the finished video obeys physics.

The appeal is easy to see. Ask a diffusion model for a video of three cars at sunset and it guesses at everything at once, including where the cars are. Ask a coding agent first and you get a scene you can read, run and edit line by line.

What Video-Coder is

The paper, "Video-Coder: Reinforcing Agentic Coding for Video Generation" (arXiv:2611.09894), describes what its authors call "the first attempt to train a code-driven video generation agent." That is their framing, and it is the sort of first that a single paper can only assert.

The ten authors come from the University of Central Florida, the University of Würzburg, Microsoft Research, Tsinghua University, UNC-Chapel Hill and UW-Madison, and the first author is Chaoyang Wang. The agent is a 9-billion-parameter model, Video-Coder-9B, fine-tuned from Qwen3.5-9B. The Hugging Face model card lists 9.41 billion parameters, BF16 weights and an Apache 2.0 license.

What a draft video actually is

Given a text prompt, Video-Coder plans a scene, then writes Blender Python to create the objects, animate their motion and set the camera. It runs the script inside the Video-Coder Harness, an environment the authors built for the job, working in a write, run and revise loop. Two tools do the checking: check_script, a fast static test, and run_blender, which executes the script and renders the draft.

Three kinds of feedback come back. Execution feedback reports syntax errors, missing cameras or runtime tracebacks. Content checks inspect what the camera actually sees, flagging empty frames, static footage or subjects that leave the frame. Visual feedback returns a few evenly spaced frames of the render, so the agent can look at its own output and adjust layout, motion timing or framing.

A submission is not taken on trust. The harness re-runs the final script from scratch in a clean directory and accepts the episode only if that independent run passes the same checks. The paper describes the accepted result as reproducible from the code alone.

Every scene renders under one shared setting: untextured gray clay, a fixed resolution and frame rate. The draft therefore encodes only structure and motion, and appearance is left entirely to the generator downstream. That is the central move of the whole system.

Why routing a generator through code could fix physics

The paper's opening argument is the familiar one about pixel-space models. Diffusion and autoregressive video models learn the video distribution without explicitly modeling the 3D world, so prompts about precise spatial relationships, multi-object interactions or physically governed motion tend to produce incorrect object counts and positions, inconsistent occlusions, implausible trajectories, and objects that deform or vanish. End-to-end video is also hard to edit or reproduce, the authors write, leaving users to rewrite prompts by trial and error.

Video-Coder's answer is to represent the scene as a program instead of letting the model infer one. A program is legible and controllable: you can read it, change one line and re-render. The authors note that GPT4Motion scripts physical motions in Blender and that Video-CoCo uses code as an intermediate chain-of-thought, then draw the distinction that those methods prompt frozen, often proprietary models inside fixed pipelines, while Video-Coder trains the coding agent against the quality of the final video.

What it improves, by the numbers

Both benchmarks are physics-focused, and every headline figure is an average on one backbone, MiniMax-H3.

On PhyGenBench, which scores physical commonsense across mechanics, optics, thermal and material dynamics, MiniMax-H3 moves from 0.61 to 0.67 with Video-Coder in front of it. The abstract describes that as "approximately 7 points" while the paper's own table shows a 6-point gain from 0.61 to 0.67. The category scores are 0.68 for mechanics, 0.79 for optics, 0.60 for thermal and 0.61 for material.

On VBench-2.0, the paper reports the physical dimensions. Average plausibility rises from 54.58% to 78.19%, which the abstract rounds to 24 points and the table puts at 23.61 percentage points. Mechanics climbs from 67.59% to 88.67%. Material improves from 48.94% to 75.02% but still trails CogVideoX-1.5 at 83.19%, which the authors attribute to material behavior depending on fine-grained appearance that the gray-clay draft deliberately leaves out.

An ablation shows which parts carry the gain. Supervised fine-tuning alone reaches 0.63 on PhyGenBench, and the full model reaches 0.67, so the reinforcement-learning stage and its annealing schedule do measurable work.

What the loop costs to run

The paper gives the hardware. Training used 8 Nvidia H100 GPUs for the agent and 16 more to serve MiniMax-H3 for the reward rollouts, with the full run taking about three days. The GitHub repository describes a lighter split for a rebuild: supervised fine-tuning on one node of 4 H100s, the RL stage on two nodes of 4, and a MiniMax-H3 server on 4 GPUs each.

The loop is expensive because every step runs real code and real renders. The RL stage samples eight programs per prompt, allows up to 100 interaction turns and a 120,000-token context, and judges each finished video with Qwen3.8-omni-flash. Each episode runs in an Apptainer sandbox with Blender installed. The setup leans on mini-swe-agent, ms-swift, verl and SGLang.

What the paper does not give is the inference cost. There is no figure for how long the agent takes to produce one draft, or what running the harness costs once it is trained.

What the paper does not report

The headline deltas are averages on a single primary backbone, and the paper runs one transfer test beyond it: applying Video-Coder to OmniWeaving lifts that generator from 0.48 to 0.51 on PhyGenBench. There is no per-prompt failure breakdown, so the paper does not say which prompts still break.

The first results table lists a dozen rival generators, among them Pika, Gen-3, Kling, Wan2.2-TI2V-5B, HunyuanVideo and LTX-Video-2B, but only as standalone baselines. Video-Coder is applied to MiniMax-H3 and OmniWeaving alone, so the table does not show what it would do for the others.

The dataset size is stated two ways. The abstract and the method section describe two sets of 6,000 and 2,000 samples, while the setup section and the conclusion call them Video-Coder-SFT-10k and Video-Coder-RL-4k. The Hugging Face model card repeats the 10k and 4k names.

There is no human evaluation. PhyGenBench uses GPT-4o as an MLLM judge and VBench-2.0 is automated, so both headline scores rest on model judges rather than viewer studies.

What a creator can do with it today

The paper, the code, the weights and the training data are all public. The main checkpoint, Video-Coder-9B, is on Hugging Face under an Apache 2.0 license, alongside an SFT-stage checkpoint and the two datasets. The project page and the GitHub repository carry the results tables, the pipeline and the full training and inference instructions, and the repository's release note is dated November 8. The repo showed 30 stars when read on November 10, and the model card reported 276 downloads in the previous month.

What it is not is a product. Running it means standing up conda environments for a supervised fine-tuning stage and a separate RL stage, building an Apptainer sandbox with Blender, and serving MiniMax-H3 with SGLang on top of it. The training recipe assumes H100-class hardware. A small studio can read the code, run the released drafts and study the approach, but there is no hosted endpoint and no per-clip price.

The open question the paper leaves is whether code-first drafts stay a research stack or reach the tools creators already use. The draft video is a real intermediate artifact, and because it is a program it can be versioned and shared in a way a prompt cannot. Whether that convenience reaches a timeline or a render farm is not something this paper answers.

The paper is on arXiv at 2611.09894, and the weights, code and data are linked from the project page and the Hugging Face model card.