Revolutionizing Video Analysis: Transforming Footage into Executable Physics Programs

MirroS has introduced Code-as-World, an agentic loop that rewrites real videos into executable MuJoCo physics programs instead of treating videos as just moving pixels. The system turns a scene into editable code, stored as scene.json, so an agent can run it, compare it with the original video, and improve it in up to five rounds. This idea matters because real videos show what happened, but they do not directly tell a model about mass, friction, gravity, or collision, and Code-as-World tries to recover exactly those hidden physical rules while also reaching strong results with Code-as-World-VL-9B on QuantiPhy.

From Video Pixels to a Real World Model

Most video AI systems are very good at guessing what the next frame should look like, but that does not mean they truly understand why an object moved the way it did.
Code-as-World starts from a simple but powerful idea: pixels are clues, not the full truth.
If you watch a short clip of a ball rolling off a table, your eyes see color and motion, but your brain also guesses gravity, weight, speed, and impact.
This project tries to give AI that same deeper layer of understanding by changing a video into executable physics code.
MirroS calls this structure an executable world representation, or EWR.
The EWR has three parts: composition, evolution, and appearance.
Composition describes what is in the scene, such as objects, shapes, size, mass, friction, floors, and walls.
Evolution explains how those objects move over time, including starting states, forces, contacts, collisions, and when the event ends.
Appearance handles visual details like camera angle, lighting, materials, background, and frame rate.
A nice way to think about it is this: composition is the toy set, evolution is the play action, and appearance is the camera used to film it.
That split is useful because changing the camera should not change the laws of physics.
In the released system, the final description becomes a scene.json file that MuJoCo can execute.
MuJoCo is a well-known physics simulator, and it lets the scene be tested again and again like a mini virtual lab.
This makes the result editable too, which is a big step above a plain caption or a hidden latent vector.
For example, a student could change the mass of a box in the code, rerun the simulation, and see how the motion changes.
That means the scene is no longer just something to watch.
It becomes something you can inspect, question, and replay.
For SEO readers searching terms like executable world representation, MuJoCo physics program, or video-to-physics AI, this is the heart of the breakthrough.

How the Agent Learns by Proposing, Testing, and Fixing

The really interesting part is that Code-as-World does not try to solve the whole problem in one shot.
Instead, it uses an agentic loop, which means it works in rounds like a careful student checking homework.
The loop follows a clear chain: propose, instantiate, execute, render, and verify.
First, the system makes a guess about what the world behind the video might be.
Next, it turns that guess into an actual scene that can run inside MuJoCo.
Then it executes the scene, renders the result back into video frames, and checks whether those frames match the original clip.
If the match is poor, the system gets structured feedback and tries again.
It can do this for up to five rounds.
This is a lot like trying to copy a trick shot from a sports video.
You try once, compare your result with the original, notice what went wrong, and adjust your next attempt.
That process is much smarter than making five random guesses and hoping one looks right.
According to the report, this loop beats Best-of-5 independent sampling even when both use the same number of total evaluations.
That means better results without asking for a bigger checking budget.
To build each guess, the system uses several tools.
SAM 3 provides instance masks and object tracks in the image plane.
VGGT-Omega estimates depth and camera geometry.
SAM 3D creates per-object meshes.
After that, candidate rollouts are projected back into the original view and compared using RGB, depth, masks, and trajectories.
The feedback is not just “good” or “bad.”
It becomes a structured signal that tells the system where the mismatch happened.
Maybe the object path is wrong, maybe the size is off, or maybe the timing of contact is late.
That matters because helpful feedback is like a coach pointing to the exact mistake instead of just saying “do better.”
For people interested in AI agents, inverse problems, or physical scene understanding, this loop is one of the strongest ideas in the whole release.

Why Open Source Release Makes This More Useful

One reason this release stands out is that it is not just a paper idea locked away in slides.
MirroS has made the project deployable at the research and internal-prototype level.
The team released a GitHub repository and two open checkpoints: Code-as-World-VL-4B and Code-as-World-VL-9B.
Both are under the Apache 2.0 license, which is good news for developers, labs, and startups that want to experiment.
The checkpoints were fine-tuned from Qwen3.5-4B and Qwen3.5-9B.
They are provided as BF16 safetensors and served with vLLM behind an OpenAI-compatible /v1 endpoint.
That setup lowers the barrier for teams that already use common LLM serving tools.
In simple words, this means researchers do not have to rebuild the whole system from zero.
They can plug it into workflows they already know.
The released configuration uses 16 sampled frames per video and a max model length of 4608.
That tells us the system is not magic dust sprinkled over full raw movies.
It works through a practical selection of frames and a manageable serving setup.
This is helpful for anyone asking, “Can I actually test this in my lab?”
The fair answer is yes, but mainly for research, benchmarking, and internal prototypes right now.
It is not yet a universal plug-and-play solution for every real-world robotics or industrial pipeline.
Still, being open makes a huge difference.
Imagine two science kits: one is shown through a locked glass box, and the other is placed on your desk with tools and instructions.
Code-as-World is much closer to the second case.
That open release also helps the broader AI community check claims, run ablations, and build new ideas on top.
People working in robotics, physical AI, simulation, and world models can treat this as a foundation instead of just a headline.
From an SEO angle, terms like Apache 2.0 AI model, Code-as-World GitHub, and MuJoCo world model checkpoint naturally fit here because they reflect what users will likely search next after hearing about the paper.

Training Data, Tutorials-Style Logic, and Why Verification Matters

A big problem in physical AI is that real video usually does not come with exact labels for force, speed, friction, or acceleration in world units.
You can watch a cup slide across a table, but the video does not hand you a neat note saying “mass equals this” or “contact force equals that.”
Code-as-World tackles this by creating verified worlds and then using them as training supervision.
This is one of the cleverest parts of the system.
First, the model was fine-tuned with 73,335 image-space question-answer pairs from datasets like RefCOCO, RefCOCO+, RefCOCOg, RefCLEF, and GOT-10K.
These examples covered things like extent, position, displacement, velocity, and acceleration in raw pixel space.
That stage is a bit like teaching a student to read rulers and track motion on graph paper before asking deeper physics questions.
Then comes the more advanced stage.
The team applied GRPO to world-space visual question answering using 1,585 text-driven and 988 video-driven executable worlds.
The reward focused on scale-normalized numerical accuracy plus correct units and output format.
That means the model was not only pushed to be close, but also to answer in the right measurable way.
Training used eight NVIDIA H100 GPUs, which shows this was a serious large-scale effort.
The reason verification matters so much is simple: bad simulated worlds would create bad lessons.
If a teacher keeps writing wrong math answers on the board, students learn the wrong rules.
By verifying worlds against source videos, the project tries to make sure the generated supervision is trustworthy enough to teach from.
This also explains why the paper focuses on exact physical labels as a key advantage.
Real footage is rich and realistic, but it is weak as direct physics supervision unless you can recover the hidden structure behind it.
For readers who like hands-on examples, imagine recording a toy car moving down a ramp.
With plain video, you can only eyeball what happened.
With Code-as-World, you aim to recover a scene you can rerun, inspect, and use to train another model to answer motion questions more precisely.
That shift from “watching” to “measuring” is what gives this work long-term value.

Benchmarks, Robotics Potential, and Current Limits

The benchmark results give this project extra weight because the gains are not tiny or vague.
On QuantiPhy validation, the 4B model scored 50.6 MRA and the 9B model scored 55.4 MRA.
The 27B reasoning version reached 58.6.
Most notably, the 9B model slightly beat Gemini-3.1 Flash at 54.8 and clearly outperformed ChatGPT-5.1 at 48.4 on the reported setup.
It also stayed far ahead of the strongest open-weight baseline mentioned, Qwen3-VL-32B-Instruct, which scored 40.2.
The ablation results are even more informative.
When using image-space-only training, scores were 44.2 for 4B and 50.9 for 9B.
After adding both world-space sources, the numbers rose to 50.6 and 55.4.
That jump suggests the verified-world supervision is doing real work, not just adding noise.
Pixel-level grounding improved too, with the 9B model moving from 63.7 to 68.3 on RefCOCO and from 20.1 to 26.6 on GOT-10K after world-space reinforcement learning.
These are strong signs that better physical supervision can also help visual grounding tasks.
So where could this matter in robotics and physical AI?
A robot that only sees pixels may react to surfaces and objects, but a robot that can infer usable physics code may plan with much more care.
Think about stacking boxes, pushing carts, or predicting whether an item will slip off a shelf.
If the video can become a tested world model, a robot may someday learn from visual experience in a more structured way.
That said, the limits are important.
The current release is rigid-body only, which means it does not fully handle soft materials like cloth, jelly, or pouring liquid.
The model also does not learn the discovery loop itself.
In other words, the loop is used around the model, not absorbed inside it as a native skill.
So this is a major research step, but not the final answer to full real-world physics understanding.
A fair summary is that Code-as-World brings us closer to editable, testable, and trainable physical scene understanding, especially for robotics and simulation, while still leaving room for future work on richer materials and more end-to-end learning.

Conclusion

Code-as-World shows a fresh way to understand video by turning real footage into executable MuJoCo physics programs instead of treating scenes as pixels alone. Its five-round agentic loop helps the system guess, test, and fix world models, and the verified results become valuable training data with physical labels that normal videos do not provide. With open Apache 2.0 releases, strong QuantiPhy results, and clear uses in physical AI and robotics, this work looks like an important step toward AI systems that do more than watch the world—they can model, test, and reason about it.

Source: https://www.marktechpost.com/2026/08/29/mirros-code-as-world-executable-world-representations/