AIGENEERS

Evaluate with surgeons, later

NVIDIA's Cosmos3-H-Dreams surgical simulator is visually impressive but lacks the physical validation and clinical testing surgeons require.

Frank Poncelet
AIGENEER
11 min. read
Evaluate with surgeons, later
Tags:
EuroStackGenAIComputer VisionAI Automation

Copy, download or open this article in ChatGPT or Claude

On 20 October NVIDIA's healthcare robotics group put Cosmos3-H-Dreams on Hugging Face. You give it one endoscopic frame from a cholecystectomy and a stream of commands for the two instruments, 28 numbers ten times a second, and it generates the next four tenths of a second of video, then the next, for as long as you keep moving the controllers. The post calls it "our most convincing in-body imagery to date" and adds: "To our knowledge, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control." It runs on one RTX PRO 6000: 30 frames per second with the four-step model, 43 with the two-step one, at 832 by 480, according to the model card. The weights are there to download.

Two sections further down, the same post says this: "Visual realism is therefore a starting point for validation, not proof of biomechanical accuracy." Then comes a list headed What is Next, in the post's own words: "Model the full procedure." "Connect generative and numerical simulation." "Support more instruments." "Develop autonomous surgical policies." And, third on the list, "Evaluate with surgeons." The paragraph under it: "These are research directions, not established clinical capabilities. Cosmos3-H-Dreams is a research simulator, not a validated system for patient care."

So this is a demo of a simulator, posted with the word simulator in the title, and the measurement that would make it a simulator, whether the tissue does what the instruments tell it to do, is the one thing the release does not contain. That is the first half of what I want to say. The second half is about who supplied what to this stack, because Europe is in it, as the patients and as the robot, and not as the simulator.

The hairline

In 2021 I came to NFTs as an artist. One of the things I did was neural style transfer on video: take an animation, repaint it frame by frame in a painted style. The flicker between frames was crazy. We built a custom framework that ironed out the differences in the static parts of the image, so the background held still while the subject moved. Other things stayed wrong. Hairlines, for instance. We accepted that. It was part of the art. The static version of that work is still my profile picture.

What I took from it: a generative model gets texture right and thin structure wrong, and between frames it drifts unless you pin down what does not move.

Five years and one architecture later, the Cosmos-H-Dreams paper from August describes the same two problems and the same fix. Against drift, a streaming cache of recent frames with an "appearance sink" that keeps tokens from the first frame, so the scene does not wander. Against thin structure, nothing yet: the paper's own failure figure is a suture, and the text says "the distilled model may hallucinate the geometry of the thread when it folds or crosses over itself". A hairline in a portrait, a thread in a knot.

The reason is one I wrote about here last month. A model like this is built to produce the most probable next frame. It does not look up what the tissue did. Invention is the mechanism, not a contaminant.

In art you accept the hairline. A cholecystectomy simulator cannot accept the cystic duct where the pattern wants it. Which is why the number that matters is the one about whether the picture follows the hands.

The number that is not there

The Cosmos3-H-Dreams model card publishes no quality metric. Under "Key performance" there is one sentence: the model "generates surgical robotic video that responds to the kinematic and control commands of both arms". The only table is frames per second. The card describes evaluation panels, "48 quality cases plus 12 action-control cases each", drawn from the held-out recordings, and reports nothing from them.

What it does report is the data. Sixty-four recordings of robotic cholecystectomy, about 526 minutes of video, listed as "Proprietary (CMR Surgical Versius clinical data)", selected from a pool of 4,792 episodes. Held out for testing: ten recordings, 10.6 minutes. The selection rules: "Only samples with exactly two instruments visible in the endoscopic view were retained", and windows with active camera commands "were excluded". A two-instrument, fixed-camera slice of one procedure.

The only quantitative evidence in this model family is the August paper on the predecessor, written by NVIDIA with three authors from CMR Surgical, and it measured tabletop suturing on a research da Vinci kit, not patients. The agreement between a policy's success rate inside the simulator and on the real robot came out at r = 0.696, pooled over four tasks, below the group's own offline baseline of 0.756. Pickup and throw: 0.67 and 0.77. Handover and knot tie: minus 0.39 and minus 0.24, the wrong sign. In their words, the system is "not yet a substitute for offline evaluation on fine bimanual manipulation". And a sentence about the whole field: "the surgical world-model field has largely measured appearance".

The base model's card, published the same day, says where the clinical domain stands. Forward-dynamics PSNR of 26.9 on the research-kit data, 18.6 on CMR's clinical recordings. And under its own table NVIDIA writes: "Pixel and distribution metrics are dominated by static tissue and do not measure directly how well the generated instruments follow the commanded actions." The metric that would matter is the one nobody has published.

There is independent work on the same question, one generation back. WorldBench, from UCLA, Sony AI, Yale and the US Army Research Laboratory, tested Cosmos-1 and Cosmos-2 on isolated physics concepts and found that "models tend to generate visually realistic scene evolutions (e.g. ball follows parabolic trajectory), but fail to adhere to physical parameters". Balls, not gallbladders, and earlier models. It is the shape of the finding that carries.

NVIDIA's other blog agrees. Its technical post on medical physics simulation, from July, says of generative world models that "they can be visually realistic without providing the explicit physical guarantees of a numerical solver", and that the deterministic environments of classical simulation are "essential" for "user trust and quantitative quality assessment". The October headline says Bringing Clinical Surgery to Life. The July fine print says what that is worth.

What evaluating with surgeons takes

The bar exists, and the field already struggles to clear it with simulators it can inspect. A systematic review in the International Journal of Surgery last year, by Walshaw and colleagues for the European Robotic Surgery Consensus group, opens with this: "The rapid adoption of robotic surgical systems has overtook the development of standardized training and competency assessment for surgeons, resulting in an unmet educational need in this field." Sixty-six robotic training curricula; virtual reality simulation in 44 of them; and "five studies (7.6%) were fully evaluated according to Messick's framework", the five kinds of validity evidence that medical education uses: content, response process, internal structure, relationship to other variables, consequences.

So the physics-based trainers, deterministic and open to inspection, mostly have not cleared the bar either. A generative simulator starts below them. You cannot read its physics off a parameter sheet, so the only check left is to test it. Which makes "Evaluate with surgeons" the product rather than an item on a roadmap.

In NVIDIA's own terms it would mean: publish the twelve action-control cases; put a blinded panel of surgeons in front of generated and real responses to the same commands; and report whether the hook goes where the console sent it and the tissue moves with it, instead of PSNR over tissue that does not move. None of that waits on a bigger GPU; it waits on a panel of surgeons and a scoring sheet.

Who supplied what

The data first. The corpus the base model was post-trained on, Open-H-Embodiment, sits on Hugging Face under NVIDIA's organisation: 750 hours, 120,000 trajectories, CC-BY-4.0. The dataset paper states that "Clinical data, predominantly contributed by CMR Surgical (489 hours of Versius procedures), accounts for 65% of the total corpus by duration." The contributor list includes TUM, TU Dresden, Turin, Óbuda, Hamlyn, Leeds and Balgrist. The cholecystectomy set that Cosmos3-H-Dreams was fine-tuned on is CMR's own proprietary recording of "clinical robotic cholecystectomy procedures on human patients", and nobody can download that part. The model's privacy subcard answers "Yes" to "Personal data used to create this model?", "No" to whether NVIDIA collected it directly, and "N/A" to whether NVIDIA holds the disclosures made to the people it came from. The dataset paper notes only that clinical collection "requires navigating stringent safety protocols, institutional review boards, and patient privacy regulations". What the patients were told is in neither.

Then the model. Cosmos3-Nano is NVIDIA's, and its card says the reasoning tower is "Qwen3-VL-8B-Instruct", frozen, and the latent space is the "Wan2.2 VAE". NVIDIA made neither; its own README lists them among the third-party materials fetched on first use. The Wan report's affiliation line reads "Wan Team, Alibaba Group", and Qwen is the Qwen team's. The runtime, FlashDreams, is NVIDIA's. The card to run it on is an RTX PRO 6000 with, per the README, more than 32 GB of VRAM.

And the robot, which is built in Cambridge. On 12 October, eight days before the release, Stryker announced "an approximately $400 million (£300 million) preferred equity investment in CMR Surgical" with "an exclusive option to acquire 100% of CMR Surgical's outstanding equity in the future", and the role of exclusive US sales agent. Healthcare IT News adds the regulatory trail: FDA De Novo authorisation for the first Versius in October 2024, a 510(k) for Versius Plus in December 2025, for gallbladder removal.

So: Europe supplied the patients' recordings, the robot and the surgeons at the console. The United States supplied the model, the runtime, the chip and the hosting. The two components NVIDIA did not make came from Alibaba's Wan team and from the Qwen team. What a European hospital can download is the weights. What it cannot download is the data. What it cannot build without NVIDIA's stack is the next version. I argued here in September that owning the code while renting the brain is not independence. This is that argument with patients in it.

To be fair to CMR: it chose this. Three of its people are on the paper, and it gets a simulator for its own console out of the deal. Nothing was taken from anyone; CMR traded its recordings for a simulator built around its own console. An open-weight checkpoint is more than most medtech vendors give away. The question is not whether CMR was wrong. It is what a European Cosmos-H would need, and who is building it. A surgical data commons that stays here, since Open-H lives on NVIDIA's organisation. A world model; Mistral's Munich hub, covered on this site this month, is the nearest candidate I can see. And the chips, where there is nothing. I expect the first two to be arguable within a few years and the third not.

Later

The wrong hairline was part of the art, and the portrait is still my profile picture. A simulator for surgeons does not get that licence. NVIDIA's own list says what comes next, in its own words. Until that item is done, Cosmos3-H-Dreams is the most convincing demo in its field, but still a demo.

For the record: I work at a Belgian systems integrator whose clients include large pharmaceutical companies, and I sat on the steering committee of a medical-diagnostics software company.

Sources

Frank Poncelet
Frank Poncelet

AIGENEER · 4 articles

Frank Poncelet is an Operations Architect at Xplore Group and co-founder of Hyperlabs specializing in AI-driven legacy modernization, agentic systems, MCP tools, and reimagining video animation through neural style transfer.

Sources:

Related articles

Google Replaces Gemini Gems With Skills Automation Engine

Google Replaces Gemini Gems With Skills Automation Engine

Google replaces Gemini Gems with "Skills," a new AI automation engine integrated directly into your chat, now requiring a paid subscription.

Yorrick SchoonheydtOct 12
Inside the Google Gemini 3.8 Live Avatar Architecture

Inside the Google Gemini 3.8 Live Avatar Architecture

Google's Gemini 3.8 Live Avatar lets enterprises build real-time digital personas. Inside the low-latency, neural rendering architecture behind it.

Yorrick SchoonheydtSep 28
Accelerating Local Air Pollution Modeling with Earth-2 CorrDiff

Accelerating Local Air Pollution Modeling with Earth-2 CorrDiff

Earth-2 CorrDiff uses AI to model air pollution faster and more accurately, enabling policy simulations, health alerts & local training. Open-source soon.

Alban CapajSep 22