pipelines/NVIDIA_WORLDGEN.md
Status: research brief, not canon. Written to resolve docs/pipeline_review/STACK_FACTS_QUESTIONS.md
Q2. Research date 2026-07-15, web-verified (training-data cutoff was Jan 2026 and NVIDIA's lineup
moved a lot since — Edify 3D alone shipped and was pulled inside that window). Every claim below is
tagged VERIFIED (checked against an NVIDIA/reputable primary source, URL inline) or INFERRED
(my synthesis bridging a gap the sources didn't state directly). Where I could not verify something
that matters, I say so instead of guessing.
**No NVIDIA product shipped as of mid-2026 is "an open-world generator" in the sense the pipeline
doc means — ingest real DEM/geodata, sculpt/populate terrain, hand back a game-ready open world.**
That thing does not exist under an NVIDIA brand. What exists instead:
1. NVIDIA's own generative-3D tooling operates at the object/prop layer, not the terrain layer.
The one piece that is real, shipped, and runs locally on a single RTX 5090 is the **AI Blueprint
for 3D Object Generation** (text prompt to Blender-ready GLB props, ~20 objects/scene) — VERIFIED,
§1.5. It populates a scene with objects. It does not create terrain, does not take a DEM, does not
output USD.
2. Nothing found does what Gaea does (heightmap sculpting, hydraulic/thermal erosion, terrain
authoring) under NVIDIA's own generative label. UE 5.8's new native Mesh Terrain system
(Epic's own feature, not NVIDIA's) is the closest 2026 development in that specific slot —
VERIFIED, §2.
3. Cesium remains the real-world-geodata leg, unchanged. NVIDIA doesn't ingest DEM/satellite data
itself; Cesium for Omniverse is the connector that streams Cesium's own tiles (terrain,
imagery, OSM buildings — sourced by Cesium, not NVIDIA) into an Omniverse/USD stage, as an
*optional alternative staging point* to feeding Cesium into UE directly. It is a Cesium product
that happens to plug into NVIDIA's platform, not an NVIDIA geodata product. VERIFIED, §1.8.
4. **NVIDIA's actual mid-2026 push (Cosmos 3, Earth-2, the GTC 2026 Omniverse announcements) is almost
entirely industrial physical-AI / robotics / autonomous-vehicle / climate infrastructure, not game
content creation.** Cosmos generates video/action data for robot and self-driving-car training, not
3D meshes for games. Earth-2 generates climate/weather simulation, not terrain meshes. VERIFIED,
§1.1, §1.6.
5. **The things that actually match the "describe it, get a populated open world" shape a solo dev
might have half-remembered are third-party, not NVIDIA** — Tencent's OpenUSD-based world pipeline,
Promethean AI (text-to-populated-3D-scene, acquired by a studio in 2025), and Overworld
(VC-funded startup, real-time local diffusion world model, demoed AT NVIDIA's GTC but is not an
NVIDIA product). These are real and worth knowing about, but "the NVIDIA open-world generator" is
the wrong name for any of them. VERIFIED, §1.9.
Practical recommendation for the pipeline correction pass: retire the premise that a single
NVIDIA product replaces any leg of the Cesium / Gaea2Unreal / SRTM chain. Keep that chain exactly as
designed. The one NVIDIA piece worth actually wiring in is the 3D Object Generation Blueprint, as a
local, RTX-5090, prop/object-population step that runs after terrain exists — i.e., it plugs in
next to UE-PCG population, not in place of Cesium/Gaea/SRTM. Full input contract in §4.
---
| Product | What it generates | Input | Output format | Local (RTX 5090) or cloud/DGX | Commercial license | Maturity (mid-2026) |
|---|---|---|---|---|---|---|
| Cosmos 3 (Predict/Transfer/Reason) | Video, images, sound, action sequences — a "world simulator," not meshes | Text, image, video, sound, action | Video/image/action tensors, not geometry | 2B variant: local on one RTX 5090 (image-only). 14B variant: 64GB+ VRAM, effectively datacenter (2xH100/H200) | Open weights, OpenMDW 1.1 (Linux Foundation) | Shipped (Cosmos 3, June 2026) — for physical AI, not games |
| Omniverse (Kit / USD Composer) | Not a generator — a USD authoring/interchange platform | N/A (SDK/apps) | USD stages | Local, RTX-class GPU | Free for dev+production+redistribution since ~May 2026 (no Enterprise subscription needed) | Actively developed, repositioned to industrial digital twins |
| Cesium for Omniverse | Streams Cesium's own geodata (terrain, imagery, OSM buildings) into an Omniverse/USD stage | Cesium ion tiles (3D Tiles) | USD-staged 3D Tiles | Local | Free, Apache 2.0 | Shipped, mature (since 2023) — but it's a Cesium product |
| Edify 3D | Text/image to 3D object (via Shutterstock/Getty-branded services) | Text or image prompt | 3D object + texture | Cloud (Shutterstock/Getty-hosted API) | Licensed through Shutterstock/Getty, not a direct NVIDIA API | Direct NVIDIA NIM preview discontinued ~June 2025 (see caveat below) |
| USD Search NIM | Natural-language / image search over existing OpenUSD/3D/image libraries | Text or image query | Search results (existing assets), not new geometry | Cloud preview (build.nvidia.com); self-hostable via NIM container | NIM dev-tier free / Enterprise for production (see §1.4 license note) | Shipped (preview→GA on NVIDIA API catalog) |
| USD Layout / USD SmartMaterial / fVDB Mesh-Gen NIMs | Assemble scenes from text (Layout); auto-material a CAD object (SmartMaterial); mesh from point-cloud (fVDB) | Text prompts / point clouds | USD scene graph / USD mesh | fVDB explicitly cloud-rendered (Omniverse Cloud APIs) | Same NIM dev/Enterprise split | Announced July 2024 as "coming soon" — current 2026 shipped status NOT independently confirmed in this pass; verify directly at build.nvidia.com before relying on any of these |
| AI Blueprint for 3D Object Generation (Microsoft TRELLIS NIM + NVIDIA SANA + Llama/Qwen) | Up to ~20 individual 3D objects/props per scene idea | Single text prompt (a scene idea, e.g. "night market") | GLB (glTF binary) per object, auto-exported into Blender | Confirmed local on RTX 5090/5080/4090/4080/RTX 6000 Ada, 48GB system RAM, ~50GB disk | Blueprint repo: Apache 2.0. TRELLIS model/code: MIT (commercial-use-permitting), but check nvdiffrast/nvdiffrec sub-dependency licenses | Shipped, real, working today |
| Earth-2 | Kilometer-scale climate/weather simulation and visualization (atmosphere, storms, cloud cover) — not terrain meshes | Climate/weather model inputs; visualized via Omniverse | Simulation datasets + interactive visualization (via Omniverse Cloud / Graphics Delivery Network) | Cloud/DGX-oriented | Enterprise/partner-oriented (Weather Company, Taiwan Weather Admin cited as early users) | Shipped, mature — wrong domain, not a terrain-for-games tool |
| DLSS 5 "3D-guided neural rendering" | Real-time pixel reconstruction using 3D structured data as a conditioning signal — a rendering technique | Rasterized 3D scene + AI reconstruction | Rendered frames | Local, RTX GPU, in-engine | Bundled with DLSS/RTX Kit SDK | Announced GTC 2026, targeted Fall 2026 launch |
| RTX Kit / Mega Geometry / RTX Remix | Rendering/geometry tooling (path-traced foliage, neural reflections, remastering old fixed-function assets) | Existing scene/assets | Rendered frames / remastered assets | Local, RTX GPU | Bundled with UE5 RTX plugins | Shipping incrementally through 2026 |
NVIDIA Cosmos 3 ("omnimodel") natively understands and generates text, images, video, ambient
sound, and actions with physics-grounded outputs
Cosmos 3 launch, NVIDIA Newsroom, June 2026;
The stated purpose is training data and simulation for robots and autonomous vehicles
(warehouses, factories, roads) — every primary source frames this as physical-AI/robotics, never
game development or terrain generation. Model matrix: Cosmos-Predict2.5 ships as 2B and 14B variants
GitHub); the 2B Text2Image variant is confirmed
runnable on one RTX 5090 (32GB), the 14B variant needs ~64GB+ VRAM (effectively dual 80GB GPUs or a
single H200) — INFERRED/aggregated from secondary GPU-requirement writeups, not an NVIDIA primary
source, flagged accordingly. Licensed under OpenMDW 1.1 (Linux Foundation) per the Cosmos product
page. **Bottom line: even the locally-runnable Cosmos variant outputs video/images, not meshes or
USD — it cannot be a terrain or asset generator for a game pipeline no matter how you wire it.**
Omniverse is not a generator; it's a USD-based platform/SDK (Kit) used to build apps like USD
Composer and USD Explorer
(Kit overview).
As of ~May 2026 NVIDIA dropped the subscription requirement — Omniverse is now free for development,
production, and redistribution, with no AI Enterprise subscription required for that free tier
GTC 2026's Omniverse announcements (DSX Blueprint, Mega Omniverse Blueprint, OpenClaw) are
industrial-digital-twin / robot-fleet / AI-factory framing, not game content
(Into the Omniverse, GTC 2026).
Game-adjacent Omniverse usage that does exist and is real: Omniverse Audio2Face for AI facial
animation, adopted by GSC for S.T.A.L.K.E.R. 2 and Fort Solis; Tencent Games adopted Omniverse to
unify USD-based asset pipelines
(Omniverse game-dev blog; note this
source reads like an older/evergreen NVIDIA blog post reused across years — treat the Tencent/GSC
claims as VERIFIED-as-stated but not necessarily current-2026 news). None of this is world/terrain
generation — it's animation tooling and asset-pipeline unification.
Edify 3D (announced GTC 2024, trained on Shutterstock's licensed 3D/image library, powering
Shutterstock's and Getty's own generative-3D services) — search evidence indicates the direct NVIDIA
NIM preview for Edify was discontinued around June 2025. I could not independently re-verify this
against a live NVIDIA source in this pass (the canonical research.nvidia.com/labs/dir/edify-3d/
page now 301-redirects to research.nvidia.com/labs/cosmos-lab/edify-3d/, which reads as Edify having
been folded under the Cosmos-lab umbrella rather than cleanly killed — INFERRED, worth a direct
follow-up check before treating as settled). What is clear: NVIDIA's own 2025-2026 developer-facing
3D-object tooling (the Blueprint in §1.5) is built on Microsoft's TRELLIS, not Edify — i.e.
NVIDIA's internal direction moved away from Edify as the generative-3D backbone. Whether
Shutterstock's/Getty's own Edify-branded commercial services are still live is unverified in this
research pass; if the pipeline ever depends on Edify specifically, re-check
shutterstock.com directly rather than trusting this brief.
The July 2024 announcement bundled several generative NIMs for OpenUSD workflows: USD Search
(text/image query over existing 3D libraries — GA/preview on the NVIDIA API catalog), USD Layout
(assemble a USD scene from text prompts via spatial intelligence), USD SmartMaterial (auto-apply
materials), fVDB Mesh Generation (point-cloud to USD mesh, explicitly cloud-rendered via
Omniverse Cloud APIs), fVDB Physics Super-Res
(NVIDIA Newsroom announcement;
USD Layout in particular is the one that sounds closest to "world generator" (spatial-intelligence
scene assembly from text) — but it was "coming soon" as of the 2024 announcement and **I could not
confirm a current mid-2026 shipped/GA status for USD Layout, USD SmartMaterial, or fVDB Mesh
Generation in this research pass.** Do not assume any of the three are live and usable — check
build.nvidia.com/explore/visual-design directly before writing a Translation Doc against them.
USD Search is the one confirmed-available piece of this set, and it is a search tool over
existing assets, not a scene/terrain generator.
This is the most concrete, currently-shipping, RTX-5090-local NVIDIA generative-3D offering found in
this research pass.
Full input contract detail is in §4. In one line: text prompt in, up to 20 individual GLB-format 3D
objects out, auto-exported to Blender — an object/prop-population tool, explicitly not a terrain
or whole-world generator (the source article's own framing: "lets 3D artists create up to 20 3D
objects to prototype a scene").
Earth-2 is NVIDIA's climate/weather digital twin: kilometer-scale atmospheric simulation (storms,
cloud cover, global weather), using the CorrDiff generative model, visualized through Omniverse
Cloud's Graphics Delivery Network
kilometer-scale climate model blog).
Every source describes outputs as weather/climate simulation datasets and visualizations — none
describes a terrain mesh, heightmap, or DEM export product. Its early adopters (The Weather Company,
Taiwan Weather Administration) are forecasting organizations, not game studios. **This is not a
terrain generator for games under any reading of the sources.**
DLSS 5's "3D-guided neural rendering" (announced GTC 2026, targeted Fall 2026) reconstructs pixels
in real time using structured 3D data as an AI conditioning signal — Jensen Huang's framing is
"fusing structured data in virtual worlds with generative AI's probabilistic computing"
(TechRadar GTC 2026 live coverage).
This is an upscaling/rendering technique applied to a scene that already exists, not a content
generator. Likewise RTX Kit (ReSTIR PT), Mega Geometry's foliage system (path-traced dense
vegetation, partnered with CD PROJEKT RED for The Witcher), and RTX Remix (remastering old
fixed-function game assets) are all rendering/geometry-processing technologies, not world/terrain
generators
Cesium for Omniverse streams Cesium's 3D Tiles — Cesium World Terrain, Bing imagery, Cesium OSM
Buildings, high-res photogrammetry — directly into an Omniverse/USD stage, with LOD streaming and
caching
NVIDIA developer blog on the integration;
GitHub). It's free/open-source, Apache 2.0. This
directly answers the pipeline doc's "does it ingest geodata natively" question about the NVIDIA
piece: **no NVIDIA generative model ingests DEM/geodata directly — Cesium does that job exactly as
today, and Cesium-for-Omniverse is only relevant if the pipeline decides to stage terrain through
Omniverse/USD before UE import, which is an optional detour, not a required or NVIDIA-native one.**
A solo dev targeting UE5.8 directly can just as well use Cesium for Unreal (the existing
UE-native Cesium plugin) and skip Omniverse entirely for this leg — INFERRED as the simpler path
given no unique NVIDIA capability is gained by routing through Omniverse for geodata specifically.
These plausibly explain where "the NVIDIA open-world generator" phrase came from — they were shown
at NVIDIA's own events, or run on NVIDIA hardware, but they are third-party products:
real-time, local-first diffusion world models for "living, adaptive worlds." Demoed at both GDC
2026 and NVIDIA's GTC 2026, runs on consumer hardware
No confirmed NVIDIA partnership beyond presenting at the conference — it is not built or sold by
NVIDIA.
textured assets with spatial logic), reported acquired by a studio in 2025. Independent company,
not NVIDIA.
Unreal-Engine-plus-OpenUSD asset pipeline (described at a GTC 2024 session,
"Build Games in the Cloud: How Tencent Streamlines 3D Content Production With OpenUSD"),
using Omniverse/USD as infrastructure. This is Tencent's own studio tooling built on top of
NVIDIA's platform, not an NVIDIA product a solo dev can download. (I found a secondary-source
reference to a Tencent "procedural content generation framework" demoed at GDC 2026 with striking
numbers — 10 km² scene generation in under 10 seconds — but could not independently confirm the
product name or details against a primary Tencent/NVIDIA source; treat that specific claim as
unverified and do not cite the "Genesis Scene" name without further confirmation.)
None of these three should be called "the NVIDIA open-world generator" — if Josh has one of these
specifically in mind (most likely candidate given the RTX-5090-local framing: Overworld, since it
explicitly targets "everyday hardware" and was the one with a live local demo), that's a
genuinely-different research task (evaluating a startup's product, not an NVIDIA SDK) and should
be flagged back rather than folded into this brief's verdict.
---
This matters regardless of which generative piece (if any) gets adopted, because the pipeline's
target engine is pinned to UE 5.8 (per docs/PIPELINE_REVIEW_MISSION.md / Josh, 2026-07-15).
support 5.4 or newer**
(Omniverse Connect requirements page).
For UE 5.4+, NVIDIA's own documentation tells you to use Epic's native USD plugin instead of
the Omniverse Connector. **This means for a UE 5.8 target, the Omniverse Connector plugin is
simply not an option — the interchange path has to run through Epic's own USD support, with or
without any NVIDIA generative piece in the loop.**
USDImporter plugin and the InterchangeOpenUSD translator
Universal Scene Description (USD) in Unreal Engine, UE 5.8 docs).
Epic's own docs flag USD support as Beta — workable, but caution before shipping on it.
I could not pull the deeper workflow docs (round-trip/live-sync specifics) past the landing page in
this pass; if the correction pass needs exact live-sync behavior, read the full UE 5.8 USD guide
pages directly rather than trusting a secondhand summary.
heightfield limitations, a genuinely new (Epic-built, not NVIDIA) terrain-authoring option that
didn't exist in earlier UE5 versions
(Unreal Engine 5.8 feature breakdown).
Worth the pipeline doc's own attention independent of anything NVIDIA — it's a second, native option
alongside Gaea2Unreal for the terrain-sculpting leg.
Blueprint) sidesteps the whole USD-compatibility question** — it outputs GLB (glTF binary), and
UE5.8 has a mature native glTF/Interchange importer independent of the USD plugin's Beta status.
This is a practical point in its favor for a solo dev who wants the shortest, most stable interchange
path.
---
Ranked by "how plausible is this as what a solo dev with a 5090 would actually reach for":
1. AI Blueprint for 3D Object Generation (TRELLIS NIM) — REAL, SHIPPED, LOCAL-5090-CONFIRMED.
Replaces: nothing in the Cesium/Gaea2Unreal/SRTM terrain chain. Slots in as an **object/prop
population step**, functionally adjacent to (not a replacement for) UE-PCG population — i.e. it's
another way to generate the props/set-dressing that populate a region once terrain exists, usable
alongside or instead of hand-placed/PCG-scattered assets for prototyping. Cannot: sculpt terrain,
take a DEM/heightmap as input, output USD, or generate anything at world/region scale (it's capped
at ~20 objects per prompt, single-object-at-a-time generation under the hood).
2. Omniverse + Cesium for Omniverse — REAL, SHIPPED, FREE, but an *optional detour*, not a
generator. Replaces: nothing that Cesium-for-Unreal doesn't already do directly for a UE-only
pipeline. The only reason to route through it is if the pipeline wants a USD-based intermediate
staging/collaboration layer (e.g. for cross-DCC asset assembly with Maya/3ds Max/Houdini before UE
import) — not needed for a solo dev going straight Cesium-to-UE. Cannot: generate new terrain or
content; it only stages/streams what Cesium already has.
3. Everything else in §1 (Cosmos, Earth-2, USD Layout/SmartMaterial/fVDB, Edify, DLSS 5/RTX Kit) —
REAL but wrong-domain or wrong-maturity for this pipeline. Cosmos and Earth-2 are physical-AI/
climate infrastructure that happen to share NVIDIA's brand and some Omniverse plumbing with game
tooling, not usable substitutes for terrain or asset generation. The USD Layout/SmartMaterial/fVDB
NIMs are the closest on paper to "describe a scene, get a populated 3D world," but their current
shipped status is unconfirmed — do not architect around them without a direct build.nvidia.com
check first. Edify is likely discontinued/absorbed. DLSS 5/RTX Kit are rendering tech, not content
generation, full stop.
What this means for the terrain_pipeline_type enum proposed in STACK_FACTS_QUESTIONS.md Q2: the
draft recommendation floated {nvidia_worldgen, cesium_feed, srtm_feed, hybrid} on the assumption
that an nvidia_worldgen value would replace the Gaea leg. Based on this research, there is no
verified NVIDIA product to put behind that enum value for terrain sculpting/generation — the honest
enum is closer to {cesium_feed, srtm_feed, gaea_sculpt, ue_mesh_terrain, nvidia_object_populate},
where nvidia_object_populate is the TRELLIS-NIM Blueprint doing prop/object population on top of
whichever terrain path (Gaea or UE 5.8's new native Mesh Terrain) is chosen — not a peer alternative
to Cesium/SRTM/Gaea, but a downstream step after them.
---
This is the one piece worth writing a Translation Doc against. Everything here is VERIFIED from the
and the GitHub repo directly.
Hardware prerequisites
datacenter GPU required)
Pipeline stages (the actual generation chain under the hood)
1. User provides one text prompt describing a scene idea (e.g. "night market," "sunny day at the
beach") via a chat interface.
2. An LLM (Qwen3-4B or Llama 3.1 8B, run locally via NIM) brainstorms up to ~20 candidate objects
that would belong in that scene, each with its own generation sub-prompt.
3. NVIDIA SANA (a fast text-to-image model) generates a 2D preview image for each candidate
object from its sub-prompt.
4. Microsoft TRELLIS (via its NIM microservice, image-to-3D) converts each 2D preview into an
actual 3D object.
5. Each object exports as an individual GLB (glTF binary) file.
6. The companion Blender add-on ("3D Object Generation" + "Asset Importer") auto-imports the GLB set
into a live Blender scene for review/refinement.
Invocation (as documented)
the N-panel in the 3D Viewport, click "Start Services," wait for READY status, then open the
"3D Object Generation UI" to enter the scene prompt.
repo's service-based architecture (NIM containers exposed locally).
Output contract
file, not terrain, not a level layout.
hand-author).
Licensing (commercial game dev)
(microsoft/TRELLIS). Caveat: two rendering dependencies
used internally (nvdiffrast, nvdiffrec) carry their own separate licenses — worth a one-time check
before shipping a product that bundles them, though for a dev-time-only asset-generation tool
(never redistributed inside the shipped game) this is unlikely to matter.
Program for "research, application development, and experimentation on up to 16 GPUs"; an NVIDIA
AI Enterprise license (~$4,500/GPU/year, or ~$1/GPU-hour cloud) is required for production use,
defined by NVIDIA as anything beyond dev/test/research/evaluation, specifically "activity serving
real end-users"
(NIM product FAQ). **Reasoned application to this
pipeline (INFERRED, not NVIDIA-stated in so many words): generating art assets locally during
development — where the NIM containers never serve a live player and the output is just baked GLB
files committed into the game's asset folder — reads as development/research use under the free
tier. If the pipeline ever calls a NIM container at runtime inside the shipped game (e.g. live
per-player procedural generation), that flips to production use and needs the Enterprise license.**
This distinction is significant enough that it's worth a direct confirmation from NVIDIA's actual
EULA text (not secondary sources) before the correction pass treats it as settled — flag as
UNVERIFIED-AT-EULA-TEXT-LEVEL.
---
Which NVIDIA product is "the open-world generator" (Earth-2 class? the GTC world-gen models? NIM
microservices?), and does it replace all three legs or only sculpting/generation?
None of the above, cleanly. Earth-2 is climate simulation (wrong domain). The GTC world-gen
models (Cosmos) are physical-AI/robotics video simulation (wrong domain, wrong output type — video,
not geometry). The NIM microservices split into: USD Search (confirmed, but it's a search tool, not
a generator), USD Layout/SmartMaterial/fVDB (announced, current status unconfirmed — don't build on
these without checking build.nvidia.com first), and the one that's actually real and running locally
today, the 3D Object Generation Blueprint (TRELLIS NIM) — which operates at the object/prop
layer, not terrain, and replaces none of Cesium/Gaea2Unreal/SRTM. It's a fourth, additive step
(object population) that runs after terrain exists, not a substitute for any of the three terrain
legs. **Recommend correcting the pipeline docs to keep the Cesium/Gaea2Unreal/SRTM chain exactly as
designed**, and separately wire in the TRELLIS-NIM Blueprint as a local RTX-5090 prop-population tool
alongside UE-PCG — with the nvidia_worldgen enum value replaced by something like
nvidia_object_populate scoped honestly to what it actually does.
If Josh had a specific different product in mind when he wrote "the NVIDIA open-world generator" —
most plausibly Overworld (§1.9), given it's the one real-time, local-hardware, "generate a
playable world" demo that actually showed up at NVIDIA's own GTC 2026 — that's worth a direct
follow-up question, since Overworld is not an NVIDIA product and evaluating it is a different research
task (startup product with its own SDK/licensing/maturity profile, not an NVIDIA-blessed pipeline
component).
---