VIDEO_TO_3D_BRIDGES.md

pipelines/VIDEO_TO_3D_BRIDGES.md

Video/generated-media → Unreal-Engine 3D bridges, and the state of splats in UE 5.8

Status: research brief, not canon. Scope: the reconstruction/lifting layer that would let

world-model or video-model outputs (e.g. NVIDIA Cosmos 3 environment generations) become real

engine content — gaussian-splatting trainers, feed-forward multi-view reconstruction, splat→mesh

conversion, UE 5.8's actual splat support, and the reverse direction (our DEM terrain conditioning

a video/world model). Sibling to NVIDIA_WORLDGEN.md (the generation-side products),

LOCAL_3D_ASSET_GEN.md (the local image-to-3D asset-gen stack this pipeline already chose), and

GEODATA_TERRAIN_CHAIN.md (the DEM→UE terrain chain this pipeline already chose). Research date

2026-07-15, web-verified — training-data cutoff for the assistant that wrote this was January

2026, and this specific field (gaussian splatting, feed-forward reconstruction, video world

models) moved enormously in the six months since, so every version/capability claim below is a

live-web number, not memory.

Every claim is tagged:

• VERIFIED — confirmed against an official/primary source this session (cited inline).
• INFERRED — a synthesis this brief is making by combining several verified facts, or a

secondary/aggregator source cross-referenced but not independently primary-confirmed. Flagged so

it can be challenged.

• UNVERIFIED / FLAG — something this pass could not confirm and that should not be assumed

true before a pipeline decision is built on it.

---

0. TL;DR verdict (read this first)

**The reconstruction bridge is real, fast-moving, and genuinely useful — but only for one specific

input class: real captured photos/video of real places. It is NOT a working path from "an AI video

model imagined an environment" to "a game-ready level," and nobody's tooling — including NVIDIA's

own newest product built specifically to solve this — has that working at our hardware tier today.**

1. Gaussian-splatting-from-real-capture is a mature, cheap, 5090-trivial local pipeline.

Postshot (Jawset), Polycam, nerfstudio/gsplat, DJI Terra, and RealityCapture/RealityScan all

turn real photos or real video walkthroughs into usable 3D at consumer-GPU cost — gsplat itself

needs only ~4-6GB VRAM for a benchmark scene, trivial on a 32GB 5090. VERIFIED, §1.1.

2. **Feed-forward reconstruction (the DUSt3R→VGGT lineage) has collapsed the "how many

consistent views do you need" number from hundreds to as few as 2-3** for a rough result

(InstantSplat), or one-pass sub-second inference for up to hundreds of views (VGGT/VGGT-Omega).

This matters for input *quantity*, but not for input *consistency* — see next point. VERIFIED,

§1.2.

3. **The actual bottleneck is not view count, it's geometric consistency — and AI-generated video

does not reliably have it.** Text/image-to-video diffusion models are not trained with 3D

geometric supervision; documented, actively-researched 2026 failure modes include surfaces

"melting" under camera motion, hallucinated disocclusion fill, the "Janus problem" (a frontal

feature duplicated on other sides of a generated scene), and cumulative geometric drift over a

sequence. A direct test (COLMAP-free 3DGS run on Sora output) found Sora's video "good enough

for 3D reconstruction" and clearly better than Pika/Gen2 — but a limited-sweep castle-rotation

test still produced a result that "doesn't make sense at all." **Quality is highly

model-dependent and not reliable enough to build a pipeline around without per-clip human

review.** VERIFIED, §1.3.

4. **NVIDIA's own newest, most directly relevant product — Lyra 2.0 (April 2026) — is built to

solve exactly this problem, and it's genuinely good news/bad news.** Good: it's Apache 2.0,

open-weight, free, and specifically engineers around video-diffusion's 3D-consistency failure

(routes per-frame geometry for information, not appearance; trains against its own degraded

outputs to resist drift), and it outputs actual gaussian splats. Bad: its stated hardware

floor is "H100/A100 GPUs with at least 43GB VRAM" — above our 5090's 32GB — with only an

unofficial, unresolved community thread attempting a Blackwell/RTX-50-series path. VERIFIED,

§1.4.2.

5. NVIDIA's other reconstruction product, Omniverse NuRec, is explicitly not for us at all: its

own hardware-requirements page lists only datacenter/professional GPUs (A100/H100/L40/RTX

Pro 6000-class) and is Linux-x86_64-only with no Windows support — a consumer RTX 5090 is not on

its supported list. VERIFIED, §1.4.1.

6. UE 5.8 has no native gaussian-splat support. Every plugin available (NanoGS, XVERSE's

XScene, MLSLabsRenderer, the official Luma AI plugin) is third-party; none of them give splats

collision, and splats bypass Lumen entirely (no GI, no reflections, no shadows through the

standard pipeline) — the universal prescription across every source checked is: **splats for

background/non-interactive dressing, real Nanite meshes for anything the player touches.**

VERIFIED, §3.

7. Splat→mesh conversion has no mature game-ready output, exactly mirroring

LOCAL_3D_ASSET_GEN.md's finding for image-to-3D meshes — SuGaR-class extraction is the

practical floor, marching-cubes-class extraction is "blobby," and true retopology/UV/LOD/

collision cleanup still has to happen in Blender by hand either way. VERIFIED, §2.

8. **The reverse direction (our DEM conditioning a video model for consistent generation) is

architecturally plausible but has no documented working recipe anywhere.** Cosmos-Transfer2.5

supports depth-map conditioning in general, but no terrain/heightmap/DEM-specific example

exists in NVIDIA's own Cosmos Cookbook or anywhere else this pass found — and even if it worked,

the output is still just *video*, which puts you straight back into problem #3 to get it into

the engine. This is a dead end for terrain *authoring* specifically (the DEM already *is* the

geometry — regenerating it as video and reconstructing back is pure loss), though it may have a

narrow legitimate use for concept/cinematic reference. VERIFIED/INFERRED, §4.

9. Where this genuinely adds value for us: real-photo/real-video-grounded reconstruction of

real cultural sites and landmarks (the §17.1-safer alternative to raw AI mesh generation that

LOCAL_3D_ASSET_GEN.md §3.4 already flagged as needed but didn't have a concrete tool for), and

non-gameplay-critical cinematic backdrops. Where it doesn't: replacing the DEM terrain chain,

replacing the already-chosen TRELLIS.2/Hunyuan3D-2.1 image-to-3D prop pipeline, or turning any

AI-*generated* (not captured) footage of a real sacred/cultural site into a shippable asset —

that last one is a sharper version of the §17.1 risk already on file, not a mitigation of it.

Full reasoning in the closing section.

---

1. Video/multi-view → 3D reconstruction: the tool landscape

1.1 Gaussian-splatting trainers (capture-based) — VERIFIED unless noted

Sources: Postshot — Radiance Fields, Postshot review — THE FUTURE 3D, Jawset Postshot User Guide, Export Gaussian Splats from Polycam to Unreal Engine, What File Types Can Polycam Export?, Luma AI review — THE FUTURE 3D, Luma WebGL Library, gsplat GitHub, gsplat 1.5.3 release, State of Gaussian Splatting 2026 — THE FUTURE 3D, RealityCapture pricing, RealityCapture review — THE FUTURE 3D, RealityScan 2.1 release — CG Channel, Video as an Input — RealityScan Help, KIRI Engine pricing.

RealityCapture/RealityScan and video, specifically (VERIFIED, load-bearing for §1.3).

RealityScan's own help docs state it **"extracts purely key frames from the videos... and ignores

interpolated frames (synthesized by a codec, which are geometrically inaccurate)"** — i.e. even

for *ordinary* smartphone video of a *real* place, Epic's own photogrammetry tool already

distrusts anything that isn't a true optical sample and throws out codec-interpolated frames as

unreliable geometry. This matters directly for the "AI-generated video" question in §1.3: if

codec-interpolated frames (a comparatively mild, well-understood synthesis artifact) are already

flagged as geometrically untrustworthy, frames from a generative video *model* (which has no

optical ground truth at all) sit on the same or a worse footing by construction, not just by

analogy.

**Capture-quality guidance (INFERRED synthesis of multiple capture-guide sources, consistent

across all of them).** 70-80% overlap between adjacent frames; roughly 80-600 stills depending on

subject/scene scale (small object ~120-140, room-scale 200-500+); **still photos are consistently

recommended over video frames** because video frames carry motion blur and compression artifacts

that structure-from-motion throws out, reducing effective data density. This is the same

"quality over quantity" and "real optical signal beats synthesized/derived frames" pattern that

shows up again in §1.3 for AI-generated video specifically — it is a consistent property of these

tools, not a one-off finding.

Gaussian splatting capture guide — Polyvia3D, How to Capture Gaussian Splatting — Radiance Fields.

1.2 Feed-forward 3D reconstruction (the DUSt3R → VGGT lineage) — VERIFIED unless noted

This is the model class that answers "how few/how unposed can the input views be": no COLMAP

pre-pass, no known camera poses, feed-forward (one pass, no iterative optimization).

• VGGT (Visual Geometry Grounded Transformer) — Meta + Oxford VGG, **CVPR 2025 Best Paper

Award**. Takes one, a few, or hundreds of views and outputs camera parameters, depth maps, point

maps (all with confidence scores), and 3D point tracks, in under a second, single feed-forward

pass. It does not itself output a mesh or gaussian splat — its COLMAP-format export is meant

to feed a downstream tool like gsplat for actual splat training. Hardware: CUDA, bfloat16 on

Ampere+ (Compute Capability 8.0+), no explicit VRAM figure published, but described as

substantially memory-optimized over its 2026 lifetime (see VGGT-Omega below). **Licensing is

split**: the original VGGT-1B checkpoint is non-commercial only; a separate

VGGT-1B-Commercial checkpoint (approval-gated, similar published accuracy — AUC@30 90.37 vs.

89.98 on Co3D) is explicitly commercial-use-licensed (excludes military applications).

VGGT GitHub, VGGT-1B-Commercial — Hugging Face, VGGT paper (CVPR 2025).

• VGGT-Omega — released as "the next step" per its GitHub description (CVPR 2026 Oral), the

headline improvement is memory efficiency — roughly 2-3x more input frames per GPU-memory budget

than the original. VGGT-Omega GitHub.

• DUSt3R → MASt3R → MV-DUSt3R(+) — the lineage this whole model class descends from. DUSt3R/

MASt3R only process image *pairs*; scaling past two views meant a combinatorial explosion of

pairwise reconstructions plus expensive, error-compounding global optimization. MV-DUSt3R

fixed this with multi-view decoder blocks that exchange information across all views in a single

feed-forward pass around one reference view; MV-DUSt3R+ (CVPR 2025 Oral) added

cross-reference-view blocks so the result doesn't depend on which view got picked as "reference,"

reconstructing single- and multi-room scenes in under 2 seconds.

MV-DUSt3R+ project page, Review: Feed-forward 3D Reconstruction, DUSt3R to VGGT.

• InstantSplat (NVIDIA Autonomous Vehicle research group) — the sparse-input extreme: builds an

accurate 3D representation from as few as 2-3 images, pose-free and SfM-free, via a

self-supervised "Gaussian Bundle Adjustment," reconstructing in seconds rather than the hundreds

of views classic 3DGS assumed.

InstantSplat project page, InstantSplat — NVIDIA AVG.

Synthesis (INFERRED). This lineage answers the *quantity* half of "input requirements" — you

no longer need 100-300 photos with known camera poses; 2-3 to a few dozen unposed views can work.

It does not answer the *consistency* half. Every one of these models is trained and evaluated

on real optical captures (or carefully-posed synthetic renders with ground-truth camera geometry)

— none of the sources checked claim robustness to views that are only *approximately* consistent

because a generative model imagined them independently frame-by-frame. That gap is exactly §1.3.

1.3 The generated-video wildcard — is any of this reliable on AI output?

**The core research problem (VERIFIED, this is an active, unsolved-in-general 2026 research area,

not a solved one).** Video diffusion models are trained on pixel-level statistical matching with no

explicit 3D geometric supervision, and multiple 2026 papers document the consequence directly:

"co-visible rigid surfaces stretch, bend, or 'melt' under camera motion" (geometric deformation),

"hallucinated content in disoccluded regions" (occlusion inconsistency), the "Janus problem"

(a frontal feature — e.g. a face — gets duplicated on other sides of a generated scene because the

model has no true 3D structure to reference), and cumulative geometric drift over longer

sequences. This is why an entire 2026 research cluster exists purely to *retrofit* geometric

consistency onto video diffusion output: GeCo (an evaluation benchmark specifically for this),

VideoWeave, VideoGPA, and GeoFlow all target exactly this failure class.

GeCo — evaluating geometric consistency for video generation, VideoWeave, VideoGPA, GeoFlow.

**A direct test on real video-model output (VERIFIED for the paper; the demo write-up is a lighter

secondary source, flagged).** A COLMAP-free 3DGS pipeline (no camera-pose pre-pass needed) was run

directly against video released from OpenAI's Sora, with the finding that Sora's output is

"good enough for 3D reconstruction," with a "significant advantage... over strong baselines"

and specifically better than Pika and Gen-2 — i.e., video-model quality for this purpose is

highly *model-dependent*, not a uniform property of "AI video." A companion write-up of the same

class of experiment reports a concrete failure case: a castle-rotation clip with a **limited camera

sweep produced a reconstruction that "doesn't make sense at all,"** attributed to inconsistency

between the AI-generated frames. Separately, Sora's original February 2024 technical report was

itself titled around "stunning geometrical consistency" relative to earlier video generators —

consistent with the "quality is model-dependent, and the best models are still improving on this

specific axis" reading, not "solved."

COLMAP-Free 3D Gaussian Splatting, 3D Scene Reconstruction from Sora Videos — demo writeup, Sora Generates Videos with Stunning Geometrical Consistency (Feb 2024).

**Purpose-built countermeasures exist precisely because naive splatting of generated video is

unreliable (VERIFIED as existing; INFERRED as evidence for the underlying problem's severity).**

VidSplat (May 2026) is explicitly a "training-free generative reconstruction framework"

that uses video-diffusion priors to *synthesize novel views that compensate for missing input

coverage* rather than trusting raw generated frames directly — i.e., it treats the generative

model as a *view-completion* tool bolted onto a real-capture-anchored reconstruction, not as the

direct source of truth. GVGS (Jan 2026) adds a visibility-aware multi-view consistency

constraint specifically because "multi-view constraints become unreliable under large geometric

discrepancies." The existence of this whole sub-field is itself strong evidence that

plain-splatting-of-raw-generated-video is not a reliable production technique as of this research

pass. VidSplat, GVGS.

1.4 NVIDIA's own reconstruction stack — the two products closest to the task's framing

1.4.1 Omniverse NuRec — VERIFIED not usable on our hardware/OS

NuRec ("agent-friendly 3D Gaussian splatting libraries that ingest real sensor data to reconstruct

and render interactive simulation in OpenUSD") converts camera-or-lidar sensor data into a

reconstructed USD scene (with trajectory metadata), built on the open-source gsplat rendering

library, with two generative cleanup models — Harmonizer (built on Cosmos, fixes artifacts and

blur) and a Fixer (addresses holes/spurious geometry from novel viewpoints) — plus an "Asset

Harvester" for pulling clean objects back out of a reconstruction.

NuRec product page.

Hardware requirements, quoted directly from NVIDIA's own docs

(Hardware Setup and Requirements — NuRec):

minimum "at least one (1) NVIDIA GPU with CUDA support (version 12.8 or higher)" and "more than

24GB of memory," recommended "more than 48GB," on a "Linux x86_64 operating system" —

the page explicitly states Linux aarch64 is unsupported and makes no mention of Windows at all.

The supported GPU list is Ampere (A100, A10, A40, RTX A6000), Ada Lovelace (L20, L40, L40S), Grace

Hopper (H20, H100), and Blackwell (RTX Pro 6000D — the professional/workstation Blackwell part,

not the consumer 5090). No consumer GeForce RTX card is on this list. Between the OS

requirement (Linux only; our dev environment is Windows) and the GPU allowlist (datacenter/

professional only), **NuRec is not runnable on our target 5090 workstation as officially

documented**, full stop. Integration is confirmed with Isaac Sim, AlpaSim, and CARLA (robotics/

autonomous-driving simulators) — no Unreal Engine integration was found in any source checked.

Licensing for the companion nurec-skills repo is dual CC-BY-4.0/Apache-2.0 for code, but this

doesn't matter if the hardware/OS floor already excludes us.

nurec-skills GitHub, gsplat license.

1.4.2 Lyra / Lyra 2.0 — VERIFIED the actual "video-model output becomes 3D" bridge, with a real hardware caveat

**This is the single closest thing found anywhere in this research pass to the exact bridge the

task asked about.** Project Lyra ("Open Generative 3D World Models," nv-tlabs/lyra on GitHub) —

Lyra 2.0 released April 15, 2026 — takes a single image, generates a long-range,

camera-controllable video walkthrough of the implied scene, and **reconstructs that walkthrough

into an explicit 3D Gaussian scene**, specifically engineered to survive the exact failure modes

documented in §1.3:

• *Spatial forgetting* is addressed by maintaining per-frame 3D geometry used **only for

information routing** — retrieving relevant past frames and establishing dense correspondence

with the target viewpoint — while the generative prior (not the geometry) handles appearance.

• *Temporal drifting* is addressed by training on "self-augmented histories" — deliberately

exposing the model to its own degraded outputs during training so it learns to correct drift

rather than compound it.

Output, quoted directly from the Hugging Face model card: **"Output Type(s): Three-Dimensional

(3D) Gaussian Scene," "Output Format: Point cloud file (e.g., .ply)"** — i.e. it hands you an

actual splat file, not just a video, closing the loop that raw Cosmos/Sora/Veo output cannot close

on its own. It ships an interactive GUI for drawing camera paths through the generated environment

as it extends. License: Apache 2.0, code and weights, on GitHub (nv-tlabs/lyra) and Hugging

Face (nvidia/Lyra-2.0) — this is the cleanest commercial license found anywhere in this entire

research pass, no MAU caps, no territory exclusions, no NVIDIA-non-commercial sub-dependency trap

of the kind LOCAL_3D_ASSET_GEN.md §1.1 found in TRELLIS.2.

nv-tlabs/lyra GitHub, nvidia/Lyra-2.0 — Hugging Face, Lyra 2.0 research page, NVIDIA open-sources Lyra 2.0 — aipedia.wiki.

The hardware caveat (VERIFIED, this is load-bearing). Lyra 2.0 is **14B parameters, built on

the WAN-14B (Wan2.1) video-diffusion base**, and its own model card and secondary reporting

converge on "H100/A100 GPUs with at least 43GB VRAM" to run locally — trained on 32 nodes of

H100, with the fast/distilled inference path targeting a GB200. Full generation is ~194 seconds per

80-frame autoregressive step at 35 denoising steps (H100/GB200-class); a distilled 4-step path cuts

this to ~15 seconds/step. Input is a single 480×832 image plus ~81 frames of camera parameters.

43GB is above our 32GB 5090. A live NVIDIA Developer Forums thread —

["Missing pure Text-to-Video (T2V) inference script for Lyra-2 / Wan2.1-14B on Blackwell (sm_120)

via WSL2"](https://forums.developer.nvidia.com/t/missing-pure-text-to-video-t2v-inference-script-for-lyra-2-wan2-1-14b-on-blackwell-sm-120-via-wsl2/367515) —

confirms someone is actively trying to get this running on consumer Blackwell (the RTX 50-series'

compute-capability family, which includes the 5090) via WSL2, but as of this research pass that

thread shows an unresolved gap, not a working recipe. **Treat "Lyra 2.0 on a single 5090" as an

open engineering spike, not a confirmed-working path** — plausibly reachable via aggressive

quantization or multi-pass/offload tricks (the model's own generation-time budget already suggests

it isn't real-time-interactive even on its target hardware, so a quality/speed-degraded 5090 path

might still be acceptable for asset-generation-at-dev-time use, as distinct from the live

interactive-exploration use case Lyra's own GUI is built for) — but this is this brief's own

extrapolation, not a documented fact.

Lyra-1.0 → Lyra 2.0 progression — Progressive Robot.

1.5 World Labs Marble — the non-NVIDIA competitor, cloud-only

Marble (World Labs, Fei-Fei Li's startup) is a "frontier multimodal world model" that generates 3D

worlds from text, a single image, multiple images/short video of a real location, a panorama, or a

coarse 3D layout (via its "Chisel" tool: boxes/planes or imported assets as a scaffold). **Marble

1.1** (2026) specifically added real-world-location 3D reconstruction and restyling, claimed

"40× faster than legacy workflows." Export formats, quoted from World Labs' own announcement:

gaussian splats ("the highest-fidelity representation for Marble worlds"), **triangle

meshes** (both physics collider meshes and separate high-quality visual meshes), and video (plus

an "enhanced video" pass that adds detail/removes artifacts). A World API exists for

programmatic/integrated access. Deployment is cloud-only — signup at marble.worldlabs.ai,

no local weights or self-hosting option found in any source checked, no game-engine-specific

export documented beyond the format list above (a mesh/PLY export would go through the same

generic UE import paths as everything else in this brief).

Marble: A Multimodal World Model, Announcing the World API, World Labs releases Marble 1.1.

Pricing (VERIFIED). Credit-based: $1.00 per 1,250 credits, $5.00/6,250-credit minimum purchase.

A standard Marble 1.0/1.1 world generation costs 1,500 credits (~$1.20); a "Draft" generation is

150 credits (~$0.12); Marble 1.1 Plus adds up to 1,500 more credits for larger worlds. Credits

don't expire; auto-refill is available. World Labs API pricing.

Positioning (INFERRED). Marble sits alongside Lyra 2.0 as the two most concrete "image/video →

explorable 3D" products found in this pass, but on the opposite end of the openness spectrum:

Marble is closed/cloud/metered (no license risk to evaluate, but no local-5090 option and a

per-generation cost that reintroduces exactly the SaaS-cost problem LOCAL_3D_ASSET_GEN.md's

whole research effort was aimed at escaping), while Lyra 2.0 is open/local-in-principle but

currently hardware-gated above our target card. Neither is a drop-in "cheap and local today" win;

both are worth a bench-test slot (Marble as a quality ceiling to compare against, exactly the role

LOCAL_3D_ASSET_GEN.md §1.8 assigned to Rodin/Tripo 3.0 for the object-gen question) rather than a

pipeline dependency today.

**Adjacent products checked and ruled out as not fitting this task (VERIFIED as a negative

finding). Google DeepMind's Genie 3** (public to Google AI Ultra subscribers in the US since

January 29, 2026) generates real-time interactive 720p worlds from text at 20-24 FPS, but it has

no API, no export to a downloadable 3D asset, and no game-engine integration — the world exists

only inside Genie's own closed runtime. It answers a different question ("can you explore an

AI-imagined space live") than this brief's question ("can you get engine-usable geometry out of

one"), and is noted here only so it isn't mistaken for a candidate.

Genie 3 — Google DeepMind, Project Genie access.

---

2. Splat → mesh conversion (the game-usable leg)

Current tools (VERIFIED unless noted).

• SuGaR (Anttwo, CVPR 2024) — the foundational method: a regularization term pulls Gaussians

onto the true surface, then Poisson reconstruction extracts a mesh from the aligned Gaussians; an

optional refinement stage binds the Gaussians to the mesh surface, which is what actually enables

editing/sculpting/rigging/animating/relighting in Blender/Unity/UE afterward. Described in a 2026

secondary source as "commonly used... to feed game engines" —

i.e. still the reference technique three years on. SuGaR GitHub.

• 2D-SuGaR (May 2026) — a 2D-Gaussian-Splatting-based successor using pretrained normal/depth

priors for initialization and regularization; reports state-of-the-art Chamfer Distance (0.67) on

the DTU benchmark. Research paper — production tooling built on it not independently confirmed.

2D-SuGaR.

• KIRI Engine 3DGS-to-Mesh — the one mobile, built-in, no-separate-post-process commercial

option: KIRI markets itself as the only mobile scanning app with native 3DGS→mesh, version 3.0

(April 2026) claiming "significantly cleaner meshes, better retention of thin structures, and 20%

faster processing" than its prior version. Free basic 3DGS-to-mesh pipeline on iOS/Android/web;

the fuller Mesh-Inclusive 3DGS Scan / PBR-material / quad-mesh / auto-rig feature set sits behind

a Pro subscription. What Is 3DGS To Mesh? — KIRI Engine, KIRI Engine pricing.

• Polyvia3D — free, browser-based, marching-cubes-class PLY/SPLAT→OBJ conversion. Its own

positioning is honest about the quality ceiling of this class of extraction: **"Marching

Cubes-based conversion produces an approximation that captures overall shape and major features

but may look 'blobby' compared to the original Gaussian render."**

3DGS to Mesh — Polyvia3D.

• **Research frontier, not yet confirmed as shipped/production tooling (flag as

ENGINEERING-SPIKE/watch-list, not adopt-today): Triangle Splatting+** (Sept 2025) reports

4-10dB higher PSNR than 2DGS/Triangle-Splatting at a similar vertex count and the best LPIPS

(the metric that best tracks human perceptual judgment) among methods compared; **2D Triangle

Splatting** (June 2025) beats 2DGS on both visual and geometric metrics while producing sparser

meshes; MeshSplatting (Dec 2025) achieves the best Chamfer distance on 5 of 15 benchmark

scenes. These represent where mesh-extraction quality is heading in the literature, but none of

the sources checked confirm a polished, shipped commercial tool built on them yet — the same

"real research, no released production tool" pattern LOCAL_3D_ASSET_GEN.md §2.1 already found

for Hunyuan3D Studio's PolyGen/SeamGPT retopology research.

Triangle Splatting+, 2D Triangle Splatting, MeshSplatting.

**Quality/topology reality (INFERRED synthesis, directly extending LOCAL_3D_ASSET_GEN.md §2's

already-established finding to this new input class).** No tool found here — commercial or

research — delivers clean, game-ready quad-flow topology, correct edge loops, and a controlled

vertex budget automatically. Marching-cubes-class extraction (the free/commodity tier) is

explicitly "blobby" by its own vendor's description; SuGaR-class surface-aligned extraction is

the practical ceiling for "actually editable in a DCC tool," which is still a starting point for

manual retopology, not a finished asset. **This is arguably a harder starting point than image-to-3D

diffusion output**, because a splat cloud is a set of soft, overlapping, anisotropic blobs rather

than a native mesh a diffusion decoder was trained to emit directly — so routing an asset through

"generate video → splat → mesh" adds an extra, lossier conversion hop compared to going straight

"generate image → mesh" via TRELLIS.2/Hunyuan3D-2.1 (LOCAL_3D_ASSET_GEN.md §1.1/§1.2), for any

asset where the direct path is available. The same Blender-headless-scripting + human-QC finishing

pipeline LOCAL_3D_ASSET_GEN.md §2.2 already specified (decimate/retopo pass, UV unwrap, LOD-chain

export, collision-hull generation) is the correct downstream step here too — nothing in this

research pass changes that recommendation, it just confirms splat-derived meshes need it at least

as much as diffusion-derived meshes do.

**Can splats skip meshing entirely in UE? (VERIFIED, answered directly in §3 — yes, but only for

non-gameplay-critical content.)** Splats can render directly in UE via any of the §3 plugins with

zero meshing step at all — this is in fact the *majority* real-world use case (architectural-vis

walkthroughs, VFX backdrop plates, digital-twin inspection). The constraint isn't technical

possibility, it's that a mesh-free splat has no collision and doesn't participate in Lumen — so

"skip meshing" is the right call exactly when nothing needs to touch, stand on, or bounce light off

the object, and the wrong call the moment gameplay geometry is needed.

---

3. UE 5.8 splat support: native or plugin

**No native Epic support exists in 5.7 or 5.8 (VERIFIED, cross-confirmed by multiple independent

sources including a direct fetch of the Epic Developer Community forum thread on the topic).**

The only Epic-hosted resource on the subject is a community-authored tutorial ("Introduction to

Gaussian Splatting in UNREAL Engine") hosted on Epic's developer portal — not an engine feature.

The Epic forum thread itself carries zero staff comment on roadmap intent; all substantive

technical discussion in it comes from community developers. Treat "Epic ships native gaussian

splatting" as not currently true and not confirmed on any near-term roadmap — a forward bet at

best, not something to architect around today.

Gaussian Splatting options — Epic Developer Community Forums.

The plugin roster (VERIFIED, capabilities/limitations per-plugin):

Sources: NanoGS puts 'Nanite-style' Gaussian Splatting in UE5 — CG Channel, NanoGaussianSplatting GitHub, XScene-UEPlugin GitHub, XScene collision/shadow issue #115, MLSLabsGaussianSplattingRenderer-UE GitHub, Luma Unreal Engine Plugin docs, Luma AI — Fab listing.

**Cross-cutting engine-level facts, true regardless of which plugin is chosen (VERIFIED, multiple

independent confirmations):**

• Lumen does not interact with gaussian splats at all — 3DGS renders as a screen-space effect

that bypasses Lumen entirely, so there is no global illumination or reflection mixing between a

splat and ordinary Unreal meshes in the same scene.

• Splats have no collision. The universal prescription found across every source: use Nanite

meshes for anything gameplay-interactive; a splat needs "a mesh under or alongside" wherever a

player can walk, stand, or fight on it.

• The current plugin generation is Windows-only — no confirmed Mac/Linux support for any of the

UE-side rendering plugins (in contrast to the *training* side — gsplat/Postshot/nerfstudio are

more platform-flexible; it's specifically the UE runtime plugins that are Windows-locked).

• **Cesium for Unreal — already load-bearing in this pipeline per GEODATA_TERRAIN_CHAIN.md — now

also streams gaussian splats. On April 27, 2026**, Cesium added 3D Gaussian Splats as a

first-class tile type to the 3D Tiles ecosystem (OGC 3D Tiles 2.0), with hierarchical LOD

streaming through CesiumJS and Cesium for Unreal, georeferenced and combinable with Cesium World

Terrain or Google Photorealistic 3D Tiles. **This is a distribution/format-interop channel, not

an authoring one** — it extends GEODATA_TERRAIN_CHAIN.md §1's existing verdict ("Cesium =

reference/preview layer, not the authoring path for the 69 playable regions") to splats

specifically: if a splat asset already exists, Cesium is a legitimate way to *stream* it

alongside real-world terrain context, but it is not a tool for *producing* the splat.

Introducing 3D Gaussian Splats with Hierarchical LOD Using 3D Tiles — Cesium, View a 3D Gaussian Splat Tileset with LODs in Cesium for Unreal.

• Cross-engine standards are emerging but not yet universal (INFERRED/partially unconfirmed).

PLY remains the de facto interchange format as of early 2026. A Khronos **glTF

KHR_gaussian_splatting** extension was reported targeting Q2 2026 ratification (this specific

ratification date is from a secondary aggregator source, not independently confirmed against

Khronos directly in this pass — flag before relying on it), and an OpenUSD "Particle Fields"

schema effort exists under AOUSD. Neither changes today's practical workflow (PLY in, per-plugin

import), but either could simplify the interchange story within the UE 5.8 cycle.

State of Gaussian Splatting 2026 — THE FUTURE 3D.

---

4. The reverse direction: DEM/geometry-conditioned generation

What Cosmos-Transfer2.5 actually supports (VERIFIED). nvidia-cosmos/cosmos-transfer2.5 is

built on Cosmos-Predict2.5 and produces conditional world (video) simulations from multiple

spatial control inputs — the documented modalities are **Edge, Depth, Segmentation, and Vis(ual

blur)**, described as covering "structural preservation, semantic replacement, lighting

consistency, and multi-control approaches." A companion recipe in NVIDIA's own Cosmos Cookbook

demonstrates depth-conditioned post-training for photorealistic synthetic imagery (an

agricultural-fleet/robotics example) — real depth-conditioning, but for a physical-AI training-data

use case, not world/terrain authoring. Depth maps for training are themselves generated from real

video via Video-Depth-Anything/DepthAnything2. A distilled low-latency "Edge" variant of Transfer2.5

shipped February 23, 2026. Licensing is split: source code is Apache 2.0, but models ship under

the separate "NVIDIA Open Model License" (distinct from the "OpenMDW 1.1" license

NVIDIA_WORLDGEN.md §1.1 found for the core Cosmos 3 omnimodel — these read as two different

license families inside the same NVIDIA Cosmos umbrella; get a direct read of the Transfer2.5

model license specifically before relying on either characterization interchangeably).

Cosmos-Transfer2.5 GitHub, Cosmos Cookbook, Cosmos-Transfer2.5 — NVIDIA Research.

Terrain/DEM conditioning specifically — VERIFIED absent. No terrain, heightmap, or DEM-specific

conditioning example was found anywhere in NVIDIA's own Cosmos Cookbook, the Cosmos-Transfer2.5

repo, or the broader web search for this brief. The framework is architecturally general enough

that a depth *image* is exactly the kind of signal it accepts — but nobody has published "feed a

real-world DEM in as the depth-conditioning signal for consistent regional generation" as a named,

working recipe. Broader ControlNet-for-video/terrain research exists in the same direction (e.g.

EarthGen, which augments a base terrain-generation model with a ControlNet so it learns

map-consistent terrain from a top-down conditioning layer) — confirming the *mechanism class* is

real and actively researched, but not that a shippable "our-DEM-in, consistent-video-out" product

exists today. EarthGen: Generating the World from Top-Down Views.

INFERRED synthesis — this brief's own reasoning on whether it's worth building. Nothing

technical stops rasterizing our own already-downloaded DEM heightmap (the 16-bit grayscale asset

GEODATA_TERRAIN_CHAIN.md §3 already produces from Copernicus GLO-30/NASADEM) into a synthetic

"depth video" — e.g. a scripted flythrough camera rendering the heightmap as a grayscale depth

pass — and feeding that into Cosmos-Transfer's depth-conditioning slot to generate a photoreal or

stylized video that respects our terrain's real silhouette. This is architecturally plausible

(depth-conditioning exists specifically to constrain geometry while letting a diffusion model

freely vary appearance), but:

1. It is untested — this specific DEM-as-depth-video technique has no documented precedent

found anywhere in this pass. It would be a from-scratch engineering spike, not a known-working

recipe with a citation behind it.

2. The output is still just video. To get playable geometry back out of that video, you are

immediately back inside §1.3's whole problem — multi-view consistency of AI-generated frames —

for a scene whose real geometry you *already had* before you generated the video. Regenerating

terrain as video and reconstructing 3D back out of it is a strictly lossy round-trip for terrain

*authoring* specifically: the DEM already is the geometry (that's the entire point of

GEODATA_TERRAIN_CHAIN.md's chain), so there is no scenario where "generate a video conditioned

on the geometry we already have, then try to rebuild the geometry from the video" beats just

using the geometry directly.

3. Where it could have real, narrower value: generating stylistically-consistent **concept

art, marketing key-frames, or a scripted cinematic flythrough** of a region that's silhouette-

locked to the real DEM — i.e., treating Cosmos-Transfer as an *art/previz tool conditioned on

real terrain*, never as a terrain-authoring shortcut. That's a legitimate, much smaller use case

than "generate our world," and it doesn't require solving the reconstruction-bridge problem at

all, because the video itself is the deliverable in that scenario, not an intermediate step

toward geometry.

This directly reinforces (does not contradict) NVIDIA_WORLDGEN.md's existing verdict that no

NVIDIA product does "DEM in, terrain out" — this pass specifically checked whether the

*conditioning* direction closes that gap, and it does not.

---

Where bridges fit our pipeline

Pulling §1-§4 together against the two chains this pipeline has already committed to

(GEODATA_TERRAIN_CHAIN.md's DEM→UE-Landscape chain, and LOCAL_3D_ASSET_GEN.md's

TRELLIS.2/Hunyuan3D-2.1 local image-to-3D chain):

Genuinely adds value:

1. Real-photo/real-video-grounded reconstruction of real cultural sites and landmarks. This is

the strongest finding in this whole brief, and it directly closes a gap LOCAL_3D_ASSET_GEN.md

§3.4 flagged but didn't solve: raw image-to-3D diffusion is the *wrong* tool for real cultural

architecture (object-scale training data, no urban/structural grounding, and a real §17.1 care

risk — a hallucinated diffusion guess at a real temple's structure can get living cultural

heritage wrong in a way a grounded reconstruction cannot), and that brief recommended

"reference-photo-grounded modeling, photogrammetry, or licensed kit assets" without naming a

concrete tool. This brief names it: Postshot/Polycam/gsplat/RealityCapture on genuine

reference photography or video of the real place (tourist photos, drone footage, museum

photogrammetry sets, Wikimedia Commons, licensed stock, or Josh's own travel captures) is a

mature, cheap, 5090-local pipeline today, with RealityCapture free under $1M revenue and gsplat

needing single-digit-GB VRAM. This only works with real captures of the real place — see the

sharp line drawn below.

2. Non-gameplay-critical cinematic/background dressing. Splats bypassing Lumen and having no

collision is a real limitation for anything a player touches, but it's a non-issue for a

distant mountain range past a region's playable edge, a scripted cutscene backdrop, or a

loading-screen flyover — exactly the same "reference/preview, not authored gameplay terrain"

role GEODATA_TERRAIN_CHAIN.md §1 already assigned to live-streamed Cesium tiles, now available

for splats specifically via Cesium's April 2026 3D Tiles integration, or via any of the §3

plugins directly.

3. A validated benchmark-gate candidate: Lyra 2.0, for the narrow case of "we have one strong

concept image of a fantastical vril-site and want an explorable/reference 3D blockout of the

*space around it*, not just the object" — a scale between what TRELLIS.2/Hunyuan3D-2.1 do

(single objects/props) and what the DEM chain does (real-world terrain). It is the only tool

found anywhere in this pass that is (a) purpose-built to survive video-diffusion's geometric-

consistency failure modes, (b) genuinely open and commercially clean (Apache 2.0, no MAU cap, no

territory exclusion — better licensing than either of LOCAL_3D_ASSET_GEN.md's two lead

candidates), and (c) outputs an actual splat file rather than just video. It belongs in the same

benchmark-gate protocol LOCAL_3D_ASSET_GEN.md §5.2 already specifies for the 5090's arrival —

with the explicit caveat that its documented hardware floor (43GB VRAM) sits above our 32GB

card, so the gate's first step for this specific tool is "does a quantized/offloaded path even

run," not "how does its output quality compare."

Where it's a worse path than what's already chosen, or actively risky:

1. Props, weapons, creatures, and vril-site objects at prop-to-structure scale — the

already-chosen TRELLIS.2/Hunyuan3D-2.1 image-to-3D pipeline is faster (seconds, not the

minutes-to-hours a video-generation-then-reconstruction round-trip costs), more mature, and

outputs a clean single-asset GLB with baked PBR in one step. Routing a prop through

"generate video → splat → mesh" adds an extra lossy conversion hop (§2's finding that

splat-derived meshes start from a blobbier representation than a diffusion decoder's native

mesh output) for zero quality benefit. Reconstruction bridges only earn their keep where no

direct image-to-3D equivalent exists at all — which is object-scale generation's known ceiling

(NVIDIA_WORLDGEN.md §1.5 found the TRELLIS NIM Blueprint caps around 20 objects/prompt,

object-scale only), i.e. specifically the structure/building/site-scale gap between "a prop"

and "a whole region."

2. The 69 regions' actual playable terrain. The DEM chain is real, deterministic, reproducible,

and already license-clean (Copernicus/NASADEM/AW3D30, per GEODATA_TERRAIN_CHAIN.md §2).

Regenerating terrain via a video/world model — even a good one — trades that determinism away

for no benefit, since (per §4) the DEM already *is* the geometry a video-generation round-trip

would only approximate more expensively and less reproducibly. This directly echoes

GEODATA_TERRAIN_CHAIN.md §6's own conclusion about a hypothetical Gaea replacement: "a

neural/generative replacement would likely trade [determinism] away unless explicitly

seeded/pinned" — nothing in this pass found a world-gen tool that pins to real DEM data closely

enough to avoid that trade.

3. **AI-*generated* (not captured) video of a real sacred/cultural site is a sharper version of the

§17.1 risk already on file, not a mitigation of it.** This is the one place this brief actively

disagrees with a naive reading of "reconstruction bridges are the care-safe alternative to raw

generation." They are — but only when the input is a genuine capture of the real place. The

moment the *video itself* is AI-generated (a diffusion model imagining what a real temple looks

like, then that imagined video gets run through a splat/reconstruction pipeline), you get an

asset that presents with all the visual trust signals of a "scanned, surveyed, real" object

(point-cloud noise, real-seeming parallax, a mesh with plausible imperfections) while actually

being exactly the same hallucinated guess LOCAL_3D_ASSET_GEN.md §3.4 already flagged as a care

risk for raw image-to-3D — just wearing photogrammetry's credibility as a costume. **The

pipeline rule this brief recommends**: reconstruction-bridge tooling is cleared for real

captures of real places; it is not a loophole around the existing "never ship a raw-generated

real sacred/cultural building" recommendation, and should not be used to launder AI-imagined

video of a real site into something that reads as surveyed.

**Residual open items, triaged the same way RESEARCHED_STACK.md triages the rest of this

research program:**

• BENCHMARK-DAY (5090 landing): whether Lyra 2.0 runs at all on a single 5090 via quantization/

offload (§1.4.2); a head-to-head of NanoGS vs. MLSLabsRenderer vs. Luma's plugin for actual

in-game splat performance at our target scene scale (§3); whether KIRI Engine's built-in 3DGS-

to-Mesh beats a manual SuGaR pass for the specific real-landmark case in §"genuinely adds value"

item 1.

• ENGINEERING-SPIKE: the untested DEM-as-depth-video conditioning idea in §4, scoped explicitly

to concept/cinematic output, never terrain authoring.

• UNKNOWNS/re-verify before relying on: the Khronos glTF KHR_gaussian_splatting Q2-2026

ratification date (secondary-sourced, §3); the exact commercial terms of the "NVIDIA Open Model

License" covering Cosmos-Transfer2.5's weights, and whether it's the same or a different license

family from Cosmos 3's core OpenMDW 1.1 (§4); KIRI Engine's exact Pro pricing (sources

disagreed, §1.1/§2).

---

Sources (all fetched/searched 2026-07-15)

• Gaussian Splatting options — Epic Developer Community Forums
• NanoGaussianSplatting — GitHub
• NanoGS puts 'Nanite-style' Gaussian Splatting in UE5 — CG Channel
• NanoGaussianSplats for UE5 — Digital Production
• XScene-UEPlugin — GitHub
• XScene collision/shadow issue #115 — GitHub
• MLSLabsGaussianSplattingRenderer-UE — GitHub
• Luma Unreal Engine Plugin (0.41) docs
• Luma AI — Fab listing
• Luma AI review — THE FUTURE 3D
• Postshot — Radiance Fields
• Postshot review — THE FUTURE 3D
• Jawset Postshot User Guide
• State of Gaussian Splatting 2026 — THE FUTURE 3D
• Export Gaussian Splats from Polycam to Unreal Engine
• What File Types Can Polycam Export?
• gsplat — GitHub
• gsplat 1.5.3 released — Radiance Fields
• gsplat documentation
• RealityCapture pricing — FlyPix
• RealityCapture review — THE FUTURE 3D
• RealityScan 2.1 released — CG Channel
• Video as an Input — RealityScan Help
• Gaussian splatting capture guide — Polyvia3D
• How to Capture Gaussian Splatting — Radiance Fields
• KIRI Engine pricing
• What Is 3DGS To Mesh? — KIRI Engine
• 3DGS to Mesh — Polyvia3D
• SuGaR — GitHub
• 2D-SuGaR — arXiv
• Triangle Splatting+ — arXiv
• 2D Triangle Splatting — arXiv
• MeshSplatting — arXiv
• VGGT — GitHub
• VGGT paper (CVPR 2025)
• VGGT-1B-Commercial — Hugging Face
• VGGT-Omega — GitHub
• MV-DUSt3R+ — project page
• Review: Feed-forward 3D Reconstruction, DUSt3R to VGGT — arXiv
• InstantSplat — project page
• InstantSplat — NVIDIA AVG
• GeCo — evaluating geometric consistency for video generation — arXiv
• VideoWeave — arXiv
• VideoGPA — arXiv
• GeoFlow — arXiv
• COLMAP-Free 3D Gaussian Splatting — project page
• 3D Scene Reconstruction from Sora Videos — vuer.ai docs
• Sora Generates Videos with Stunning Geometrical Consistency — arXiv
• VidSplat — arXiv
• GVGS — arXiv
• NVIDIA Omniverse NuRec — product page
• NuRec Hardware Setup and Requirements
• nurec-skills — GitHub
• nv-tlabs/lyra — GitHub
• nvidia/Lyra-2.0 — Hugging Face
• Lyra 2.0 — NVIDIA Research
• NVIDIA open-sources Lyra 2.0 — aipedia.wiki
• Lyra 2.0 facts — Progressive Robot
• Lyra-2/Wan2.1-14B Blackwell/WSL2 thread — NVIDIA Developer Forums
• Marble: A Multimodal World Model — World Labs
• Announcing the World API — World Labs
• World Labs releases Marble 1.1 — Radiance Fields
• World Labs API pricing
• Genie 3 — Google DeepMind
• Project Genie access announcement — Google
• Cosmos-Transfer2.5 — GitHub
• Cosmos Cookbook
• Cosmos-Transfer2.5 — NVIDIA Research
• Cosmos 3 technical report — NVIDIA Research
• NVIDIA Launches Cosmos 3 — NVIDIA Newsroom
• What Is NVIDIA Cosmos3-Nano? — WaveSpeed
• nvidia/Cosmos3-Nano — Hugging Face
• EarthGen: Generating the World from Top-Down Views — arXiv
• Introducing 3D Gaussian Splats with Hierarchical LOD Using 3D Tiles — Cesium
• View a 3D Gaussian Splat Tileset with LODs in Cesium for Unreal

Cross-referenced against this repo's docs/pipeline_review/tech_research/NVIDIA_WORLDGEN.md,

LOCAL_3D_ASSET_GEN.md, and GEODATA_TERRAIN_CHAIN.md throughout — see inline references above

for exactly which findings extend, refine, or reinforce each sibling brief.

Generated by harness/site/structure_site.py — the URL path is the repo path. review root