pipelines/GAP_MAP.md
The survey deliverable of docs/PIPELINE_REVIEW_MISSION.md Phase A. Five survey lanes; each
section lands as its reader returns (compaction-safe increments). Gap axes per doc: ROLE ·
V3_GAPS (pre-v3-spine assumptions) · STACK_STALE (retired-tool citations) · UNFINISHED ·
HW_ASSUMPTIONS · REGION_PAGE_IMPLICATIONS (feeds the Phase-C template contract) · KEEP.
1. The built RAG harness has outrun every design doc that describes it. harness/rag/ runs
fastembed (384-dim ONNX, no daemon) over sqlite-vec — NOT the Ollama nomic-embed-768/ChromaDB
stack the proposal docs specify — and Scope A (book substrate) is BUILT + TESTED (a real Batak
PDF, 418 chunks; auto-sort gazetteer title_region_map.json) though three of the five docs
still call it unbuilt. The proposals' "open flags" are partly resolved-in-code without
ratification records (canon_authority_tier 1.5/2/3/4 silently live in config.json).
2. The live RAG index is v2.5-shaped. chunk.py still chunks docs/spine/CH_* as v2.5
##-block sections; the 2,676 spine_entry chunks carry NO beat/fork/site/boss/SQ/scene typed
metadata — "every fork in Ch 43" has nothing to filter on. A v3 typed-unit chunking + metadata
pass on the spine collection is a first-order correction (cheap: the store-adapter abstraction
held; the index rebuilds in seconds).
3. Three stances on retrieval, one built reality. AUTONOMOUS_BUILD_ARCHITECTURE (L32) calls
vector-RAG the single HIGHEST-leverage gap; PIPELINE_ADAPTER (L168/242) records embeddings/
ChromaDB as RETIRED for book processing ("pin never"); the actual harness/rag/ was then built
anyway on a third stack (fastembed+sqlite-vec) and works. The 5090+128GB local-compute plan
changes the cost calculus that produced the retirement verdict. The correction pass must
reconcile this explicitly — one ruled retrieval posture — not silently pick a side.
4. The v3 spine converted much of the "RAG problem" into a plain FK-join problem. Typed
beats/forks/§3b sites/§9 bosses/§11 SQs/§13 scenes are deterministic, gate-verified structured
data now; the architecture doc's Layer-3 framing (deterministic-FK solved vs semantic-RAG
gap) predates this and over-scopes the semantic gap.
5. The strongest region-page lever: the deterministic join path `region_id → chapter_id(s) →
site_id(s) → {boss, SQ, scene}` that the v3 §3b/§9/§11/§13 blocks now enable WITHOUT
prose-scraping — blocked only by the unformalized site/chapter→region mapping and the
UNSETTLED region_id format (three incompatible formats coexist: live snake_case slugs vs
T8_REGION_NNN_NAME vs REG_NNN — a Josh ruling in the adapter's roll-up; the template should
abstract region_id behind one resolver until ruled).
6. **Book processing (the 5090 gate) is closer than the docs say AND less done than the status
lines claim.** The chunk/embed/store path runs today, CPU-only, verified real (fastembed,
is_real=true) — but the claimed Batak ingest left NO durable trace (no book_substrate manifest
record; .rag/book_src/processed/ empty; T0_Source_Material_Registry = 5 placeholder rows), and
the documented OCR fallback (ocrmypdf/tesseract) is NOT INSTALLED on this machine. The
critical path to "book processing complete" is: install OCR, re-run + persist the pilot,
wire the registry, then bulk-drain the inbox.
7. The book→spine binding is chapter-grain only. Sources join whole chapters (.meta.json
"chapters" + --chapters); nothing binds a source to a v3 typed unit (beat/fork/site/mark) —
fine for Phase 1, but the region-page citation contract (Phase C) should decide the target
grain deliberately.
8. The pipeline docs' "empty registry" premises are stale in the FAVORABLE direction. The
cascade populated what the docs call stubs: T0_Boss_Encounter 282 rows (with exactly the
verb/axis/counters/reward columns COMBAT_ENCOUNTER_SYSTEM §3.2 says don't exist),
T0_Creature_Roster 132, T0_School_Registry 13; recall.py + check_spine_graph exist and are
gate-wired. Still genuinely empty: T0_Questline_Index, T0_Familiar_Registry, the region-page
scaffold, engines E2-E8 (0%). No milestone/ms-* tag exists — Milestone 1 never ran; Josh's
full-spine-first path superseded the slice-first premise, so the milestone loop needs an
"E1 pre-done" variant before it ever runs.
9. Ship-hardware vs dev-hardware must be split explicitly. The runtime-layer docs' 3-8B/
2-5GB-VRAM figures are SHIP targets (console-tier); nothing in the corpus sizes Josh's own
dev-side compute — the 5090 is the dev-prototyping machine for exactly that runtime layer,
and the correction pass should state the two tiers side by side so nobody reads ship
constraints as build guidance.
10. The 20-section region-page template ALREADY EXISTS — `T1_Region_Page_Template [ACTIVE
v1.0] (last updated May 3, 2026), one tier up from the empty _source/02_Tier_2_Region_Pages/`
scaffold. Phase C is a template-v2 REVISION, not a from-scratch authoring: it needs the
retired-registry ref fixed (its "T0 Registry Feeds" still derives from the absorbed
T0_Folklore_Entity_Registry), the Section-19 stale ending vocabulary v3 names by line
("World State 3 L3 Neutral" / "L5 Truly Bad" → the ratified band/world-state names), the
§3b-site-row consumption boundary (the spine now types terrain/functions/threat/vril/
affordance natively — the page consumes, never re-authors; only geometry stays page-side),
and the per-citation use_class field (cross-cutting from lane 4).
11. **T1_Build_Pipeline_Contracts §9 is wholly superseded; §5/§6/§7 partially — and v3 names the
exact lines.** §9's body-prose-dominant model + §9.6 scatter is replaced by the executed
graph; §6's spatial_requirements/quest_beat_structure/branching_logic are supplied by §3b
sites + the typed beat/fork graph; §5's scene input mechanism should parse the spine's
ratified 7-col scene rows (its 11-value scene_type enum is v3-adopted verbatim — keep); §7's
canonical_zones object is near-verbatim what §3b now types (only area_bounds geometry stays
page-side). The stale path_bucket enum (convergence_neutral→ALL, pre_branch→UNLOCKED) and
the "Neutral/Bad ending" vocabulary appear at the exact cited lines (1331, 1450, 1670, 2280).
12. Contract hygiene findings: seven live registries the contract never mentions (Dwelling,
Environment_Grammar, Hollow_Codex, Minigame, Reputation_Ladder, Rune, Vehicle — the L1
fun-systems layer); nine T99 citations at v2.3 vs the real v3.2; TWO different Drive paths
for region pages inside one contract; the §4.11 duplication (adapter-flagged) still live;
neither T1_Build_Pipeline_Contracts nor T1_Region_Page_Template has a DOC_MAP row
(anti-orphan violations — fixed at this landing).
names/paths (§2.0), the Scope A/B split, chunking/embedding/retrieval (§2), storage doctrine
(§3), the 9-step build order (§4), 5 open flags (§5).
(L216-227) has no beat_id/fork_id/site_id/boss_id/sq_id/scene_id fields; never mentions
SPINE_SCHEMA_v3.
fastembed (config.json/requirements.txt/index_manifest.json agree); §2's body stays
ChromaDB-voiced (L150/162/208) against its own L18 reconciliation note; L195 cites the
pre-migration script 09 instead of the built harness.
without a ratification record; Flag 5's §8 contract-text edits NOT applied (20+ Drive-pathed
lines verified live in T1_Build_Pipeline_Contracts); §1.3's acquisition figures superseded by
BOOK_ACQUISITION_MINIMIZED (buy list ~7, floor 5).
local-7B/3B routing, no 3x-Max-x20 rotation.
ChunkResults → §7 Region Build / §6 Quest; retrieve() fuses governing_canon +
supporting_substrate labelled/separate) — sound SHAPE, but keyed to the OLD 20-section
T1_Region_Page_Template; re-key to template v2 at Phase C.
coarse-vs-fine chunking asymmetry; manifest + index_freshness discipline; canon-first sequencing.
adopted). V3_GAPS: none-aware (generic "§9 Story Spine = bidirectional", L141). STACK_STALE:
L140 names the retired chain verbatim ("UE5/Meshy/AIVA/ElevenLabs/Cinematic + §10 SFX =
tertiary") — a direct rewrite target; §2.3 calls §8's Ollama/768-dim "a real, specified
runtime" (superseded); §4.6's $150-1000/$700-1000/$200-500 API-cost envelopes are in tension
with the 3x-Max-x20 subscription model. UNFINISHED: the §3 "0% built" ledger is stale twice
over (harness built; Scope A tested); §4.1/§4.4 (Drive repoint, registry reconciliation) still
genuinely open — verified live. REGION_PAGE_IMPLICATIONS: §2.7 (L136-144) is the clearest
absorption map in the set (§7 Region Page primary consumer via 06_chapter_research/
08_build_region_page; §8.6 provenance columns travel into the registries the page reads) —
prime Phase-C raw material once re-pointed. KEEP: the grounded §8 walkthrough (still verbatim-
accurate); the retraction-cascade + cultural-authenticity-audit strengths; the still-true
Drive-path/registry corrections.
3-entrypoint query API, 8-step order). V3_GAPS: L68-76 v2.5-sectioning is the LITERAL source
of the live chunk.py staleness; §5 metadata has no typed-unit fields. STACK_STALE: §3's
nomic-embed decision and §4's ChromaDB trade-off table both inverted by the real build; the
L112-123 example config is a path/schema fossil (rag_config.json/chromadb_data vs the real
config.json/.rag/). UNFINISHED: build steps 4-6 superseded by real Scope-A progress; the
research_corpus_ref link flag unlanded. REGION_PAGE_IMPLICATIONS: §6.4 names the enrich.py
--semantic seam (still unwired); §6.1 is the literal absorption signature. KEEP: chunking-
boundary-drives-citation-exactness; the per-unit canon-chunking rules (add v3 unit types
alongside); the ChunkResult shape + 3-entrypoint API; the store-adapter abstraction (it is WHY
the sqlite-vec swap was cheap — the design's own principle validated against its stated choice).
V3_GAPS: minimal surface; offers zero v3 guidance. STACK_STALE: repeats ChromaDB + proposes a
THIRD file-naming convention (superseded by §2.0's canonical names before anything was built);
L36 Ollama/768 assumption. UNFINISHED: the Flores vertical slice (Step 5) not started; the §5
contract deltas still unapplied. REGION_PAGE_IMPLICATIONS: weakest of the set (gestures only).
KEEP: the store-vs-window/"retrieval is the bridge" framing (the best exposition of it) + the
forced-order dependency reasoning — durable and stack-agnostic.
typed-unit metadata); add the one-line qualifier + the v3 re-chunk item.
primary). UNFINISHED: L40-42 self-declared items — the enrich.py --semantic seam (still
unwired) and a stale "Scope A ... Needs Ollama" claim (Scope A is built+tested on fastembed,
per file mtimes + the manifest's embedding_is_real: true).
calls; the substrate absorption lives in BOOK_PROCESSING_WORKFLOW.
trust-content-not-labels model the four proposal docs should be corrected TO (state what runs,
cite the manifest, never let a status label overstate).
(Full reader report: the session task archive; this is the load-bearing distillate.)
Pipelines → Validation/Orchestration) + the prioritized 7-gap list + Fable handoff items. Meta,
not operational.
re-statement (completeness = typed beat/fork/site/boss/SQ/scene coverage); L19's Layer-3 binary
over-scopes the semantic gap (see cross-cutting #4); L20 doesn't credit the quest-binding
problem the v3 rewrite resolved.
Cesium = REVISIT not drop (real-world geodata ingestion is a different function than
procedural generation); MetaHuman/PCG are UE-native (not stale); the NeoStack plugin mention is
ALREADY aligned. AIVA/ElevenLabs outside the pivot.
list carries no status deltas (its own item 3 is substantially discharged by PIPELINE_ADAPTER
without cross-reference).
plan changes it.
this... else decoration"); the engine/game separation (L44-46); the gap CATEGORIES.
autonomy-readiness verdicts, the 38-value unpinned-config ledger, and the 10-item
[DECISION NEEDED - JOSH] roll-up.
doc with heading anchors (reality: 79 files + in-file § anchors); L194 "the 200-400-word body
spec ... stands" is FALSE (Block 4 demoted to a short emotional lens; the typed graph is the
event spec); L189's flat ~20-column scatter is mis-sized vs the v3 child-table shapes; L105/126
T0_Scene_Spec_Registry (31 cols) vs the spine's ratified 7-col §13 rows = two schemas for one
concept, unreconciled; L126's condition_expression grammar gap is PLAUSIBLY ALREADY CLOSED by
the v3 §5.3 gate mini-grammar (check before inventing); L153's "behind the spine" framing is
cleared; L157 region_id-as-free-text still true BUT v3 §3b sites are the new join surface.
pin L230, the schema pin L247) — re-run its readiness against a 5090-local backend (no API key,
no per-generation cost, different failure modes); the REGISTRY schema work in §2 is
tool-agnostic and survives. L262 Gaea2Unreal questioned; Cesium revisit. NeoStack pins aligned.
Mesh-import-format pin (L251) survives any backend.
Weapon_Registry column contradiction (§2.3 vs §10.3); the Folklore_Entity archived-vs-live
contradiction; the SFX registry spec-vs-live divergence (19 vs 20 cols, mojibake); the 10-item
decision roll-up — none resolved as of the doc.
L274 terrain-affordance is 5090-changed; L169's dormant OCR path is worth re-costing (local);
L242's "embeddings: pin never" is the retirement the cross-cutting #3 conflict re-opens.
canon/region_pages/... needs re-siting (no canon/ root exists); region_page_path + legendary_quest_path FKs mean
pages are referenced FROM ≥2 registries; corpus_query_log_path = pages carry their own
retrieval-provenance trail; T8_section_templates/ JSON is where the field list is meant to
live; retrieval is grep/read + deep-research by design (no vector-DB dependency hardcoded);
region_id FKs fan out to 13 registries; two real upstream CONTENT gaps (per-region
canonical_zones mostly unauthored; the Combat predator-prey leg unwritten); 67 live vs 69
canonical regions; and the cross-cutting-#5 join path is the new deterministic backbone.
pin-when-load-bearing ledger policy; the decision-roll-up mechanism (needs a fresh pass).
engines E0-E8 in dependency order with owning contracts, recall.py query shapes, the 8-stage
loop + machine/human exit gate, cost staggering, 3 open Josh flags.
(§3d thin-index; live CH_13 confirms); E3 modeled as a downstream engine reading a locked spine
— boss rows were actually authored chapter-by-chapter across the whole cascade (the 282-row
registry is the evidence); loop stage 3 presumes E1 unrun (all 79 done — the loop needs the
E1-pre-done variant); the locked/drafted state model needs reconciling post-rewrite (CH_13
still reads spine_status drafted); the recall.py-as-aspirational framing is favorably stale.
L49 AIVA/ElevenLabs = outside the ruled pivot, status UNVERIFIED (a Phase-B question); the
cost model never priced E6/E7 SaaS spend at all — the local-5090 swap removes an axis the doc
never modeled.
model pin — now sequences against the 4-8-week workstation gate); the loop has never run once;
the central slice-first cost premise superseded in practice — re-argue or retire it.
roster folded from every citing chapter's §3b rows (E4/E6 gate on zone bounds), (2) a
per-region boss/sub-boss index (BE ids + verbs + threads) so E3 never re-derives the spine,
(3) the region's SQ hooks (spatial_requirements feed), (4) asset-hook fields re-pointed to the
Meshy replacement, (5) the recall-node binding.
cost-staggering principle; the per-engine recall query shapes (now MORE true); the
machine+human+provenance exit-gate discipline.
layer (vril-density dial, Composure, evolve-not-swap, ~30 clocks, Sigma-bounded RNG,
win-AND-death economy), Parts C/D index the runtime + milestone docs, the Appendix = the
2026-07-04 canon audit.
no verb/axis/win columns) is FALSE — 282 rows with those exact columns; the §3.7 three-taxonomy
mode reconciliation ALREADY HAPPENED (the v3 §4 closed 12-value gameplay_function enum + the
Pillar-1.12 projection); §3.3-3.6's "not specified" items are narrowed by the v3 beat/fork
grammar (Part B's deeper mechanics remain its real contribution); the recall.py "does not yet
exist" note was wrong on the day it was written.
(cross-cutting #9).
written; one appendix of two; the §12/Part-B numeric coefficients still deferred to the unrun
balance audit; the D-3/D-4/D-5/D-9 ledger superseded by DECISIONS_PENDING_JOSH in practice;
the L93 [DECISION NEEDED] rows now exist in-registry (signed off in practice, unreflected).
permitted-vs-protected flags (the 132-row roster + care tiers), environmental-boss-protocol
state for the 6 canon instances with arena-level vril-site layout (maps onto §3b fields), the
region-level thread/boss index for multi-chapter regions, school/weapon-legendary home-base
anchors (13 schools + 72 weapons land in home regions), rolled-up mode-variety rotation state
(the §11 anti-staleness rules).
taxonomy (matches numerical canon); the Appendix's what-is-CANONIZED catalog.
SRC_NNNNN→rename/move), the 4-step pipeline, three special-format recipes (sacred-texts DVD,
NatGeo .cng, Sámi drum corpus).
binding — cross-cutting #7).
OCR claim NOT REAL on this machine (ocrmypdf + tesseract both absent — verified by version
check); the region gazetteer is a 24-entry exact-substring map with no fallback for no-hit
titles (→ unassigned/) — exactly the local-7B/3B triage class the standing policy earmarks,
unmentioned.
SCHEMA_VALIDATION_PLACEHOLDER rows, zero real; the claimed Batak ingest (L58 status) left no
persisted artifact (no book_substrate manifest record; processed/ empty — the trust-content-
not-labels violation class); recipe B has nothing staged (nationalgeographic/ EMPTY); no
ingestion path for the staged .docx/.doc/.jpg/.webp/.rdf.
real blocker is OCR-binary installation, not compute tier; Ollama listed as secondary but dead
(not installed) — harmless.
citation (query.py --collection book_substrate --chapter CH_NN → (title,author,year,page));
the where-does-a-citation-LAND-in-a-region-page schema is explicitly deferred — Phase C
defines it.
SIGNED OFF + PURCHASED 2026-07-15; acquisition closed, handed to processing.
66-77 chapter/topic mapping during the O-1 Opus re-key.
individually-pending while the §8 banner says ratified-as-listed (label-vs-table tension); the
Hewlett registry-vs-RCM ownership conflict correctly deferred to the RCR cleanup.
spot-verified accurate).
AUTHORIAL-REBUTTAL / REFERENCE / TONE / FRINGE-SKIP); the declared gazetteer seed.
regions of named primaries — expand by hand OR route no-hit titles through a local-7B/3B
classifier fed this doc's vocabulary (the local-first move).
into a region page — SUBSTRATE cites as primary evidence; ESOTERIC-PRIMARY only in the L3
esoteric-grounding layer; AUTHORIAL-REBUTTAL (Bulwer-Lytton, Blavatsky, Donnelly) may NEVER be
cited as truth (the vril-firewall operationalized). Dropping the tag at absorption is a
canon-fidelity failure class, not a style slip. The Phase-C template carries a per-citation
use_class field.
the Abimbola BUY, ~20 per-article Sámi Scripta PDFs + tracings + thesis, Batak, Tamil/Tantra,
Sri Lanka, Ibn Battuta ×5, Flores/Komodo/hominin, San/Khoisan, Swahili/Kilwa, wootz steel, an
anti-fringe cluster incl. Goodrick-Clarke).
the recipe must SKIP run.exe/autorun.inf (never execute) and the DVD chrome.
described files; 4 .jpg + 1 .webp museum-ID images staged loose at top level belong in it.
two SRC_ ids; dedupe pre-ingest); .docx/.doc/.rdf need a defined path or manual triage.
(The full 10-section contract walkthrough lives in the reader report; this is the distillate.)
_source/02_Tier_2_Region_Pages/ = README (the tier's purpose + theexplicit "Empty scaffold awaiting (1) the v3 rewrite [DONE] and (2) the region-page-building
pipeline engine") + the inbox. 0 of 69 pages exist. THE TEMPLATE LIVES ONE TIER UP (SUPERSEDED by v2.0, 2026-07-15):
T1_Region_Page_Template [ACTIVE v2.0] — 20 sections (Geography · Flora · Fauna · People ·
Spiritual/Occult · Weapons/Military · Architecture · Politics/Factions · Minerals/Trade ·
Music/AIVA brief · Velheim Presence · Characters · Quest Framework · Trade Skills ·
Cross-Regional · Academic Refs · UE5 Build Specs · Time Period/Fluctuation · World-State
Versions · Session Planning) + Region Identity and Character Quick Reference headers.
Template defects: the retired Folklore_Entity feed; the §19 stale vocabulary (cross-cutting
#10); no per-citation use_class field; no consumption boundary vs the v3 §3b site rows.
§0 schema/runtime split): §0 KEEP (the split is the future engine API; the six Translation
Docs remain unauthored). §1 UE5: 14-subsystem roster KEEP; terrain modeled as Cesium+Gaea2UE+
SRTM (+ enum) — unreconciled vs the NVIDIA world-gen fact; PCG should read §3b sites; Save
State = the ratified 39 ws.* keys. §2 Meshy: RETIRED end-to-end (SaaS/API/credential model);
the 7-category taxonomy + mesh_tags schema + state machine + §2.8 cultural discipline are
vendor-agnostic KEEPs for the 5090-local replacement. §3 AIVA / §4 ElevenLabs / §10 SFX
(ElevenLabs-SFX, Stable Audio Open, Boom/PSE/Soundly): vendors UNCONFIRMED vs the pivot — the
Phase-B question; the voice-clone authorization gate (§4.4.1) and both cultural-authenticity
disciplines are unconditional KEEPs. §5 Cinematic: the scene_type 11-enum is v3-adopted
verbatim (KEEP); the input mechanism + path_bucket enum re-key per cross-cutting #11; the
blocked_on_dependencies state machine KEEP. §6 Quest: the v3 thesis consumer — beat/fork/
spatial shape now originates in the spine; the six-option framework v3-preserved (KEEP); the
L1331/L1450 stale vocabulary. §7 Region Build: Mode A/Mode B + cross-region reciprocation
KEEP; canonical_zones→§3b consumption; the two-Drive-paths inconsistency; L1670 stale
vocabulary. §8 Book Processing: the three-layer model + 12-category taxonomy KEEP; the whole
path convention is Drive-era (the real inbox is elsewhere and live); reconcile with the BUILT
fastembed/sqlite-vec harness (lane 3). §9 Story Spine: WHOLLY SUPERSEDED by the executed v3
(body-prose-dominant + scatter + the 7/8-state machine + the 9-grid RR → the typed graph +
check_spine_graph gate); civilizational_marks preserved as v3 Block 6; L2280 stale path
vocabulary; re-base is the correction pass's biggest single rewrite. NeoStack naming (§1.14/
§5) plausibly = the ratified Neostack AI — Phase-B confirm, not assumed.