RAG_AND_BOOK_PIPELINE.md

pipelines/RAG_AND_BOOK_PIPELINE.md

RAG + Book-Processing Pipeline — Master Proposal

CANON SUBORDINATION — this document is PROPOSAL-TIER: it serves canon and never outranks it.

The canon: the CVD · the T1 foundation docs · the T0 registries (registries/) · the spine

(docs/spine/CH_*.md) · the region pages (_source/02_Tier_2_Region_Pages/). Authority order: docs/DOC_MAP.md § 0.

Canon served (scanned from this document's own citations — widen it by hand where it is thin):

T1_Build_Pipeline_Contracts · T0_Research_Corpus_Registry · T0_Schema_Dictionary · T0_Source_Material_Registry

READ THAT CANON FIRST — open it and derive from it before you build anything from this document.

If this document disagrees with canon, CANON WINS and this document is the defect — fix the

document, never the canon. Nothing here is applied until it is ratified into canon.

Status: PROPOSAL (build-infrastructure, proposal-first per CLAUDE.md). Owner: Josh. Authored 2026-07-13

from the three deep-dive drafts in docs/proposals/rag_book/ (book_pipeline_assessment.md,

rag_implementation_design.md, storage_sequencing.md), critic-corrected and reconciled.

This governs nothing canonical. The RAG index and the book pipeline are a CONSUMER layer. Canon

stays authoritative in registries/*.csv + _source/**/*.md + docs/** + user-memory. The vector

store only indexes a rebuildable copy of canon; it is a build-artifact, never a source of truth. This is

the constitutional-vs-consumer separation held exactly (T1_Build_Pipeline_Contracts §0/§8.1;

docs/AUTONOMOUS_BUILD_ARCHITECTURE.md Layer 2).

STORE SUBSTRATE — DECIDED (2026-07-13): sqlite-vec single-file + structure-first router

The Continuo sibling review (docs/proposals/CONTINUO_COMPARISON.md) settled this against the earlier ChromaDB+Ollama plan below. Continuo — a solo-built sibling brain under the same constraints — ships its vector-RAG as sqlite-vec in the SAME single (encryptable) SQLite file as the structured rows + FTS + provenance FKs, with a structure-first router (structured queries answer with ZERO prose tokens; the vector KNN is a bounded lane, not the default), asymmetric doc/query embedding at compile-time, and an L1-KNN→L0-drill — a proven, production posture.

• DECISION: the RAG substrate is sqlite-vec in one encryptable SQLite file (vectors + FTS5/BM25 + structured rows + provenance FKs together), fronted by a structure-first router — a deterministic structured/FK query answers first (zero prose tokens) and the vector KNN is a bounded fallback lane, never the default. Embedding stays a local model (Ollama/FastEmbed) at compile-time. The port is direct (both projects Windows/Python/local-first), removes the separate ChromaDB service, and inherits Continuo's real migration story. ChromaDB is the documented fallback only if the Ch-2-5 pilot surfaces a blocker sqlite-vec cannot meet.
• Supersedes the "one ChromaDB store, two collections" phrasing in the design section below — read that as "one SQLite file, two namespaces (canon, book_substrate)"; the chunking / metadata / retrieval-contract design is unchanged.
• Companion Continuo takes now landed or planned: build-lifecycle state machine = DONE (harness/build_lifecycle.py, the lifecycle gate — property-tested by exhaustive enumeration, catches commit-before-critic). Still planned (in docs/FOUNDATION_COMPLETION_PLAN.md): the entailment-verify pass (beyond anchor-exists to source-entailment; drop/demote unsupported claims) and a hard per-node budget + compaction on the retrieval layer (the one gap the sibling table found).

Two RAG scopes run through this whole doc, never conflated:

• Scope A — BOOK-SUBSTRATE RAG. The §8 corpus query over processed source books + deep-research

synthesis (ethnographic / historical / folklore / occult). Feeds region/chapter/quest authoring.

Non-canon evidence (authority tier 0). CONTRACTED in §8; largely unbuilt.

• Scope B — CANON-REASONING RAG. The canon itself embedded (registries + tier docs + CVD + RULINGS +

memory + spine) so a build agent semantically retrieves the right rules/decisions/precedent for a task.

The "enrich your reasoning" goal. NOT in any contract; buildable today against the repo as-is.

---

1. BOOK-PIPELINE VERDICT — how good is the §8 pipeline, and what is actually built

1.1 As a contract, §8 is sound and above the bar for its layer

The §8 Book Processing Pipeline Contract (`_source/01_Tier_1_Foundation/T1_Build_Pipeline_Contracts

[ACTIVE v1.0].md`, lines 1888–2183, sub-sections 8.1–8.11) is ~300 lines of genuine architecture, not a

stub. It specifies, correctly:

• Identity + doctrine (§8.1): book processing is the upstream content-acquisition pipeline serving

the corpus-query layer to Region Page (§7), Quest (§6), and validation pipelines — and it gets the

consumer separation exactly right: "ChromaDB chunks are build-asset, regenerable from

T0_Source_Material_Registry rows at any time" (line 1894). Canon lives in the registry; the store is

a rebuildable derivative. Preserve this doctrine verbatim on repo-port.

• Ingest (§8.4–8.7): ocrmypdf/Tesseract multilingual OCR → pypdf manifest + a manual author pass

assigning primary_category / region_tags / chapter_scope_hints /

cultural_authenticity_classification / thread_relevance_tags; a 7-state source lifecycle

(`pending → ocr_in_progress → ocr_failed → manifest_enrichment_pending → ready_for_indexing →

query_ready → regeneration_required`) — this IS the "pending generation_status queue" the downstream

asset pipelines described.

• Chunk + embed (§8.4/§8.5/§8.10): 512-token chunks, 75-token overlap ("15%, NVIDIA-optimal");

Ollama-served nomic-embed-text v1.5 or :v2-moe multilingual, 768-dim; ChromaDB local SQLite-backed;

single-model discipline enforced. The runtime is pinned, not hand-waved.

• Extract (§8.2): a 12-value primary_category taxonomy + 12 named Phase-3 extraction streams

(05_extract_category.py) that read indexed chunks and emit cited JSON build-assets appended to T0

content registries (Folklore_Entity, Cross_Cultural_Link_Graph, School_Registry, Vril_Site,

Language_Script, Creature_Roster, Quest_Seed, Antagonist_Network, per-region

cosmology/music/herbal streams). Every derived row gains primary_source_array +

primary_source_citation_text + corpus_query_provenance (§8.6) — real citation discipline.

• Retrieve (§8.4/§8.9): 04_query_corpus.py returns ranked chunks with full citations filtered by

query keywords, top_k, region filter, primary_category filter, chapter scope filter; per-section

query_keywords/top_k live in T8_section_templates; a citation audit (07_audit_citations.py)

validates every downstream claim traces to a chunk.

• Two things it does especially well: a retraction cascade (canonical_retraction_status auto-

removes chunks from ChromaDB + re-routes downstream citations — the Natawidjaja 2023 Gunung Padang

worked example is real), and a cultural-authenticity audit cadence (§8.8, Care Doctrine wired into

the substrate layer, high-care Ch 29/42/57 set enumerated).

A competent engineer could build the book-substrate RAG from this document. The design work is largely

done. Verdict: strong, buildable blueprint — for half the problem.

1.2 The honest built-vs-contract ledger (CORRECTED)

A prior draft asserted "no scripts 00–09 exist; only harness/ exists." **That is wrong, and it drove a

duplicate-build plan.** Re-find before asserting absence. The corrected ledger:

ALREADY PRESENT in-repo (under

_source/99_Tier_99_Operational/T99_Project_2_Book_Processing_Workflow_Files/):

Contract-only (specified, not yet coded):

• Scripts 01–08 (OCR, manifest, index, query, extract dossier, region build, citation audit) — **fully

specified in the T99 DRAFT §8 + the §8.4 contract**, but not yet present as .py files. Script 09

IS coded (above).

• ChromaDB store artifact, T8_section_templates JSON — not yet created.
• T0_Source_Material_Registry — 5 placeholder rows (SCHEMA_VALIDATION_PLACEHOLDER_001…005, all

identified_not_acquired/manifest_enrichment_pending). Schema is locked + round-trips; content empty.

So the accurate state is: **schema locked, runtime PROVEN (script 09 runs the real chromadb+ollama stack),

one of ten scripts coded, the operational spec for the other nine present in-repo, the corpus 0% filled.**

The build plan is therefore PORT + MINE, not build-from-scratch: port/repoint 09_drift_audit.py off

its Drive assumptions, mine the T99 DRAFT for the real chunking/pilot/script specs, then code the missing

01–08 against that spec.

1.3 The four corrections §8 needs before it is buildable-as-written

1. Repo-port the access pattern (Drive → git). Every §8 path is a stale Drive path

(/02-Book Processing/…, /Humanity/00_Tier_0_Master_Indices/…) and the registry-of-record is

"canonical Drive sheets via Google Sheets API." Canon migrated to registries/*.csv + _source/**.

Re-point: registry-of-record → registries/<name>/<tab>.csv; schema authority →

_source/** + T0_Schema_Dictionary; operational dirs → repo-local (§3 of this doc). §8.11 itself

lists the Drive location as an OPEN item — this is a mechanical, canon-free repoint. Edit the §8 text

in-place (living-source) so no later agent re-hallucinates Drive.

2. Add the canon-reasoning RAG (Scope B). §8 embeds only SRC_NNNNN book chunks; it never embeds a

registry row, a tier-doc section, a CVD hard line, a ruling, or a spine node.

AUTONOMOUS_BUILD_ARCHITECTURE.md calls Scope B "the single biggest missing piece for

reasoning-enrichment at scale" (gap #1). Extend §8's store with a second collection sharing the

embedder + query harness (§2 below).

3. Add a deep-research ingestion lane. docs/BOOK_ACQUISITION.md (ratified) makes deep-research

synthesis the DEFAULT substrate path — ~764 of ~922 sources are "deep-researchable, do NOT buy," only

~158 MUST-ACQUIRE. §8 ingests OCR'd PDFs only; it has no state/path/provenance for deep-research output

as an indexable unit. Add a second ingest lane (source_type = deep_research_synthesis, native-text,

its own cited provenance) so the ~764-source majority reaches the index.

4. Reconcile the two corpus registries + re-slice the phasing.

T0_Research_Corpus_Registry (RCM mirror, 2113 T0_RC_* acquisition/needed rows) and

T0_Source_Material_Registry (§8's SRC_* processed rows) both exist; §8 mentions only the latter and

never specifies the handoff. Add research_corpus_ref on T0_Source_Material_Registry (SRC → its

T0_RC row) so "identified → acquired/researched → processed → indexed" is one traceable chain. And

re-slice §8's monolithic full-corpus batch into (a) stand-up-the-runtime-empty + (b) chapter-

incremental fill, so the mandatory pilot gate becomes the Ch 2–5 pilot (§4).

Net: **§8 is the right blueprint for book-substrate RAG, written for the pre-migration Drive world, and

silently omits the canon-reasoning RAG the parent goal most needs. Port its schema + query contract

verbatim (they are good), bolt on the canon collection + the deep-research lane, repoint the paths, and

re-slice to RAG-runtime-first + chapter-incremental.**

---

2. THE RAG DESIGN — the concrete repo-native stack

2.0 Canonical names (RECONCILED — these win over either source draft)

The two source drafts disagreed on paths/names. **This table is the single authority; ignore the

divergent names in rag_book/*.md.**

2.1 Scope decision — one store, two collections

**One ChromaDB persistent store, two collections (canon, book_substrate), one embedder, one query

harness, one rebuild driver, one freshness gate.** Not two stores; not one blended collection.

• Rejected: one blended collection with a scope flag — a naive semantic query could rank a non-canon

ethnographic chunk above a CVD hard line with nothing structural preventing an agent from weighting them

equally. That is the "invented canon at scale" failure this repo exists to prevent. Authority must be

physically separable.

• Rejected: two entirely separate stores — doubles the ingest/query/rebuild/gate surface for a solo

builder and blocks a fused build-time query without cross-store plumbing.

• Chosen: two collections in one store gives physical authority separation (each chunk carries

canon_authority_tier; a book_substrate-scoped query never sees canon and vice versa), shared infra,

independently rebuildable collections (--scope canon never touches a PDF), and a clean fused query

that returns the two result sets labelled and un-mixed. The strongest objection — a huge Scope A

dominating a small Scope B — is answered by per-collection rebuild, which Chroma already supports.

2.2 Chunking — asymmetric by source type (the core move)

The chunk boundary is chosen so the citation the pipeline must emit is exact.

Scope A (books) — fixed window, page-anchored. Honor §8's 512-token / 75-overlap, but split

structure-aware: page → heading/section → paragraph → 512-token window, **never merging across a page

boundary**. Each chunk records page_start/page_end so the (title, author, year, page) citation is

exact. Deep-research synthesis dossiers are treated as Scope A sources (source_type = deep_research,

ocr_method = native_text), chunked by markdown heading then the 512-window, carrying SRC_* rows so

citations resolve uniformly.

Scope B (canon) — one chunk = one already-authored retrievable unit (NOT a 512-window; a window would

fragment a hard line or fuse two rows and destroy the citation):

• Registry rows: one chunk PER ROW, serialized as a natural-language card (`"<PK> — <registry>:

field=value; …"); metadata carries registry_name, row_pk, fk_refs[], doc_path` — preserving the

FK graph inside the vector layer so a hit hands off to recall.py/enrich.py for the exact bindings.

• Tier-doc prose (_source/):** section-granular on the CVD-§17.14 H2/H3/H4 discipline (reliable

split points); sub-split >~800 tokens with the heading path prepended to every sub-chunk.

• CVD: §-granular; hard lines (§17) get one chunk PER HARD LINE (atomic, highest-value retrieval

unit), carrying hard_line_id + applies_to_pipeline[].

• RULINGS: one chunk per ruling (ruling_id).
• Memory files: one chunk per file (already atomic).
• Spine entries: section-granular per the v2.5 schema.

Because the 1M window can hold big chunks (§3), Scope B is indexed COARSE by design — rules must not be

severed from their exceptions.

2.3 Embedding — local Ollama, one multilingual model, both collections

nomic-embed-text:v2-moe (Ollama-served, 768-dim), local, for BOTH collections.

• Local, not API: the substrate will be re-indexed repeatedly (every chunk-param/model migration =

full re-index per §8.7); API embedding would bill on every rebuild of millions of chunks — untenable

for a solo builder. Ollama is free/offline/re-runnable, exactly what "rebuildable at any time"

presumes. §8 already assumes Ollama; script 09 already calls ollama.embed.

• Multilingual from the start: the substrate is deeply non-English (Indonesian, Sámi, Swahili,

Sanskrit, Nahuatl, Greek/Latin — the §8 ocr_languages enum). Starting on English-primary v1.5

guarantees a forced full re-index when non-English recall proves weak; start on :v2-moe (768-dim —

matches the §8.5 schema exactly, ~100 languages, Matryoshka-capable).

• One model across A + B: one embedding function, one dimension, one migration surface, comparable

A/B distances for the fused query. Each chunk still carries embedding_model_version, so a future

divergence is controlled, not blocked. Query vectors MUST use the same model+version as the target

collection (asserted from the manifest — a mismatch is a hard error, not silent bad recall).

• Pilot-gated fallback: bge-m3 (1024-dim, dense+sparse+multi-vector) is the pre-approved fallback IF

:v2-moe non-English recall is weak at the pilot gate. It is 1024-dim, so adopting it is a MAJOR

migration — chosen only on pilot evidence, never speculatively.

2.4 Vector store — ChromaDB (honoring §8), repo-native

ChromaDB, persistent local SQLite-backed client — as §8 already locks and as 09_drift_audit.py

already uses (chromadb.PersistentClient). Local-first, zero-server, per-collection embedding functions,

rich where metadata filtering (which §6's filter contract needs). The store adapter (`harness/rag/

store.py) hides Chroma behind upsert(collection, chunks) / query(collection, vec, where, k)` /

rebuild_collection(...), so if Chroma's un-versionable binary store dir ever hurts at scale, **LanceDB

is the pre-approved migration target** (versions cleanly alongside git) touching one adapter module.

2.5 Metadata schema — one schema unifying §8 + canon

Every chunk carries: chunk_id, collection, source_type

(book_page|deep_research|registry_row|tier_doc_section|cvd_section|hard_line|ruling|memory|spine_entry),

doc_path (repo-relative — the repo-native replacement for the Drive path), canon_authority_tier,

embedding_model_version, chunk_params_version, content_hash, plus shared scoping

region_tags[] / chapter_scope_hints[] / thread_relevance_tags[] (§8.5 vocabulary, reused).

• Scope-A adds [§8]: source_id (FK), denormalized source_title/author/year/publisher,

page_start/page_end, chunk_index_in_source, primary_category,

cultural_authenticity_classification, canonical_retraction_status (so the query layer can

exclude_retracted per §8.7).

• Scope-B adds: registry_name+row_pk+fk_refs[], section_anchor/heading_path, cvd_section,

hard_line_id, applies_to_pipeline[], ruling_id.

The shared scoping axes are what make the fused query work: "everything about Ch 2 Flores, substrate +

governing canon" is one filtered query across both collections.

2.6 Retrieval contract — matches §8.4, adds Scope B + the fused query

One module harness/rag/query.py, three entrypoints, all returning the same ChunkResult, all obeying

the recall.py output contract (CITED · COMPLETE-for-top-k · NO-HALLUCINATION — a filtered/missing

result is reported, never fabricated):

• **`query_substrate(query, top_k, region_filter, primary_category_filter, chapter_scope_filter,

thread_filter, cultural_tier_filter, exclude_retracted=True)`** — the 1:1 repo-native realization of

§8.4's 04_query_corpus.py. Consumed by §7 Region Build (Mode A/B) and §6 Quest.

• **`query_canon(query, top_k, authority_tier_max, source_type_filter, region_filter,

chapter_scope_filter, thread_filter, applies_to_pipeline)`** — Scope B. Answers "what canon/rules govern

this task" when no FK names them (e.g. "does anything restrict fighting this culture's serpent?" →

governing hard line + the rated-R monster ruling + the deities-portrayable memory chunk, each cited).

• **`retrieve(task_context) → { governing_canon: query_canon(...), supporting_substrate:

query_substrate(...) }` — the fused build-agent query, returning the two sets labelled and

separate**: the agent authors *from* substrate, *under* canon, and can never blend a tier-0 chunk into a

canonical decision. This is the constitutional-vs-consumer separation enforced at retrieval time.

Integration point: enrich.py chapter CH_NN --semantic — after the deterministic exact-FK bindings

(unchanged, still first-class), append a clearly-delimited "semantically-related canon (retrieved, cited)"

block = query_canon(<themes+region+threads>, authority_tier_max=3). The /spine-chapter cascade, the §7

Region Builder, and the §6 Quest Writer all read that one enriched pack — Scope B wired through the

existing enrichment seam, not bolted beside it. RAG is the semantic COMPANION to enrich.py, never a

replacement: enrich.py retrieves what the FK graph can name; RAG retrieves what it cannot (semantic

neighbours, thematic precedent, the untagged governing rule).

---

3. STORAGE + THE 1M RELATIONSHIP

3.1 Store vs window — different organs, both required

• The vector store is the DURABLE, ever-growing INDEX — the whole corpus (30M+ words of canon plus a

multiplied volume of book substrate), on disk, persistent across every session and build phase, growing

monotonically. Many multiples of any context window.

• The 1M context window is the per-task WORKING SET — volatile, per-invocation. Opus 4.8's 1M ≈ ~750K

words; even a 1M window sees < ~3% of a 30M-word game at once, before any book substrate. The model has

zero knowledge of the corpus between calls.

• Retrieval is the bridge: store → retrieve → window → generate → gate → commit. The query CLI selects

the right slice OUT of the store INTO the window each task. Anyone who says "1M means we don't need RAG"

is wrong at this scale.

What the 1M window changes: fewer, larger, cleaner chunks (whole rows / whole sections, not 512-token

shards — hence the coarse Scope-B chunking in §2.2), and more generous top_k with less aggressive

re-ranking (pull 30 candidates, let attention do final selection). What it does NOT change: the corpus

is still orders of magnitude larger than any window, still durable, still invisible between calls. The

store is mandatory; the window just makes each retrieval fatter and forgiving.

3.2 Where the index physically lives

**A local, gitignored BUILD ARTIFACT at .rag/, rebuilt from canon — NOT repo-tracked — plus a small

git-tracked manifest.** This follows §8.1's own doctrine ("chunks are build-asset, regenerable at any

time") applied to both collections.

• .rag/ (gitignored) — the ChromaDB SQLite store (.rag/chromadb/) + processed PDFs

(.rag/book_src/_processed/). Never enters version control: a binary vector DB in git would bloat the

repo, produce unmergeable binary conflicts, and silently drift from the canon it was built from — and a

stale committed index "lies with confidence," which the trust-content-not-labels discipline forbids.

• harness/rag/index_manifest.json (git-tracked) — per collection: embedding_model_version, chunk

params, chunk/doc counts, indexed source list, and a canon_content_hash over the inputs (for

canon: the concatenated content-hashes of every indexed repo-tracked registry/doc; for

book_substrate: the source_id+chunk_count+embedding_model_version set of query_ready sources).

The exact analog of docs/fidelity_baseline.json: a tiny tracked declaration governing a large derived

artifact.

• index_freshness gate (in run_gates.py via gates_config.json) — recompute the input hash,

compare to the manifest; if canon changed but the manifest did not, the gate goes RED ("index stale,

rebuild required"), exactly like the fidelity gate on undeclared registry drift. You cannot land a canon

edit and leave the reasoning index silently stale. (Run run_gates.py UNPIPED so $? is trustworthy —

per the standing gate-verification note.)

• Rebuild is cheap + local: python harness/rag/index.py --scope canon|substrate|all; Ollama

on-device embedding, no per-embed API cost; Scope B cheap enough to run on every canon-affecting commit,

Scope A incremental (only regeneration_required/newly ready_for_indexing sources) with a full

rebuild only on model/chunk-param migration.

Memory-file edge case (out-of-repo — see Open Flag 4). User-memory lives at

~/.claude/projects/C--dev-humanity-forgotten/memory/*.md, OUTSIDE the git repo. It is INDEXED into the

canon collection (its content is valuable standing guidance) but is **excluded from the RED

index_freshness gate hash**, because a repo-scoped gate cannot treat an out-of-repo, un-git-tracked file

as a gated source of truth. Memory freshness is tracked ADVISORY-only (a re-index reminder), not a hard

gate — otherwise an edit outside the repo boundary could redden a repo gate that a repo-only builder

cannot diagnose.

3.3 How an Ultracode agent queries it

Identical SHAPE to how the cascade already shells out to enrich.py/recall.py: a Workflow build agent

runs `harness/rag/query.py "<q>" --collection canon|book_substrate|all --top-k N --region <id> --chapter

CH_NN --category <cat>` as a step, gets cited chunks back on stdout, and the retrieved slice enters its 1M

working window for that node's authoring. The Workflow engine orchestrates

(AUTONOMOUS_BUILD_ARCHITECTURE.md Layer 5); the store is the knowledge; the 1M window is where the node

is built. The semantic arm is folded into the existing enrichment pack (enrich.py --semantic), not

bolted beside it — deterministic recall stays the backbone (it guarantees COMPLETE binding sets a

similarity search can miss); semantic retrieval adds the relevant context the FK graph cannot name.

---

4. BUILD ORDER + SEQUENCING

Josh's corrected order — **RAG-index-infra + ingest-existing-canon FIRST, then book processing fills it,

then pages/quests, then the concurrent rest-of-corpus stream, then Flores last.** "RAG before book

processing" does NOT mean a big empty machine idles: it means stand up the store and immediately fill its

canon side from the repo, so the reasoning-enrichment win is real on day one and the machinery is

battle-tested before the expensive book run writes into it.

Step 1 — RAG index infra (store + embedding + query CLI), empty. [~0.5–1 session]

harness/rag/config.json; gitignore .rag/; harness/rag/store.py (Chroma adapter); pull

nomic-embed-text:v2-moe, assert 768-dim, round-trip a fixture. Zero content dependency.

Step 2 — Ingest EXISTING canon → the canon collection. → Scope B works IMMEDIATELY. [~2–3 days]

harness/rag/index.py --scope canon walks registries/*.csv (row chunks), _source/**/*.md tier docs +

CVD (section/hard-line chunks), RULINGS + memory + docs/spine/* → embed → canon. Emit

index_manifest.json. Depends on nothing but the repo. Highest-leverage, lowest-cost win; de-risks the

whole store before a single PDF exists. This is the literal answer to "RAG before book processing."

Step 3 — query_canon + enrich.py --semantic + index_freshness gate. [~1 day]

Ship query.py::query_canon; wire the semantic block into enrich.py; add index_freshness to

gates_config.json + run_gates.py. Scope B is now live end-to-end through the cascade's grounding seam.

Step 4 — Scope A scaffold: PORT + MINE, then code the gaps. [~2–3 days]

PORT/repoint the existing 09_drift_audit.py off its Drive assumptions to the §3 repo paths; MINE the

in-repo T99_Project_2_Book_Processing_Workflow [DRAFT v0.1].md (§§3–8) for the real chunking/pilot/script

specs; then code the missing 01_batch_ocr → 02_build_manifest (→ T0_Source_Material_Registry rows,

source_filepath repointed to .rag/book_src/_processed/) → 03_index_corpus (512/75 → book_substrate)

→ query.py::query_substrate (the 04 realization). Add the deep-research ingest lane

(source_type=deep_research) and the research_corpus_ref RCM↔Source_Material link. Validated on a

handful of sources; the corpus is still empty.

Step 5 — Pilot validation gate, re-scoped to the Ch 2–5 books. [~2 days]

Run the T99 DRAFT §5.4 pilot procedure on the SAME books Ch 2–5 pages will need (not a separate 50-book

detour): process them, run retrieval-quality validation queries, **decide :v2-moe vs bge-m3 on this

evidence.** Must pass before wider extraction.

Step 6 — Book-process Ch 2–5 → fill book_substrate.

Run §8 Phase 1→2 for the Ch 2–5 source set; the state machine (pending → query_ready) tracks fill; Scope

A comes alive per-book.

Step 7 — Build Ch 2–5 region/chapter pages + quests (consume query_substrate).

§7 Region Build (Mode A dossier / Mode B section) + §6 Quest, scoped by region_tags +

chapter_scope_hints. The build pack now blends both collections: deterministic enrich.py +

canon-semantic (B) + book_substrate (A).

Step 8 — Concurrent stream: book-process Ch 6–77 + Epilogue + Prologue + Ch 1.

A separate, parallel stream fills the rest of the substrate while Ch 2–5 pages proceed.

Step 9 — FLORES vertical slice — the teach-and-correct loop. [LAST]

The loop over the first ~4 chapters connecting the schema to the plugins/code that actually build the

game. It exercises retrieve() end-to-end (canon + substrate) into authored region/quest output and

TUNES the retrieval params (top_k per section, chunk granularity, collection-blend weight, filter

tightness) against real quality — the empirical tuner. Last because it needs a filled store and a running

author to correct against.

Why forced, not arbitrary: infra (1) has no content dependency → first; canon ingest (2) depends only on

the repo → immediate, delivers Scope B before any book cost; book indexing (4–6) is the WRITER → nothing

retrieves substrate until it runs; pages/quests (7) are the READER → strictly follow the writer; Flores

(9) tunes the loop → needs 6–7 producing real output.

---

5. OPEN FLAGS for Josh (genuine forks only)

1. [MODEL — pilot-gated] nomic-embed-text:v2-moe (768-dim) vs bge-m3 (1024-dim). Recommendation:

start on :v2-moe (matches the §8.5 768-dim schema with no change, avoids a forced non-English

re-index), and let the Step-5 Ch 2–5 pilot gate make the final call on retrieval-quality data. A wrong

choice = a full re-index of the whole substrate. Flagged so the gate does not forget to make the call.

2. [STORE — pre-approved fallback] ChromaDB now, LanceDB reserved. Recommendation: build on ChromaDB

(contracted, already proven by 09_drift_audit.py); keep LanceDB as the pre-approved migration target

(git-versionable file) behind the store.py adapter if the binary store dir hurts at scale. No action

needed now — a documented escape hatch, not a fork.

3. [SCHEMA — build-infra, apply at populate] canon_authority_tier scale needs Josh's ratification.

The design uses a retrieval-weighting scale (0 substrate · 1 CVD · 1.5 rulings + memory · 2 T1

docs · 3 T0 registries + spine · 4 T99). Placing rulings/memory at 1.5, ABOVE tier-2 T1 docs, is a

NEW consumer-layer weighting — it is NOT derived verbatim from the CLAUDE.md authority hierarchy

(CVD > T1 > T0 > T99, which does not formally rank Josh's rulings or user-memory). This is a reasonable

infra weighting (a ruling is Josh's standing word), but it is proposed, not asserted canon. Ratify or

adjust the numbers. (Consumer-layer only — does not touch world canon.)

4. [FRESHNESS GATE — out-of-repo input] How should user-memory participate in index_freshness?

Scope B indexes ~/.claude/projects/C--dev-humanity-forgotten/memory/*.md, which live OUTSIDE the git

repo, while the gate hashes repo-tracked inputs (mirroring fidelity_baseline.json). Recommendation:

INDEX memory into canon but EXCLUDE it from the RED gate hash — track its freshness ADVISORY-only (a

re-index reminder), since a repo-scoped gate cannot fail on an un-git-tracked file a repo-only builder

can't diagnose. Alternative if you want it gated: snapshot memory into a repo-tracked

docs/memory_snapshot/ on a cadence and hash the snapshot. Pick which.

5. [CONTRACT DELTAS — living-source edits, apply on acceptance] Edit §8 in place. On acceptance of

this design: (a) repoint every §8 Drive path to the §2.0 repo names; (b) add a short §8 (or sibling)

section defining the canon collection + its metadata schema + the enrich.py --semantic seam;

(c) record the store-is-a-build-artifact + manifest + index_freshness discipline so the §8.7

regeneration_required triggers have a home. Mechanical/canon-free — flagged not applied, proposal-first.

None of the above invents world canon; all are consumer-layer build-infrastructure decisions.

Generated by harness/site/structure_site.py — the URL path is the repo path. review root