book_pipeline_assessment.md

pipelines/book_pipeline_assessment.md

§8 Book Processing Pipeline — Honest Architect's Assessment

CANON SUBORDINATION — this document is PROPOSAL-TIER: it serves canon and never outranks it.
The canon: the CVD · the T1 foundation docs · the T0 registries (registries/) · the spine
(docs/spine/CH_*.md) · the region pages (_source/02_Tier_2_Region_Pages/). Authority order: docs/DOC_MAP.md § 0.
Canon served (scanned from this document's own citations — widen it by hand where it is thin):
CVD §6 · T1_Build_Pipeline_Contracts · T0_Research_Corpus_Registry · T0_Schema_Dictionary · T0_Source_Material_Registry
READ THAT CANON FIRST — open it and derive from it before you build anything from this document.
If this document disagrees with canon, CANON WINS and this document is the defect — fix the
document, never the canon. Nothing here is applied until it is ratified into canon.

Assessment date: 2026-07-13. Scope: the §8 Book Processing Pipeline Contract inside

_source/01_Tier_1_Foundation/T1_Build_Pipeline_Contracts [ACTIVE v1.0].md (lines 1888–2183,

sub-sections 8.1–8.11). This is a build-infrastructure assessment — proposal/design, not world canon.

Everything below cites the actual contract, the repo state, and the two companion docs

(docs/AUTONOMOUS_BUILD_ARCHITECTURE.md, docs/BOOK_ACQUISITION.md).

---

1. Bottom line up front

cross-reference — it is ~300 lines of genuine architecture (11 sub-sections) with a source-material

registry schema, a 7-state machine, a 12-value category taxonomy, 12+ extraction streams, a corpus-query

access pattern, a cultural-authenticity audit cadence, a retraction-cascade, and version discipline.

As a specification of a book-substrate RAG pipeline, it is sound.

T0_Source_Material_Registry holds 5 placeholder rows (SCHEMA_VALIDATION_PLACEHOLDER_001…005),

not real sources. T8_section_templates/ does not exist. It is a contract-only pipeline.

(/02-Book Processing/..., /Humanity/00_Tier_0_Master_Indices/...) and the registry-of-record is

described as "canonical Drive sheets" read via "Google Sheets API." Canon has since migrated to

registries/*.csv + _source/**/*.md in git. The access pattern must be re-pointed at the repo.

It does not embed the canon itself (Scope B: canon-reasoning RAG — the "enrich your reasoning"

goal). §8's vector store is keyed to SRC_NNNNN source rows; there is no canon-embedding lane anywhere

in it. Scope B is a genuine architectural gap §8 does not address.

synthesis for ~764 of the sources; buy only ~158). §8 has no ingestion path for deep-research output.

Verdict in one line: **§8 is a strong, buildable blueprint for book-substrate RAG that (a) has not been

built, (b) is written against the pre-migration Drive world, and (c) silently omits the canon-reasoning

RAG the parent task most cares about.**

---

2. What §8 actually specifies (grounded walk-through)

2.1 Identity and doctrine (§8.1, lines 1890–1898)

§8 positions Book Processing as the upstream content-acquisition pipeline — it "builds and maintains

the canonical research corpus… and serves the corpus query layer to downstream Region Page Build, Quest

Writing, and validation pipelines" (line 1892). Critically, it gets the **constitutional-vs-consumer

separation exactly right: "Substrate is canonical, derived content is build-asset… ChromaDB chunks

are build-asset, regenerable from T0_Source_Material_Registry rows at any time**" (line 1894). Canon

lives in the registry; the vector store is a rebuildable consumer artifact. This doctrine is correct and

must be preserved on repo-port — it is exactly what keeps the RAG index a consumer layer and canon

authoritative.

The three-layer operational model (line 1898): Layer 1 = normalized OCR'd corpus (processed PDFs),

Layer 2 = ChromaDB vector store + structured T0 indices, Layer 3 = the query layer used during

Projects 3–7.

2.2 Ingest (§8.4 Phase 1–2, §8.5, §8.7)

ocrmypdf with worker parallelization, routes to _processed/ or _failed/. OCR engines: ocrmypdf /

Tesseract / Czur-native, with a multilingual Tesseract language-pack list (ocr_languages, line 2004).

manual author pass assigns primary_category, region_tags, chapter_scope_hints,

cultural_authenticity_classification, thread_relevance_tags.

ocr_failed → manifest_enrichment_pending → ready_for_indexing → query_ready → regeneration_required`.

This IS the "pending generation_status queue" the downstream pipelines described — for source material

it is source_material_status.

incl. canonical_retraction_status (a genuinely sophisticated field — see §2.6) and acquisition_status.

2.3 Chunk + embed (§8.4 Phase 2 indexing, line 1974; §8.5, §8.10)

(multilingual 100+ languages), 768 dims** (lines 1974/2013/2023/2142).

(lines 1982/2130). "Local SQLite-backed instance for Author seat operation."

primary_category, region_tags, chapter_scope_hints, thread_relevance_tags, embedding_vector.

discipline holds across full corpus." Correct and load-bearing.

This is a real, specified runtime — embedder, store, chunk params, and dims are all pinned. The

AUTONOMOUS_BUILD_ARCHITECTURE.md claim (line 19) that "there is no… runtime" is true of the *repo*

(nothing is built), but the *contract* does specify the runtime. That is better than the summary implies.

2.4 Extract — the 12 categories + the Phase-3 streams (§8.2)

occult_primary_source, regional_history, ethnobotany_ethnomedicine, archaeology_archaeoastronomy,

linguistic_corpus, primary_source_translation, scholarly_synthesis, ethnomusicology,

folklore_collection, ritual_practice. Each source carries exactly one; overlap surfaces via tags +

query (line 1919). PATCH-extensible; MAJOR migration on rename/removal.

that read the indexed corpus + a target T0 schema, compose an authoring prompt with retrieved chunks,

call the Anthropic API (or emit a paste-file), and output JSON build-assets that an author appends to a

canonical T0 registry. Targets: Folklore_Entity, Cross_Cultural_Link_Graph, School_Registry,

Vril_Site (expansion), Inscription_Spine (expansion), Language_Script, Creature_Roster,

Quest_Seed, Antagonist_Network (institutional anchors), plus per-region **cosmology / music /

herbal-medicinal** streams feeding T8 region-page sections.

target_t0_registry_stream_scope.json, carrying header metadata + extracted_entries[]; the author

reviews against schema, validates citations resolve to chunks, appends to the canonical T0 sheet.

primary_source_array (array of SRC_NNNNN), primary_source_citation_text, and

corpus_query_provenance — so every book-derived row traces back to its source + the query that

surfaced it. This is exactly the citation discipline a substrate RAG needs.

2.5 The corpus-query (retrieval) contract (§8.4 Phase 4, §8.9)

citations per query parameters covering **query keywords, top_k, region filter, primary_category filter,

chapter scope filter**. Read-only against canonical store."

**T8_section_templates JSON provides per-section query_keywords, top_k, and the output template

skeleton.** This is the parameterization layer — and it does not exist yet (see §3).

citation audit (07_audit_citations.py, line 1977) validates every downstream claim traces back to

a corpus chunk via semantic match — a §7.7 region-page quality gate.

This is a competent RAG retrieval contract: top-N + metadata filters + citation-carrying chunks + a

downstream claim-to-chunk audit. What it lacks: a re-ranker, an explicit similarity/eval metric, and the

top_k defaults themselves (deferred to the unbuilt T8_section_templates).

2.6 Two things §8 does especially well

(retracted_by_publisher | retracted_by_author | contested | superseded_by_newer_edition) triggers

automatic chunk removal from ChromaDB + a downstream citation re-routing audit through region pages

and T0 rows. The worked example — the Natawidjaja 2023 Gunung Padang retraction dropping a vril-site

anchor — is real and correct. Few RAG designs think about source retraction at all.

completion, bound to CVD §6/§17.1 hard lines, with the high-care set (Ch 29 Sámi, Ch 42 Haiti, Ch 57

Australia) and elevated set enumerated, primary sources named, and a surface-to-author-on-uncertainty

discipline. This is the Care Doctrine wired into the substrate layer — a strength.

2.7 Downstream consumption (§8.9)

Consumption fan-out is fully mapped: §7 Region Page = primary consumer (Mode A dossier via

06_chapter_research.py, Mode B section build via 08_build_region_page.py); §6 Quest = secondary

(via T8 region-page substrate + Phase-3-populated registries); **§1–5 UE5/Meshy/AIVA/ElevenLabs/Cinematic

hard-line chapter authoring, writes spine canon that §7 reads back). The "pending generation_status queue"

pattern the parent task referenced is real: book processing surfaces new entities into content registries

with a pending status that queues the asset pipelines.

---

3. Contract-only vs actually built — the honest ledger

ElementContract saysRepo reality
Scripts 00–09 (OCR, manifest, index, query, extract, audit, drift)/02-Book Processing/scripts/None exist. find returns nothing; only harness/ (canon gates) exists.
ChromaDB collection/02-Book Processing/chromadb_data/Does not exist. No chroma artifact anywhere.
T0_Source_Material_Registry2500–3900+ SRC_NNNNN rows5 placeholder rows (SCHEMA_VALIDATION_PLACEHOLDER_001…005), all identified_not_acquired, manifest_enrichment_pending. Schema is validated; content is empty.
T8_section_templates JSON (top_k, query_keywords per section)/02-Book Processing/schemas/T8_section_templates/Does not exist. The retrieval parameterization layer is unbuilt.
Ollama + nomic-embed-text runtimespecifiednot installed/wired in-repo
The 50-book pilot validation gate (§8.4, line 1986)mandatory before full-corpusnot run (no corpus)

So: **schema is locked and validated (the 5 placeholder rows prove the columns round-trip), but the

pipeline runtime and the corpus are 0% built.** §8 is a blueprint awaiting a builder.

---

4. Gaps, drift, and sequencing problems (specific)

4.1 Drive→repo drift — pervasive and load-bearing

Every operational path in §8 is a stale Drive path, and the **registry-of-record is described as Drive

sheets**:

/Humanity/00_Tier_0_Master_Indices/T0_Source_Material_Registry.

This is precisely the Layer-2 gap AUTONOMOUS_BUILD_ARCHITECTURE.md line 18 names ("the access pattern

still says fetch T0 sheets via Google Sheets API from Drive… must be re-pointed at the repo"). Fix:

re-point the registry-of-record to registries/<name>/<tab>.csv (git), the schema authority to

_source/** + T0_Schema_Dictionary, and re-home the operational paths (scripts/, chromadb_data/,

category_extractions/, region_research_outputs/, schemas/T8_section_templates/) under a repo-local

directory (e.g. pipelines/book_processing/). None of this touches canon content — it is a path/interface

repoint. Note the "Author seat / local SQLite" framing (line 1982) is *already* local-first and composes

cleanly with a repo-native, git-durable index; the only thing that must change is the Drive-sheet

registry-of-record and the Drive paths.

4.2 It is book-substrate RAG ONLY — the canon-reasoning RAG is absent (the big one)

The parent task names two scopes. §8 addresses exactly one:

region_tags/chapter_scope_hints/primary_category to author region/chapter/quest content.

docs, CVD, docs/** RULINGS/memory/spine — so a build agent semantically retrieves the right rules,

decisions, and prior canon for a task): §8 does not touch this. §8's collection is keyed to

SRC_NNNNN source-book chunks; its metadata is book metadata; nothing in §8.2–8.10 embeds a registry

row, a tier-doc section, a CVD hard line, a RULING, or a spine node.

This is not a nitpick: AUTONOMOUS_BUILD_ARCHITECTURE.md calls Scope B "**the single biggest missing

piece for reasoning-enrichment at scale**" (line 19) and the #1 gap for autonomy (line 32). §8, read

literally, would let a builder conclude "we have a RAG contract" while the reasoning-enrichment RAG remains

unbuilt and unspecified. Design decision to force now: one store with two namespaced collections

(substrate keyed to SRC_NNNNN; canon keyed to registry/doc anchors) vs two stores. Recommendation

(to argue in the sibling RAG-design proposals): **one ChromaDB instance, two collections/namespaces,

shared embedder + query harness** — because a build agent frequently needs BOTH at once (retrieve the CVD

hard line AND the ethnographic source that grounds a scene), and a single query surface with a

collection filter is cheaper than two subsystems. §8's own "chunks are rebuildable build-assets"

doctrine (line 1894) extends cleanly to a canon collection (canon chunks are equally rebuildable from the

git registries/docs). §8 should be extended, not replaced, to add the canon collection — the embedder,

store, chunker, query API, and citation discipline are reusable verbatim.

4.3 Ingestion lane ≠ ratified substrate strategy (deep-research has no path in)

§8 ingests OCR'd PDFs. But docs/BOOK_ACQUISITION.md (ratified, Josh 2026-07-03) makes

deep-research synthesis the DEFAULT substrate path: of ~922 "needed" sources, ~764 are

"DEEP-RESEARCHABLE (do NOT buy)" and only ~158 are MUST-ACQUIRE. §8 has **no ingestion contract for

deep-research output** — there is no state, no path, no SRC_ provenance convention for "an agent

deep-research synthesis with citations" as an indexable unit. As written, §8 can only RAG over the ~158

bought-and-scanned books, leaving the ~764-source majority of the substrate outside the index. Fix:

add a second ingest lane — deep-research synthesis dossiers (Markdown + citation list) chunked and indexed

with a source_type = deep_research_synthesis and their own provenance (the synthesis's cited sources),

distinct from pdf_ocr. This is a schema + Phase-1/2 extension.

4.4 Two "corpus" registries, unreconciled

There are two corpus registries and §8 references only one:

book roster (author, title, tier, ch_primary, filing_destination, status=needed_t1, etc.).

§8 never mentions the RCM. The relationship — an RCM "needed" row, once acquired/deep-researched, becomes

a SRC_NNNNN processed row — is unspecified. Without it there is no join between "what we planned to

source per chapter" and "what is actually indexed," and the chapter_corpus_coverage_summary /

primary_source_count fields (§8.6, lines 2049–2051) can't be computed against the plan. Fix: specify

the RCM→Source_Material handoff (an FK or a status transition), so acquisition-planning (RCM) and

processed-corpus (Source_Material) reconcile.

4.5 Sequencing: §8's monolithic full-corpus phasing vs Josh's RAG-first, chapter-incremental order

Josh's corrected sequence: **build the RAG layer → process books for Ch 2–5 → build region/chapter pages

slice LAST. §8 as written assumes a full-corpus batch**: Phase 2 indexes the whole corpus, a

mandatory 50-book pilot gate (line 1986) precedes "Phase 3 full-corpus extraction." That batch framing

does not match a chapter-incremental, RAG-first rollout. Two concrete mismatches:

1. "Build the store" is conflated with "fill the store." The RAG-first requirement is to stand up the

*runtime* (embedder + ChromaDB + chunker + 04_query_corpus API + T8_section_templates) **empty and

working**, THEN feed it Ch 2–5 books. §8 describes the runtime only as a step inside a full-corpus

indexing pass. Re-slice §8 into (a) a landable "index runtime, empty" milestone and (b) an

incremental "process Ch N books into it" loop keyed to chapter_scope_hints.

2. The 50-book pilot gate should be re-scoped to a Ch 2–5 pilot (the same books the first pages

need), so the pilot and the first real work are the same motion rather than a separate 50-book detour.

Note §8's core dependency logic is actually correct and supports RAG-first: retrieval (Phase 4) can't

run until indexing (Phase 2) fills the store — "RAG must land before book processing because book

processing fills the index" is honored. The gap is only the batch-vs-incremental granularity.

4.6 Runtime specifics that are deferred or thin

(§8.11, line 2159) — the single most important retrieval-tuning knob is entirely deferred.

queries" (line 1986) but never defines the pass/fail metric (recall@k, MRR, human-rated relevance).

100+-language folklore/ethnography corpus; the v1.5-vs-v2-moe choice is left to "build pipeline

configuration." Fine for a contract; needs a decision before build.

configuration per session-log retrospective" (line 1984). Cost envelopes are given ($150–1000 extraction;

~$700–1000 region build; $200–500 drift) — useful, but the model + the on-device-vs-API split is unpinned.

orchestration concern, but the durable-index/working-window split the parent task names maps cleanly:

ChromaDB local SQLite = the durable index (correct per line 1982); retrieval feeds the 1M window. No §8

change needed, just note it in the RAG-storage proposal.

---

5. Verdict

As a contract, §8 is sound and above the bar for its layer. It specifies ingest, a state machine,

chunking, an embedder, a vector store, a category taxonomy, extraction streams, a citation-carrying

retrieval API, provenance columns, a retraction cascade, a cultural-authenticity audit, and version

discipline — with the constitutional-vs-consumer separation correctly preserved (the vector store is a

rebuildable consumer of canon, not canon). A competent engineer could build the book-substrate RAG from

this document. That is a real asset; the design work is largely done.

But it is not buildable as-written without four corrections, and it silently under-scopes the parent

goal:

1. Repo-port the access pattern (Drive sheets/paths → git registries/*.csv + _source/** +

repo-local pipelines/book_processing/). Load-bearing; §8.11 itself lists the Drive location as open.

2. Add the canon-reasoning RAG (Scope B). §8 is book-substrate-only; the "enrich the reasoning" RAG —

the #1 autonomy gap — is absent. Extend §8's store to a second canon collection sharing the embedder +

query harness; decide one-store-two-namespaces (recommended) now.

3. Add a deep-research ingestion lane. The ratified default substrate path (~764 deep-researched vs

~158 bought) has no way into §8's PDF-only ingest.

4. Reconcile the two corpus registries (RCM T0_Research_Corpus_Registry acquisition roster ↔

T0_Source_Material_Registry processed corpus) and re-slice the phasing from full-corpus-batch to

RAG-runtime-first + chapter-incremental (Ch 2–5 pilot = the first real work), matching Josh's sequence.

Net: **§8 is the right blueprint for half the problem (book-substrate RAG), written for the wrong

(pre-migration Drive) world, and needs to be extended to cover the other half (canon-reasoning RAG) that

the parent task most needs.** Build the runtime empty and repo-native first; port §8's schema and query

contract verbatim (they are good); bolt on the canon collection and the deep-research lane; then feed it

Ch 2–5.

---

Appendix — primary citations

(8.1 identity 1890; 8.2 categories/streams 1900; 8.3 canon deps 1938; 8.4 access pattern/scripts 1966;

8.5 source+manifest+chunk schema 1988; 8.6 per-registry column adds 2025; 8.7 status/regeneration 2055;

8.8 cultural-authenticity 2083; 8.9 cross-pipeline linkages 2116; 8.10 version discipline 2132;

8.11 open items 2150).

harness/ (canon gates only — no book-processing scripts); no chromadb/T8_section_templates artifacts.

gap line 19; gap ranking lines 32–37); docs/BOOK_ACQUISITION.md (~158 must-acquire vs ~764

deep-researchable); registries/T0_Research_Corpus_Registry [DRAFT v0.1]/Sheet1.csv (2113 RCM rows);

_source/00_Tier_0_Master_Indices/T0_Schema_Dictionary [ACTIVE v1.3].md (SRC_NNNNN lock line 81, §8.6

column derivations).

Generated by harness/site/structure_site.py — the URL path is the repo path. review root