pipelines/DEEP_BOOK_PROCESSING_CONTRACT.md
CANON SUBORDINATION — this document is PROPOSAL-TIER: it serves canon and never outranks it.
The canon: the CVD · the T1 foundation docs · the T0 registries (registries/) · the spine
(docs/spine/CH_*.md) · the region pages (_source/02_Tier_2_Region_Pages/). Authority order:docs/DOC_MAP.md§ 0.
Canon served (scanned from this document's own citations — widen it by hand where it is thin):
CVD §17·T0_Antagonist_Network_Registry·T0_Boss_Encounter_Registry·T0_Character_Index·T0_Creature_Roster· (+13 more t0)
READ THAT CANON FIRST — open it and derive from it before you build anything from this document.
If this document disagrees with canon, CANON WINS and this document is the defect — fix the
document, never the canon. Nothing here is applied until it is ratified into canon.
Status: PROPOSAL — DRAFT v0.1, awaiting director adjudication. Authored 2026-07-20. This governs nothing
canonical and edits no canon doc; it proposes a new population leg for the §8 Book Processing pipeline and
lists the precise contract edits its acceptance would carry. Provenance: Josh's 2026-07-21 order — read each
book and process it across all of our canon, knowing what the game actually is (a 10x-scale AAA action-
adventure fantasy RPG showing players all three layers through immersion), and treat book processing as the
time to enhance the story, identify seeds and weavings and connections, and wire everything in with a named
consumer. It designs against the real contracts and vision (T1 Build Pipeline Contracts §8, §7, §6; the AAA
Re-Architecture Brief; the 33 Significance Lenses; the Esoteric and Science grounding laws; the RAG and Book
Pipeline master proposal), and it holds the cardinal rules: do not invent canon, extractions PROPOSE into the
existing ratification lanes, no output is done until it has a named consumer and a DOC_MAP row.
The corpus is built and indexed. The drain landed roughly 1,741 sources and 568,000 page-anchored chunks into
the book_substrate collection; the sources are region-sorted through the gazetteer; and the reject-audit gave
each a shallow story-seed tag — a one-line what-this-seeds-in-the-game index annotation. That index layer says
which books are seed-rich. It does not do the reading.
What does not yet exist is the per-book deep read. The §8 Phase-3 category-extraction streams are corpus-wide
and query-first: they ask the whole indexed corpus one question per registry (every folkloric entity in
Indonesia, every cross-cultural link on Thread 5) and rank chunks across all books. That is the right shape for
populating a registry breadth-first, and it stays. It is the wrong shape for the thing Josh is asking for:
nothing reads a single book cover-to-cover against the live canon and asks what THIS book, specifically, offers
the game — its concrete quest seeds, its weave-link proposals to what the arc already carries, its entity
candidates, and its Layer-3 rabbit-holes for the players who research to connect the dots. A query-first stream
skims the top-k of a book on one axis; a book-first deep read exhausts one book on every axis. The two are
complements, and the deep read is the missing one.
This contract specifies the per-book deep read as a new leg inside §8 — call it the §8 per-book deep-read pass
(Phase 3B), the book-first complement to the query-first Phase-3 streams. Its unit of work is the per-book
extraction record. Its outputs land through the same category_extractions build-asset home, the same T0
registry column adds (§8.6), and the same ratification lanes the query-first streams already use; it invents no
new canonical surface, only a new way to fill the surfaces that exist. It is region-batched and incremental,
riding each region's pass-2 per the ruled depth model, and it never gates on the whole library.
read reconstructs the book from its ordered page-anchored chunks (or reads the processed text directly) with
the target region's and chapter's canon pack resident, and mines the book against that pack.
an existing ratification lane and becomes a canonical registry row only on sign-off. A record that would
imply a chapter assignment, a numerical, a name, a hard line, or a new system does not resolve it — it flags
it. Do-not-invent binds every row.
source's page-anchored chunk. A row with no page anchor fails audit and never appends. This is the §8.6
citation discipline (primary_source_array, primary_source_citation_text, corpus_query_provenance) applied at
the row grain.
fun and systems, L1.5 ambient immersion, L2 completionist depth, L3 the rabbit-hole connections — so the
deep read directly feeds the three-layer mandate rather than dumping undifferentiated notes.
filter (real relational tie, lens registration, arc-service, layer-fit, care-and-Frame-A) within the
distance-decay budget before it queues. A finding that is merely interesting with no real tie and no arc
service is demoted to Layer-2 texture or cut, exactly as §4 of the lenses doc requires.
collective, never the individual antagonist or the myth's own monster; deities are real and portrayable;
a cryptid gets distinguished from an authentic living tradition; protected property is referenced not
reproduced; the living faithful are never combat targets. Care means weight and authenticity, never
omission, and never sanitizing toward rated-G.
book, the full Opus deep read on the seed-rich set); the verification bar stays at full strength (a fresh-
context adversarial critic plus the §8.8 batch cultural-authenticity audit on every record before any
append). Efficiency is tiering, never thinness.
The record is one structured object per source, named EXR_<source_id>, serialized as a JSON build-asset. It
carries a header, four typed output arrays, a book-level care summary, and a lifecycle status. Field-level
authority for any value that resolves to a downstream schema (a quest_type, a link_type, a target registry, a
layer) is the downstream contract named in the field; the record restates nothing it can cross-reference.
source_year travel with the record for human-readable audit.
re-decided here (the substrate-region ruling holds — no blanket slug-to-chapter baking).
additional chapter_scope_hints as a source-row correction (a manifest-enrichment feedback, not a chapter
assignment).
record (§4 rollout).
improving-loops provenance chain.
Each seed is a typed object naming a thing the player would do, not a theme. A seed carries:
encounter, region_page_section, mini_arc_situation, trade_expression, vehicle_or_mount, festival_or_daily_
immersion. These are the AAA-brief opportunity categories made into an enum (games and contests, monsters,
otherworlds, daily-life immersion, crafts, vehicles, situations, superstitions, music and performance).
faces, or learns-that-pays-off.
folklore_combat, school_curriculum, grimoire_link, seed_defense, regional_npc_dialogue, and the rest), else
NULL. A seed that fits no existing quest_type is flagged for a category-mint ruling, never force-fit — this
is the §7.5 preserved veto surface.
EXTENDED 2026-07-22b (D-QUEST-TYPE-MINT): "and the rest" now reaches twenty-one members, the three added
being minigame_or_contest, festival_or_daily_immersion, and trade_expression — the surfaces 89 seeds across
all 11 shelves of the queue had no home for. They are side-only surface members in the established
folklore_combat / grimoire_link / seed_defense pattern: a main-graph core or climax beat stays
main_story_per_chapter under CG-21 regardless of contest, festival, or craft flavour. Assigning the value
remains a §7.5 Section-13 decision, not an extraction-time one, and a seed whose honest home is a
T0_Minigame_Registry row, a region-page ambient section, or trade crafting substrate takes no quest_type at
all.
questlines or a trade line, else NULL.
§6.5 encounter_facet so a folklore boss keeps its facet without bending its primary type).
region_tags and chapter_scope_hints, never a new assignment. A seed whose natural home is a region with no
page yet flags the slug, per the reject-audit doctrine.
the §7.5 Section-13 seed stub's zone_ref or site_ref; NULL where region-ambient.
Each link is a both-ends-named, page-cited proposal typed to the live Cross_Cultural_Link_Graph vocabulary, so
it enters the graph's ratification lane as a well-formed row rather than free prose. A weave_link carries:
catalog-convergence, catalog-membership, continuity, living-tradition-continuity, motif-recurrence,
knowledge-transmission, diaspora-transmission, reception-transmission, linguistic-cognate, pattern-parallel,
contrast-inversion, weaving-progression, sets-up, callback. The enum is observed-pinned and extends only on
attested critic-verified demand.
link with an unnamed end is not a link.
weavings the link realizes (verified against the canonical thread roster, since the substrate carries
recurring thread-number mislabels).
type the graph enforces (a real tie, never a single-superior-source-taught-the-world claim, §19.7).
(the live column vocabulary; reveal discipline binds — an L1 or L2 link is player-perceivable, an L3 link is
design-only and never declared).
field, or regional_specialist per §8.6, so the graph carries its evidence tier.
Each candidate is a proposed being, place, person, school, or object, typed to the registry that owns it, so it
enters that registry's stream and its §8.8 audit as a well-formed row. A candidate carries:
Schema-Dictionary entity_id spec.
weapon, familiar_candidate, or boss_candidate.
T0_Vril_Site_Registry, T0_Language_Script_Registry, T0_Inscription_Spine, T0_Character_Index,
T0_Antagonist_Network_Registry, T0_Weapon_Registry, T0_Familiar_Registry, or T0_Boss_Encounter_Registry.
CORRECTED 2026-07-22 against the LIVE registries/ tree. Two members of the original list were wrong.
(a) T0_Folklore_Entity_Registry NEVER EXISTED and is retired: the project had already ruled the folklore
layer into T0_Creature_Roster under a species_class discriminator and declared it across 41
docs/fidelity_baseline.json expected_cell_diffs ([POPULATE→T0_Folklore_Entity_Registry] →
[POPULATE→T0_Creature_Roster (species_class=folklore)]), and that table is purpose-built for it —
folklore_behavior_substrate, folklore_attribution_source, cultural_depiction_required,
cultural_care_register — with species_class=folklore already carrying live rows. Josh ruled 2026-07-22
that deities land there too, under species_class=deity, the discriminator carrying what the table name
does not. So folkloric_being and deity both resolve to T0_Creature_Roster. (b) T0_Boss_Encounter was
simply the wrong directory name; the live registry is T0_Boss_Encounter_Registry. Provenance and the
full reasoning: docs/proposals/PHANTOM_REGISTRY_ROUTING_FINDING.md. The pipeline enforces this in
harness/rag/exr_conform.py (RETIRED_REGISTRY_MAP + KIND_SPECIES_CLASS) and harness/rag/deep_read.py.
categories, never a stereotyped descriptor).
what the source attests.
of reverence (entity_kind deity, folkloric_being, or an ancestor or guardian-spirit the record treats as
divine or venerated — divine or deity-adjacent PORTRAYAL is the trigger, not the entity_kind == deity token
alone), the portrayability handling per the deities-are-real ruling: whether the being is a good deity usable
as a disguised quest-giver or rescuer, or an evil being staged as a boss. Alongside it, emit contact_register
on every such row — the §2.6b attested-contact default, verbatim: embodiment and the mortal-disguise
quest-giver device are the DEFAULT for every deity and deity-adjacent being (value embodied_disguise); they
are set aside ONLY where the tradition's own record POSITIVELY ATTESTS a different mode (value mediated_envoy,
contact through the source's own named messengers or diviners), with a page-level citation; absence of an
attested theophany motif is NEVER grounds to un-body; contact_register is a CRAFT field and attaches NO
care-tier escalation. The hard limits are unchanged: never a deity-name on a mortal NPC, never a revered or
real figure cast as villain. See §2.6b for the standing rule.
myth's own dragon or chaos-serpent is a fightable boss, not a protected animal); the cryptid-versus-authentic-
tradition check (a modern Western cryptozoology construct is framed authorially, never credited as Indigenous
folklore); the protected-property check (joik referenced as form never composed, AIATSIS-registered art
referenced not reproduced, Vodou as living religion never Hollywood-zombie-trope); the Vril-Society firewall
where the source touches polar or Thule or Hyperborea or precursor content.
Each L3 connection is a real-lore dot the rabbit-hole player could connect, grounded and anti-fringe, so the
deep layer rewards research rather than credulity. An l3_connection carries:
cross-tradition resonance the book documents).
tradition-as-historical-fact, Layer-2-phenomenological-worldview, Layer-3-perception-unlock, artistic-bridge-
Tier-3 (documented history presented as history with the metaphysical read held as the game's interpretation,
the two never conflated on a player-facing artifact), or anti-fringe-reframe.
hyperdiffusion, the Nazi-occult Vril-Society conflation, cryptozoology-as-tradition); authorial-only, never
a player-facing debunk.
often the design-only twin of an L3-visibility link).
The record carries a civilizational_marks array — the L2 lane the four original classes missed, and a live
first-class surface: SPINE_SCHEMA_v3 Block 6 carries a civilizational_marks field per chapter, enrich.py
already pulls [SRC:]-cited marks into every build pack, and the Care Doctrine (§17.1) treats
builders-of-knowledge representation as a protective floor, not decoration. Each entry: the mark (a real
attested achievement — engineering, astronomy, medicine, navigation, agronomy, writing — of the book's
culture), the layer tag (L2, surfacing as discoverable world-texture), the page-anchored citation, and a
care note where the mark corrects a defamatory trope. Landing and consumer per the wiring table row:
the spine civilizational_marks field and the RCM verified-canon paragraph, through the same critic-gated
authoring pass that owns the page's marks material.
classification, naming the discipline that binds the whole book (the Sámi, Haiti, Australia high-care set and
the Congo, East African, South Africa, Mesoamerica, Amazon elevated set).
checklist for this record.
(the Gunung Padang worked example).
Three standing rules, ratified from three independent audits that converged on them before any ruling
existed. They bind every record entering this pipeline, every region pass, and the §8.8 audit. They are
CONTRACT-tier so they cannot become a note on one seed and rot.
CALIBRATION FIRST. The §17 floor is the only absolute (no genocide-as-goal, no real-world targeted hate or
atrocity-glorification, no sexual violence, no child harm). Above it, care means WEIGHT and AUTHENTICITY,
never omission; care-tier INFLATION is a defect; true history is never hidden or scrubbed; and where
material reads insensitive the instrument is RENAME or RE-ORIGINATE, never removal. Per CVD §17.1 the build
loop rules these and does not escalate them.
RULE 1 — PSEUDOHISTORY VERSUS HISTORY IS THE LINE, NOT SUBJECT MATTER. Fabricated occult or racial
pseudohistory (the Vril Society, Base-211 and hollow-earth, Thule mysticism, Ariosophy root-races,
hyperdiffusionist race-science, cryptozoology dressed as Indigenous tradition) is AUTHORIAL-ONLY because it
is FALSE, and may never surface as truth on a player-facing artifact. The documented historical record of an
atrocity, including the Third Reich and the Holocaust, is TRUE HISTORY and is admitted as ordinary substrate;
it may be named player-facing at the CH_41 apartheid and TRC tier, with elevated care, the affected people's
own agency shown and not only their victimhood, and antagonism routed through the institution or apparatus
rather than the community — wherever a ratified chapter's ground already carries it. No new node is minted to
hold it; the chapter count is locked cardinal canon. NO LIVING TRADITION IS UNIQUELY EXCLUDED: a tradition's
association with a real atrocity does not remove it from ordinary playable substrate, and a protective
exclusion applied to one tradition and no other is care-tier inflation and a defect. Authorial-only is scoped
PER-RECORD BY CONTENT, never per-shelf and never per-subject, and must name the record, its declared
consumers, and the specific adjacency it guards. THE ONE HARD HOLD is the adjacency rule: real Holocaust
property, restitution, looted-asset or financial-persecution material never feeds the House of Velheim
banking-dynasty engine or any antagonist row, and no WWII beat stages adjacent to that apparatus. A wrong
region tag is an ACCURACY defect fixed immediately and without a ruling, in both the registry row and the
chunk meta_json.
RULE 2 — LIVING-PRACTITIONER AND NAMED-INFORMANT SUBSTRATE. Published technique, materials, real species and
plant names, constraints, timings and craft grammar are carried at FULL SPECIFICITY — thinning them is the
failure §17.1 names, not care. Real public toponyms and settlements ship by their real names; a place name is
never withheld as a de-identification measure. Withheld, at minimum scope: a living private individual's
name; a village-plus-biography combination that pins one individual; and a specific living family's
bereavement staged as a game event. Where such a beat is wanted, RE-ORIGINATE it — author a different person
and circumstance, keep the motif and the full rated-M weight — rather than renaming or cutting it. Content a
tradition holds closed is referenced, never reproduced, triggered by a POSITIVE recorded restriction from any
record; silence in one source is not evidence of openness, and absence of any recorded restriction anywhere
is openness. This is a representation rule about real living people and about knowledge a tradition holds
closed; it is NOT an intellectual-property rule and asserts no property right. Care is collective: a
benevolent practitioner or community leader is never the story antagonist and never a quest target, and is
not thereby mechanically invulnerable.
RULE 3 — CONTEMPORARY HARM, REAL VICTIMS, AND TRUE CAUSE. Real named victims of violence within living memory
never reach a player-facing surface, and de-identification is at CASE-SIGNATURE grain: no reproducible
combination of place, date, method and distinctive ritual detail. Names stay in the citation trail as
provenance. The gate is ATTESTED-BELIEF, not attested-incident: the belief and its idiom must be attested for
the stratum, but a fictional incident built on that belief needs no attested instance of its own; the bar is
presentational — never depict lethal accusation as the era's customary practice, and never assert it as
something the culture did. THE TRUE-CAUSE CLAUSE: where the record shows community-internal harm arising from
sincere belief, grief, poverty, or the availability of a marginal person as scapegoat, THAT is the cause that
gets built. A corrupt broker or manufactured conspiracy may be ONE discoverable branch and must never be the
mandatory explanation — inventing a puppet-master to spare a community its own agency is a scrub, not care,
and teaches the player something false. COMMUNITY AUTHORSHIP: the community keeps authorship of its own
repair; its documented non-lethal mechanism is staged ALONGSIDE the harm, never only after a win and never
only the harm.
creature vs folkloric_being is decided by the SOURCE'S OWN FRAMING, not by the reader's judgement
of whether the being is "real": if the source presents it as a being of a tradition (a spirit, an
ancestor, a wildman, a guardian), it is folkloric_being; if the source presents it as fauna
(taxonomy, ecology, behaviour), it is creature. Two independent audits reached this rule before it
was ruled. species_class on T0_Creature_Roster carries the distinction downstream, so both land
in the same table without the routing label having to encode authenticity.
THE AUTHENTICITY FIREWALL IS AN ENFORCEMENT, NOT A CITATION. Saying the cryptid/authentic-tradition
distinction "rides in the care fields" was FALSE when checked: folklore_behavior_substrate is
populated on 0 of 132 roster rows, folklore_attribution_source on 2, and 9 of 12 folklore-class rows
— including Ebu Gogo and Mokele-mbembe, the two rows the firewall exists for — carry an EMPTY
cultural_care_register. A firewall stored in an empty field protects nothing. Therefore:
**Any row at species_class = folklore MUST carry a non-empty cultural_care_register, and where the
being is a modern Western cryptozoology construct rather than a living tradition, that field must say
so explicitly.** This is a required-non-empty append condition, checkable by gate — not a note.
Until the check exists, the §8.8 audit enforces it per-row and records the enforcement.
The per-book lifecycle is a small state machine, mirroring the §8.7 source-material machine so a book is
resumable and a stalled read is visible:
are queued into their ratification lanes.
legal terminal state; not every row need land).
migration since the record was authored triggers a re-read of the affected scope.
Per-row status (proposed, queued, ratified, rejected, superseded) tracks each output independently under the
book-level state, so one book can have ratified seeds and still-queued links.
Every output has a build-asset home, a ratification lane, a canonical landing, a named consumer, and a pipeline
step at which the consumer reads it. The table is the anti-orphan contract for this doc.
| Output | Build-asset home | Ratification lane | Canonical landing | Named consumer | Pipeline step (when) |
|---|---|---|---|---|---|
| The per-book extraction record | .rag/book_src/extractions/EXR_<source_id>.json (gitignored build-asset, tracked by an extractions_manifest.json) — OR docs/book_extractions/EXR_<source_id>.json (repo-tracked) [OPEN Q1] | — (the record is the carrier) | — (regenerable from the source + canon pack) | harness/rag/deep_read.py writes it; the merge step reads it | §8 Phase 3B per-book pass, riding the region's pass-2 |
| story_seeds[] | the record's story_seeds array | queued to docs/proposals/quest_seed_queue/<region>.md → the §7.5 Section-13 seed-stub typing → the §6.5 seed contract | T0_Quest_Seed_Registry rows + the region page Section 13 side-quest seed stub (proposed_quest_type, suggested_questline_id, zone_ref/site_ref, realized_beat_ref or region_ambient) | §7 Region Builder Section 13; §6 Quest Writer standard track (reads T0_Quest_Seed_Registry + region page Section 13) | §7 region-page authoring consumes at Section 13; §6 quest authoring consumes at standard-track initiation |
| weave_links[] | the record's weave_links array | the documented lane: §8.2 Cross_Cultural_Link_Graph stream proposal rows → §8.8 batch cultural-authenticity audit + the RR connective-web verify → author append | T0_Cross_Cultural_Link_Graph rows (the LIVE columns link_type, integrity_visibility, evidence_tier, evidence_basis, lore_pointer, primary_source; scholarly_basis_classification is the §8.6-PLANNED add, landing at Phase-3 stream population) | enrich.py thread/weave pull; §7 Region Builder Section 15 Cross-Regional Connections; the RR connective-web verify | §7 region-page Section 15 authoring; every Range Review's connective-web closure pass |
| entity_candidates[] (Phase-3-stream targets: folklore_being, creature, school, vril_site, language_script, inscription, antagonist) | the record's entity_candidates array | queued into the matching §8 Phase-3 stream's category_extractions/<target>_<region>.json → §8.8 batch cultural-authenticity audit → author append | the target T0 registry (Creature_Roster — which houses folkloric_being under species_class=folklore and deity under species_class=deity per the §2.4 correction — School_Registry, Vril_Site, Language_Script, Inscription_Spine, Antagonist_Network), each gaining primary_source_array + corpus_query_provenance per §8.6 | §7 Region Builder (Section 3 fauna, Section 5 schools, Section 11 vril, Section 12 characters); §2 Meshy (Sections 3/6/11/12 mesh); §6 Quest (folklore_combat, school_curriculum, grimoire_link) | §7 region-page authoring; §2/§4/§5 downstream mesh/voice/scene at region-driven dispatch |
| entity_candidates[] (cross-pipeline targets: boss_candidate, weapon, familiar_candidate, persona_npc) | the record's entity_candidates array | NO §8 Phase-3 stream exists for these owners — they route as CROSS-PIPELINE FEED PROPOSALS to the owner lane's own ratification (boss candidates → the BOSS_PERSONAS/Boss_Encounter design lane; weapons → the Weapon_Registry bond-quest lane; familiars → the Familiar_Registry population leg; persona NPCs → the Character_Index persona lane), each entering as a flagged proposal in that lane's queue doc | the owner T0 registry, only after the owner lane's own ratification | the owner lane (boss design, weapon bond quests, familiar population, persona system) | at the owner lane's next pass over its region/chapter |
| l3_connections[] | the record's l3_connections array | queued to the grounding_doc appendix (ESOTERIC_GROUNDING / SCIENCE_GROUNDING / thread doc) [OPEN Q4] + link-graph L3 rows where links_to resolves | ESOTERIC_GROUNDING.md / SCIENCE_GROUNDING.md L3 substrate note + T0_Cross_Cultural_Link_Graph rows at integrity_visibility L3_canonical_truth_unnamed with a lore_pointer | PROPOSED (not yet extant): an enrich.py L3 pull — today enrich.py reads SCIENCE_GROUNDING only and has no L3/rabbit-hole section; this consumer is contingent on resolving Q4 (a queryable L3-substrate home), with the link-graph L3 rows the already-wired half | /spine-chapter Step-1 L3 grounding; the RR reveal-discipline re-verify |
| civilizational_marks[] (the builders-of-knowledge L2 lane) | the record's civilizational_marks array (seed_kind = civilizational_mark; [SRC:]-cited; §17.1 Care-Doctrine lane) | the region-page/RCM verified-canon lane (the same critic-gated authoring pass that owns the page's marks material) | the spine civilizational_marks field (SPINE_SCHEMA_v3 Block 6) at the chapter build + the RCM verified-canon paragraph | the EXISTING enrich.py civilizational_marks pull (already live in build packs); §7 Region Builder | the chapter/page authoring pass that consumes the enrich pack |
| care_flags | the record's care_summary + each row's care_flag | the §8.8 batch cultural-authenticity audit (high-care routes to author review per §8.8) | the relevant registry row's care fields or a T0_Hard_Lines affected-region flag | the §8.8 audit; the normalize-doc critic; Josh at high-care | the §8.8 batch audit, before any row appends |
| extraction_status (per book) | T0_Source_Material_Registry.extraction_status (a new column, or an extensions JSON field) [OPEN Q2] | — | the source row | harness/rag/deep_read.py (resumable driver) + the rollout prioritizer | resumability across sessions; rollout tracking |
The deep read is only wired when the build pack carries it. enrich.py gains one section, the same shape as its
existing per-chapter pulls: for the chapter being built, it pulls the RATIFIED deep-read outputs that map to
the chapter — the ratified quest seeds for the chapter's region, the ratified link rows whose ends touch the
chapter, the ratified entity rows homed to the chapter, and the L3 grounding notes for the chapter's traditions
— and emits them in the cited pack the cascade author and the fresh-context critic both read. Ratified-only is
the firewall: a proposed-but-unratified row never enters a build pack, so the deep read can never smuggle
unratified canon into authoring. This reuses the existing grounding seam (enrich.py chapter CH_NN --json), not
a new one, exactly as the RAG semantic block folds into the same pack.
docs/proposals/DEEP_BOOK_PROCESSING_CONTRACT.md — this contract — consumer: the §8 book pipeline anddirector adjudication — status PROPOSAL.
harness/rag/deep_read.py (proposed) — the per-book deep-read driver and the merge step — consumer: therollout; writes the extraction records and queues rows — status PROPOSED.
.rag/book_src/extractions/ (or docs/book_extractions/) — the per-book extraction record home — consumer:the merge step and enrich.py's ratified-harvest pull — status PROPOSED.
docs/proposals/quest_seed_queue/ — the per-region quest-seed proposal queue — consumer: §7.5 Section-13typing and the §6.5 seed contract — status PROPOSED.
The rollout is region-batched and incremental, riding each region's pass-2 per the ruled depth model, and it
never gates on the whole library. The story-seed tag index — which as of the 2026-07-21 corpus-wide weave-tag
pass covers ALL registered sources (1,761 tagged, Opus-verified, the deity-roster and chapter-hint families
audit-corrected), not only the reject-audit set — is the prioritizer within and across batches; the
local-model triage floor's seed-richness score refines ordering inside a batch. Cost is tiered per the
global policy.
skeleton and a seed-richness score. This is the honest floor for do-not-skip: even a thin book gets a triage
record, so the corpus is fully swept, and the score tells the Opus lane where the value is. The reject-audit
tags seed this floor (the what-this-seeds annotations are the first triage pass already done).
cover-to-cover read against the canon pack, authoring the extraction record. Opus for volume authoring per
the tiering policy.
critic verifies every record: do the seeds clear the five-gate lens filter, are the links Frame-A and anti-
hyperdiffusion with both ends named and cited, are the entity candidates neither defanged nor mis-credited,
do all citations resolve to real page anchors, is the reveal discipline held (no L3 leak to an L1 or L2 row).
The §8.8 batch cultural-authenticity audit runs in the same pass; high-care records route to author review.
resolves foreign keys, and appends ratified rows to their registries with a fidelity-baseline refresh in the
same commit. Registry edits are never trivial-mechanical for review purposes — the content took the critic
and the audit; the merge is only the safe apply vector.
new canon (a new slug, a chapter-assignment implication, a locked numerical, a name in the naming canon, a
hard line, the §17 floor, a new system) route to the flag — never changed silently by the loop.
removed from the list above and one only: the clause this bullet used to carry — "a new quest_type routes to
the flag, never coined by the loop" — is retired-rule apparatus predating the cardinal-rule reframe. A
quest_type member is not a pillar and not a locked numerical; it names a surface rather than asserting new
state, so extending the §6.5 enum is execution within the locked vision, and it lands the same way every
other execution-tier change lands: proposal-tier, cited to attested evidence volume, adversarially verified,
and written into the ratified home with a dated inline provenance tag. This matches the contract's own
observed-enum-pin discipline (T1 §6.5 CG-25), that a domain extends only on attested critic-verified demand.
§6.5 now carries twenty-one members (minigame_or_contest, festival_or_daily_immersion,
trade_expression added on 89 unhoused seeds across all 11 shelves of docs/proposals/quest_seed_queue/).
Every other term in the bullet above still binds unchanged — a new slug, a chapter assignment, a locked
numerical, a name, a hard line, the §17 floor, and a new SYSTEM all still route to the flag, and this
reconciliation grants no licence beyond the one term it names.
member is recorded in mint_flags rather than force-fit, and per-seed typing stays at the §7.5 Section-13
stub, so an enum extension never commits a seed to becoming a quest row.
Komodo, Bali, Sumatra and Java, and the Angkor spread), which the RAG pipeline already names as the pilot,
so the deep read proves out on the same books the first region pages need.
seeds). These drain into book_substrate first (the INGEST verdict is decided but not yet ingested), then get
the deep read. The 9 JOSH_REVIEW items wait for Josh before either ingest or read.
The pilot proves the record schema, the five-gate lens scoring, the four ratification lanes, and the enrich.py
harvest wire before any scale spend. It is the empirical tuner for top-k, depth-tier thresholds, and the
critic rubric, exactly as the Flores vertical slice tunes the retrieval params.
PILOT SCOPE UNDER THE SLICE RESCOPE (Josh RULED 2026-07-27, fifth sitting). The vertical slice is now
PROLOGUE→CH 13, so the roadmap's P1 entry criterion covers 13 regions rather than four, and this pilot moves
with it — but the pilot set itself is UNCHANGED and stays the Ch 2-5 books. That is deliberate, and it is
the cheaper reading of the ruling:
regions keeps the tuner small and fast, exactly as authored.
region's deep read riding its own pass-2 and feeding its own region page. Those eight pages do not exist
yet and are the slice's longest ungated pole, so the batches start as early as their books allow (nine
counting the fairy_realm page, which is outside this lane per the bullet below).
fairy_realm (Prologue + Ch 1) is authored canon with no book substrate and therefore sits OUTSIDEthe deep-read lane entirely — it has no pilot batch, no pass-2 batch, and no triage score, and its
absence from every region-batch list is correct rather than a gap.
parallelize cheaply; it paces its own batch and must not be sampled (OPEN Q7's review-load question now
has a named high-care instance inside the slice, not only later in the arc).
After the pilot, each region's pass-2 carries its deep-read batch: the region's books, prioritized seed-rich-
first by the tag index and the triage score, deep-read against that region's canon pack, ratified, and the
harvest wired into that region's page build. This is the ruled incremental model — deep research rides each
region's pass-2, never the whole library at once — so the deep read produces usable output for the region being
built rather than idling behind a monolithic corpus run.
extraction_status on the source row plus a resumable JSONL log (the drain_inbox pattern, at
.rag/book_src/deep_read_log.jsonl) make the rollout survive session death: a killed batch resumes at the
first not_deep_read or deep_read_in_progress source, and a landed record is git-visible provenance. This is the
land-as-you-go discipline the repo already runs on.
These are the precise edits acceptance would carry. This doc applies none of them; it stages them for the
director.
the query-first Phase-3 streams: one paragraph naming the deep read, its unit (the per-book extraction
record), its four output classes, and the note that it feeds the SAME stream targets and lanes, not new ones.
extensions field, per OPEN Q2), and add the per-book extraction record JSON structure alongside the existing
category_extractions JSON structure, with the EXR_<source_id> naming convention.
primary_source_array and corpus_query_provenance column adds already specified; no new target-registry
columns are minted (the deep read fills the surfaces §8.6 already defines).
change, source correction, retraction, migration) to the source-material state discipline.
region pages (Sections 3, 5, 11, 12, 13, 15) and §6 quest seeds, alongside the query-first streams already
listed.
per-book deep read, the tier stack, the tag-index prioritizer, the rollout model, and the resumable log, so
the operational flow carries the new leg.
engine that produces the lens-scored candidates the §4 filter runs against (the lenses doc already names
enrich.py surfacing candidates; the deep read is where the book-first candidates come from).
extraction records are derived non-copyrighted structured records, and provenance argues for tracking them.
The fork: .rag/book_src/extractions/ (gitignored build-asset, tracked by a manifest, matching the store-
is-a-build-artifact doctrine) versus docs/book_extractions/ (repo-tracked, git-visible provenance for
every ratification). Recommendation: repo-tracked under docs/, because the record is the audit trail for a
canonical row and should be git-visible like the ratification lanes; the copyright constraint is on the PDF
text, not on a cited structured extraction. Flagged because it is a build-infrastructure choice with a
provenance-versus-repo-weight tradeoff, not a mechanical default.
Registry shifts the fidelity gate and needs a deliberate re-baseline; an extensions JSON field avoids the
re-baseline but is less first-class for a rollout prioritizer to query. Recommendation: a real column, with
the re-baseline in the same commit (the standing scatter ritual), because the rollout queries it constantly
and a first-class lifecycle field earns its baseline shift. Flagged as a registry-schema call.
The same folkloric entity or the same link can surface from both lanes. Who owns the merge, and does a
book-first row supersede, complement, or defer to a query-first row on conflict. Recommendation: the
deterministic merge keys on the canonical anchor (entity name plus culture, or link source-plus-target-plus-
thread) and unions the citations across lanes into one row, so two lanes strengthen one row rather than
minting duplicates; a genuine content conflict (two different combat profiles for one being) routes to the
critic, not to a silent last-writer-wins. Flagged because the merge policy is load-bearing and currently
unspecified in §8.
lands there at L3 visibility with a lore_pointer). But a research-players grounding note that is not a
cross-cultural link — a real-scholarship anchor for an occult mechanic, an anti-fringe reframe — has no
obvious registry home; the grounding docs (ESOTERIC_GROUNDING, SCIENCE_GROUNDING) are prose, not a queryable
registry. The fork: an appendix in the grounding docs versus a new L3-substrate note field or registry.
Recommendation: an appendix in the named grounding doc for the prose note, plus (where it also seeds a
perceivable connection) the link-graph L3 row, so the design layer has the grounding and the graph has the
pointer; a new registry is not warranted until the volume proves it. Flagged as an L3-home design choice.
dozens of cultures and belongs to no single region's pass-2. If it waits for every region, its weave-density
(its whole value) is deferred to the end. The fork: force-fit it to one region's pass-2, or give the
comparative_mythology and folklore_collection primary_category books a separate cross-region deep-read lane
prioritized by weave-density. Recommendation: a separate cross-region lane, run early and re-run as regions
land (its links target region pairs, so it re-queues cheaply as each end's region ratifies). Flagged because
it is a rollout-sequencing choice the ruled region-batched model does not cover.
read-each-book: the local-model triage floor genuinely reads EVERY book (do-not-skip honored, zero paid
tokens) and promotes any book to the full Opus deep read on evidence. The original question, for the record:
book. A full Opus deep read of every one of ~1,741 sources is a large spend, and the tag index says many are
thin. The reconciliation this doc proposes — a local-model triage floor over EVERY book (so every book is
read at least once, do-not-skip honored) plus the full Opus deep read on the seed-rich set — is a tiering
interpretation of read-each-book, not the literal every-book-at-full-depth. Recommendation: the tiered
reading, because it honors do-not-skip while routing the spend to value, and the triage floor is a genuine
read that can promote any book to the full tier on evidence. Flagged for Josh's confirmation that tiered
reading satisfies the order, since it is a reading of his own words.
audit, and §6.8 ruled elevated-care quest routing (CG-26). Whether the high-care deep-read records (Sámi,
Haiti, Australia) need per-record author review or the batch audit suffices is the same question at the
record grain. Recommendation: per-record author review for the high-care three, batch audit for elevated and
standard, matching the §8.8 high-care discipline; flagged so the director confirms the review load rather
than the loop deciding it.