pipelines/books/BOOK_PROCESSING_WORKFLOW.md
How a book Josh buys/scans becomes canon substrate — organized by region, kept tidy.
_source/02_Tier_2_Region_Pages/_ready_for_processing/ — Josh drops scanned/purchased PDFs here (643 staged 2026-07-14). The index.py --process-folder default was repointed to this visible T2 location
(it used to be .rag/book_src/ready_for_processing/, which stayed empty). A sidecar <name>.meta.json
({"source_id":"SRC_00042","title":"...","region":"west_africa","chapters":"CH_12,CH_39","category":"folklore"})
is optional — see auto-sort below. Only TOP-LEVEL PDFs are processed by the auto-sort loop; the
special-format subfolders (sacredtexts/, nationalgeographic/, samidrum_reference/) are handled by the
recipes at the bottom, not the PDF loop.
.rag/book_src/processed/<region>/ — after processing, the renamed PDF lands here, organized by region (SRC_00042__The-Power-of-Writing-Batak.pdf), gitignored. (Kept in .rag/ so the processed copies
stay out of git; only the inbox moved to the T2 tree.) VISIBILITY (2026-07-21): browse it at
_source/02_Tier_2_Region_Pages/_processed/books/ — a directory junction onto this tree, so the organized
per-region library is visible next to the other done-homes without moving the pipeline's real paths
(_processed/natgeo_jpg/ is the same junction pattern onto the NatGeo JPG product library).
State. The live store holds 14,577 canon chunks and 0 book_substrate chunks, and
.rag/book_src/ does not exist here. Nothing wiped it: the 5090 migration ruled the vector store
rebuilt rather than copied (docs/5090_SETUP_RUNBOOK.md D-12) and parked the separate
book_src corpus copy in the same session's closing list (*"book_src copy (books never gate)"*), so
this box only ever ran --scope canon. All 1,914 T0_Source_Material_Registry rows point at
.rag/book_src/processed/<region>/… and none resolve. Full evidence: docs/PIPELINE_LEDGER.md
§RAG-2. Consequence: query_substrate() returns nothing, so the region-page substrate lane is
unbacked until the restore runs.
The corpus lives on DESKTOP-0AQBV43 (tailnet 100.98.85.18). SMB there is credentialed and
SSH/WinRM/RDP are closed, so step 1 is Josh's — a lane cannot authenticate.
python -c "import sqlite3;print(list(sqlite3.connect(r'C:\dev\humanity-forgotten\.rag\humanity_rag.sqlite').execute('select collection,count(*) from chunks group by collection')))"
A line containing ('book_substrate', <a big number>) means Step 1 is the right move. If it prints
only ('canon', …), the old store was rebuilt there too — skip to the fallback at the bottom and
send .rag\book_src\ instead.
Prefer the STORE over the PDFs: it is one file, it needs no re-OCR and no re-embedding, and it is
the only carrier of the store-side repairs (the 2026-07-19 region re-tag rewrote region_tags
in chunk meta_json for 7,648 chunks / 240 sources; the SRC_00567 orphan cleanup deleted rows the
source PDFs would re-emit). Re-ingesting from PDFs silently discards both.
"C:\Program Files\Tailscale\tailscale.exe" file cp "C:\dev\humanity-forgotten\.rag\humanity_rag.sqlite" milks5090:
Send .rag\book_src\ too when convenient (it is the fallback for anything the donor store lacks,
and the re-OCR/VLM and deep-read lanes need the source files) — an external disk is fine for that
leg; drop it at E:\book_intake\, which exists for exactly this.
"C:\Program Files\Tailscale\tailscale.exe" file get E:\book_intake python harness/rag/import_substrate.py --from E:\book_intake\humanity_rag.sqlite --dry-run python harness/rag/import_substrate.py --from E:\book_intake\humanity_rag.sqlite python harness/rag/sparse.py --build --collection book_substrate python harness/rag/query.py "Manggarai mbaru niang architecture" --collection book_substrate --region flores_island python harness/run_gates.py
--dry-run surveys the donor and runs every guard without writing. The import is additive:
it touches only the named collection, re-packs the vec0 blob from the donor's stored JSON vector so
nothing is re-embedded, and refuses (never half-imports) on a dim mismatch, on an
embedding_model_version that is not the live index's — the e5 migration is a separate pass and one
store never holds two models — on a donor id that is canon here, or on an empty target collection.
It asserts the canon row count across the run. Prove the tool before trusting it:
python harness/rag/import_substrate.py --selftest (12 checks, 3 of them mutation checks).
Land the absence-tooth in the same commit as the restore. store_integrity passes today with
the substrate wholly absent — its contract is *"a collection with chunks must have vectors"*, and a
collection with no chunks satisfies it vacuously. That is the gate-side half of how this went
unnoticed for a week. It is left unarmed deliberately (it would go red on a blocker only Josh can
clear, stopping every other lane) and arms with the restore, where it can be green from birth.
If the donor store is unavailable, fall back to the corpus: copy .rag/book_src/ and re-run
python harness/rag/drain_inbox.py, then re-apply the 2026-07-19 re-tag remediation from the
registry's surviving region_tags column before trusting any region-filtered query.
You do NOT have to rename or tag the PDFs first. When a file has no .meta.json, the pipeline:
1. Extracts a title — from the PDF's embedded metadata (/Title, /Author), falling back to the first
two pages of text, then the cleaned filename. So scan_00123.pdf still gets a real human title.
2. Infers the region — scores that text + the filename against harness/rag/title_region_map.json (a
distinctive-keyword→region gazetteer seeded from docs/BOOKS_OWNED.md). Highest-scoring region wins;
a tie or zero hits → unassigned/ (printed with an [UNSORTED] flag so you can drop a one-line
.meta.json or move it by hand). Verified: batak_scan_0012.pdf→sumatra_batak, a Skeet Malay-magic
scan→malay_world, an unrelated file→unassigned.
3. Allocates a stable SRC_NNNNN by scanning the registry for the next free id (no collisions), and
renames + moves to processed/<region>/SRC_NNNNN__<clean-title>.pdf.
So the answer to "will it figure out the region and rename properly even though it's named however the
printer/online PDF named it?" — yes, for any book whose region is in the gazetteer (all the owned + BUY
titles are). For a book the gazetteer doesn't recognize, it lands in unassigned/ with a clear flag rather
than guessing wrong. Extend the gazetteer (add a keyword row) whenever a new region/book shows up — then run
python harness/rag/test_gazetteer.py (the 90-title regression battery; the matcher is substring-based, and
the battery pins the known traps — short keywords like 'edda'/'iberia'/'budge' hide inside common words).
Expanded 2026-07-15 to 41 buckets covering every BOOKS_OWNED + sacred-texts-USB region (Persia, China,
Siberia, Japan, Vedic India, Norse, Celtic/fairy, British Isles, Slavic, Basque/Iberia, Native North
America, Haiti, Mesoamerica, Andes, Inuit, Australia + additions to eight existing buckets).
# one file:
python harness/rag/index.py --scope substrate --source ".rag/book_src/ready_for_processing/foo.pdf" \
--source-id SRC_00042 --source-title "..." --region west_africa --chapters CH_12,CH_39 \
--category folklore --move-processed
# the whole ready_for_processing folder (reads each <name>.meta.json for per-file region/chapters):
python harness/rag/index.py --scope substrate --process-folder
# the BULK DRAIN (600+-file unattended runs) — use the driver, not the bare folder loop:
python harness/rag/drain_inbox.py --dry-run # classify + region-preview everything first
python harness/rag/drain_inbox.py --limit 25 # bounded pilot, then drop --limit for the full drain
# the OCR leg (tesseract installed 2026-07-15): quarantined scans -> text layer -> back to the inbox:
python harness/rag/ocr_quarantine.py --limit 10 # then the next drain stage ingests them
OCR-leg state (2026-07-16): the full quarantine was OCR'd and drained — 27/28 batch + the
Abimbola Ifá recovered via a pikepdf repair-roundtrip (ocrmypdf's "generated PDF is INVALID"
verdict on it was archival pedantry — bad indirect refs readers treat as null; the text layer
probes clean and it ingested). Standing remainder in _needs_ocr/: the three Oxford Ancient
Egypt volumes (~220 chars post-OCR in both probe windows — tesseract cannot rasterize their
encoding) — parked for the ruled VLM-OCR lane (PaddleOCR-VL/DeepSeek-OCR class) on the 5090,
per RESEARCHED_STACK's OCR slot. Non-Latin-script sources (the Arabic Ibn Battuta vol 2) OCR
to garbage under eng-only tessdata and land in the triage queue — add language packs or route
to the VLM lane before trusting their text.
Bulk drains go through harness/rag/drain_inbox.py (2026-07-15), which adds what an unattended
600+-file run needs and the bare loop lacks: a TEXT-LAYER GATE (a no-text scan would otherwise ingest
0 chunks yet still be moved to processed/ and registered query_ready — falsely done; gated files
quarantine to _needs_ocr/ until ocrmypdf gives them a text layer), PER-FILE ERROR ISOLATION (one
encrypted/corrupt PDF no longer kills the run; failures log and stay in place), the NOISE GATE
(gazetteer-unresolved files are SKIPPED as the triage queue for the local-model relevance lane or a
manual .meta.json — never bulk-vectorized as noise; --ingest-unassigned overrides deliberately),
and a resumable JSONL log at .rag/drain_log.jsonl. After any real drain: refresh
docs/fidelity_baseline.json (registry_fidelity.py --emit-baseline), rebuild indexes if gates ask,
gates 15/15, commit — the standing scatter ritual.
1. Vector DB — chunks the PDF page-anchored (pypdf native text; OCR upstream with ocrmypdf only for
image-only scans) → embeds (fastembed multilingual) → the book_substrate collection (tier-0 evidence,
separate from canon). Citations resolve as (title, author, year, page).
2. Research corpus — registers the source in T0_Source_Material_Registry (lifecycle → query_ready,
with region_tags / chapter_scope_hints / chunk_count). *(Next: link research_corpus_ref to its
T0_RC_* row so identified → acquired → processed → indexed is one traceable chain.)*
3. Region pages — the region/chapter authoring step then consumes it via
python harness/rag/query.py "<q>" --collection book_substrate --chapter CH_NN (Scope A) alongside the
deterministic enrich.py + the canon-semantic block. *(Region-page authoring = Milestone-1 downstream.)*
4. Rename + move — the PDF is renamed SRC_id__sanitized-title.pdf and moved to processed/<region>/.
The four steps above are the ingest legs: they get a book indexed, region-sorted, and query-ready. They do
NOT read a book. The per-book deep read is the separate book-first leg that reads one source cover-to-cover
against the live canon and mines what THIS book offers the game — its quest seeds, its weave-links to what the
arc already carries, its entity candidates, its Layer-3 rabbit-holes, and its builders-of-knowledge civilizational
marks. It is the book-first complement to the query-first Phase-3 category streams (which ask the whole corpus one
question per registry); the two are complements, not substitutes. Governing contract:
docs/proposals/DEEP_BOOK_PROCESSING_CONTRACT.md (lands inside T1_Build_Pipeline_Contracts §8 as Phase 3B).
EXR_<source_id>.json, at docs/book_extractions/(repo-tracked — the record is derived, non-copyrighted, cited structured data, so it stays git-visible as the
audit trail for every canonical row it seeds; the copyright gitignore covers the PDF text, not the extraction).
Each record: a header, four output arrays (story_seeds / weave_links / entity_candidates split by owner
lane / l3_connections), the civilizational_marks L2 array, and a book-level care_summary. Every proposed
row carries a [p. N] page anchor (cited-or-cut) and is proposal-tier — it queues into an existing
ratification lane and becomes canon only on sign-off. A docs/book_extractions/ manifest tracks the set.
enrich.py surfaces ONLY the RATIFIED deep-read outputs into a chapter build pack,so a proposed-but-unratified row can never smuggle unratified canon into authoring.
book — a candidate skeleton + a seed-richness score, so the corpus is fully swept (do-not-skip honored) and the
score tells the Opus lane where the value is. (2) Opus deep read on the seed-rich set — the full
cover-to-cover read that authors the EXR record. (3) Opus critic + the §8.8 cultural-authenticity audit on
every record (the verification bar, full strength; the high-care three — Sámi / Haiti / Australia — route to
per-record author review, elevated and standard to the batch audit). (4) Deterministic merge (zero-token)
dedups against existing rows, resolves FKs, appends ratified rows, refreshes the fidelity baseline in the same
commit. (5) Director / Josh for contested findings and any row that would assert new canon.
books within and across batches; the triage seed-richness score refines ordering inside a batch.
(seed-rich-first), read against that region's canon pack, ratified, and wired into that region's page build. It
NEVER gates on the whole library. The cross-cutting comparative-mythology / folklore-collection volumes (Frazer,
Eliade, Campbell, Briggs) that belong to no single region run in a separate cross-region lane prioritized by
weave-density, run early and re-run as each end's region lands.
extraction_status lifecycle (per book). not_deep_read → triaged → deep_read_in_progress → deep_read_extracted → deep_read_audited → deep_read_ratified,
with regeneration_required on a canon change / source correction / retraction / migration. It's a real column
on T0_Source_Material_Registry, distinct from the ingest source_material_status (a book is query_ready
before it is deep-read).
.rag/book_src/deep_read_log.jsonl (the drain_inbox pattern) makes a killed batch resume at the first not_deep_read / deep_read_in_progress source; a landed record is git-visible
provenance — land-as-you-go.
Sumatra/Java, the Angkor spread — the same books the first region pages need) plus the **138 reject-audit INGEST
keepers** (seed-rich by construction). It proves the record schema, the five-gate lens scoring, the four
ratification lanes, and the enrich.py harvest wire before any scale spend. The 9 JOSH_REVIEW reject-audit
items wait for Josh before either ingest or read.
The pipeline is BUILT + TESTED (a real Batak PDF ingested = 418 page-anchored chunks, retrieved correctly).
The FULL corpus run waits on the minimized purchase list (docs/BOOK_ACQUISITION_MINIMIZED.md) — the point
of the §4 test round was to prove the pipeline works, NOT to clear the whole list now.
The auto-sort loop above handles standard PDFs. Three staged sources are NOT plain PDFs; each has its own recipe.
../_processed/sacredtexts/)STATE: the mirror is FULLY CONVERTED and the resolved set FULLY DRAINED — 1,247 books /
568,323 chunks ingested all-time (the resume pass 2026-07-21 converted the last 269 deeper-nested
books and drained the final 109 resolved ones; per-source reported-vs-stored reconciliation runs
CLEAN at 1,741/1,741 across the whole store after the SRC_00567 partial-ingest orphan cleanup).
The remaining product is the TRIAGE QUEUE: 282 gazetteer-unresolved converted books staged at
.rag/book_src/sacredtexts_md/ awaiting the local-model relevance lane (never bulk-vectorized).
The mirror itself is the frozen source archive at ../_processed/sacredtexts/ per the
completed-staging-tree lifecycle; the recipe below is retained for re-runs against that path.
The full Internet Sacred Text Archive. Clean simple HTML (<TITLE>/<H1>) and much of it plain .txt,
organized by tradition folder (tao/ Taoism, zor/ Zoroastrianism, egy/ Egyptian, hin/ Hinduism,
aus/ Australia, pac/ Pacific, sha/ Shamanism, sbe/ Sacred Books of the East, and so on). Recipe:
(1) map each chapter/region to its tradition folder(s); (2) walk ONLY those folders; (3) strip HTML to text
(the tradition folder name is free category metadata); (4) register each as a substrate source (public-domain
primary; cite by tradition and text). Do NOT ingest all 140K — pull the relevant traditions per node, using
sacredtexts/index.htm (the catalog) as the map. This puts the actual primary texts in hand for most of the
arc's traditions.
nationalgeographic/, .cng images) — convert, then choose.cng files are JPEGs XOR-obfuscated with a single-byte key. Recipe:
(1) python harness/book_processing/cng_to_jpg.py <cng_dir> --recursive --out <jpg_dir> — auto-detects the
key and validates the decoded JPEG before writing; (2) THEN either keep the JPGs as the visual-reference
library (NatGeo's primary value: art direction, terrain, cultural imagery for the region pages), OR OCR the
JPGs with Tesseract if you want the article text in the corpus, OR bundle a magazine's pages to a searchable
PDF (img2pdf + ocrmypdf). Do NOT run the third-party .xlsm macro — the Python tool is an auditable,
macro-free replacement.
samidrum_reference/ + the owned tracings PDF) — mostly already in handJosh already holds the tracings (Sámi drums – Tracings of the drumskin designs.pdf), the figure-to-cosmology
decode (Joy_Francis_ActaE_234pdfA.pdf), and the Scripta drum-journal volume — so Manker himself (unobtainable)
is not needed. The added samidrum_reference/manker_drum_provenance.md is the museum-location table for all
~71 drums; samidrum_reference/image_urls.txt lists the 71 full-resolution tracing PNGs
(old.no/samidrum/drums/mankerNN.png) to download only if the drum imagery is wanted as visual reference.
Recipe: ingest the tracings + Joy + Scripta PDFs through the normal loop (region sapmi, Ch 29); treat the
provenance table as reference metadata; download the 71 PNGs only if needed.