SUBSTRATE_REGION_RETAG_2026-07-19.md

pipelines/SUBSTRATE_REGION_RETAG_2026-07-19.md

Substrate region re-tag — the substring-poisoning remediation (2026-07-19)

Status: APPLIED. The companion harness fix is the left-word-boundary gazetteer matcher

(harness/rag/index.py _infer_region, regression-pinned in harness/rag/test_gazetteer.py, battery

100/100). Provenance: the sumatra_java region-page build surfaced one leaked cross-region source in a

region-filtered query; the diagnosis generalized to the whole auto-sorted class.

The gazetteer matcher scored keywords by pure lowercase SUBSTRING (kw in text), so short tokens hid

inside common words: "malay/malaya/malayan" all match inside "Himalaya(n)" (a Garhwal pilgrimage paper

beat tibet_himalaya 3-2 and was filed malay_world — SRC_00025, the proven leak); "san" matches inside

"thousand/sanskrit/sanctuary" (the san_khoisan epidemic); "bali" inside "kabbalistic"; "iberia" inside

"Siberia". The exposed class is the 307 auto-sorted junk-titled sources (numeric filenames, largely

the Donner Institute / Scripta comparative-religion paper dump), which had no human-readable title for

the gazetteer to verify and were bucketed on first-two-page probe text at min_score 1.

Matcher: keyword matches now require a LEFT word boundary (may not start mid-word; the right side

stays open so the curated stem tokens keep matching their inflections — himalaya→himalayan). Three

new adversarial regression cases pin the class in test_gazetteer.py.

Store: an expression index (idx_chunks_source_id) added to the RAG store for per-source queries.

Re-scored all 307 junk-titled sources against their STORED chunk text (first 5 chunks, richer than

the original probe) with the fixed matcher. A keyword matcher must not pretend to classify

comparative-religion academic prose (a spot-check caught two plausible-looking-but-wrong low-score

re-tags: a French Gayomart/Iran paper scoring "bali", a Huns-and-Scandinavia paper scoring

"gabon_bwiti" — both correctly refused under the thresholds):

KEEP (66): the old tag scores >= 2 on stored text and sits within 1 of the winner — evidenced.
RETAG (15, unambiguous only): winner >= 3, margin over runner >= 2, old tag scored 0. Applied, and

the PDFs moved to their correct processed/ bucket folders: SRC_00025 malay_world→tibet_himalaya ·

SRC_00037 →amazon_colombia · SRC_00078/00079/00089/00091/00351/00368/00369/00374/00375 →sami_sapmi

(the Donner/Åbo Sámi-studies cluster) · SRC_00141/00143 →tibet_himalaya · SRC_00322 →angkor_khmer ·

SRC_00353 →norse_germanic.

STRIP → unassigned (225 + 1 no-text = 226): the old tag is unevidenced in the stored text and no

honest keyword winner exists. Region tags REMOVED (a wrong tag poisons region-filtered builds; an

absent tag is honest) — these sources are invisible to region-filtered queries until re-classified.

Store: 7,648 chunks re-tagged in .rag/humanity_rag.sqlite; T0_Source_Material_Registry region_tags

cells updated on all 240 changed rows; fidelity baseline re-emitted (extension-zone hash).

The 226 stripped sources are the semantic re-classification queue: junk-titled rows in

T0_Source_Material_Registry with an EMPTY region_tags cell (query: source_title has no alphabetic

word of length >= 3 AND region_tags = ''). They join the standing ~370-file never-ingested triage

queue as the same job class — content-level classification by a model that reads meaning, not

keywords. Until then, absence-from-region-queries is the correct, noise-gate-conformant state.

Region-filtered substrate queries (the load-bearing correctness mechanism for region pages and all

downstream builds) no longer surface the mis-bucketed class. Coverage note per the no-silent-caps

rule: region query recall over this paper dump is REDUCED until the triage lane re-classifies the 226

— the earlier state was not better coverage, it was silent cross-region pollution.

Generated by harness/site/structure_site.py — the URL path is the repo path. review root