SAN_KHOISAN_SUBSTRATE_REPAIR_2026-07-28.md

pipelines/SAN_KHOISAN_SUBSTRATE_REPAIR_2026-07-28.md

san_khoisan substrate repair — the second-scorer boundary defect (task #21, 2026-07-27/28)

Status: APPLIED to the store + T0_Source_Material_Registry + the processed/ buckets. NOT committed

(director owns the commit). run_gates.py not run (director owns the ritual). The companion harness

fix is the propagation of the ONE left-word-boundary matcher into the second scorer, plus a

bare-surname purge in the san_khoisan keyword list; regression-pinned in

harness/rag/test_gazetteer.py (108/108 + 9/9 strict-path, exit 0).

Provenance: the region-page planner's south_africa (Ch 13) build was HARD-BLOCKED because the

san_khoisan slug read ~54% off-region (ledger 2026-07-27w). Method reused verbatim:

docs/pipeline_review/SUBSTRATE_REGION_RETAG_2026-07-19.md.

1. The diagnosis — why the 07-19 fix did not hold

The 2026-07-19 left-word-boundary fix landed in harness/rag/index.py::_infer_region ONLY.

harness/book_processing/sacredtexts_convert.py::infer_region_strict — the scorer that assigns

every book on the bulk sacred-texts leg (and, via triage_unassigned.py, the deep-probe inbox

leg) — carried its own independent copy of the rule as sum(2 for kw in kws if kw in head) /

sum(1 for kw in kws if kw in body): pure lowercase SUBSTRING. A second implementation of a rule

is a second place for the fix to miss, and the entire defect class was re-created on the leg that

ingested the largest bulk archive.

Two tokens did the damage, both legitimate San vocabulary:

• eland (the antelope at the centre of San cosmology) hides inside Iceland, Ireland and

Leland. Ten sacred-texts books were filed san_khoisan on a head-hit worth 2 points each.

• lloyd (Lucy Lloyd of the Bleek-Lloyd archive) is a bare common surname. It matched

Arthur Lloyd's translation of *Shinran and His Work* and J. William Lloyd's

*The Karezza Method*. This one SURVIVES the boundary fix — it is a real standalone word — so the

matcher fix alone would not have cleared it. Same latent risk on barnard (Alan Barnard).

Positive control (the mis-tag reproduced, not inferred)

Ran the SHIPPED infer_region_strict against each book's real title/author/body and the

left-boundary variant beside it (scratchpad/diag3.py):

Two sources (SRC_01396 South-African Folk-tales, SRC_01399 Xhosa Folk-lore) did NOT come from the

gazetteer at all — the drain log shows them SKIP_UNASSIGNED five times and then INGESTED with

region=san_khoisan on 2026-07-17 06:20, i.e. from a model-lane triage verdict. Both are genuinely

southern-African and both re-score as KEEP on evidence.

The remaining PDFs in the polluted set (SRC_00156/00161/00219/00245/00246/00247/00215/00202) are

survivors of the 2026-07-19 sweep: that sweep re-scored only the 307 junk-titled sources against

their first FIVE chunks and left these tagged.

A second, independent defect found in passing

SRC_01186's registry cell already read celtic_fairy while all 320 of its chunks still read

san_khoisan — a prior partial fix that updated the registry and never touched the vector store.

Registry/store divergence is invisible to every current gate; only the store drives retrieval.

2. The fix (tagger level)

• harness/rag/index.py — the matcher is now the exported kw_hit(kw, hay), THE one

implementation. _infer_region calls it. Docstring names the drift class explicitly.

• harness/book_processing/sacredtexts_convert.py — infer_region_strict now calls ix.kw_hit

instead of kw in head / kw in body. triage_unassigned.py imports this function, so the

inbox deep-probe leg is fixed by the same edit.

• harness/rag/title_region_map.json — BARE-SURNAME purge on san_khoisan (the boundary-proof half

of the defect): lloyd → lucy lloyd + bleek and lloyd; barnard → alan barnard.

Distinctive single surnames (bleek, vinnicombe) stay; eland stays and is now safe. The

header note's stale "SUBSTRING match" wording is corrected to LEFT-WORD-BOUNDARY and carries the

new purge rule.

• harness/rag/test_gazetteer.py — 12 new regression cases: 8 Ireland/Iceland/Leland/Lloyd titles

through _infer_region, an 8-case STRICT-PATH battery running the real infer_region_strict

(the second scorer is now pinned so the two can never drift apart again), a positive control

(Bleek/Lloyd *Specimens of Bushman Folklore* must still resolve san_khoisan — a battery that

cannot say yes proves nothing), and an eland/Drakensberg case proving the real tokens still win.

108/108 + 9/9 strict, exit 0.

Note on expectations: the eight formerly-mis-tagged titles now resolve to unassigned, not to

celtic_fairy/norse_germanic/japan. That is CORRECT and honest — Ireland/Iceland/gypsy/Shinran are

not gazetteer keywords, so a bare title names no distinctive token and triage is the right answer.

Routing to the real owner happens on stored text in the re-tag below.

3. The re-tag (score-thresholded; the noise gate governs)

Re-scored all 24 san_khoisan-tagged sources against their STORED chunk text with the fixed matcher,

then applied the 2026-07-19 thresholds unchanged:

• KEEP — the old tag scores >= 2 on stored text and sits within 1 of the winner.
• RETAG — winner >= 3, margin over runner >= 2, old tag scored 0.
• STRIP → tags REMOVED (the triage queue); a wrong tag poisons region-filtered builds, an absent

tag is honest.

Method amendment (recorded because it changed the outcome): the 07-19 pass probed the first 5

chunks. On books rather than papers those are the title page and the table of contents. The narrow

window STRIPped **SRC_00265 *Specimens of Bushman folklore*** — the Bleek/Lloyd flagship of the

whole slug — on old=1, and scored *The Mind in the Cave* at 1. A remediation that deletes its own

best source is measuring the window, not the book. The probe is now a 30-chunk SPREAD sample

(front/middle/tail, evenly indexed), mirroring triage_unassigned.py's two-window deep probe.

Under it SRC_00265 scores 3 and KEEPs, SRC_00215 scores 5, SRC_00202 scores 7. Both runs are on

disk (sk_rescore.json narrow, sk_rescore2.json spread).

Dispositions — 24 sources / 4,439 chunks, reconciled

KEEP (8 sources, 1,542 chunks) — evidence on stored text:

RETAG (3 sources, 282 chunks) — re-routed to their correct slug, files moved to the matching

processed/ bucket:

RECONCILE (1 source, 320 chunks) — SRC_01186 *Heroic Romances of Ireland*: the registry already

said celtic_fairy and the re-score's own winner is celtic_fairy (3, margin 1 — under the RETAG

bar). Two independent judgements agreeing is not a new judgement; the store was reconciled to the

registry rather than stripped.

STRIP → unassigned (12 sources, 2,295 chunks) — SRC_01187 (515) · SRC_01173 (445) · SRC_01243 (291)

· SRC_01159 (253) · SRC_01163 (210) · SRC_01209 (207) · SRC_00675 (161) · SRC_01254 (80) ·

SRC_00246 (45) · SRC_00161 (42) · SRC_00247 (41) · SRC_00245 (5). Files moved to

.rag/book_src/processed/unassigned/; registry region_tags cleared and source_filepath

updated. These join the standing semantic re-classification queue (the local-7B lane) — the same

job class as the 226 stripped on 2026-07-19. Nothing was deleted.

**Reconciliation: 1,542 KEEP + 282 RETAG + 320 RECONCILE + 2,295 STRIP = 4,439 = the pre-repair

san_khoisan chunk count (measured, not assumed).** Store writes: 2,897 chunks re-tagged

(everything but the KEEPs). Registry: exactly 16 rows changed (git diff --stat = 16 insertions,

16 deletions; CRLF endings preserved, 1,915 lines before and after).

Post-repair slug counts, measured: san_khoisan 4,439 → 1,542; celtic_fairy 20,965;

swahili_coast 526; pacific_oceania 3,906.

4. Before / after — the probes

Same script, same five south_africa queries + two raw-slug queries, one embedder load, run against

the store before and after the re-tag (scratchpad/sk_probe.py, outputs sk_probe_before.txt /

sk_probe_after.txt). --region south_africa expands to exactly one slug, san_khoisan, so the

region probe and the slug probe are the same shelf seen two ways.

Probe A — "ancestors, naming, and the spirits of the dead" (top 10)

8 of 10 off-region before; 0 of 10 after.

Probe B — "healing ritual, the shaman, and the spirit world" (top 5)

The reported Yoruba case is gone — and it was worse than "3rd": on this probe the Yoruba

naming paper held rank 1 and 6 before the repair. It is now absent from the shelf entirely.

Probe C — "folk tales and storytelling traditions" (top 10): before, ranks 4/5/6/8/9/10 were

Heroic Romances of Ireland, Visions and Beliefs, and Gypsy Sorcery. After, all ten are Xhosa

Folk-lore, South-African Folk-tales, Specimens of Bushman folklore, Lost World of the Kalahari and

The Mind in the Cave.

Probes D/E ("San rock art, the eland, and trance dance in the Drakensberg"; "sacred landscape,

mountains, and water places") were already clean at the top before, and dropped their two polluted

tail hits (Shinran ranks 4/5/6, Swahili rank 7) after.

Across all seven probes, counted mechanically against the moved-source set: **BEFORE 38 of 70

ranked hits off-region (54%); AFTER 0 of 70.** Per probe, before → after: A 0/10 → 0/10 ·

B 8/10 → 0/10 · C 6/10 → 0/10 · D 6/10 → 0/10 · E 4/10 → 0/10 · slug-B 8/10 → 0/10 ·

slug-D 6/10 → 0/10. The 54% matches the planner's independent ~54% chunk-level estimate.

5. Verification

• harness/rag/test_gazetteer.py — exit 0, 108/108 gazetteer cases + 9/9 strict-path cases

(including the positive control that a genuine San title still resolves).

• harness/check_index_freshness.py — PASS, exit 0. Note: the registry edit is a canon input,

so the gate went RED on write, exactly as designed. Confirmed with a controlled swap that the

gate was PASS before the repair and that the repair was the sole cause (pre-repair registry

reproduces the declared hash 9ea522c1…; the repaired tree hashes 19f7072a…). Re-greened by

the mandated python harness/rag/index.py --scope canon (10,135 chunks, 386.5s, fastembed,

embedding_is_real=true); harness/rag/index_manifest.json now declares 19f7072a….

• harness/check_store_integrity.py — PASS, exit 0: `book_substrate 661,081 chunks /

661,081 vectors OK, canon 10,135 / 10,135 OK`. This is the standing-law check: the canon

rebuild did NOT drop the shared vec table (the 2026-07-16 class). Substrate vector count is

byte-identical to the pre-rebuild reading taken for this purpose.

• docs/fidelity_baseline.json re-emitted via registry_fidelity.py --emit-baseline. The diff is

exactly one line: the T0_Source_Material_Registry extension hash

(7f1476cd9b074da7… → abe0107f260a91d9…). No other lane's edits were swept in.

• Registry write scoped and non-destructive: git diff --stat = 16 insertions / 16 deletions on

1,915 rows; CRLF line endings and quoting preserved (1,915 CRLF before and after, 0 bare LF).

• run_gates.py NOT run and nothing committed — both belong to the director.

Modified tree at handoff: harness/rag/index.py · harness/rag/title_region_map.json ·

harness/rag/test_gazetteer.py · harness/book_processing/sacredtexts_convert.py ·

harness/rag/index_manifest.json · registries/T0_Source_Material_Registry/…csv ·

docs/fidelity_baseline.json · harness/scorecard.json + .md (fidelity run artefacts).

6. Blast radius — what was deliberately NOT touched

The matcher change is behaviour-preserving-or-stricter: it can only REMOVE false keyword hits, never

add one. Token edits were confined to the san_khoisan list, and lloyd / barnard / eland are

san_khoisan-exclusive in the gazetteer (verified: no other slug lists them), so no other slug's

scoring moved. Only san_khoisan-tagged sources were re-tagged; the 3 RETAG destinations

(celtic_fairy, swahili_coast, pacific_oceania) are additive.

Sweep for the same defect in every other slug (scratchpad/sk_blast2.py): scored all 1,654

tagged (source, tag) pairs in T0_Source_Material_Registry, flagging any whose tag has substring-only

evidence in title+author. 3 pairs, all benign, none needing action:

• SRC_00258 china_daoist — 'taoist' inside "SchipperTaoistBody" (a space-stripped filename title

of Schipper's *The Taoist Body*). Tag is correct; only the head match is boundary-blocked.

• SRC_00451, SRC_00485 india_vedic — 'vedic' inside "Ayurvedic" (Vasant Lad, David Frawley).

This IS the substring shape, but the resulting tag is defensible and was already hand-settled in

the 2026-07-16 short-token purge. Flagged, not changed.

Positive control on that sweep (it must fire on known-bad input, or 3 proves nothing): re-run

against the pre-repair registry with the pre-repair token list, it detects 8 rows — the same 3

plus five san_khoisan 'eland'-in-Ireland/Iceland titles. The detector works; the post-repair

result is real.

Scope limit stated honestly: that sweep is head-level (title + author), which is where the

2× head weight made the defect decisive. A full-text sweep over all 661k stored chunks was started

and stopped — it ran at ~200 sources per 15 minutes against the 9.4 GB store, too slow for this

window. It remains available (scratchpad/sk_blast.py) as a local-compute overnight job.

7. For the director — three items, none applied

1. south_africa reaches only ONE slug. query.py::REGION_TO_SLUGS maps

south_africa → ["san_khoisan"]; zulu_nguni and bantu_kingdoms map to no region_id at all,

so their sources are unreachable from any region-filtered query. The SLICE_SCOPE ledger already

counted "south_africa ~25 (san_khoisan 23 + zulu + bantu)" — that arithmetic is not what the

filter actually returns. Recommendation: add zulu_nguni (and bantu_kingdoms if Ch 13 owns

it) to south_africa. NOT applied — that file's own doctrine says entries are added only where

the mapping asserts nothing about chapter scope, and this one touches Ch 13's shelf directly.

It is a one-line consumer-side change with no re-tag required.

2. **SRC_01399 *Xhosa Folk-lore* KEEPs under san_khoisan** (evidence: 'bushmen', 'eland'; runner

zulu_nguni scored 1). Theal's collection is Xhosa-primary with San material in it, so the

thresholds and the consumer both favour keeping it — but if item 1 lands, zulu_nguni becomes

the more accurate slug without losing the Ch 13 shelf.

3. Folder ≠ tag residue. 54 files still sit in .rag/book_src/processed/san-khoisan/; only 10

of them carry the tag. The other 44 are junk-titled PDFs the 2026-07-19 sweep stripped without

moving. Cosmetic and gitignored, but it makes the folder tree a misleading index of the corpus.

Also worth a line in the ledger: the record above is written to scratchpad per the work order. Its

natural home, matching the 07-19 precedent, is docs/pipeline_review/ with a docs/DOC_MAP.md

row — the anti-orphan rule applies to repair records too.

Generated by harness/site/structure_site.py — the URL path is the repo path. review root