systems/CANON_INTEGRITY_ARCHITECTURE.md
<!-- Chief-architect synthesis of the 9-domain canon-integrity audit (workflow wilyils3n, 2026-07-04). The buildable spec; Section E is the staged build plan. -->
Chief-architect synthesis of the 9 domain audits into one buildable, deterministic system. Every failure class is a real finding cited by domain (D1–D9). The system encodes those failure patterns as code so Opus 4.8 recalls canon with perfect precision instead of re-deriving it. Nothing invents canon; where canon is genuinely open, the system surfaces the open status, never picks a side.
Builds ON the existing harness (verified): registry_fidelity.py (cell fidelity + FK coverage via docs/fk_spec.json), run_gates.py (config-driven runner whose exit code the autonomy loop trusts, driven by harness/gates_config.json — current gates: stress, fidelity, scan, topography, lint, anchors), scan_migration_integrity.py, check_chapter_topography.py, lint_canon_format.py, check_source_anchors.py, and the declared baselines. Two new modules proposed: harness/build_canon_graph.py and harness/lint_canon_graph.py, plus harness/recall.py.
---
No catastrophic data loss. Prose T1/CVD docs and populated T0 registries migrated substantively intact. Real losses split into genuine byte-level damage (repair now) and empty stubs never authored (declared cascade holes, several carrying misleading [ACTIVE] labels).
| # | Item | Location | Severity | Owner |
|---|---|---|---|---|
| 1 | T0_Vril_Site_Registry broken CSV quoting — multi-value cells (thread_anchors/vril_function) spill across fields; two conventions coexist ("""T13","T18""" malformed vs clean pipe). ~20/31 rows mis-parse. Found independently by D4, D7, D9. | Vril_Site Sheet1.csv | MED-HIGH | Grounded — re-serialize to one convention, verify vs frozen xlsx, refresh fidelity_baseline.json same commit |
| 2 | T0_SFX_Registry double-encoded UTF-8 — § as raw c2 a7 (renders §), byte-verified, 3+ rows in live canon CSV | SFX populated_from | MED | Grounded — re-encode; add mojibake gate rule |
| 3 | Duplicated + corrupted workflow copy — 2nd copy in _Files/ scratchpad with v-.1 (0→-); AaS says _Files/ is scripts only | Book-Processing Workflow _Files/ | MED | Grounded — remove dup |
Environment_Manifest → reconstruct from PIPELINE_ADAPTER.md ledger; T6/T7 = cascade authoring.[ACTIVE]-on-empty-stub lying labels (HIGH — top hallucination vector): T1_AaS L292/294/296/297 describe 4 empty T99 stubs as substantive ACTIVE canon (only Normalization honestly annotated). Grounded — annotate the 4 roster lines.[POPULATE→…] (77 Inscription, ~20 Chapter_Index cols × 79) and [DECISION NEEDED - JOSH] (145 link rows) are legal sentinels — never resolved as FK targets, never passed as content. The two .docx legal files are substantive but outside the markdown layer (needs text-derivation, low priority).
Verdict: repair A.1 #1–3 + annotate A.2 #7 now; populate staged registries on sign-off; rest is declared backlog. No content silently lost — every risky gap is self-labeled.
---
Nodes = entities parsed from registries + docs; edges = FKs + typed doc-mentions. Hardest cross-domain fact: almost nothing FK-joins without a normalization layer — the same entity carries 2–4 incompatible encodings. The builder's first job is to normalize every ID and record the alias.
Key forks the builder must reconcile:
CH_NN+CH_PROLOGUE/CH_EPILOGUE (79) — vs Ch NN/Ch_NN, float 52.0, schema CH_00/CH_78.THREAD_NN (22) — vs T01, packed T01:E, THREAD_01:CH_05. (Tnn↔THREAD_nn verified 0-mismatch → safe to lock.)REG_NNN, SFX R_EGYPT (3rd encoding).TRADE_NN — vs schema TR_NN, ability-ref Trade_3_Blacksmithing (3rd form).Adeptus Minor/Adeptus_Minor.range_id (R_NN_MM), WEAVING_NN (20). Unenumerated: 12 elven species. No FK bridge: Research_Corpus (2112 real) ↮ Source_Material (5 fake, but the declared FK target).Full entity roster: Chapter, Thread, Thread-cell, Weaving, Region, Range, Hard-line (HL_NNNN, 120), CVD-section, Component, Pillar (+Gate alias), Integrity-band (3 surface forms), Realm-name, Character (CHAR, 118), Operative (NW, 62), Familiar (FAM — needs scheme), Legendary-species, Ability (185), Mastery-tier, Trade, Trade-gate, Weapon, Vril-site (31), Script (84), Inscription (98), Link (145), Worldstate (25), Scene/Quest/Voice/Theme/SFX, Corpus-source, Doc-artifact, Cardinal-count.
Clean structured FKs (regression-guard): Thread_Grid.chapter_id→Chapter_Index (77/77); Chapter_Index.thread_anchors⇄Thread_Grid.progression_stage (0 mismatch); Inscription.script_ids→Language_Script; Link.region/chapter→Region/Chapter; SFX.hard_line_anchors→Hard_Lines.
Broken/orphan FKs: Chapter_Index.primary_boss_id→Boss (79/79 unresolved, empty table); Chapter_Index.region_primary (no FK col); Story_Spine.region_id (false REG_FLORES ref); SFX.region_id (R_EGYPT); Trade home (Ch_12 case fork); Ability.trade_dependency (3rd form + FishingNav/Fishing_Navigation dup); Research_Corpus↮Source_Material (no bridge).
Dangling grouping-labels: 34 WEAVING_NN:CH_NN edges; 10 link edges to nonexistent Grid cells; range_id; 30 script-genealogy edges (3 both-sided: ARAMAIC, LATIN_ALPHABET, CHINESE_TRADITIONAL_LOGOGRAPHIC).
Typed doc-mention edges (straggler detection): Ch NN→node; §N.M→section (mis-citation class); HL_NNNN→rule; realm-name→current name (§17.7 straggler class — the flagship); count-literal→constant; Pillar/Gate/Component alias.
harness/canon_graph.json (+.md): node table (canonical_id, type, source_file, source_row, raw_aliases[], sentinel_flags[]) + edge table (src, dst, edge_type, resolved:bool, resolver_note). Unresolvable edges retained with reason, never dropped. Companion docs/id_normalization.json declares every alias→canonical mapping (auditable canon, not hidden code).
---
New rules land in harness/lint_canon_graph.py (new graph gate) unless marked "extends." Reads canon_graph.json+id_normalization.json, fails exit≠0, writes graph_scorecard.{md,json} (one line per violation: file:row, rule, expected vs found). Carries docs/graph_baseline.json (declared-open set) exactly like fidelity_baseline.json — green when reality matches the declared-open set, RED the instant a *new* straggler appears or a declared one is fixed without refreshing the baseline.
Tier 1 (build first):
registry_fidelity.py). Every FK resolves after normalization; non-NULL non-sentinel non-resolving = FAIL; allow-NULL list for grouping-labels + declared-empty targets. Catches orphan FKs + the 3-encoding-region/4-encoding-thread/TRADE-vs-TR forks; FAILs if a *new* encoding lacks a declared mapping.check_source_anchors.py). Catches the mis-citation class (§13→§10 voice, §17.7→§17.3 astral, §15→§17.9 Grand-Sage).Tier 2:
[DECISION NEEDED - JOSH] with both sides + authority chain.Tier 3 (hygiene):
lint): no [ACTIVE] title-only stub; AaS roster matches disk (4 lying labels); version-string vs header (SSBL v3.0/v3.2); no dup canonical doc.scan): no double-encoded UTF-8 (SFX mojibake); sentinels legal but flagged at DRAFT→ACTIVE flip; archived-vs-live registry contradiction (Folklore).Wiring: one graph gate + graph-build prerequisite (failure=FAIL) in gates_config.json; G2/G3/G4/G8/G9 extend existing gates; docs/graph_baseline.json+id_normalization.json join the baseline set.
---
harness/recall.py reads canon_graph.json, gives perfect-precision recall (the /dev/continuo lesson operationalized).
Query shapes: (1) by-entity; (2) all-canon-for-a-node recall node CH_34 — union across every registry+doc touching it (the load-bearing cascade query); (3) neighborhood/edges; (4) by-hard-line (--chapter/--pipeline/--topic); (5) retired-name reverse index (recall aliases Maatherion reaches still-retired-named rows — the rename-safe guarantee); (6) open-canon surface; (7) count-sheet; (8) integrity-chain.
Output contract: (a) cited (source_file:row per field), (b) complete (full binding set, never a sample), (c) no-hallucination (missing→NOT_POPULATED (declared-empty, Phase-N) or [DECISION NEEDED - JOSH], never fabricated). Alias-normalized; unresolved edges returned with resolved:false+reason.
Pipeline consumption: Book→node+hardlines --topic cultural + corpus via Research_Corpus (not the 5 placeholders). Chapter/region/quest/lore→node enrichment + open halt + aliases guard. Art(Meshy)→node+band+hardlines. Music/SFX/Voice→band+cultural hardlines (no-joik/songline/Vodou). UE→node coords/biome + §9.6 scatter write-map + counts. Highest-value guarantee: the all-canon-for-a-node query returns a complete, cited, alias-normalized binding set that the graph gate has already proven internally consistent.
---
docs/id_normalization.json (all alias→canonical mappings). No code depends on it → zero risk, Josh-reviewable.build_canon_graph.py → canon_graph.json, retains unresolved edges, recognizes sentinels; add as graph-build step (failure=FAIL). Never fails on canon defects — it records them.graph_baseline.json (CVD-body 8× stay declared-open, Josh-exclusive), flip to RED-on-new.recall node CH_NN first.Exit code stays trustworthy at every stage: new gates are declared-empty-SKIP or report-only until their baseline is set, matching the existing hardening discipline. recall.py becomes the enrichment entrypoint the /spine-chapter cascade skill calls.
---