systems/SITE_RELIABILITY.md
Status: ACTIVE (2026-08-10) · Owner: the site lane · Tool: harness/site/site_ritual.py
· Debt ledger: build/ops/SITE_DEBTS.md · Seat channel: build/ops/SEAT_REPORTS.md
Josh, 2026-08-09: *"the website pushes have been entirely unreliable."* He was right, and the night
that produced that sentence produced four named failures. This page is what each one actually was,
what the ritual does about it, and the one line a seat runs.
python harness/site/site_ritual.py # build + gates + deploy + PROVE it is live python harness/site/site_ritual.py --check # build + gates + compare production, deploy nothing
Options: --slice (Prologue→Ch 13 only, a faster loop), --seat NAME (who owes a debt this run
writes), --project / --branch (defaults humanity-review / main — **anything but main
deploys a preview and production does not move**), --force (take the single-writer lock),
--attempts N (live-verify retries while Pages propagates).
Exit codes, so a runner loop can branch without reading prose:
| code | meaning |
|---|---|
| 0 | deployed AND proven live — or --check and production already IS this build |
| 2 | a gate refused the build or the deploy; a debt is now open |
| 3 | environment refusal — credentials, wrangler, or another ritual holds the lock |
| 4 | curation firewall — factory state on a published board page |
| 5 | deployed but NOT proven live — the F3 class; treat the push as unlanded |
| 6 | --check: the gates pass and production is behind this build — a deploy is owed |
| 7 | the ritual itself broke; treat as unproven |
Nothing about the build is re-implemented. build_progress_site.main() is called with the argv the
shell entry point parses (site_ritual.py:run_build), so every gate, every positive control and
every refusal is the one already written in harness/site/build_progress_site.py. The ritual is the
surface, the proof and the ledger around it.
A seat scored an L12 into repo records; its commit said the site was deployed with it; the live page
still served L12_REFUSES_without_a_ruled_ship_scale. Josh looked, found nothing, and was right.
Nothing re-read production. The mechanism is already documented as a scar in the build script
itself — build_progress_site.py:3831-3834: without --branch main, wrangler infers the branch
from git (master), **every deploy lands as a PREVIEW, and the production URL keeps serving the old
build.** A zero exit from wrangler is not evidence that production moved.
What the ritual does: verify_live() runs three arms after a deploy, and each one reports
whether it RAN — an arm that could not run never reports clean.
(/accounts/{id}/pages/projects/humanity-review) and checks environment, branch, stage status,
and that the canonical deployment is the one this run just made (compared against
deployment_url in build/site_deploy_record.json). This is what catches the preview-deploy lie
directly.
/build_report.json (written by build_progress_site.py:3657, so it ships inside the deployed tree) and compares four stamp
fields against the local build: generated, canon_sha, game_sha, pages.
generated + canon_sha)is this build's. This is the arm that reads what Josh reads.
A failure on any arm exits 5, writes an unproven debt, and says in plain words: treat the push
as unlanded.
A thing this lane had to discover to make the arms work: https://humanity-review.pages.dev is
behind Cloudflare Access — an anonymous fetch 302s to continuo.cloudflareaccess.com, so the
pages.dev URL cannot be byte-verified by a seat. The custom domains on the same project
(review.humanityrpg.com, humanityrpg.com) answer 200 to anyone and serve the identical tree.
VERIFY_BASES therefore tries the custom domain first and prints which base spoke; if none can be
read, the run is unproven, never "probably fine". (Two consequences for the director, below.)
A seat recorded *"the site deploy is owed, not done — the photobook prose gate refused six
pre-existing Flores captions"* in a commit message and nowhere else. No surface tracked owed
deploys, so the next seat could not know one was owed.
What the ritual does: every refusal writes a structured block to build/ops/SITE_DEBTS.md —
tracked in git, unlike the site tree itself:
## OPEN 2026-08-10 10:22 UTC deploy-owed key:production-behind seat:lane-a-site-reliability - refused: production is behind this build: generated: local '2026-08-10 10:21 UTC' vs live '2026-08-10 08:32 UTC'; canon_sha: local '790e79f7' vs live '6cdb86b3' - owed: a deploy -- the gates pass at this commit and production does not serve it - clear it: run `python harness/site/site_ritual.py` - seen: 1 (latest 2026-08-10 10:22 UTC) - local build: canon 790e79f7 pages 1298 generated 2026-08-10 10:21 UTC - verify base: https://review.humanityrpg.com
Blocks carry the gate that refused, the pages named in the refusal (extracted from the refusal
text, and labelled as extracted rather than inferred), who owes it, and the captured refusal folded
into a <details>. Keys name the CONDITION, not the commit (production-behind, deploy-unproven, or the gate slug), so repeats bump a seen-count and refresh the facts in place rather than spamming the file. A
verified deploy closes every OPEN block automatically with the sha that closed it.
The freshness hook: the ritual prints OPEN SITE DEBTS (at start) and (at end) on every run,
so a seat that never opens the ledger still learns the site owes a push.
The build refuses for real reasons (a link gate on .png bytes behind a .jpg URL — the no-PIL
fallback made honest at build_progress_site.py:956-960; a missing Pillow in the audio rig venv), and
build/site_deploy_record.json is written only on success (land_on_disk,
build_progress_site.py:3795). So the one artifact a verifier reads describes the last *successful*
deploy and says nothing about the three refusals since.
What the ritual does: it writes on FAILURE — the case that had no surface — and it appends **one
block per run**, success or failure, to build/ops/SEAT_REPORTS.md:
2026-08-10 10:22 UTC | seat: lane_a_site_reliability | commits: none | escalations: 0 | blocked-on: deploy-owed -- production is behind this build - site: deploy-owed -- 1298 pages -- canon 790e79f7 -- production is behind this build - owed: 1 OPEN block(s) in build/ops/SITE_DEBTS.md
That block is an append, never a read-modify-write: seats share the file and are not
synchronised. (Verified the hard way — a second lane wrote to SEAT_REPORTS.md while this lane's
own check was running. A concurrent-append *merge* no longer loses lines either: .gitattributes
gives the ops channels the union driver, because pull --rebase -X theirs was silently discarding
one side — build/ops/COMMS_PROTOCOL.md §7.)
The format was wrong for its first six hours, and that mattered more than it looks. The original
line led with - SITE RITUAL …, which the site's seat reader renders undated, and
seat:lane-a-site-reliability truncates at the first hyphen into a seat called "lane"
(harness/site/seats.py:82 _TIME_LEAD, :88 _SEAT_LABEL). A writer that lands a block on every run
buries the human reports faster than any seat can — at the time it was caught, SEAT_REPORTS.md
held five unparseable robot blocks and zero seat closes. Fixed at `site_ritual.py:608
report_line (+ :596 seat_token`), and **self-test arm 5 now checks the emitted head line against
seats.py's own regexes** rather than by eye.
build_progress_site.py:3070-3071 empties build/site whenever a build renders somewhere else. The
reasoning is sound (a tree that describes neither this build nor production is a verification trap)
but the cost is real: the only local copy of what IS live gets deleted, and the *tracked*
build/site_deploy_record.json is left standing beside an empty directory, still describing a
mirror that no longer exists. That is exactly the state this lane found the repo in at 10:05 UTC on
2026-08-10 — build/site empty, the record claiming a 1,298-page deploy, and a second seat's check
tree sitting in build/tmp/seat22_sitecheck that had caused it.
Two answers, both landed:
1. The ritual's --check holds the mirror aside (hold_mirror / release_mirror: a rename, not
a copy, so it is instant) and puts it back byte-for-byte, with an orphan recovery on the next
run if a ritual dies in between. A check can no longer cost you the local copy of production.
2. build_progress_site.py now stamps the tracked record with mirror_invalidated and why,
whenever it empties the mirror. Emptying the tree stays right; leaving a record that silently
over-claims does not.
Two rituals deploying at once is a way for production to end up serving neither. The ritual takes a
single-writer lock (build/.site_ritual.lock, pid + seat + start), refuses with exit 3 while a
live one is held, and clears it automatically when it is stale (90 min) or its process is gone. The
liveness probe uses OpenProcess, not os.kill(pid, 0) — on Windows the latter opens
PROCESS_TERMINATE and would kill the very run it was asking about.
The Cloudflare token is read from the user environment by build_progress_site.deploy() and never
passes through the ritual's output. Everything the ritual writes to disk goes through scrub()
first, which redacts the live values of CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID,
CLOUDFLARE_ACCESS_TOKEN, CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET by exact match,
then applies a shape rule for anything credential-like from a tool we do not control. The exact-match
leg is fired on a planted credential in the ritual's own self-test, because a shape rule alone would
let an unusual credential through.
A checker that has never been shown failing proves nothing. self_test() runs before anything else
on every invocation and prints its arm count. Eight arms today; the load-bearing one is the **stamp
comparator shown FAILING on a changed canon_sha** before it is trusted to report a match — that
comparator *is* the F3 detector, and an unarmed one would report every stale site as fresh. The
others: the comparator stays quiet on identical builds; the classifier names the prose gate; the
classifier calls unrecognised output unclassified-refusal and never clean; the scrubber
redacts a planted credential; the shape rule catches a token-shaped string; the surface extractor
reads pages out of a real refusal; and the seat-report head line is parsed by **the site's own
reader** (seats._TIME_LEAD, seats._SEAT_LABEL), so a format regression fails here rather than on
Josh's phone.
The whole ritual rests on the live fetch being live, so the question is worth answering with
citations rather than confidence. No, on both legs:
build_progress_site.py:2752 sets HTML_CACHE = "no-store" and :2791 headers_text writes it as the FIRST rule of _headers, /* — a catch-all that covers
/build_report.json too, since only the media extensions get an override. A build that emitted a
cacheable value for HTML is refused (:2850 FRESHNESS HEADER REFUSED), and the rule was
written because Josh reloaded the review page on his phone and got the page from before the fix.
?ritual=<epoch> query (site_ritual.py:368 _request), which changes the cache key. The
Cache-Control: no-cache request header on the same line is belt-and-braces and should not be
relied on — Cloudflare ignores client no-cache headers by design; the query bust is the mechanism
that actually works.
The consequence worth keeping: because the origin is no-store, **what the ritual proves and what
Josh's phone loads are the same bytes**. If a future change makes HTML cacheable, the ritual would
keep passing (it busts the cache) while Josh saw a stale page — which is F3 rebuilt out of headers.
The freshness gate is what stops that, and it is armed.
--check, 2026-08-10 10:20–10:22 UTC)
RITUAL SELF-TEST: 7 arms armed (the stamp comparator was shown FAILING on a changed sha before it
was trusted to report a match).
OPEN SITE DEBTS (at start): none.
progress-site [full-arc]: 79 nodes, 1298 pages, 40 images, 98.6 MB, firewall rules armed: 16
REVEAL FIREWALL: clean (positive control fired on the canary first)
PROSE GATE: 0 visible block(s) over 200 characters across 1298 pages; 16 arms armed …
LINK GATE: 0 dead link(s) on the review surfaces and the front door …
OPSEC SWEEP: 0 hit(s) across 6 rules …
LIVE VERIFY [api]: production is deployment f052c3c4 on branch 'main' (production, deploy success),
created 2026-08-10T08:34:44.743539Z
LIVE VERIFY [bytes]: MISMATCH against https://review.humanityrpg.com -- production is not this build:
generated: local '2026-08-10 10:21 UTC' vs live '2026-08-10 08:32 UTC'
canon_sha: local '790e79f7' vs live '6cdb86b3'
LIVE VERIFY [html]: DID NOT RUN -- the bytes arm never got there
SITE DEBT WRITTEN: build/ops/SITE_DEBTS.md -- key:production-behind.
SITE RITUAL: DEPLOY OWED -- production is behind this build. The gates passed; nothing was deployed
because this was a --check.
OPEN SITE DEBTS (at end): 1 -- production owes a push.
Exit 6, in about two minutes. Every gate passed at HEAD (790e79f7), production was 20 commits and ~1h50m behind, and that fact is now
on a tracked surface instead of in nobody's head. The refusal machinery was fired separately on a
real prose-gate refusal text against temp ledgers: classified prose-gate → exit 2, both named
pages extracted, a repeat bumped seen: 2 instead of duplicating, a planted token-shaped string did
not reach the file, and close_debts cleared it on a verified deploy.
1. RULED 2026-08-10 (5c9ce40e), and the ruling has a live consequence for this lane.
review.humanityrpg.com and humanityrpg.com are both attached to the humanity-review Pages
project and both answer 200 unauthenticated, while the pages.dev URL is the one behind
Cloudflare Access; the standing "blocked on unauthenticated Cloudflare MCP" note is stale. The
director's ruling: **the apex detaches from the Pages project, and review.humanityrpg.com joins
the Access app.** Both are Josh's-hand dashboard clicks and are tracked as IN YOUR COURT
(docs/DIRECTOR_RUNBOOK.md §3.1). Until they land, treat every "gated" site surface as PUBLIC.
When they land, this ritual's byte arm loses its anonymous witness — see 2, and set the
service token in the same sitting or every push becomes unprovable the moment the wall goes up.
2. Byte-level verification depends on a public base. If Access is ever extended to the custom
domains, every push becomes unprovable and the ritual will start exiting 5. The fix then is a
Cloudflare Access service token: set CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET in
the user environment and the ritual sends them automatically (site_ritual.py:_request).
build_progress_site.py --deploy. The old command still works and still gates; what it cannot do
is prove the push landed or leave a trace when it did not.
build/ops/SITE_DEBTS.md says so in its SEAT_REPORTS.md close. The ritual has already written the line.
## OPEN blocks mean production owes a push, and only averified deploy closes them.