Commitca839505Recorded23 Aug 2026Repositorysigil-doc

test: harden the SD-P8 gate and rework two sabotage legs after running them

Message

Running the sabotage suite found three things, all in the instruments.

LEG 02 predicted a red at the language arm and got one at arm 1. That is the expectation being wrong rather than the gate: arm 1 reads both profiles through the public modules, so it catches every PROFILE-LEVEL sabotage first. Recorded in the leg rather than quietly renumbered, and LEG 03 rewritten to reach past it -- it now sabotages the parser's multiplicity ENFORCEMENT while both profiles still say one, which separates "the profile is configured correctly" from "the parser obeys the profile". Only the second is what a required red is about.

New LEG 08 attacks the pinned literals from the DERIVATION side, by rewording the message formatter -- which is where a reword would actually come from. Arm 1 is blind to it and so is the differential; only the pins see it, and both languages' pins are required to.

Arm 5's root-only rewrite is now guarded. It substitutes the old root symbol for the new one and requires byte equality, and a substitution that could match anything else would MASK a difference. It refuses to rewrite unless the two spellings do not overlap, and dies if the rewrite matched nothing.

Also: arm 3's not checked assertion was a four-argument test ... -o ..., which is ambiguous in POSIX; it counts once and compares once now. And gate-sd-p3.sh's course_dir is dead since its arm 3 retired -- documented in place so that setting SD_P3_COURSE_DIR is not silently ignored.

Changed
 test/gate-sd-p3.sh     |   6 ++++++
 test/gate-sd-p8.sh     |  29 +++++++++++++++++++++++++++--
 test/sabotage-sd-p8.sh | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------
 3 files changed, 128 insertions(+), 31 deletions(-)
Diff
test/gate-sd-p3.shmodified
@@ -42,6 +42,12 @@ test -f test/transcripts/sd-p3/lowering-before.txt
42
test -f test/transcripts/sd-p3/boundary-census-before.txt
43
"$sigil" --version
44
+45
# DEAD SINCE THE ARM 3 RETIREMENT (2026-08-22), and kept only so that setting
+46
# `SD_P3_COURSE_DIR` is not silently ignored by a variable that no longer exists.
+47
# Arm 3 was the only reader; it widened the lowering corpus to the Guile course
+48
# conversion. Nothing else in this gate uses it, so setting the variable today
+49
# does nothing. Its successor, `test/gate-sd-p8.sh` arm 5, derives its corpus
+50
# from the repository rather than taking a directory.
51
course_dir="${SD_P3_COURSE_DIR:-}"
52
53
# `test-semantic-structures` reaches `(sigil doc render html)` and therefore
test/gate-sd-p8.shmodified
@@ -185,8 +185,13 @@ rows=$(wc -l <"$census")
185
test "$rows" -eq "$total" || die "arm 03: examined $total manuscripts but printed $rows rows"
186
test "$total" -ge 27 || die "arm 03: the census found only $total manuscripts; the population is wrong"
187
test "$bad" -eq 0 || { rg '^ILLEGAL' "$census" >&2; die "arm 03: $bad manuscripts declare a language that does not admit their root"; }
−188
test "$( { rg -c '^UNCHECKED' "$census" || true; } )" = "" -o "$( { rg -c '^UNCHECKED' "$census" || echo 0; } )" -eq 0 \
−189
|| { rg '^UNCHECKED' "$census" >&2; die "arm 03: some manuscripts were not classified at all"; }
+188
# The `not checked` class, asserted explicitly rather than inferred from the
+189
# absence of complaint. A `test A -o B` compound with four-plus arguments is
+190
# ambiguous in POSIX and easy to get subtly wrong, so this counts once and
+191
# compares once.
+192
unchecked=$( { rg -c '^UNCHECKED' "$census" || true; } )
+193
test "${unchecked:-0}" -eq 0 \
+194
|| { rg '^UNCHECKED' "$census" >&2; die "arm 03: ${unchecked} manuscripts were not classified at all"; }
195
test "$base_lang" -ge 1 || die "arm 03: no manuscript exercises the base language"
196
test "$book_lang" -ge 15 || die "arm 03: only $book_lang manuscripts moved to the genre language"
197
pass 3 "census $total manuscripts, 0 illegal, 0 unclassified :: $base_lang base, $book_lang book, $other_lang other-language, $negative negative fixtures"
@@ -410,7 +415,27 @@ for f in "$gate_dir/sec-after"/*; do
415
# section byte for byte -- a far stronger statement than "it changed, as
416
# predicted", and the one that would catch a migration that also moved content.
417
old="${root_b#@}"; new="${root_a#@}"
+418
# THE SUBSTITUTION MUST BE UNAMBIGUOUS BEFORE IT IS TRUSTED. A rewrite that
+419
# could match something other than the root symbol would MASK a real
+420
# difference, which is the one direction a differential must never fail in.
+421
# Two conditions make it unambiguous: after the change the old spelling must
+422
# be gone entirely, and before it the new spelling must not have existed. If
+423
# either fails, the manuscript is reported UNEXPECTED rather than rewritten --
+424
# a refusal to judge, not a pass.
+425
if [ "$( { rg -cF "]] $new>" "$b" || true; } )" != "" ] \
+426
|| [ "$( { rg -cF "]] $old>" "$f" || true; } )" != "" ]; then
+427
unexpected=$((unexpected + 1))
+428
printf 'UNEXPECTED %-40s root=%s -> %s, but the two spellings overlap; refusing to rewrite\n' \
+429
"$name" "$root_b" "$root_a" >>"$ledger"
+430
continue
+431
fi
432
sed "s/\]\] $old>/]] $new>/g" "$b" >"$gate_dir/rewritten-$name"
+433
# ...and the rewrite must actually have done something. A sed that matched
+434
# nothing would compare the base section to the after section unchanged, and
+435
# a manuscript whose only difference WAS the root would then read as
+436
# UNEXPECTED for the wrong reason.
+437
cmp -s "$b" "$gate_dir/rewritten-$name" \
+438
&& die "arm 05: the root rewrite for $name changed nothing, so its root symbol is not where this arm thinks it is"
439
if cmp -s "$f" "$gate_dir/rewritten-$name"; then
440
expected_change=$((expected_change + 1))
441
printf 'ROOT-ONLY %-40s root=%s -> %s, nothing else moved\n' "$name" "$root_b" "$root_a" >>"$ledger"
test/sabotage-sd-p8.shmodified
@@ -79,6 +79,23 @@ done
79
pristine_hash() { cat "$work/pristine-$(echo "$1" | tr / _)"; }
80
pristine_copy() { echo "$work/copy-$(echo "$1" | tr / _)"; }
81
+82
plant() { # $1 = file, $2 = pattern to replace, $3 = replacement
+83
local before after matches
+84
before=$(sha256sum "$1" | awk '{print $1}')
+85
test "$before" = "$(pristine_hash "$1")" || die "plant: $1 is not pristine"
+86
# UNIQUENESS FIRST. `sed` replaces the first match on every matching line and
+87
# says nothing about how many there were, so a pattern that drifted into
+88
# matching two sites would sabotage more than the leg describes -- and a leg
+89
# whose red cannot be attributed to one edit is not evidence.
+90
matches=$(grep -c -- "$2" "$1" || true)
+91
test "${matches:-0}" -eq 1 \
+92
|| die "plant: '$2' matches ${matches:-0} lines of $1; a sabotage must name exactly one site"
+93
sed -i "s|$2|$3|" "$1"
+94
after=$(sha256sum "$1" | awk '{print $1}')
+95
test "$before" != "$after" \
+96
|| die "plant: $1 did not change -- the sabotage never landed, so a red below would mean nothing"
+97
}
+98
99
# RETURNS non-zero rather than calling `die`. `die` ends in `exit`, and an
100
# `exit` inside a function still ends the SHELL -- so an `if ! restore_all`
101
# guard around a dying function can never take its own failure branch. That is
@@ -171,31 +188,48 @@ restore_all || die "restore failed; the worktree is still sabotaged"
188
# ---------------------------------------------------------------- LEG 02
189
# PRODUCT. Give the genre language the BASE language's name.
190
#
−174
# The `sigil/doc/book:` prefix is the part of the genre's diagnostic that sends
−175
# an author to the right documentation, and it is derived from one string in one
−176
# place. This is the edit that would silently undo it.
+191
# The `sigil/doc/book:` prefix is what sends an author to the right
+192
# documentation, and it comes from one string in one place.
+193
#
+194
# EXPECT THE RED AT ARM 01, NOT AT ARM 04. Arm 1 reads both profiles' name,
+195
# roots and multiplicity through the public modules, so it catches every
+196
# PROFILE-LEVEL sabotage before anything downstream runs. That layering was not
+197
# obvious until this leg was run: the first version of it predicted a red at arm
+198
# 4 and got one at arm 1, which is the expectation being wrong rather than the
+199
# gate. Recorded rather than quietly renumbered -- an arm that reddens somewhere
+200
# unpredicted is a finding either way
+201
# (topics/an-expectation-names-which-check-must-fail).
202
echo "=== LEG 02 -- the genre profile reports the base language's name"
203
plant "$BOOK" \
204
'(make-document-profile "sigil/doc/book" .(chapter) .one)' \
205
"(make-document-profile \"sigil/doc\" '(chapter) 'one)"
206
run_gate "$work/leg02.log" && die "LEG 02: the gate passed with the genre language misnamed"
−182
leg_red 02 "$work/leg02.log" "arm 04: the genre language did not refuse @article naming sigil/doc/book"
+207
leg_red 02 "$work/leg02.log" "arm 01: the genre profile is not D8's"
208
restore_all || die "restore failed; the worktree is still sabotaged"
209
210
# ---------------------------------------------------------------- LEG 03
−186
# PRODUCT. Widen the genre's multiplicity.
+211
# PRODUCT, AND IT HAS TO REACH PAST ARM 01.
212
#
−188
# A REQUIRED RED becoming green is the failure mode a pass/fail count cannot
−189
# see: with `one-or-more`, the two-chapter manuscript is ACCEPTED and the gate
−190
# must say which case stopped being refused.
−191
echo "=== LEG 03 -- the genre accepts several chapters"
−192
plant "$BOOK" \
−193
"'(chapter) 'one)" \
−194
"'(chapter) 'one-or-more)"
−195
run_gate "$work/leg03.log" && die "LEG 03: the gate passed with the genre accepting two chapters"
+213
# The point is to make a REQUIRED RED go green -- the failure mode a pass/fail
+214
# count cannot see -- and to do it where arm 1 is blind. Widening the profile's
+215
# multiplicity would not work: arm 1 asserts `mult=one` for both languages and
+216
# would catch it first, exactly as leg 2 just showed. So this sabotages the
+217
# ENFORCEMENT instead, in `parse-document-parts-with-profile`, where `several?`
+218
# gates whether a second root is accepted. Both profiles still SAY `one`; the
+219
# parser stops meaning it.
+220
#
+221
# That is a strictly better sabotage than the one it replaces: it separates
+222
# "the profile is configured correctly" from "the parser obeys the profile",
+223
# and only the second is what the required reds are about.
+224
echo "=== LEG 03 -- the parser stops enforcing multiplicity, while both profiles still say one"
+225
plant "$PARSER" \
+226
'(or several? (null? root-nodes))' \
+227
'(or #t (null? root-nodes))'
+228
run_gate "$work/leg03.log" && die "LEG 03: the gate passed with multiplicity unenforced"
229
leg_red 03 "$work/leg03.log" "arm 04: not every case matched its expected verdict"
−197
rg -qF 'SD_P8_LANG book-refuses-two want=refuse got=accept MISMATCH' "$work/leg03.log" \
−198
|| die "LEG 03: the gate reddened, but not because the two-chapter manuscript was accepted"
+230
rg -qF 'want=refuse got=accept MISMATCH' "$work/leg03.log" \
+231
|| { rg -n 'SD_P8_LANG' "$work/leg03.log" >&2
+232
die "LEG 03: the gate reddened, but no required-refusal case flipped to accept"; }
233
restore_all || die "restore failed; the worktree is still sabotaged"
234
235
# ---------------------------------------------------------------- LEG 04
@@ -256,34 +290,66 @@ leg_red 07 "$work/leg07.log" "arm 04: the probe declares only 5 cases"
290
restore_all || die "restore failed; the worktree is still sabotaged"
291
292
# ---------------------------------------------------------------- LEG 08
+293
# PRODUCT, AND IT IS THE ONE THE BRIEF ASKED FOR BY NAME.
+294
#
+295
# SD-P8 rewrote pinned diagnostic literals. The risk the brief named is a
+296
# migration that updates a pin to match new output, leaving pin and product
+297
# moved together and neither tested against the other. Leg 1 attacks that from
+298
# the PROFILE side; this leg attacks it from the DERIVATION side, which is where
+299
# a reword would actually come from -- someone tidying the message formatter.
+300
#
+301
# Arm 1 is blind to it (both profiles still report the right name, roots and
+302
# multiplicity) and so is arm 5 (the lowering of a legal manuscript does not
+303
# change). Only the pinned literals can see it, which is the whole point.
+304
echo "=== LEG 08 -- the derived misplaced-root message is reworded"
+305
plant "$PARSER" \
+306
'": top-level author forms must precede "' \
+307
'": top-level author forms must come before "'
+308
run_gate "$work/leg08.log" && die "LEG 08: the gate passed with a shipped diagnostic reworded"
+309
leg_red 08 "$work/leg08.log" "arm 04: the base language did not refuse @chapter with its own diagnostic"
+310
restore_all || die "restore failed; the worktree is still sabotaged"
+311
# ...and the SUITE's pins see it too, which is what makes them pins rather than
+312
# decoration. Run separately because the gate stops at its first failing arm.
+313
plant "$PARSER" \
+314
'": top-level author forms must precede "' \
+315
'": top-level author forms must come before "'
+316
run_suite "$work/leg08-suite.log" && die "LEG 08: the suite passed with a shipped diagnostic reworded"
+317
rg -qF "the base profile is D8's policy and D8's diagnostics, exactly" "$work/leg08-suite.log" \
+318
|| { tail -25 "$work/leg08-suite.log" >&2; die "LEG 08: the suite reddened somewhere other than the base pins"; }
+319
rg -qF "the book genre language is a sibling profile with its own diagnostics" "$work/leg08-suite.log" \
+320
|| { tail -25 "$work/leg08-suite.log" >&2; die "LEG 08: the GENRE language's pins did not see the reword"; }
+321
printf 'SD_P8_SABOTAGE_08 RED both languages pinned literals saw a reworded derivation\n'
+322
restore_all || die "restore failed; the worktree is still sabotaged"
+323
+324
# ---------------------------------------------------------------- LEG 09
325
# THE POSITIVE CONTROL, and it is not a formality.
326
#
−261
# Seven reds prove the gate can fire. They say nothing about whether it can
+327
# Eight reds prove the gate can fire. They say nothing about whether it can
328
# STAND DOWN -- a gate wedged red by leg 1's restore having gone wrong would
329
# score every leg above as a success. This is also what proves the tree came
330
# back: `restore_all` checks hashes, and this checks behaviour.
−265
echo "=== LEG 08 -- the restored tree is green"
+331
echo "=== LEG 09 -- the restored tree is green"
332
restore_all || die "restore failed; the worktree is still sabotaged"
333
# THE FULL GATE, with SD_P8_SKIP_REGRESSION deliberately UNSET. Every leg above
334
# ran with it set; if this control ran the same way, nothing here would ever have
335
# executed the regression arm and "green" would be a claim about five arms
336
# described as six.
337
set +e
−272
bash test/gate-sd-p8.sh >"$work/leg08.log" 2>&1
+338
bash test/gate-sd-p8.sh >"$work/leg09.log" 2>&1
339
control_rc=$?
340
set -e
−275
sed -i 's/\x1b\[[0-9;]*m//g' "$work/leg08.log"
+341
sed -i 's/\x1b\[[0-9;]*m//g' "$work/leg09.log"
342
test "$control_rc" -eq 0 || {
−277
tail -40 "$work/leg08.log" >&2
−278
die "LEG 08: the restored tree is NOT green; every red above is uninterpretable"; }
−279
rg -q '^SD_P8_GATE_GREEN$' "$work/leg08.log" \
−280
|| { tail -20 "$work/leg08.log" >&2; die "LEG 08: the gate exited 0 without printing its green marker"; }
−281
rg -q 'SD_P8_GATE_PARTIAL' "$work/leg08.log" \
−282
&& die "LEG 08: the control run skipped an arm; it does not establish a full green"
+343
tail -40 "$work/leg09.log" >&2
+344
die "LEG 09: the restored tree is NOT green; every red above is uninterpretable"; }
+345
rg -q '^SD_P8_GATE_GREEN$' "$work/leg09.log" \
+346
|| { tail -20 "$work/leg09.log" >&2; die "LEG 09: the gate exited 0 without printing its green marker"; }
+347
rg -q 'SD_P8_GATE_PARTIAL' "$work/leg09.log" \
+348
&& die "LEG 09: the control run skipped an arm; it does not establish a full green"
349
for n in 01 02 03 04 05 06; do
−284
rg -q "^SD_P8_GATE_$n " "$work/leg08.log" \
−285
|| die "LEG 08: arm $n printed no marker in the control run"
+350
rg -q "^SD_P8_GATE_$n " "$work/leg09.log" \
+351
|| die "LEG 09: arm $n printed no marker in the control run"
352
done
−287
printf 'SD_P8_SABOTAGE_08 GREEN the restored tree passes all six arms\n'
+353
printf 'SD_P8_SABOTAGE_09 GREEN the restored tree passes all six arms\n'
354
−289
echo "SD_P8_SABOTAGE_COMPLETE 7 reds (5 product, 1 instrument, 1 coverage) and 1 positive control"
+355
echo "SD_P8_SABOTAGE_COMPLETE 8 reds (6 product, 1 instrument, 1 coverage) and 1 positive control"