Commit15732c5eRecorded23 Aug 2026Repositorysigil-doc

test: act on the adversarial review, and retire+replace gate-sd-p7 arm 02

Message

The review (read-only over 0b7c5ca..HEAD) found six live defects. All are fixed here, and running the fixes found three more in the fixes themselves.

RETIRED test/gate-sd-p7.sh arm 02 and REPLACED it. It is the same class as the gate-sd-p3 arms retired earlier -- a differential pinned to an endpoint this phase moves -- and it was missed the first time. Characterised by measurement before the notice was written: the exclusion set, the record count (22 -> 27) and the baseline's ABSOLUTE byte positions each retire it independently. With the location numbers normalised away, 19 of 22 manuscripts are structurally identical and the 3 that differ are D8 itself -- two kind flips and sd-p1-article-root going from a raise to a real document. The property is intact and the arm cannot say so.

Its replacement is gate-sd-p8 arm 06, both sides in one run, asserting on STRUCTURE rather than positions -- deliberately, since this retirement is the demonstration of why a baseline pinned to offsets is a trap. The weakening is named in the arm: it cannot see a pure location regression, and arm 05 plus the root-profile suite cover that.

Arm 05 now asserts only_after and the skipped SETS. A ten-line content rewrite of sd-p1-article-root.sgd had been passing as "the base tree could not lower it". The first skipped check compared COUNTS and reddened falsely -- a count cannot tell "a manuscript regressed" from "a manuscript was added to be refused", so it compares names, in both directions, deriving the added set from git.

Arm 05's root-only rewrite is bounded to at most two generated heads, so a manuscript that changed its root AND a nested command of the same name is reported UNEXPECTED rather than "nothing else moved".

Arm 03's UNCHECKED class is reachable now; it was defaulted and overwritten on every branch, so "0 unclassified" was vacuous. The class that exists is "a language this arm does not know", which is not the same as a manuscript being wrong.

The dump's make-srcloc predicate guards on the manuscript's own name -- its comment had claimed a check the code did not make -- and an unrecognised binding suffix is reported rather than crashing the emitter.

sabotage LEG 06 is distinguishable from LEG 05: both reddened with the same sentence, so either would have scored the other's success.

The gate transcript now lists uncommitted paths. It printed head: while measuring the working tree.

Also: src/sigil/doc/reader.sgl said the base language admits "exactly one @chapter" and that its diagnostics are reproduced "byte for byte" -- true when SD-P1 wrote them, falsified by D8 two weeks later. And the "nineteen manuscripts" figure in four places is 17; the counts now reconcile to the census's independently derived 32.

Changed
 src/sigil/doc/reader.sgl     |  30 +++++++++----
 test/gate-portability.sh     |  10 +++++
 test/gate-sd-p3.sh           |  19 ++++----
 test/gate-sd-p7.sh           | 138 ++++++++++++++++++++++++---------------------------------
 test/gate-sd-p8.sh           | 397 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------
 test/sabotage-sd-p8.sh       |  24 +++++++++-
 test/sd-p8-lowering-dump.sgl |  38 ++++++++++++----
 test/test-passive-parse.sgl  |  11 +++--
 8 files changed, 511 insertions(+), 156 deletions(-)
Diff
src/sigil/doc/reader.sglmodified
@@ -2,11 +2,24 @@
2
;;; surface a DELEGATING language profile builds on.
3
;;;
4
;;; SD-P1 (review finding B2): the root container is a profile parameter rather
−5
;;; than a constant in the parser. `#lang sigil/doc` uses `sigil-doc-profile`,
−6
;;; which is exactly the historical policy -- exactly one `@chapter` -- spelled
−7
;;; as a value. A language such as `#lang folio/doc`, whose roots are `@note`
−8
;;; and `@project`, constructs its own profile and delegates the whole grammar
−9
;;; and every location to this module. It must never fork the parser.
+5
;;; than a constant in the parser.
+6
;;;
+7
;;; SD-P8 then used that parameter on `sigil/doc` itself, executing ruling D8
+8
;;; (2026-08-22): `#lang sigil/doc` admits exactly one `@article` -- the generic
+9
;;; standalone document -- and `@chapter` moved to the sibling genre language
+10
;;; `#lang sigil/doc/book` in `(sigil doc book reader)`. So the base profile is
+11
;;; NO LONGER the historical policy spelled as a value, and its two root-policy
+12
;;; diagnostics are deliberately no longer byte-identical to the pre-SD-P1
+13
;;; strings: they are DERIVED from the profile and followed its root set when it
+14
;;; moved, with no edit to the formatter.
+15
;;;
+16
;;; (The two sentences those two paragraphs replace were TRUE when SD-P1 wrote
+17
;;; them and were falsified by a ruling two weeks later that did not touch this
+18
;;; file. Prose beside code goes stale silently and no gate covers it.)
+19
;;;
+20
;;; A language such as `#lang folio/doc`, whose roots are `@note` and
+21
;;; `@project`, constructs its own profile and delegates the whole grammar and
+22
;;; every location to this module. It must never fork the parser.
23
;;;
24
;;; The profile constructors are re-exported here rather than left in
25
;;; `(sigil doc private parser)` so that a delegating reader depends only on a
@@ -25,9 +38,10 @@
38
(let ((entry (assq key context)))
39
(if entry (cdr entry) (error "sigil/doc reader: incomplete language context"))))
40
−28
;; `#lang sigil/doc`. Behaviour is unchanged by SD-P1: the profile it passes
−29
;; reproduces the previous hardcoded policy, including both diagnostics
−30
;; byte for byte.
+41
;; `#lang sigil/doc`. SD-P1 made the root container a parameter without
+42
;; changing behaviour; SD-P8 then changed the behaviour deliberately, by
+43
;; changing only the profile this passes. The form sequence emitted below is
+44
;; unchanged by both.
45
(define (read-language-source source context)
46
(read-language-source-with-profile sigil-doc-profile source context))
47
test/gate-portability.shmodified
@@ -25,6 +25,16 @@ if rg -n -- '-L[[:space:]]+\.\./sigil-sxml/src' "${active_scripts[@]}"; then
25
exit 1
26
fi
27
+28
# SD-P8 added `test/gate-sd-p8.sh`, which resolves sigil-sxml and the dev
+29
# compiler exactly as these do. Adding it here rather than leaving it to the
+30
# absolute-path sweep alone is the population check this file exists to be: a
+31
# new gate that took a sibling path unconfigurably would pass the regex above
+32
# and still be unportable.
+33
rg -q 'SIGIL_SXML_ROOT' test/gate-sd-p8.sh
+34
rg -q 'SIGIL_D1_COMPILER' test/gate-sd-p8.sh
+35
rg -q 'SIGIL_D1_COMPILER' test/sabotage-sd-p8.sh
+36
rg -q 'SIGIL_SXML_ROOT' test/sabotage-sd-p8.sh
+37
38
for gate in test/gate-d4a.sh test/gate-d6.sh test/gate-d6-customization.sh \
39
test/gate-semantic-structures.sh test/gate-glossary.sh; do
40
rg -q 'SIGIL_SXML_ROOT' "$gate"
test/gate-sd-p3.shmodified
@@ -135,7 +135,9 @@ pass 2 "equality over $texts text / $codes code / $refs reference / $urls url-li
135
#
136
# WHY IT IS GONE. Ruling D8 (David, 2026-08-22) moves `@chapter` out of
137
# `#lang sigil/doc` into `#lang sigil/doc/book`, which edits the `#lang` line of
−138
# nineteen manuscripts. All 16 of this arm's corpus are among them. A header
+138
# 17 manuscripts (plus 3 more whose ROOT command changed with the header left
+139
# alone, and 1 rewritten outright). All 16 of this arm's corpus are among the 17.
+140
# A header
141
# line growing from 16 bytes to 21 moves every byte offset in the manuscript by
142
# 5, so every one of those hashes changes -- measured before the migration was
143
# written, by editing one fixture and re-running the dump:
@@ -166,13 +168,14 @@ pass 2 "equality over $texts text / $codes code / $refs reference / $urls url-li
168
# BECAUSE it is ruled and recorded -- laundering is hiding a red, not retiring a
169
# differential whose endpoints have moved.
170
#
−169
# SUCCESSOR, and it is strictly stronger: `test/gate-sd-p8.sh` arm 3. It
+171
# SUCCESSOR, and it is strictly stronger: `test/gate-sd-p8.sh` arm 5. It
172
# generates BOTH sides in one run -- a detached worktree at SD-P8's own base and
173
# the working tree -- so it has no committed baseline to go stale, and it
174
# refuses with "baseline superseded" if HEAD stops descending from that base. It
−173
# compares 21 manuscripts rather than 16, on a DERIVED population rather than an
+175
# compares 22 manuscripts rather than 16 (21 of them in the intersection with the
+176
# base tree), on a DERIVED population rather than an
177
# inherited list, and its claim is not "the hashes match" but "the only thing
−175
# that changed is the root symbol, and for the nineteen header-only manuscripts
+178
# that changed is the root symbol, and for the 17 header-only manuscripts
179
# not even that" -- a hash equality tells you two numbers differ and nothing
180
# about why.
181
#
@@ -185,8 +188,8 @@ pass 2 "equality over $texts text / $codes code / $refs reference / $urls url-li
188
# STANDING PATTERN, restated for whoever changes lowering next: ship your own
189
# before/after differential scoped to your own base, embed the base SHA in the
190
# arm, and make a superseded base fail with "baseline superseded -- see ruling"
−188
# rather than with a bare diff. `test/gate-sd-p8.sh` arm 3 is a worked example.
−189
pass 3 "RETIRED 2026-08-22 at SD-P8 under Fable's 2026-08-11 ruling; superseded by test/gate-sd-p8.sh arm 3 (see comment)"
+191
# rather than with a bare diff. `test/gate-sd-p8.sh` arm 5 is a worked example.
+192
pass 3 "RETIRED 2026-08-22 at SD-P8 under Fable's 2026-08-11 ruling; superseded by test/gate-sd-p8.sh arm 5 (see comment)"
193
194
# ---------------------------------------------------------------- ARM 04
195
# RETIRED 2026-08-22, same disposition and same reasoning as arm 3 above, of
@@ -212,10 +215,10 @@ pass 3 "RETIRED 2026-08-22 at SD-P8 under Fable's 2026-08-11 ruling; superseded
215
# structurally by `test/test-prose-runs.sgl` (arm 1 of this gate, whose count is
216
# derived from that suite) and by arm 2's node-by-node equality between the bare
217
# and wrapped spellings, neither of which depends on a pre-SD-P3 hash table.
−215
# `test/gate-sd-p8.sh` arm 3 additionally shows that the D8 migration moved
+218
# `test/gate-sd-p8.sh` arm 5 additionally shows that the D8 migration moved
219
# NOTHING in the lowering of the manuscripts SD-P3 cared about, header aside,
220
# which is the closest standing equivalent of what this arm certified.
−218
pass 4 "RETIRED 2026-08-22 at SD-P8 with arm 3, whose changed-set was its only input; property covered by arm 1, arm 2 and test/gate-sd-p8.sh arm 3 (see comment)"
+221
pass 4 "RETIRED 2026-08-22 at SD-P8 with arm 3, whose changed-set was its only input; property covered by arm 1, arm 2 and test/gate-sd-p8.sh arm 5 (see comment)"
222
223
# ---------------------------------------------------------------- ARM 05
224
# Regression: the suites that own the frozen behaviour, with their counts
test/gate-sd-p7.shmodified
@@ -113,89 +113,65 @@ test "$witnesses" = "$disjuncts" \
113
pass 1 "section-profile suite ${suite_total}/${suite_total}, ${sections_built} sections built through a profile, ${witnesses} core witnesses = ${disjuncts} block-node? disjuncts"
114
115
# ---------------------------------------------------------------- ARM 02
−116
# DEFAULT BEHAVIOUR UNCHANGED, MEASURED. Every manuscript in the tree is built
−117
# and its canonical document -- every field AND every node's full source
−118
# location -- compared against the pre-change baseline. "The suites still pass"
−119
# only covers the properties someone thought to assert; this is an equality.
−120
before=test/transcripts/sd-p7/construction-before.txt
−121
after="$gate_dir/construction-after.txt"
−122
−123
# The baseline must carry its own provenance, so a superseded one fails as
−124
# "baseline superseded" rather than as a bare diff.
−125
grep -q '^SD_P7_CONSTRUCTION_DUMP v1$' "$before" \
−126
|| die "arm 02: the baseline carries no provenance marker"
−127
base_sha=$(grep '^base-sha: ' "$before" | sed 's/^base-sha: //')
−128
git cat-file -e "$base_sha^{commit}" 2>/dev/null \
−129
|| die "arm 02: baseline names commit $base_sha, which is not in this repository"
−130
git merge-base --is-ancestor "$base_sha" HEAD \
−131
|| die "arm 02: baseline base $base_sha is not an ancestor of HEAD -- baseline superseded, regenerate and say so"
−132
−133
bash test/sd-p7-construction-dump.sh "$after" >/dev/null 2>&1 \
−134
|| die "arm 02: could not produce an AFTER dump"
−135
−136
# Compare the RECORDS only. Headers legitimately differ (different base sha,
−137
# different dirty-path count) and so does the completion line, which now reports
−138
# the exclusion the baseline predates. The documents must not differ.
−139
before_body="$gate_dir/before.body"; after_body="$gate_dir/after.body"
−140
sed -n '/^=== /,$p' "$before" | grep -v '^SD_P7_CONSTRUCTION_DUMP_COMPLETE' > "$before_body"
−141
sed -n '/^=== /,$p' "$after" | grep -v '^SD_P7_CONSTRUCTION_DUMP_COMPLETE' > "$after_body"
−142
−143
# ...and the completion markers are asserted separately rather than dropped, so
−144
# a truncated run cannot pass by having had its marker filtered away.
−145
grep -q '^SD_P7_CONSTRUCTION_DUMP_COMPLETE ' "$before" || die "arm 02: baseline has no completion marker"
−146
grep -q '^SD_P7_CONSTRUCTION_DUMP_COMPLETE ' "$after" || die "arm 02: AFTER has no completion marker"
−147
−148
# A diff of two empty files compares equal (taxonomy shape 4). Assert the
−149
# compared artifact is non-trivial, against a denominator READ FROM THE AFTER
−150
# ARTIFACT'S OWN HEADER rather than recomputed here or imagined.
−151
# Field position, not "the first number on the line": these header lines carry
−152
# prose that contains digits (`SD-P7`, `*.sgd`), and a number extractor over
−153
# prose is a coin flip that happens to be right today.
−154
compared=$(grep '^compared: ' "$after" | awk '{print $2}')
−155
found=$(grep '^population: ' "$after" | awk '{print $2}')
−156
excluded=$(grep '^excluded: ' "$after" | awk '{print $2}')
−157
test -n "$compared" || die "arm 02: AFTER header reports no compared count"
−158
test "$((compared + excluded))" = "$found" \
−159
|| die "arm 02: AFTER partition does not account for its population ($compared + $excluded != $found)"
−160
# THE EXCLUSION IS BOUNDED BY THE COMMIT, NOT BY A NAME PREFIX.
+116
# RETIRED 2026-08-23 BY THE LEADER, AT SD-P8, UNDER FABLE'S RULING OF
+117
# 2026-08-11. This arm has NOT silently disappeared and its red has NOT been
+118
# hidden -- read this before concluding anything.
119
#
−162
# The first version of this check compared the dump's excluded count against
−163
# `find test/fixtures -name 'sd-p7-*.sgd' | wc -l` -- which is the SAME
−164
# PREDICATE RESTATED. Add another `sd-p7-*.sgd` later and both sides grow
−165
# together, the gate stays green, and the manuscript silently leaves the
−166
# differential. A tautology cannot bound anything.
+120
# WHAT IT DID. It regenerated `test/sd-p7-construction-dump.sh` and required the
+121
# CONSTRUCTED document of every manuscript in the tree -- `document->string`,
+122
# which carries every field and every node's full source location -- to be
+123
# byte-identical to `test/transcripts/sd-p7/construction-before.txt`. That is a
+124
# strong statement, and a genuinely different one from a lowering differential:
+125
# SD-P7 changed CONSTRUCTION (`make-section` and the author runtime's node
+126
# admission) and touched lowering not at all, so a lowering differential would
+127
# have been byte-identical under a completely broken SD-P7.
128
#
−168
# The real bound is: an excluded manuscript must be one THIS BRANCH ADDED since
−169
# the baseline's base. That set cannot grow by naming convention.
−170
added_manuscripts="$gate_dir/added-manuscripts"
−171
git diff --name-only --diff-filter=A "$base_sha" HEAD -- 'book/*.sgd' 'test/*.sgd' \
−172
| sort > "$added_manuscripts"
−173
excluded_paths="$gate_dir/excluded-paths"
−174
grep '^excluded-path: ' "$after" | sed 's/^excluded-path: //' | sort > "$excluded_paths"
−175
test "$excluded" -ge 1 || die "arm 02: the exclusion matched nothing, so it is not describing what it claims"
−176
test "$excluded" = "$(wc -l < "$excluded_paths")" \
−177
|| die "arm 02: the dump's excluded COUNT and its excluded PATHS disagree"
−178
diff -u "$added_manuscripts" "$excluded_paths" > "$gate_dir/exclusion.diff" \
−179
|| { cat "$gate_dir/exclusion.diff"
−180
die "arm 02: the exclusion set is not exactly the manuscripts this branch added since $base_sha"; }
−181
while read -r p; do
−182
test -f "$p" || die "arm 02: excluded path $p does not exist"
−183
done < "$excluded_paths"
−184
−185
before_records=$(grep -c '^=== ' "$before_body")
−186
after_records=$(grep -c '^=== ' "$after_body")
−187
test "$before_records" -gt 0 || die "arm 02: the baseline body holds no records"
−188
test "$after_records" = "$compared" \
−189
|| die "arm 02: AFTER wrote $after_records records for $compared compared manuscripts"
−190
test "$after_records" = "$before_records" \
−191
|| die "arm 02: AFTER covered $after_records manuscripts, baseline covered $before_records"
−192
before_bytes=$(wc -c < "$before_body")
−193
test "$before_bytes" -gt 10000 \
−194
|| die "arm 02: baseline body is only $before_bytes bytes -- too small to be the real dump"
−195
−196
diff -u "$before_body" "$after_body" > "$gate_dir/construction.diff" \
−197
|| { head -40 "$gate_dir/construction.diff"; die "arm 02: constructed documents changed"; }
−198
pass 2 "constructed documents identical across $after_records manuscripts ($before_bytes bytes compared; $excluded added by this phase excluded and exercised by the suite instead; baseline $base_sha)"
+129
# WHY IT IS GONE. Ruling D8 moves `@chapter` out of `#lang sigil/doc`, and this
+130
# arm is pinned to its endpoint in THREE independent ways, each of which alone
+131
# retires it. Measured, not argued -- `test/sd-p7-construction-dump.sh` was run
+132
# against this tree and compared to the committed baseline:
+133
#
+134
# 1. THE EXCLUSION SET is `git diff --diff-filter=A <base> HEAD` compared for
+135
# EXACT equality against the dump's excluded paths. SD-P8 adds five
+136
# fixtures, so this dies first, before any document is compared.
+137
# 2. THE RECORD COUNT: the baseline header says `population: 22 manuscripts`;
+138
# this tree has 32 `.sgd` files and the dump compares 27.
+139
# 3. THE EMBEDDED POSITIONS. The baseline is full of ABSOLUTE byte offsets --
+140
# `("book/practical-sigil/01-useful-first-script.sgd" 78 4 1 2143)` -- and
+141
# a `#lang` line growing from 16 to 21 bytes shifts every one of them by 5
+142
# on all 17 header-migrated manuscripts.
+143
#
+144
# WHAT THE MEASUREMENT ACTUALLY SHOWS, which is the part that matters for
+145
# whether anything is being hidden. Of the 22 manuscripts common to both sides,
+146
# once the location NUMBERS are normalised away:
+147
#
+148
# 19 of 22 structurally IDENTICAL
+149
# 3 of 22 structurally different, and all three are D8 itself:
+150
# sd-p3-bare-prose.sgd (kind chapter) -> (kind article)
+151
# sd-p3-wrapped.sgd (kind chapter) -> (kind article)
+152
# sd-p1-article-root.sgd RAISED ... -> a real document, because D8 is
+153
# exactly what makes `@article` legal in the base
+154
# language; this is the phase's headline effect
+155
# showing up in the construction dump
+156
#
+157
# So the property this arm protects is INTACT, and the arm cannot say so because
+158
# its baseline is pinned to offsets. That is endpoint drift, not a defect, and
+159
# it is the same disposition as `test/gate-sd-p3.sh` arms 3 and 4.
+160
#
+161
# REGENERATING THE BASELINE IS THE WRONG ANSWER, for the same reason it was
+162
# wrong there: `construction-before.txt` is a PRE-SD-P7 endpoint. Regenerating
+163
# it against today's tree destroys the transition it certifies and leaves a
+164
# differential comparing the present to the present.
+165
#
+166
# SUCCESSOR: `test/gate-sd-p8.sh` arm 06, which generates BOTH sides in one run
+167
# (a detached worktree at SD-P8's own base, and the working tree) and asserts on
+168
# STRUCTURE rather than on absolute byte positions -- deliberately, because this
+169
# retirement is the demonstration of why a baseline pinned to offsets is a trap.
+170
#
+171
# EVIDENCE PRESERVED: `test/transcripts/sd-p7/construction-before.txt` stays in
+172
# the tree unmodified, and the `test -f` preconditions still assert it is
+173
# present. It is SD-P7's reviewed evidence.
+174
pass 2 "RETIRED 2026-08-23 at SD-P8 under Fable's 2026-08-11 ruling; superseded by test/gate-sd-p8.sh arm 06, which asserts on structure rather than offsets (see comment)"
175
176
# ---------------------------------------------------------------- ARM 03
177
# THE DELEGATING-LANGUAGE CONTRACT. A profile is installed by binding the
test/gate-sd-p8.shmodified
@@ -10,7 +10,7 @@
10
# here. The exceptions are stated where they occur and are FLOORS, not
11
# restatements of a value.
12
#
−13
# ARM 3 IS A DIFFERENTIAL AND IS SCOPED TO ITS OWN BASE. It generates BOTH sides
+13
# ARM 5 IS A DIFFERENTIAL AND IS SCOPED TO ITS OWN BASE. It generates BOTH sides
14
# in one run -- a detached worktree at $base_sha, and this working tree -- so it
15
# has no committed baseline to go stale, and it fails with "baseline superseded"
16
# rather than a bare diff if its base is no longer an ancestor of HEAD. It is
@@ -43,6 +43,14 @@ mkdir -p "$repo_dir/build"
43
# PROVENANCE. This differential's meaning is bound to this commit: it is the
44
# tree D8 was executed against. If HEAD ever stops descending from it, the arm
45
# says so in those words instead of printing a diff nobody can interpret.
+46
# THE BASE IS PINNED AND IS DELIBERATELY NOT `master`. `0b7c5ca` is the tree D8
+47
# was executed against; master moved past it on 2026-08-23 while this branch was
+48
# in flight. This differential certifies the D8 TRANSITION, not "everything since
+49
# last Tuesday", and re-pointing it at a moving master would reintroduce exactly
+50
# the endpoint drift that retired `gate-sd-p3.sh` arms 3-4 and `gate-sd-p7.sh`
+51
# arm 2. The is-ancestor check below is what keeps it honest: it still holds
+52
# after a rebase onto a later master, and refuses with "baseline superseded" if
+53
# it ever stops.
54
base_sha="${SD_P8_BASE_SHA:-0b7c5ca}"
55
56
sigil="${SIGIL_D1_COMPILER:-$repo_dir/../task-sigil-language-readers/build/dev/bin/sigil}"
@@ -71,7 +79,16 @@ test -f test/sd-p8-language-probe.sgl
79
echo " compiler: $sigil"
80
echo " redirects: $redirects"
81
echo " base_sha: $base_sha"
+82
# THE TREE STATE, NOT JUST THE COMMIT. This gate measures the WORKING TREE, and
+83
# printing only `head:` would name a commit it is not testing -- a differential
+84
# whose transcript is one `git stash` away from describing a different tree than
+85
# it measured. `test/sd-p7-construction-dump.sh` already does this properly and
+86
# it is where the shape is borrowed from. Uncommitted paths are LISTED, not
+87
# counted, because "3 modified paths" tells a later reader nothing about whether
+88
# the modification was in the product or in the gate.
89
echo " head: $(git rev-parse HEAD)"
+90
echo " tree: $(git status --porcelain | wc -l) uncommitted path(s)"
+91
git status --porcelain | sed 's/^/ tree-dirty: /'
92
93
sxml_root="${SIGIL_SXML_ROOT:-$repo_dir/../sigil-sxml}"
94
test -d "$sxml_root/src" || die "sigil-sxml source not found at $sxml_root/src"
@@ -138,16 +155,84 @@ rg -q '^SD_P8_KIND article :: "article"$' "$kind_log" \
155
|| { cat "$kind_log" >&2; die "arm 02: article is no longer constructible -- folio's publication lowering depends on it"; }
156
rg -q '^SD_P8_KIND chapter :: "chapter"$' "$kind_log" \
157
|| { cat "$kind_log" >&2; die "arm 02: chapter is no longer constructible"; }
−141
# THE POPULATION, not the diff. Every `make-document` call site in shipped source
−142
# and in the test tree, classified; any that names `'book` is a finding.
+158
# THE POPULATION, NOT THE DIFF -- and it needs an exception class, which the
+159
# first version of this sweep did not have.
+160
#
+161
# WHAT WENT WRONG, because it is the interesting part. The sweep was
+162
# "every `(make-document` site in the tree; any naming `'book` is a finding",
+163
# and it fired on `test/test-root-profile.sgl`, where two sites construct
+164
# `'book` DELIBERATELY inside `assert-error` to prove the kind is refused. The
+165
# sweep could not tell a violation from the test that guards against
+166
# violations, so it was red on a healthy tree. It was never seen green: the
+167
# sabotage suite reached this arm before any full gate run did, on a leg aimed
+168
# at something else entirely.
+169
#
+170
# The repair is not to exclude `test/`, which would gut the sweep. It is to
+171
# CLASSIFY every site and print one row each, so the output length is the
+172
# denominator and an unclassified site fails loudly:
+173
#
+174
# * in `src/`, `tools/` or `book/` -- shipped or product code -- a `'book`
+175
# site is a violation, full stop;
+176
# * in `test/` it is legitimate ONLY as a refusal test, which means the site
+177
# sits inside an `assert-error` or a captured-message form;
+178
# * anything else is UNCLASSIFIED and fails, because "I could not tell" and
+179
# "it is fine" must not print the same way.
+180
#
+181
# And at least one NEGATIVE-TEST site is REQUIRED. Without one, nothing in the
+182
# tree exercises the refusal at a call site and this arm would be asserting an
+183
# absence with no positive control -- which passes for free.
184
sites="$gate_dir/make-document-sites.txt"
185
{ rg -n --no-heading "\(make-document " src test tools book || true; } >"$sites"
186
site_count=$(wc -l <"$sites")
−146
test "$site_count" -ge 1 || die "arm 02: found no make-document call sites at all; the sweep examined nothing"
−147
book_sites=$( { rg -c "\(make-document '?book" "$sites" || true; } )
−148
test "${book_sites:-0}" -eq 0 \
−149
|| { rg -n "\(make-document '?book" "$sites" >&2; die "arm 02: a call site still constructs the dropped kind"; }
−150
pass 2 "kind book refused by name with 2 positive controls; $site_count make-document sites swept, 0 construct it"
+187
# A FLOOR, labelled: this is not a restatement of the current count, it is the
+188
# level below which the sweep has plainly lost its population.
+189
test "$site_count" -ge 20 \
+190
|| die "arm 02: the sweep found only $site_count make-document sites; it is not examining the tree"
+191
+192
book_ledger="$gate_dir/book-kind-sites.txt"
+193
: >"$book_ledger"
+194
violations=0; negative_tests=0; unclassified=0
+195
while IFS= read -r hit; do
+196
test -n "$hit" || continue
+197
path="${hit%%:*}"
+198
rest="${hit#*:}"
+199
lineno="${rest%%:*}"
+200
case "$path" in
+201
src/*|tools/*|book/*)
+202
violations=$((violations + 1))
+203
printf 'VIOLATION %s:%s (product code must not construct the dropped kind)\n' \
+204
"$path" "$lineno" >>"$book_ledger" ;;
+205
test/*)
+206
# The site's own line plus one either side: a refusal test wraps the
+207
# construction in a form that CAPTURES the raise.
+208
window=$(sed -n "$((lineno > 1 ? lineno - 1 : 1)),$((lineno + 1))p" "$path")
+209
if printf '%s' "$window" | rg -q 'assert-error|message \(lambda'; then
+210
negative_tests=$((negative_tests + 1))
+211
printf 'NEGATIVE-TEST %s:%s (constructs it to prove it is refused)\n' \
+212
"$path" "$lineno" >>"$book_ledger"
+213
else
+214
unclassified=$((unclassified + 1))
+215
printf 'UNCLASSIFIED %s:%s (names the dropped kind and is not guarded by a refusal assertion)\n' \
+216
"$path" "$lineno" >>"$book_ledger"
+217
fi ;;
+218
*)
+219
unclassified=$((unclassified + 1))
+220
printf 'UNCLASSIFIED %s:%s (outside every classified directory)\n' \
+221
"$path" "$lineno" >>"$book_ledger" ;;
+222
esac
+223
done < <( { rg -n --no-heading "\(make-document '?book" src test tools book || true; } )
+224
cat "$book_ledger"
+225
book_rows=$(wc -l <"$book_ledger")
+226
book_hits=$( { rg -c "\(make-document '?book" "$sites" || true; } )
+227
test "$book_rows" -eq "${book_hits:-0}" \
+228
|| die "arm 02: found ${book_hits:-0} sites naming the dropped kind but classified $book_rows of them"
+229
test "$violations" -eq 0 \
+230
|| { rg '^VIOLATION' "$book_ledger" >&2; die "arm 02: $violations product call sites still construct the dropped kind"; }
+231
test "$unclassified" -eq 0 \
+232
|| { rg '^UNCLASSIFIED' "$book_ledger" >&2; die "arm 02: $unclassified sites name the dropped kind and could not be classified"; }
+233
test "$negative_tests" -ge 1 \
+234
|| die "arm 02: no test constructs the dropped kind to prove it is refused, so this sweep's absence claim has no positive control"
+235
pass 2 "kind book refused by name with 2 positive controls; $site_count make-document sites swept, ${book_hits:-0} name the dropped kind and all $negative_tests are refusal tests, 0 in product code"
236
237
# ---------------------------------------------------------------- ARM 03
238
# (ARM 05 below carries the differential; arms 2 and 3 are the cheap ones and
@@ -160,22 +245,39 @@ pass 2 "kind book refused by name with 2 positive controls; $site_count make-doc
245
# this arm prints one row per manuscript so the output length is the second one.
246
census="$gate_dir/census.txt"
247
: >"$census"
−163
total=0; base_lang=0; book_lang=0; other_lang=0; negative=0; bad=0
+248
total=0; base_lang=0; book_lang=0; other_lang=0; negative=0; bad=0; unknown_lang=0
249
while IFS= read -r m; do
250
total=$((total + 1))
251
lang=$(head -1 "$m" | sed -n 's|^#lang \(.*\)$|\1|p')
252
root=$( { rg -o -m1 '^@[a-z][a-z-]*' "$m" || echo "@NONE"; } | head -1)
+253
# THREE CLASSES, AND `UNCHECKED` HAS TO BE REACHABLE OR THE PASS LINE LIES.
+254
#
+255
# The first version defaulted `verdict=UNCHECKED` and then overwrote it on
+256
# EVERY branch -- the inner catch-all made everything either OK or ILLEGAL --
+257
# so "0 unclassified" was vacuously true. A guard against a state the code
+258
# makes unreachable is not a denominator, it is decoration.
+259
#
+260
# The class that genuinely exists is: a language THIS GATE DOES NOT KNOW.
+261
# That is different from a manuscript being wrong, and conflating the two is
+262
# the actual risk here -- a `#lang folio/doc` fixture added later would be
+263
# reported ILLEGAL, which is false, or would need this arm silently widened.
+264
# So an unknown language is UNCHECKED and FAILS, which forces whoever adds a
+265
# language to say what its legal roots are.
266
verdict=UNCHECKED
267
case "$m" in
268
*/sd-p8-base-*|*/sd-p8-book-article-root.sgd|*/sd-p8-book-two-chapters.sgd)
269
verdict=NEGATIVE-FIXTURE; negative=$((negative + 1)) ;;
270
*)
−173
case "$lang::$root" in
−174
"sigil/doc::@article") verdict=OK; base_lang=$((base_lang + 1)) ;;
−175
"sigil/doc/book::@chapter") verdict=OK; book_lang=$((book_lang + 1)) ;;
−176
"sigil/doc::@note") verdict=OK-EXPLICIT-PROFILE; other_lang=$((other_lang + 1)) ;;
−177
"sd-p7/folio::@note") verdict=OK-OTHER-LANGUAGE; other_lang=$((other_lang + 1)) ;;
−178
*) verdict=ILLEGAL; bad=$((bad + 1)) ;;
+271
case "$lang" in
+272
"sigil/doc"|"sigil/doc/book"|"sd-p7/folio")
+273
case "$lang::$root" in
+274
"sigil/doc::@article") verdict=OK; base_lang=$((base_lang + 1)) ;;
+275
"sigil/doc/book::@chapter") verdict=OK; book_lang=$((book_lang + 1)) ;;
+276
"sigil/doc::@note") verdict=OK-EXPLICIT-PROFILE; other_lang=$((other_lang + 1)) ;;
+277
"sd-p7/folio::@note") verdict=OK-OTHER-LANGUAGE; other_lang=$((other_lang + 1)) ;;
+278
*) verdict=ILLEGAL; bad=$((bad + 1)) ;;
+279
esac ;;
+280
*) verdict=UNCHECKED; unknown_lang=$((unknown_lang + 1)) ;;
281
esac ;;
282
esac
283
printf '%-22s %-16s %-9s %s\n' "$verdict" "$lang" "$root" "$m" >>"$census"
@@ -185,16 +287,18 @@ rows=$(wc -l <"$census")
287
test "$rows" -eq "$total" || die "arm 03: examined $total manuscripts but printed $rows rows"
288
test "$total" -ge 27 || die "arm 03: the census found only $total manuscripts; the population is wrong"
289
test "$bad" -eq 0 || { rg '^ILLEGAL' "$census" >&2; die "arm 03: $bad manuscripts declare a language that does not admit their root"; }
−188
# The `not checked` class, asserted explicitly rather than inferred from the
−189
# absence of complaint. A `test A -o B` compound with four-plus arguments is
−190
# ambiguous in POSIX and easy to get subtly wrong, so this counts once and
−191
# compares once.
+290
# The `unknown language` class, counted from the rows and cross-checked against
+291
# the loop's own counter -- two independently produced numbers, because a loop
+292
# cannot audit itself.
293
unchecked=$( { rg -c '^UNCHECKED' "$census" || true; } )
+294
test "${unchecked:-0}" -eq "$unknown_lang" \
+295
|| die "arm 03: $unknown_lang unknown-language manuscripts counted but ${unchecked:-0} rows printed"
296
test "${unchecked:-0}" -eq 0 \
−194
|| { rg '^UNCHECKED' "$census" >&2; die "arm 03: ${unchecked} manuscripts were not classified at all"; }
+297
|| { rg '^UNCHECKED' "$census" >&2
+298
die "arm 03: ${unchecked} manuscripts declare a language this arm does not know, so it cannot say whether their root is legal; widen the arm deliberately or fix the manuscript"; }
299
test "$base_lang" -ge 1 || die "arm 03: no manuscript exercises the base language"
300
test "$book_lang" -ge 15 || die "arm 03: only $book_lang manuscripts moved to the genre language"
−197
pass 3 "census $total manuscripts, 0 illegal, 0 unclassified :: $base_lang base, $book_lang book, $other_lang other-language, $negative negative fixtures"
+301
pass 3 "census $total manuscripts, 0 illegal, 0 of an unknown language :: $base_lang base, $book_lang book, $other_lang other-language, $negative negative fixtures"
302
303
# ---------------------------------------------------------------- ARM 04
304
# END TO END through the real `#lang` path: the two REQUIRED GREENS and the four
@@ -303,7 +407,7 @@ pass 4 "$printed of $declared #lang cases matched: 4 required reds named by lang
407
#
408
# The claim: across the whole manuscript corpus, the ONLY manuscripts whose
409
# lowering changed are the ones whose ROOT COMMAND this phase changed, and for
−306
# each of those the change is EXACTLY the root symbol. The nineteen manuscripts
+410
# each of those the change is EXACTLY the root symbol. The 17 manuscripts
411
# that took a header edit alone lower identically.
412
#
413
# What makes that assertable is `test/sd-p8-lowering-dump.sgl`, which emits every
@@ -415,13 +519,24 @@ for f in "$gate_dir/sec-after"/*; do
519
# section byte for byte -- a far stronger statement than "it changed, as
520
# predicted", and the one that would catch a migration that also moved content.
521
old="${root_b#@}"; new="${root_a#@}"
−418
# THE SUBSTITUTION MUST BE UNAMBIGUOUS BEFORE IT IS TRUSTED. A rewrite that
−419
# could match something other than the root symbol would MASK a real
−420
# difference, which is the one direction a differential must never fail in.
−421
# Two conditions make it unambiguous: after the change the old spelling must
−422
# be gone entirely, and before it the new spelling must not have existed. If
−423
# either fails, the manuscript is reported UNEXPECTED rather than rewritten --
−424
# a refusal to judge, not a pass.
+522
# THE SUBSTITUTION MUST BE UNAMBIGUOUS BEFORE IT IS TRUSTED, and it must be
+523
# BOUNDED. A rewrite that could match something other than the root symbol
+524
# would MASK a real difference, which is the one direction a differential must
+525
# never fail in. Three conditions:
+526
#
+527
# * after the change the old spelling must be gone entirely, and before it
+528
# the new spelling must not have existed -- so the mapping is a bijection
+529
# on this pair of sections;
+530
# * the rewrite must change SOMETHING, or the comparison below is between
+531
# the two sections unmodified and a genuine root-only change would read as
+532
# UNEXPECTED for the wrong reason;
+533
# * and it must change AT MOST as many occurrences as a root can account
+534
# for. A `/g` substitution over the whole section would also rewrite a
+535
# NESTED generated `chapter` head, so a manuscript that changed its root
+536
# AND some other command of the same name would report "nothing else
+537
# moved". The root appears twice in a lowered manuscript -- once in the
+538
# generated command head and once in the `author-flatten` command-name
+539
# argument -- so more than two is a second construct, not the root.
540
if [ "$( { rg -cF "]] $new>" "$b" || true; } )" != "" ] \
541
|| [ "$( { rg -cF "]] $old>" "$f" || true; } )" != "" ]; then
542
unexpected=$((unexpected + 1))
@@ -429,11 +544,14 @@ for f in "$gate_dir/sec-after"/*; do
544
"$name" "$root_b" "$root_a" >>"$ledger"
545
continue
546
fi
+547
occurrences=$( { rg -oF "]] $old>" "$b" || true; } | wc -l)
+548
if [ "$occurrences" -lt 1 ] || [ "$occurrences" -gt 2 ]; then
+549
unexpected=$((unexpected + 1))
+550
printf 'UNEXPECTED %-40s root=%s -> %s, but %s generated `%s` heads is not a root alone; refusing to rewrite\n' \
+551
"$name" "$root_b" "$root_a" "$occurrences" "$old" >>"$ledger"
+552
continue
+553
fi
554
sed "s/\]\] $old>/]] $new>/g" "$b" >"$gate_dir/rewritten-$name"
−433
# ...and the rewrite must actually have done something. A sed that matched
−434
# nothing would compare the base section to the after section unchanged, and
−435
# a manuscript whose only difference WAS the root would then read as
−436
# UNEXPECTED for the wrong reason.
555
cmp -s "$b" "$gate_dir/rewritten-$name" \
556
&& die "arm 05: the root rewrite for $name changed nothing, so its root symbol is not where this arm thinks it is"
557
if cmp -s "$f" "$gate_dir/rewritten-$name"; then
@@ -462,6 +580,75 @@ test "$compared" -eq "$after_compared" \
580
|| die "arm 05: the dump lowered $after_compared manuscripts but the ledger holds $compared"
581
echo " denominator: after found $after_found, lowered $after_compared; before lowered $before_compared; ledger rows $((compared + only_before))"
582
+583
# `only_after` and `skipped` ARE ASSERTED, and the first version of this arm did
+584
# not assert either. A count nobody asserts on is a count nobody is checking:
+585
# `test/fixtures/sd-p1-article-root.sgd` had its whole body rewritten by this
+586
# phase (+10 -9) and passed through this ledger as `AFTER-ONLY ... the base tree
+587
# could not lower it`, which is TRUE and says nothing about its content. The
+588
# after-only class is legitimate but it is not free: each member must be named
+589
# in advance, so a manuscript joining it is a decision rather than an accident.
+590
#
+591
# Both expected members are here for a REASON, not merely listed:
+592
# sd-p1-article-root.sgd root `@article`; the base profile at the base SHA
+593
# was `(chapter)` and refused it. D8 is exactly the
+594
# change that makes it lowerable, so its appearance
+595
# here is the phase's headline effect, not a gap.
+596
# sd-p8-book-chapter-root.sgd added by this phase; no base-side counterpart
+597
# can exist.
+598
expected_after_only="$gate_dir/expected-after-only"
+599
printf 'sd-p1-article-root.sgd\nsd-p8-book-chapter-root.sgd\n' | sort >"$expected_after_only"
+600
actual_after_only="$gate_dir/actual-after-only"
+601
{ rg '^AFTER-ONLY' "$ledger" || true; } | awk '{print $2}' | sort >"$actual_after_only"
+602
diff -u "$expected_after_only" "$actual_after_only" >"$gate_dir/after-only.diff" \
+603
|| { cat "$gate_dir/after-only.diff" >&2
+604
die "arm 05: the set of manuscripts only the current tree can lower is not the set this phase intends"; }
+605
test "$only_after" -eq "$(wc -l <"$expected_after_only")" \
+606
|| die "arm 05: $only_after after-only rows for $(wc -l <"$expected_after_only") expected names"
+607
# A CONTENT REWRITE OF AN AFTER-ONLY MANUSCRIPT IS STILL A CHANGE, and the
+608
# differential structurally cannot see it (there is no before-side section to
+609
# compare). Saying so is the honest disposition; hiding it behind a passing
+610
# ledger row is not. The suite is what covers these: `test/test-root-profile.sgl`
+611
# evaluates `sd-p1-article-root.sgd` through the shipped base profile and asserts
+612
# its kind and title, and `test/sd-p8-language-probe.sgl` loads
+613
# `sd-p8-book-chapter-root.sgd` through the real `#lang` path.
+614
echo " after-only: $only_after manuscripts, named in advance; their CONTENT is not covered by this differential and is covered by the suite and the language probe instead"
+615
# THE SKIPPED SETS, COMPARED BY NAME. The first version of this check compared
+616
# COUNTS and required `after <= before`, which was simply wrong: this phase adds
+617
# four manuscripts that are DELIBERATELY unlowerable (the negative fixtures the
+618
# language probe drives), so the after side legitimately skips more. The gate
+619
# caught it loudly on its first run with "something stopped being lowerable",
+620
# which was false -- a count cannot tell "a manuscript regressed" from "a
+621
# manuscript was added to be refused". Only the names can.
+622
#
+623
# Two directions, both asserted:
+624
# * a manuscript the base tree could LOWER must not have become skippable --
+625
# that is the regression this check exists for;
+626
# * a manuscript newly skipped must be one this phase ADDED, which is derived
+627
# from git rather than listed here.
+628
skipped_names() { rg -o '^--- SKIPPED \S+' "$1" | awk '{print $3}' | sort; }
+629
skipped_names "$after_dump" >"$gate_dir/skipped-after"
+630
skipped_names "$before_dump" >"$gate_dir/skipped-before"
+631
comm -13 "$gate_dir/skipped-before" "$gate_dir/skipped-after" >"$gate_dir/newly-skipped"
+632
comm -23 "$gate_dir/skipped-before" "$gate_dir/skipped-after" >"$gate_dir/no-longer-skipped"
+633
git diff --name-only --diff-filter=A "$base_sha" HEAD -- '*.sgd' \
+634
| sed 's|.*/||' | sort >"$gate_dir/added-manuscripts"
+635
while read -r n; do
+636
test -n "$n" || continue
+637
grep -Fxq -- "$n" "$gate_dir/added-manuscripts" \
+638
|| { echo "newly skipped: $n" >&2
+639
die "arm 05: $n is skipped by this tree and was not skipped by the base tree, and this phase did not add it -- it stopped being lowerable"; }
+640
done <"$gate_dir/newly-skipped"
+641
# The other direction is a GAIN, not a fault, but it must still be named: a
+642
# manuscript that stopped being skipped is one D8 made lowerable, and it has to
+643
# be in the after-only set the arm already pins.
+644
while read -r n; do
+645
test -n "$n" || continue
+646
grep -Fxq -- "$n" "$expected_after_only" \
+647
|| { echo "no longer skipped: $n" >&2
+648
die "arm 05: $n became lowerable and is not in this arm's expected after-only set"; }
+649
done <"$gate_dir/no-longer-skipped"
+650
echo " skipped: after $(wc -l <"$gate_dir/skipped-after"), before $(wc -l <"$gate_dir/skipped-before"); $(wc -l <"$gate_dir/newly-skipped") newly skipped (all added by this phase), $(wc -l <"$gate_dir/no-longer-skipped") newly lowerable"
+651
652
# A FLOOR, deliberately, not a restatement: adding a manuscript must not
653
# desynchronise this, and a corpus that collapsed must not read as a pass.
654
test "$identical" -ge 15 \
@@ -475,6 +662,126 @@ test "$only_before" -eq 0 \
662
pass 5 "differential vs $base_sha: $identical identical, $expected_change root-only, $only_after newly lowerable, 0 unexpected, over $compared lowered of $after_found found"
663
664
# ---------------------------------------------------------------- ARM 06
+665
# THE CONSTRUCTION DIFFERENTIAL, replacing `test/gate-sd-p7.sh` arm 02.
+666
#
+667
# Arm 5 above compares LOWERING -- emitted syntax. This compares the CONSTRUCTED
+668
# document, which is a different property and the one SD-P7 owns: a lowering
+669
# differential is byte-identical under a completely broken content profile. When
+670
# arm 02 of gate-sd-p7 was retired (its baseline is pinned to absolute byte
+671
# offsets that a `#lang` edit shifts), that property lost its standing check, and
+672
# retiring without replacing loses a property.
+673
#
+674
# THE ONE DESIGN DECISION, and it is the lesson of the retirement it replaces:
+675
# THIS ARM ASSERTS ON STRUCTURE, NOT ON POSITIONS. `document->string` embeds
+676
# every node's `(file pos line col span)`. Comparing those verbatim is precisely
+677
# what made the old baseline unmaintainable, so every location TUPLE belonging
+678
# to a manuscript is normalised to the marker `L` before comparison. What
+679
# remains is the document's shape and every non-positional field: kinds, titles,
+680
# authors, ids, text, the tree.
+681
#
+682
# That is deliberately weaker than the arm it replaces, and the weakening is
+683
# named rather than glossed: THIS ARM CANNOT SEE A PURE LOCATION REGRESSION.
+684
# Locations are covered instead by arm 5 (which asserts them header-relative,
+685
# so it sees any shift that is not the header's) and by
+686
# `test/test-root-profile.sgl`, whose location assertions recompute every
+687
# expected byte offset, line and column independently.
+688
before_c="$gate_dir/construction-before.txt"
+689
after_c="$gate_dir/construction-after.txt"
+690
setsid -w bash test/sd-p7-construction-dump.sh "$after_c" >"$gate_dir/construction-after.log" 2>&1 \
+691
|| { tail -20 "$gate_dir/construction-after.log" >&2; die "arm 06: the construction dump failed on this tree"; }
+692
( cd "$base_tree" \
+693
&& cp "$repo_dir/test/sd-p7-construction-dump.sh" test/ \
+694
&& SIGIL_D1_REDIRECTS="$gate_dir/base-redirects.sgl" \
+695
SIGIL_D1_COMPILER="$sigil" \
+696
setsid -w bash test/sd-p7-construction-dump.sh "$before_c" ) \
+697
>"$gate_dir/construction-before.log" 2>&1 \
+698
|| { tail -20 "$gate_dir/construction-before.log" >&2; die "arm 06: the construction dump failed on the base tree"; }
+699
for side in "$before_c" "$after_c"; do
+700
rg -q 'SD_P7_CONSTRUCTION_DUMP_COMPLETE' "$side" \
+701
|| { tail -10 "$side" >&2; die "arm 06: a construction dump printed no completion marker"; }
+702
done
+703
+704
# The location normaliser, and the marker line excluded for the reason SD-P3's
+705
# gate learned the hard way: it names the corpus size, so leaving it in ties the
+706
# LAST manuscript's record to how many manuscripts were dumped.
+707
split_construction() { # $1 = dump, $2 = out dir
+708
mkdir -p "$2"
+709
grep -v '^SD_P7_CONSTRUCTION_DUMP_COMPLETE' "$1" \
+710
| sed -E 's/("[^"]*\.sgd") [0-9]+ [0-9]+ [0-9]+ [0-9]+/\1 L/g' \
+711
| awk -v o="$2" '/^=== /{n=$2; gsub("/","_",n)} n!=""{print > (o "/" n)}'
+712
}
+713
split_construction "$before_c" "$gate_dir/con-before"
+714
split_construction "$after_c" "$gate_dir/con-after"
+715
+716
# THE NORMALISER MUST BE SHOWN TO HAVE FIRED. A `sed` that matched nothing would
+717
# make this a verbatim comparison wearing a normaliser's name -- and it would
+718
# then fail for exactly the reason the retired arm did, with a message about
+719
# structure.
+720
#
+721
# CHECK THE NORMALISED OUTPUT, NOT THE RAW DUMP. The first version of this guard
+722
# grepped `$after_c`, which is the dump BEFORE `split_construction` runs sed over
+723
# it, so the marker could never appear there and the guard reddened on a healthy
+724
# tree. Caught by the guard itself on its first run, which is the only reason a
+725
# check aimed at one file rather than another is visible at all.
+726
normalised=$( { rg -c ' L\)' "$gate_dir"/con-after/* || true; } | awk -F: '{n+=$NF} END{print n+0}')
+727
test "${normalised:-0}" -ge 1 \
+728
|| die "arm 06: the location normaliser matched nothing in the split records; it is not normalising, so this comparison is not the one described"
+729
+730
con_ledger="$gate_dir/construction-ledger.txt"
+731
: >"$con_ledger"
+732
con_same=0; con_diff=0; con_after_only=0; con_before_only=0
+733
for f in "$gate_dir/con-before"/*; do
+734
name="${f##*/}"
+735
a="$gate_dir/con-after/$name"
+736
if [ ! -f "$a" ]; then
+737
con_before_only=$((con_before_only + 1))
+738
printf 'BEFORE-ONLY %s (this tree no longer constructs it)\n' "$name" >>"$con_ledger"
+739
elif cmp -s "$f" "$a"; then
+740
con_same=$((con_same + 1))
+741
printf 'IDENTICAL %s\n' "$name" >>"$con_ledger"
+742
else
+743
con_diff=$((con_diff + 1))
+744
printf 'DIFFERS %s\n' "$name" >>"$con_ledger"
+745
fi
+746
done
+747
for f in "$gate_dir/con-after"/*; do
+748
name="${f##*/}"
+749
test -f "$gate_dir/con-before/$name" || {
+750
con_after_only=$((con_after_only + 1))
+751
printf 'AFTER-ONLY %s (added by this phase)\n' "$name" >>"$con_ledger"; }
+752
done
+753
cat "$con_ledger"
+754
+755
# THE EXPECTED DIFFERENCES, NAMED. Anything else is a finding. All three are D8
+756
# itself: two manuscripts whose root command changed, so their document KIND
+757
# changed with it, and one that the base tree could not construct at all because
+758
# its `@article` root was illegal there -- which is the whole point of the phase.
+759
expected_con="$gate_dir/expected-construction-diffs"
+760
printf 'test_fixtures_sd-p1-article-root.sgd\ntest_fixtures_sd-p3-bare-prose.sgd\ntest_fixtures_sd-p3-wrapped.sgd\n' | sort >"$expected_con"
+761
{ rg '^DIFFERS' "$con_ledger" || true; } | awk '{print $2}' | sort >"$gate_dir/actual-construction-diffs"
+762
diff -u "$expected_con" "$gate_dir/actual-construction-diffs" >"$gate_dir/construction.diff" \
+763
|| { cat "$gate_dir/construction.diff" >&2
+764
die "arm 06: the constructed documents that changed are not the ones this phase intends"; }
+765
# And the two kind flips are asserted BY NAME, so "it differs" cannot stand in
+766
# for "it differs in the way D8 predicts".
+767
for m in sd-p3-bare-prose sd-p3-wrapped; do
+768
rg -q '\(document \(kind chapter\)' "$gate_dir/con-before/test_fixtures_$m.sgd" \
+769
|| die "arm 06: $m was not a chapter at the base, so the kind-flip claim is not about what changed"
+770
rg -q '\(document \(kind article\)' "$gate_dir/con-after/test_fixtures_$m.sgd" \
+771
|| die "arm 06: $m is not an article now"
+772
done
+773
rg -q 'RAISED' "$gate_dir/con-before/test_fixtures_sd-p1-article-root.sgd" \
+774
|| die "arm 06: the base tree DID construct sd-p1-article-root.sgd, so D8 is not what made it constructible"
+775
rg -q '\(document \(kind article\)' "$gate_dir/con-after/test_fixtures_sd-p1-article-root.sgd" \
+776
|| die "arm 06: sd-p1-article-root.sgd still does not construct"
+777
test "$con_before_only" -eq 0 \
+778
|| { rg '^BEFORE-ONLY' "$con_ledger" >&2; die "arm 06: this tree stopped constructing a manuscript the base tree built"; }
+779
# A FLOOR, labelled: not a restatement of the current count.
+780
test "$con_same" -ge 15 \
+781
|| die "arm 06: only $con_same constructed documents are unchanged; the migration moved more than D8 intends"
+782
pass 6 "construction differential vs $base_sha: $con_same identical, $con_diff changed and all named, $con_after_only added, 0 lost; locations normalised (see comment: this arm cannot see a pure location regression, arm 5 and the suite cover that)"
+783
+784
# ---------------------------------------------------------------- ARM 07
785
# REGRESSION. The full suite with the FULL load path -- four entries, because a
786
# short path degrades into "failed to load", which the runner reports WITHOUT
787
# failing the process. The file count is its own assertion for that reason.
@@ -487,8 +794,8 @@ pass 5 "differential vs $base_sha: $identical identical, $expected_change root-o
794
# the gate with this UNSET and requires the GREEN marker, so no leg can be
795
# satisfied by a run that skipped anything.
796
if [ -n "${SD_P8_SKIP_REGRESSION:-}" ]; then
−490
printf 'SD_P8_GATE_06 SKIPPED (SD_P8_SKIP_REGRESSION is set)\n'
−491
echo "SD_P8_GATE_PARTIAL arms 1-5 only; the regression arm did not run"
+797
printf 'SD_P8_GATE_07 SKIPPED (SD_P8_SKIP_REGRESSION is set)\n'
+798
echo "SD_P8_GATE_PARTIAL arms 1-6 only; the regression arm did not run"
799
exit 0
800
fi
801
regression_log="$gate_dir/regression.log"
@@ -497,31 +804,31 @@ setsid -w "$sigil" -L "$sxml_root/src" -L src -L test -L book test \
804
>"$regression_log" 2>&1 || regression_status=$?
805
sed -i 's/\x1b\[[0-9;]*m//g' "$regression_log"
806
grep -qE 'Tests: +[0-9]+ passed' "$regression_log" \
−500
|| { tail -40 "$regression_log" >&2; die "arm 06: could not read a suite summary -- the run may not have executed anything"; }
+807
|| { tail -40 "$regression_log" >&2; die "arm 07: could not read a suite summary -- the run may not have executed anything"; }
808
files_line=$(grep -oE 'Files: +[0-9]+ of [0-9]+ run' "$regression_log" | tail -1 || true)
−502
test -n "$files_line" || { tail -40 "$regression_log" >&2; die "arm 06: the runner reported no file count"; }
+809
test -n "$files_line" || { tail -40 "$regression_log" >&2; die "arm 07: the runner reported no file count"; }
810
files_run=$(echo "$files_line" | grep -oE '[0-9]+' | head -1)
811
files_found=$(echo "$files_line" | grep -oE '[0-9]+' | tail -1)
−505
test "$files_run" = "$files_found" || die "arm 06: only $files_run of $files_found test files ran"
+812
test "$files_run" = "$files_found" || die "arm 07: only $files_run of $files_found test files ran"
813
if grep -qE '[0-9]+ failed' "$regression_log"; then
814
grep -nE '✗|failed' "$regression_log" | head -30 >&2
−508
die "arm 06: a test failed"
+815
die "arm 07: a test failed"
816
fi
817
test "$regression_status" = "0" \
−511
|| { tail -40 "$regression_log" >&2; die "arm 06: the runner exited $regression_status with no failing test reported"; }
+818
|| { tail -40 "$regression_log" >&2; die "arm 07: the runner exited $regression_status with no failing test reported"; }

Showing the first 500 of 521 diff lines for this file. This diff is INCOMPLETE; read the file or clone the repository for the rest.

test/sabotage-sd-p8.shmodified
@@ -272,6 +272,26 @@ plant "$DUMP" \
272
'(and #f (pair? value)'
273
run_gate "$work/leg06.log" && die "LEG 06: the gate passed with the position normalization disabled"
274
leg_red 06 "$work/leg06.log" "manuscripts changed for a reason this phase did not intend"
+275
# CORROBORATION, and it is not optional here. LEG 05 and LEG 06 both redden arm
+276
# 5 with the SAME sentence, so on that string alone the two legs cannot be told
+277
# apart by their own evidence -- either one would score the other's success.
+278
# They must be distinguished by WHICH manuscripts move:
+279
#
+280
# LEG 05 changes one manuscript's prose -> exactly minimal.sgd is UNEXPECTED
+281
# LEG 06 disables the normalization -> every HEADER-MIGRATED manuscript
+282
# is UNEXPECTED, because their
+283
# embedded make-srcloc positions are
+284
# absolute again
+285
#
+286
# So this leg requires a WALL, not a row: at least ten unexpected manuscripts,
+287
# and minimal.sgd among them for a reason opposite to leg 5's.
+288
unexpected_rows=$( { rg -c '^UNEXPECTED' "$work/leg06.log" || echo 0; } )
+289
test "${unexpected_rows:-0}" -ge 10 \
+290
|| { rg '^UNEXPECTED|^IDENTICAL' "$work/leg06.log" >&2
+291
die "LEG 06: disabling the normalization produced only ${unexpected_rows:-0} unexpected manuscripts; a header-shift leak should move every migrated one, so this red is probably not the one this leg is about"; }
+292
rg -qF 'UNEXPECTED minimal.sgd' "$work/leg06.log" \
+293
|| die "LEG 06: the wall of unexpected manuscripts does not include minimal.sgd"
+294
printf 'SD_P8_SABOTAGE_06 CORROBORATED %s manuscripts unexpected, not the single row leg 5 produces\n' "$unexpected_rows"
295
restore_all || die "restore failed; the worktree is still sabotaged"
296
297
# ---------------------------------------------------------------- LEG 07
@@ -346,10 +366,10 @@ rg -q '^SD_P8_GATE_GREEN$' "$work/leg09.log" \
366
|| { tail -20 "$work/leg09.log" >&2; die "LEG 09: the gate exited 0 without printing its green marker"; }
367
rg -q 'SD_P8_GATE_PARTIAL' "$work/leg09.log" \
368
&& die "LEG 09: the control run skipped an arm; it does not establish a full green"
−349
for n in 01 02 03 04 05 06; do
+369
for n in 01 02 03 04 05 06 07; do
370
rg -q "^SD_P8_GATE_$n " "$work/leg09.log" \
371
|| die "LEG 09: arm $n printed no marker in the control run"
372
done
−353
printf 'SD_P8_SABOTAGE_09 GREEN the restored tree passes all six arms\n'
+373
printf 'SD_P8_SABOTAGE_09 GREEN the restored tree passes all seven arms\n'
374
375
echo "SD_P8_SABOTAGE_COMPLETE 8 reds (6 product, 1 instrument, 1 coverage) and 1 positive control"
test/sd-p8-lowering-dump.sglmodified
@@ -5,7 +5,7 @@
5
;;;
6
;;; SD-P3's dump is the AFTER half of a differential whose BEFORE half is a
7
;;; committed table of sha256s taken from a pre-SD-P3 tree. D8 migrates the
−8
;;; `#lang` line of nineteen manuscripts, which moves every byte offset in each
+8
;;; `#lang` line of 17 manuscripts, which moves every byte offset in each
9
;;; of them by the header's length delta, so every one of those hashes changes.
10
;;; The BEFORE half stores hashes and not sections, so no normalization can be
11
;;; applied to it retroactively -- the differential cannot be rescued, only
@@ -85,12 +85,21 @@
85
(define (syntax-datum-of value)
86
(if (syntax? value) (syntax-datum value) value))
87
+88
;; THE GUARD IS THE MANUSCRIPT'S OWN NAME, not merely "not the generated
+89
;; marker". An earlier version accepted any six-element `make-srcloc` call whose
+90
;; second element was any string, while its comment claimed a check the code did
+91
;; not make: an author writing `(make-srcloc "anything.sgd" 1 1 100 5)` inside a
+92
;; top-level `@(...)` would have had element 4 silently rewritten, hiding a real
+93
;; change to that literal. Nothing in the corpus does that today -- which is
+94
;; exactly the condition under which an over-broad predicate goes unnoticed.
+95
(define current-manuscript "")
+96
97
(define (generated-srcloc-call? value)
98
(and (pair? value)
99
(eq? (syntax-datum-of (car value)) 'make-srcloc)
100
(= (length value) 6)
101
(let ((file (syntax-datum-of (cadr value))))
−93
(and (string? file) (not (string=? file "<sigil/doc-generated>"))))))
+102
(and (string? file) (string=? file current-manuscript)))))
103
104
(define expression-binding-prefix "#%sigil/doc-expression:")
105
(define (expression-binding? name)
@@ -142,13 +151,25 @@
151
;; same reason as the other two, and it is worth knowing on its own: a
152
;; manuscript's generated binding names are NOT stable under an edit to its
153
;; `#lang` line.
+154
;;
+155
;; A SUFFIX THAT IS NOT A NUMBER IS REPORTED, NOT SUBTRACTED. `string->number`
+156
;; returns `#f` on a name this code did not expect, and `(- #f header-end)`
+157
;; would raise from inside the emitter -- killing the dump and surfacing in
+158
;; the gate as "the dump printed no completion marker", a red about a missing
+159
;; marker for what is actually an unrecognised binding name. Emitting the
+160
;; symbol verbatim instead makes it a DIFFERENCE the differential reports
+161
;; against the other side, which is the correct place for a surprise.
162
((and (symbol? value) (expression-binding? (symbol->string value)))
−146
(display "#%sigil/doc-expression:rel:")
−147
(display (- (string->number
−148
(substring (symbol->string value)
−149
(string-length expression-binding-prefix)
−150
(string-length (symbol->string value))))
−151
header-end)))
+163
(let* ((name (symbol->string value))
+164
(suffix (substring name
+165
(string-length expression-binding-prefix)
+166
(string-length name)))
+167
(offset (string->number suffix)))
+168
(if (and offset (integer? offset))
+169
(begin (display "#%sigil/doc-expression:rel:")
+170
(display (- offset header-end)))
+171
(begin (display "#%sigil/doc-expression:UNRECOGNISED-SUFFIX:")
+172
(display suffix)))))
173
(else (write value))))
174
175
;; ----------------------------------------------------------------- profiles
@@ -245,6 +266,7 @@
266
(id (language-header-id header))
267
(profile (profile-for id)))
268
(set! header-end (language-header-end header))
+269
(set! current-manuscript name)
270
;; The MANIFEST line carries everything that legitimately differs between
271
;; the two trees -- the language id, the header length, the root command --
272
;; and is deliberately OUTSIDE the compared section. Putting it inside would
test/test-passive-parse.sglmodified
@@ -266,10 +266,13 @@
266
(tree-node-text child))
267
(assert-true (not (string=? slice (tree-node-text child)))))))
268
−269
(test "the passive tree does not enforce the single-chapter root policy"
−270
;; "exactly one @chapter" is a LOWERING rule, not a fact about the grammar.
−271
;; folio roots are @note/@project and must be inspectable before SD-P1
−272
;; parameterizes the root container.
+269
(test "the passive tree does not enforce the single-root policy"
+270
;; A root policy is a LOWERING rule, not a fact about the grammar. It was
+271
;; "exactly one @chapter" when this arm was written and is "exactly one
+272
;; @article" since ruling D8 (SD-P8) -- and the point is that the passive path
+273
;; is indifferent to BOTH. The manuscript below declares `#lang sigil/doc` and
+274
;; two @note roots, which no sigil/doc profile has ever admitted, and the
+275
;; passive tree reports it regardless.
276
(let* ((source "#lang sigil/doc\n@note[\"topics/x\"]{@p{body}}\n@note[\"topics/y\"]{@p{body}}\n")
277
(tree (inspect-document-source source "roots.sgd" 16))
278
(commands (let loop ((xs (document-tree-roots tree)) (out '()))