Commit46e1fd80Recorded17 Nov 2026Repositorysigil-doc
Let examples carry hidden checks
Message
An example can now name a check, inline with check: or in a file with check-file:. The check runs after the example and must return true, but only the example's own source is shown. This lets a manual show just the code a reader would write while still proving that it works.
An example with a check shows its source only: make-example refuses 'source+expected with a check, and example defaults to 'source when one is given. A failed check names the check and its file in the build error. Debug and datum outputs include the check by design.
Release 0.2.1.
Changed
README.md | 3 +++
docs/examples.md | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
package.sgl | 2 +-
src/sigil/doc.sgl | 2 +-
src/sigil/doc/author.sgl | 28 +++++++++++++++++++++----
src/sigil/doc/model.sgl | 28 +++++++++++++++++++++----
src/sigil/doc/traversal.sgl | 19 +++++++++++------
src/sigil/doc/verify.sgl | 52 ++++++++++++++++++++++++++++++++++++----------
test/fixtures/example-check.sgd | 19 +++++++++++++++++
test/fixtures/example-check/failing.check.sgl | 3 +++
test/fixtures/example-check/settings.check.sgl | 3 +++
test/fixtures/example-check/settings.sgl | 4 ++++
test/gate-d4a.sh | 7 +++++++
test/test-example-check.sgl | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14 files changed, 391 insertions(+), 27 deletions(-)Diff
README.mdmodified
@@ -44,6 +44,9 @@ language-labelled block, and `@code[#f]{...}` is an unlabelled block. Named and
44
colon `@code` bodies are also blocks. Subject source inside these forms is data 45
and is not executed during document authoring. 46
+47
Executable examples, their expectations and hidden checks are described in+48
[`docs/examples.md`](docs/examples.md).+49
50
Evaluation binds the resulting semantic value as `document`. The package 51
includes backend-neutral model/traversal/publication modules, static HTML 52
fragments, and a backend-neutral print adapter with a Typst source backend.docs/examples.mdadded
@@ -0,0 +1,84 @@
+1
# Checked examples+2
+3
An `@example` is Sigil code that `sigil doc verify` (or `verify-document`)+4
runs, in a fresh process, and compares against one expectation. Renderers+5
show an example only with a verified result, so a published page never+6
shows code that didn't run.+7
+8
```text+9
@example["greeting" expect-stdout: "hello\n"]{+10
@code["sigil"]{(display "hello\n")}+11
}+12
+13
@example["config" source-file: "examples/config.sgl" expect-value: "42"]{}+14
```+15
+16
Keywords:+17
+18
| Keyword | Meaning |+19
| --- | --- |+20
| `source-file:` | Read the source from a file under the verification root instead of the body. |+21
| `session:` | Inline examples with the same session run cumulatively: each one runs after every earlier one in the session. File examples can't join a session. |+22
| `expect-stdout:`, `expect-value:`, `expect-error:`, `expect-timeout:` | Exactly one expectation. `expect-value:` compares the written form of the program's last value. |+23
| `expect-stderr:` | Expected standard error, `""` by default. |+24
| `timeout:` | Seconds, 1 to 60. 5 by default. |+25
| `display:` | `'source+expected`, `'source` or `'hidden`. The default is `'source+expected`, or `'source` for an example with a check. |+26
| `check:`, `check-file:` | A hidden check; see below. |+27
+28
## Hidden checks+29
+30
Sometimes the code a reader should see isn't the code that proves it works.+31
A configuration example should show only the configuration, while checking+32
it means calling functions the reader never would. `check:` and+33
`check-file:` keep that checking code out of the page.+34
+35
```text+36
@example["settings" expect-value: "#t" check: '(equal? (cdr (assq 'theme settings)) "dark")]{+37
@code["sigil"]{(define settings (list (cons 'theme "dark")))}+38
}+39
+40
@example["web-server" source-file: "examples/web.sgl"+41
check-file: "examples/web.check.sgl" expect-value: "(80 443)"]{}+42
```+43
+44
- **`check:`** takes a quoted Sigil expression, or its text as a string.+45
Use `begin` for several forms. A quoted expression is written back as+46
text with `write`, so it must hold only data that reads back the same:+47
keywords such as `name:` don't, so use a string or `check-file:` for+48
checks that pass keyword arguments. The reader doesn't allow `@` commands+49
inside `[...]`, so a code block can't be written there; code that builds+50
examples with `make-example` can pass any of these.+51
- **`check-file:`** takes a path under the verification root, resolved the+52
same way as `source-file:`. Use it for anything longer than one+53
expression. Give `check:` or `check-file:`, not both.+54
- **How it runs.** In one process, the verifier loads the shown source,+55
binds its last value as `example-result`, then loads the check in the+56
same environment. The check sees every definition and import the shown+57
source made, plus `example-result`, so it can check an example whose+58
last expression is a value with no name, such as a configuration. The+59
expectation applies to the whole run: `expect-value:` compares the+60
check's last value, and `expect-stdout:` compares the output of both+61
parts.+62
- **`example-result` is an ordinary binding.** If the shown source+63
defines `example-result` itself, the verifier's `set!` replaces that+64
value before the check runs, so don't use the name in shown code.+65
- **What readers see.** Only the shown source. An example with a check+66
can't use `display: 'source+expected`, because the expected value is+67
the check's, not the shown code's: `'source` is the default and+68
`'source+expected` is refused. The check isn't part of the+69
verified result's effective source, so the HTML, plain text and Typst+70
renderers, copy buttons and any tool that extracts example source never+71
see it. A session's later examples include earlier examples' shown+72
source, not their checks.+73
- **Caching and contracts.** The cache key and the result's specification+74
contract cover the check, so changing a check re-verifies the example.+75
An example without a check has the same document datum, cache key and+76
contract it had before checks existed. `document->datum`, and the debug+77
output built from it, include the check by design: they describe the+78
whole document, not what readers see.+79
+80
From Sigil code, pass the check to `make-example` as keys after the+81
location: `check-kind: 'inline check: "(expression)"`, or+82
`check-kind: 'file check: "path/to/check.sgl"`. `example-check-kind`+83
returns `#f`, `'inline` or `'file`, and `example-check` returns the text or+84
path.package.sglmodified
@@ -1,6 +1,6 @@
1
(package 2
name: "sigil-doc"−3
version: "0.2.0"+3
version: "0.2.1" 4
sigil: "^0.18.4" 5
description: "A programmable document language for Sigil" 6
license: "BSD-3-Clause"src/sigil/doc.sglmodified
@@ -17,7 +17,7 @@
17
admonition? admonition-kind admonition-title admonition-children admonition-location make-admonition 18
exercise? exercise-id exercise-title exercise-children exercise-location make-exercise 19
example? example-id example-source-kind example-source example-language example-session example-target−20
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location make-example+20
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location example-check-kind example-check make-example 21
project-check? project-check-id project-check-root project-check-files project-check-config project-check-target 22
project-check-expected-stdout project-check-timeout project-check-location make-project-check 23
anchor? anchor-id anchor-location make-anchor reference? reference-target reference-label reference-location make-referencesrc/sigil/doc/author.sglmodified
@@ -94,18 +94,38 @@
94
(define (example id 95
(keys: (source-file #f) (session #f) (target 'bytecode) 96
(expect-stdout #f) (expect-value #f) (expect-error #f) (expect-timeout #f)−97
(expect-stderr "") (timeout 5) (display 'source+expected))+97
(expect-stderr "") (timeout 5) (display #f)+98
(check #f) (check-file #f)) 99
(rest: body))−99
(let ((expectation (one-expectation expect-stdout expect-value expect-error expect-timeout)))+100
(let ((expectation (one-expectation expect-stdout expect-value expect-error expect-timeout))+101
(hidden (example-check check check-file)))+102
;; With a check, the result belongs to the check, so the default+103
;; shows the source alone.+104
(set! display (or display (if (car hidden) 'source 'source+expected))) 105
(if source-file 106
(if (null? body) 107
(make-example id 'file source-file #f session target (car expectation) (cadr expectation)−103
expect-stderr timeout display (author-location))+108
expect-stderr timeout display (author-location)+109
check-kind: (car hidden) check: (cadr hidden)) 110
(error "example: file source requires an empty body")) 111
(if (and (= (length body) 1) (code-block? (car body))) 112
(make-example id 'inline (code-block-content (car body)) (code-block-language (car body)) session target−107
(car expectation) (cadr expectation) expect-stderr timeout display (author-location))+113
(car expectation) (cadr expectation) expect-stderr timeout display (author-location)+114
check-kind: (car hidden) check: (cadr hidden)) 115
(error "example: inline source requires exactly one sigil code block")))))+116
;; check: is a quoted Sigil expression, its text, or a sigil code+117
;; block; check-file: is a path under the verification root. Neither+118
;; is ever rendered.+119
(define (example-check check check-file)+120
(cond ((and check check-file) (error "example: give check: or check-file:, not both"))+121
(check-file (list 'file check-file))+122
((not check) (list #f #f))+123
((string? check) (list 'inline check))+124
((code-block? check)+125
(if (member (code-block-language check) '(#f "sigil"))+126
(list 'inline (code-block-content check))+127
(error "example: check: code must be sigil")))+128
(else (list 'inline (format "~s" check))))) 129
(define (project-check id 130
(keys: root files (config 'dev) (target 'bytecode) 131
(expect-stdout "") (timeout 5))src/sigil/doc/model.sglmodified
@@ -19,7 +19,7 @@
19
admonition? admonition-kind admonition-title admonition-children admonition-location make-admonition 20
exercise? exercise-id exercise-title exercise-children exercise-location make-exercise 21
example? example-id example-source-kind example-source example-language example-session example-target−22
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location make-example+22
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location example-check-kind example-check make-example 23
project-check? project-check-id project-check-root project-check-files project-check-config project-check-target 24
project-check-expected-stdout project-check-timeout project-check-location make-project-check 25
anchor? anchor-id anchor-location make-anchor@@ -51,7 +51,8 @@
51
(define-struct admonition-record (kind) (title) (children) (location)) 52
(define-struct exercise-record (id) (title) (children) (location)) 53
(define-struct example-record (id) (source-kind) (source) (language) (session) (target)−54
(expectation-kind) (expected) (expected-stderr) (timeout) (display) (location))+54
(expectation-kind) (expected) (expected-stderr) (timeout) (display) (location)+55
(check-kind) (check)) 56
(define-struct project-check-record (id) (root) (files) (config) (target) (expected-stdout) (timeout) (location)) 57
(define-struct anchor-record (id) (location)) 58
(define-struct reference-record (target) (label) (location))@@ -115,6 +116,11 @@
116
(define example-timeout example-record-timeout) 117
(define example-display example-record-display) 118
(define (example-location x) (safe-location (example-record-location x)))+119
;; A check is hidden code the verifier runs after the shown source, in+120
;; the same program. CHECK-KIND is #f (no check), inline (CHECK is+121
;; Sigil source) or file (CHECK is a path under the verification root).+122
(define example-check-kind example-record-check-kind)+123
(define (example-check x) (copy-string-or-false (example-record-check x))) 124
(define project-check? project-check-record?) 125
(define (project-check-id x) (string-copy (project-check-record-id x))) 126
(define (project-check-root x) (string-copy (project-check-record-root x)))@@ -397,7 +403,20 @@
403
(define (make-exercise id title children loc) 404
(if (not (valid-id? id)) (fail "make-exercise" "invalid id") (exercise-record id: (string-copy id) title: (copy-list (require-inline "make-exercise" title #f)) children: (copy-list (require-block "make-exercise" children #f)) location: (require-location "make-exercise" loc)))) 405
(define (valid-timeout? value) (and (integer? value) (>= value 1) (<= value 60)))−400
(define (make-example id source-kind source language session target expectation-kind expected expected-stderr timeout display loc)+406
(define (valid-check? kind check)+407
(case kind+408
((#f) (not check))+409
((inline) (string? check))+410
((file) (nonempty-string? check))+411
(else #f)))+412
(define (make-example id source-kind source language session target expectation-kind expected expected-stderr timeout display loc+413
(keys: (check-kind #f) (check #f)))+414
(unless (valid-check? check-kind check)+415
(fail "make-example" "check kind must be #f, inline with Sigil source, or file with a path"))+416
;; A check's value is the example's result, so showing the expected+417
;; result under the shown source would show the check's value instead.+418
(when (and check-kind (eq? display 'source+expected))+419
(fail "make-example" "an example with a check can't use display source+expected")) 420
(if (not (valid-id? id)) (fail "make-example" "invalid id") 421
(if (not (memq source-kind '(inline file))) (fail "make-example" "source kind must be inline or file") 422
(if (or (not (string? source)) (and (eq? source-kind 'inline) (not (string=? language "sigil"))))@@ -413,7 +432,8 @@
432
language: (copy-string-or-false language) session: (copy-string-or-false session) target: target 433
expectation-kind: expectation-kind expected: (string-copy expected) 434
expected-stderr: (string-copy expected-stderr) timeout: timeout display: display−416
location: (require-location "make-example" loc)))))))))))+435
location: (require-location "make-example" loc)+436
check-kind: check-kind check: (copy-string-or-false check))))))))))) 437
(define (make-project-check id root files config target expected-stdout timeout loc) 438
(if (not (valid-id? id)) (fail "make-project-check" "invalid id") 439
(if (or (not (nonempty-string? root)) (not (proper-list? files)) (null? files)src/sigil/doc/traversal.sglmodified
@@ -59,12 +59,19 @@
59
((glossary? n) (list 'glossary (loc->datum (glossary-location n)))) 60
((admonition? n) (list 'admonition (list 'kind (admonition-kind n)) (cons 'title (if (admonition-title n) (map-node (admonition-title n)) '())) (cons 'children (map-node (admonition-children n))) (loc->datum (admonition-location n)))) 61
((exercise? n) (list 'exercise (list 'id (exercise-id n)) (cons 'title (map-node (exercise-title n))) (cons 'children (map-node (exercise-children n))) (loc->datum (exercise-location n))))−62
((example? n) (list 'example (list 'id (example-id n)) (list 'source-kind (example-source-kind n))−63
(list 'source (example-source n)) (list 'language (example-language n))−64
(list 'session (example-session n)) (list 'target (example-target n))−65
(list 'expectation (example-expectation-kind n) (example-expected n))−66
(list 'expected-stderr (example-expected-stderr n)) (list 'timeout (example-timeout n))−67
(list 'display (example-display n)) (loc->datum (example-location n))))+62
;; A check entry appears only when there is a check, so an example+63
;; without one keeps the datum (and cache key) it always had.+64
((example? n) (append+65
(list 'example (list 'id (example-id n)) (list 'source-kind (example-source-kind n))+66
(list 'source (example-source n)) (list 'language (example-language n))+67
(list 'session (example-session n)) (list 'target (example-target n))+68
(list 'expectation (example-expectation-kind n) (example-expected n))+69
(list 'expected-stderr (example-expected-stderr n)) (list 'timeout (example-timeout n))+70
(list 'display (example-display n)))+71
(if (example-check-kind n)+72
(list (list 'check (example-check-kind n) (example-check n)))+73
'())+74
(list (loc->datum (example-location n))))) 75
((project-check? n) (list 'project-check (list 'id (project-check-id n)) (list 'root (project-check-root n)) 76
(cons 'files (project-check-files n)) (list 'config (project-check-config n)) 77
(list 'target (project-check-target n)) (list 'expected-stdout (project-check-expected-stdout n))src/sigil/doc/verify.sglmodified
@@ -34,11 +34,16 @@
34
(define (example-specification-contract example) 35
(sha256 36
(format "~s"−37
(list 'example−38
(example-id example) (example-source-kind example) (example-source example)−39
(example-language example) (example-session example) (example-target example)−40
(example-expectation-kind example) (example-expected example)−41
(example-expected-stderr example) (example-timeout example) (example-display example)))))+37
(append+38
(list 'example+39
(example-id example) (example-source-kind example) (example-source example)+40
(example-language example) (example-session example) (example-target example)+41
(example-expectation-kind example) (example-expected example)+42
(example-expected-stderr example) (example-timeout example) (example-display example))+43
;; Present only with a check, so existing contracts don't change.+44
(if (example-check-kind example)+45
(list (list 'check (example-check-kind example) (example-check example)))+46
'()))))) 47
(define (project-specification-contract check) 48
(sha256 49
(format "~s"@@ -115,21 +120,29 @@
120
(define (location-prefix location) 121
(format "~a:~a:~a" (srcloc-file location) (srcloc-line location) (srcloc-column location))) 122
(define (example-fail example detail)−118
(error (format "~a: sigil/doc verify: @example ~a ~a; subject ~a"+123
(error (format "~a: sigil/doc verify: @example ~a ~a; subject ~a~a" 124
(location-prefix (example-location example)) (example-id example) detail 125
(if (eq? (example-source-kind example) 'file) 126
(example-source example)−122
(string-append "inline:" (example-id example))))))+127
(string-append "inline:" (example-id example)))+128
(case (example-check-kind example)+129
((file) (string-append "; check " (example-check example)))+130
((inline) (string-append "; check inline:" (example-id example)))+131
(else ""))))) 132
(define (project-fail check detail) 133
(error (format "~a: sigil/doc verify: @project-check ~a ~a" 134
(location-prefix (project-check-location check)) 135
(project-check-id check) detail))) 136
(define (bounded-output who text) 137
(if (> (string-length text) 1048576) (error (string-append "sigil/doc verify: " who " exceeds 1 MiB")) text))−129
(define (write-wrapper dir)+138
;; With a check, the runner loads the shown source, binds its last value+139
;; as example-result, then loads the check in the same environment; the+140
;; check's value is the result. Without one the runner is unchanged.+141
(define (write-wrapper dir with-check?) 142
(write-file-string (path-join dir "runner.sgl") 143
(string-append 144
"(import (sigil core) (sigil fs) (sigil io))\n"+145
(if with-check? "(define example-result #f)\n" "") 146
"(let ((exception-path (read-line (current-input-port)))\n" 147
" (exception-nonce (read-line (current-input-port))))\n" 148
" (close-input-port (current-input-port))\n"@@ -137,7 +150,9 @@
150
" (write-file-string exception-path\n" 151
" (format \"~s\" (list exception-nonce (exception-kind exn) (exception-message exn))))\n" 152
" (raise exn)))\n"−140
" (write-file-string \"result.txt\" (format \"~s\" (load \"subject.sgl\")))))\n")))+153
(if with-check?+154
" (write-file-string \"result.txt\" (format \"~s\" (begin (set! example-result (load \"subject.sgl\")) (load \"check.sgl\"))))))\n"+155
" (write-file-string \"result.txt\" (format \"~s\" (load \"subject.sgl\")))))\n")))) 156
(define (read-captured path who) 157
(if (not (file? path)) "" 158
;; The active drainers cap this file at 1 MiB + one sentinel byte.@@ -278,7 +293,19 @@
293
(example-location example) 294
(string-append "@example " (example-id example)))) 295
(cumulative-example-source examples example)))−281
(key (cache-key document example source executable))+296
;; The check runs after the shown source in the same program, so+297
;; the expectation applies to its result. Results keep SOURCE as+298
;; their effective source: the check is never presented.+299
(check-source+300
(case (example-check-kind example)+301
((inline) (example-check example))+302
((file) (read-file-string (contained-file root (example-check example)+303
(example-location example)+304
(string-append "@example " (example-id example) " check-file"))))+305
(else #f)))+306
(key (cache-key document example+307
(if check-source (string-append source "\n;; check\n" check-source) source)+308
executable)) 309
(record (cache-path cache-dir key))) 310
(if (and (not no-cache?) (cache-hit? record key)) 311
(make-result example source (example-expectation-kind example) #t@@ -297,7 +324,10 @@
324
(let* ((nonce (make-exception-nonce)) 325
(exception-path (path-join channel-dir "result"))) 326
(write-file-string (path-join dir "subject.sgl") source)−300
(write-wrapper dir)+327
(if check-source+328
(begin (write-file-string (path-join dir "check.sgl") check-source)+329
(write-wrapper dir #t))+330
(write-wrapper dir #f)) 331
(let* ((run (run-subject executable dir (example-timeout example) exception-path nonce)) 332
(stdout (run-stdout run)) (stderr (run-stderr run)) 333
(value (if (file? (path-join dir "result.txt"))test/fixtures/example-check.sgdadded
@@ -0,0 +1,19 @@
+1
#lang sigil/doc/book+2
@chapter["Hidden Checks" authors: '("David Wilson")]{+3
@example["inline-check" expect-value: "#t" timeout: 30 check: '(equal? greeting "hello")]{+4
@code["sigil"]{(define greeting "hello")}+5
}+6
@example["file-check" source-file: "example-check/settings.sgl" check-file: "example-check/settings.check.sgl" expect-value: "(\"dark\" 14)" timeout: 30 display: 'source]{}+7
@example["result-check" expect-value: "3" timeout: 30 check: '(length example-result)]{+8
@code["sigil"]{(list 'alpha 'beta 'gamma)}+9
}+10
@example["session-base" session: "doubling" expect-value: "#t" timeout: 30 check: '(= base 10)]{+11
@code["sigil"]{(define base 10)}+12
}+13
@example["session-doubled" session: "doubling" expect-value: "20" timeout: 30 check: 'doubled]{+14
@code["sigil"]{(define doubled (* base 2))}+15
}+16
@example["plain" expect-value: "3" timeout: 30]{+17
@code["sigil"]{(+ 1 2)}+18
}+19
}test/fixtures/example-check/failing.check.sgladded
@@ -0,0 +1,3 @@
+1
;; A check that fails: the shown settings are fine, but this expects+2
;; a theme they don't have.+3
(equal? (cdr (assq 'theme settings)) "light")test/fixtures/example-check/settings.check.sgladded
@@ -0,0 +1,3 @@
+1
;; Hidden: runs after settings.sgl in the same program.+2
(list (cdr (assq 'theme settings))+3
(cdr (assq 'font-size settings)))test/fixtures/example-check/settings.sgladded
@@ -0,0 +1,4 @@
+1
;; What a reader writes: settings only.+2
(define settings+3
(list (cons 'theme "dark")+4
(cons 'font-size 14)))test/gate-d4a.shmodified
@@ -16,6 +16,13 @@ export SIGIL_REDIRECTS="$redirects"
16
rg -q '37 passed' "$cache_dir/publication.log" 17
echo D4A_GATE_01_D0_D2_D3_PUBLICATION_37_37 18
+19
# Hidden example checks run real subjects, so they need the executable.+20
SIGIL_BIN="$sigil" "$sigil" -L "$sigil_sxml_root/src" -L src -L test test \+21
test/test-example-check.sgl --report \+22
| tee "$cache_dir/example-check.log"+23
rg -q '10 passed' "$cache_dir/example-check.log"+24
echo D4A_GATE_01B_EXAMPLE_CHECKS_10_10+25
26
"$sigil" -L "$sigil_sxml_root/src" -L src -L test eval -f test/d4a-trusted-presentation.sgl \ 27
| tee "$cache_dir/trusted.log" 28
rg -q D4A_TRUSTED_PRESENTATION_GREEN "$cache_dir/trusted.log"test/test-example-check.sgladded
@@ -0,0 +1,164 @@
+1
;; Hidden example checks: check: and check-file: run after the shown source+2
;; in the same program, and no renderer or extracted source shows them.+3
(import (sigil test) (sigil fs) (sigil string) (sigil process) (sigil list)+4
(sigil doc) (sigil doc examples) (sigil doc verify)+5
(sigil doc presentation) (sigil doc publication) (sigil doc print)+6
(sigil doc render html) (sigil doc render typst)+7
(only (sigil doc author) example code))+8
+9
(load "test/fixtures/example-check.sgd")+10
(define fixture (eval 'document))+11
(define examples (document-examples fixture))+12
(define (example-named id) (find (lambda (e) (string=? (example-id e) id)) examples))+13
(define loc (document-location fixture))+14
(define root "test/fixtures")+15
(define sigil-bin+16
(or (getenv "SIGIL_D1_COMPILER") (getenv "SIGIL_BIN")+17
(error "set SIGIL_BIN to the Sigil executable the examples run under")))+18
+19
(define (raised-message thunk)+20
(guard (e (else (exception-message e))) (thunk) #f))+21
+22
(define (verify-one doc id)+23
(let ((xs (document-examples doc)))+24
(verify-example doc xs (find (lambda (e) (string=? (example-id e) id)) xs)+25
root sigil-bin #f #t)))+26
+27
(define (present doc)+28
(map (lambda (r)+29
(let ((spec (find (lambda (e) (string=? (example-id e) (verify-result-id r)))+30
(document-examples doc))))+31
(make-verified-presentation spec r (verify-result-effective-source r))))+32
(verify-document doc root: root sigil-bin: sigil-bin no-cache: #t)))+33
+34
(test "the author commands record checks, and an example without one is unchanged"+35
(let ((inline (example-named "inline-check"))+36
(file (example-named "file-check"))+37
(plain (example-named "plain")))+38
(assert-equal 'inline (example-check-kind inline))+39
(assert-equal "(equal? greeting \"hello\")" (example-check inline))+40
(assert-equal "(define greeting \"hello\")" (example-source inline))+41
(assert-equal 'file (example-check-kind file))+42
(assert-equal "example-check/settings.check.sgl" (example-check file))+43
;; With a check the default display is the source alone; without one+44
;; it is unchanged.+45
(assert-equal 'source (example-display inline))+46
(assert-equal 'source+expected (example-display plain))+47
(assert-equal #f (example-check-kind plain))+48
(assert-equal #f (example-check plain))+49
;; No check entry in the datum of an example without a check.+50
(assert-equal #f (assq 'check (cdr (document->datum plain))))+51
(assert-true (assq 'check (cdr (document->datum inline))))))+52
+53
(test "a passing inline check, a check-file and example-result all verify"+54
(let ((inline (verify-one fixture "inline-check"))+55
(file (verify-one fixture "file-check"))+56
(plain (verify-one fixture "plain")))+57
(assert-equal "#t" (verify-result-value inline))+58
(assert-equal "(define greeting \"hello\")" (verify-result-effective-source inline))+59
(assert-equal "(\"dark\" 14)" (verify-result-value file))+60
(assert-false (string-contains? (verify-result-effective-source file) "assq"))+61
(assert-equal "3" (verify-result-value plain))+62
;; The check sees the shown program's last value as example-result.+63
(assert-equal "3" (verify-result-value (verify-one fixture "result-check")))+64
(assert-equal "(+ 1 2)" (verify-result-effective-source plain))))+65
+66
(define (failure-of . examples)+67
(raised-message+68
(lambda ()+69
(verify-document (make-document 'chapter "Failing" '() examples loc)+70
root: root sigil-bin: sigil-bin no-cache: #t))))+71
+72
(define (inline-with-check id source check expected timeout)+73
(make-example id 'inline source "sigil" #f 'bytecode 'value expected "" timeout 'source loc+74
check-kind: 'inline check: check))+75
+76
(test "a failing check fails verification and names the check"+77
(let ((message (failure-of (inline-with-check "failing" "(define greeting \"hi\")"+78
"(equal? greeting \"hello\")" "#t" 30))))+79
(assert-true (string? message))+80
(assert-true (string-contains? message "value mismatch"))+81
(assert-true (string-contains? message "check inline:failing"))))+82
+83
(test "the check decides even when the shown source alone would pass"+84
;; The shown program's own value is #t, which matches the expectation;+85
;; the check's #f is what is compared.+86
(let ((message (failure-of (inline-with-check "shown-passes" "(define ready #t)\nready"+87
"(not ready)" "#t" 30))))+88
(assert-true (string? message))+89
(assert-true (string-contains? message "got \"#f\""))))+90
+91
(test "a check that raises or times out fails verification"+92
(let ((raised (failure-of (inline-with-check "raises" "(define x 1)" "(error \"check broke\")" "#t" 30)))+93
(looped (failure-of (inline-with-check "loops" "(define x 1)" "(let loop () (loop))" "#t" 2))))+94
(assert-true (string-contains? raised "raised unexpected Sigil exception"))+95
(assert-true (string-contains? raised "check broke"))+96
;; The run is killed at the timeout with no value, so the value+97
;; comparison is the first to fail.+98
(assert-true (string? looped))+99
(assert-true (string-contains? looped "check inline:loops"))))+100
+101
(test "a failing check-file fails verification and names the file"+102
(let ((message (failure-of (make-example "failing-file" 'file "example-check/settings.sgl" #f #f 'bytecode+103
'value "#t" "" 30 'source loc+104
check-kind: 'file check: "example-check/failing.check.sgl"))))+105
(assert-true (string-contains? message "value mismatch"))+106
(assert-true (string-contains? message "check example-check/failing.check.sgl"))))+107
+108
(test "a session runs earlier examples' shown source but not their checks"+109
(let ((doubled (verify-one fixture "session-doubled")))+110
(assert-equal "20" (verify-result-value doubled))+111
(assert-true (string-contains? (verify-result-effective-source doubled) "(define base 10)"))+112
(assert-false (string-contains? (verify-result-effective-source doubled) "(= base 10)"))))+113
+114
(test "constructors and author commands refuse malformed checks"+115
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc+116
check-kind: 'inline check: #f))+117
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc+118
check-kind: 'file check: ""))+119
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc+120
check-kind: 'hidden check: "1"))+121
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc+122
check: "1"))+123
;; The expected value would be the check's, so it can't be shown.+124
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source+expected loc+125
check-kind: 'inline check: "1"))+126
(assert-error (example "x" expect-value: "1" check: "1" display: 'source+expected+127
(code "sigil" "1")))+128
(assert-error (example "x" expect-value: "1" check: "1" check-file: "a.sgl"+129
(code "sigil" "1"))))+130
+131
(test "a check-file outside the verification root is refused"+132
(let* ((escaping (make-example "escaping" 'inline "1" "sigil" #f 'bytecode+133
'value "1" "" 30 'source loc+134
check-kind: 'file check: "../escape.sgl"))+135
(doc (make-document 'chapter "Escape" '() (list escaping) loc)))+136
(assert-true (string? (raised-message+137
(lambda () (verify-document doc root: root sigil-bin: sigil-bin no-cache: #t)))))))+138
+139
(test "HTML, plain text and typst show only the shown source, not the check or its value"+140
(let* ((presentations (present fixture))+141
(publication (build-publication-index (list (make-publication-document "checks" fixture))))+142
(context (make-html-render-context publication "checks"+143
(lambda (target) (string-append "#" (publication-target-id target)))+144
presentations 2))+145
(html (document->html-fragment fixture context))+146
(plain (document->plain fixture))+147
(typst (print-publication->typst+148
(make-print-publication "Checks" #f '("David Wilson")+149
(list (make-publication-document "checks" fixture))+150
presentations))))+151
(for-each+152
(lambda (output)+153
(assert-true (string-contains? output "greeting"))+154
(assert-false (string-contains? output "equal?"))+155
(assert-false (string-contains? output "assq"))+156
(assert-false (string-contains? output "(= base 10)")))+157
(list html plain typst))+158
;; The checks return #t and no shown example's own value is #t, so #t+159
;; on a page would be a check's value. (Typst markup uses #text, so+160
;; there the value is looked for as a string.)+161
(assert-false (string-contains? html "#t"))+162
(assert-false (string-contains? plain "#t"))+163
(assert-false (string-contains? typst "\"#t\""))+164
(assert-true (string-contains? html "font-size"))))