Commit46e1fd80Recorded17 Nov 2026Repositorysigil-doc

Let examples carry hidden checks

Message

An example can now name a check, inline with check: or in a file with check-file:. The check runs after the example and must return true, but only the example's own source is shown. This lets a manual show just the code a reader would write while still proving that it works.

An example with a check shows its source only: make-example refuses 'source+expected with a check, and example defaults to 'source when one is given. A failed check names the check and its file in the build error. Debug and datum outputs include the check by design.

Release 0.2.1.

Changed
 README.md                                      |   3 +++
 docs/examples.md                               |  84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 package.sgl                                    |   2 +-
 src/sigil/doc.sgl                              |   2 +-
 src/sigil/doc/author.sgl                       |  28 +++++++++++++++++++++----
 src/sigil/doc/model.sgl                        |  28 +++++++++++++++++++++----
 src/sigil/doc/traversal.sgl                    |  19 +++++++++++------
 src/sigil/doc/verify.sgl                       |  52 ++++++++++++++++++++++++++++++++++++----------
 test/fixtures/example-check.sgd                |  19 +++++++++++++++++
 test/fixtures/example-check/failing.check.sgl  |   3 +++
 test/fixtures/example-check/settings.check.sgl |   3 +++
 test/fixtures/example-check/settings.sgl       |   4 ++++
 test/gate-d4a.sh                               |   7 +++++++
 test/test-example-check.sgl                    | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 14 files changed, 391 insertions(+), 27 deletions(-)
Diff
README.mdmodified
@@ -44,6 +44,9 @@ language-labelled block, and `@code[#f]{...}` is an unlabelled block. Named and
44
colon `@code` bodies are also blocks. Subject source inside these forms is data
45
and is not executed during document authoring.
46
+47
Executable examples, their expectations and hidden checks are described in
+48
[`docs/examples.md`](docs/examples.md).
+49
50
Evaluation binds the resulting semantic value as `document`. The package
51
includes backend-neutral model/traversal/publication modules, static HTML
52
fragments, and a backend-neutral print adapter with a Typst source backend.
docs/examples.mdadded
@@ -0,0 +1,84 @@
+1
# Checked examples
+2
+3
An `@example` is Sigil code that `sigil doc verify` (or `verify-document`)
+4
runs, in a fresh process, and compares against one expectation. Renderers
+5
show an example only with a verified result, so a published page never
+6
shows code that didn't run.
+7
+8
```text
+9
@example["greeting" expect-stdout: "hello\n"]{
+10
@code["sigil"]{(display "hello\n")}
+11
}
+12
+13
@example["config" source-file: "examples/config.sgl" expect-value: "42"]{}
+14
```
+15
+16
Keywords:
+17
+18
| Keyword | Meaning |
+19
| --- | --- |
+20
| `source-file:` | Read the source from a file under the verification root instead of the body. |
+21
| `session:` | Inline examples with the same session run cumulatively: each one runs after every earlier one in the session. File examples can't join a session. |
+22
| `expect-stdout:`, `expect-value:`, `expect-error:`, `expect-timeout:` | Exactly one expectation. `expect-value:` compares the written form of the program's last value. |
+23
| `expect-stderr:` | Expected standard error, `""` by default. |
+24
| `timeout:` | Seconds, 1 to 60. 5 by default. |
+25
| `display:` | `'source+expected`, `'source` or `'hidden`. The default is `'source+expected`, or `'source` for an example with a check. |
+26
| `check:`, `check-file:` | A hidden check; see below. |
+27
+28
## Hidden checks
+29
+30
Sometimes the code a reader should see isn't the code that proves it works.
+31
A configuration example should show only the configuration, while checking
+32
it means calling functions the reader never would. `check:` and
+33
`check-file:` keep that checking code out of the page.
+34
+35
```text
+36
@example["settings" expect-value: "#t" check: '(equal? (cdr (assq 'theme settings)) "dark")]{
+37
@code["sigil"]{(define settings (list (cons 'theme "dark")))}
+38
}
+39
+40
@example["web-server" source-file: "examples/web.sgl"
+41
check-file: "examples/web.check.sgl" expect-value: "(80 443)"]{}
+42
```
+43
+44
- **`check:`** takes a quoted Sigil expression, or its text as a string.
+45
Use `begin` for several forms. A quoted expression is written back as
+46
text with `write`, so it must hold only data that reads back the same:
+47
keywords such as `name:` don't, so use a string or `check-file:` for
+48
checks that pass keyword arguments. The reader doesn't allow `@` commands
+49
inside `[...]`, so a code block can't be written there; code that builds
+50
examples with `make-example` can pass any of these.
+51
- **`check-file:`** takes a path under the verification root, resolved the
+52
same way as `source-file:`. Use it for anything longer than one
+53
expression. Give `check:` or `check-file:`, not both.
+54
- **How it runs.** In one process, the verifier loads the shown source,
+55
binds its last value as `example-result`, then loads the check in the
+56
same environment. The check sees every definition and import the shown
+57
source made, plus `example-result`, so it can check an example whose
+58
last expression is a value with no name, such as a configuration. The
+59
expectation applies to the whole run: `expect-value:` compares the
+60
check's last value, and `expect-stdout:` compares the output of both
+61
parts.
+62
- **`example-result` is an ordinary binding.** If the shown source
+63
defines `example-result` itself, the verifier's `set!` replaces that
+64
value before the check runs, so don't use the name in shown code.
+65
- **What readers see.** Only the shown source. An example with a check
+66
can't use `display: 'source+expected`, because the expected value is
+67
the check's, not the shown code's: `'source` is the default and
+68
`'source+expected` is refused. The check isn't part of the
+69
verified result's effective source, so the HTML, plain text and Typst
+70
renderers, copy buttons and any tool that extracts example source never
+71
see it. A session's later examples include earlier examples' shown
+72
source, not their checks.
+73
- **Caching and contracts.** The cache key and the result's specification
+74
contract cover the check, so changing a check re-verifies the example.
+75
An example without a check has the same document datum, cache key and
+76
contract it had before checks existed. `document->datum`, and the debug
+77
output built from it, include the check by design: they describe the
+78
whole document, not what readers see.
+79
+80
From Sigil code, pass the check to `make-example` as keys after the
+81
location: `check-kind: 'inline check: "(expression)"`, or
+82
`check-kind: 'file check: "path/to/check.sgl"`. `example-check-kind`
+83
returns `#f`, `'inline` or `'file`, and `example-check` returns the text or
+84
path.
package.sglmodified
@@ -1,6 +1,6 @@
1
(package
2
name: "sigil-doc"
−3
version: "0.2.0"
+3
version: "0.2.1"
4
sigil: "^0.18.4"
5
description: "A programmable document language for Sigil"
6
license: "BSD-3-Clause"
src/sigil/doc.sglmodified
@@ -17,7 +17,7 @@
17
admonition? admonition-kind admonition-title admonition-children admonition-location make-admonition
18
exercise? exercise-id exercise-title exercise-children exercise-location make-exercise
19
example? example-id example-source-kind example-source example-language example-session example-target
−20
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location make-example
+20
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location example-check-kind example-check make-example
21
project-check? project-check-id project-check-root project-check-files project-check-config project-check-target
22
project-check-expected-stdout project-check-timeout project-check-location make-project-check
23
anchor? anchor-id anchor-location make-anchor reference? reference-target reference-label reference-location make-reference
src/sigil/doc/author.sglmodified
@@ -94,18 +94,38 @@
94
(define (example id
95
(keys: (source-file #f) (session #f) (target 'bytecode)
96
(expect-stdout #f) (expect-value #f) (expect-error #f) (expect-timeout #f)
−97
(expect-stderr "") (timeout 5) (display 'source+expected))
+97
(expect-stderr "") (timeout 5) (display #f)
+98
(check #f) (check-file #f))
99
(rest: body))
−99
(let ((expectation (one-expectation expect-stdout expect-value expect-error expect-timeout)))
+100
(let ((expectation (one-expectation expect-stdout expect-value expect-error expect-timeout))
+101
(hidden (example-check check check-file)))
+102
;; With a check, the result belongs to the check, so the default
+103
;; shows the source alone.
+104
(set! display (or display (if (car hidden) 'source 'source+expected)))
105
(if source-file
106
(if (null? body)
107
(make-example id 'file source-file #f session target (car expectation) (cadr expectation)
−103
expect-stderr timeout display (author-location))
+108
expect-stderr timeout display (author-location)
+109
check-kind: (car hidden) check: (cadr hidden))
110
(error "example: file source requires an empty body"))
111
(if (and (= (length body) 1) (code-block? (car body)))
112
(make-example id 'inline (code-block-content (car body)) (code-block-language (car body)) session target
−107
(car expectation) (cadr expectation) expect-stderr timeout display (author-location))
+113
(car expectation) (cadr expectation) expect-stderr timeout display (author-location)
+114
check-kind: (car hidden) check: (cadr hidden))
115
(error "example: inline source requires exactly one sigil code block")))))
+116
;; check: is a quoted Sigil expression, its text, or a sigil code
+117
;; block; check-file: is a path under the verification root. Neither
+118
;; is ever rendered.
+119
(define (example-check check check-file)
+120
(cond ((and check check-file) (error "example: give check: or check-file:, not both"))
+121
(check-file (list 'file check-file))
+122
((not check) (list #f #f))
+123
((string? check) (list 'inline check))
+124
((code-block? check)
+125
(if (member (code-block-language check) '(#f "sigil"))
+126
(list 'inline (code-block-content check))
+127
(error "example: check: code must be sigil")))
+128
(else (list 'inline (format "~s" check)))))
129
(define (project-check id
130
(keys: root files (config 'dev) (target 'bytecode)
131
(expect-stdout "") (timeout 5))
src/sigil/doc/model.sglmodified
@@ -19,7 +19,7 @@
19
admonition? admonition-kind admonition-title admonition-children admonition-location make-admonition
20
exercise? exercise-id exercise-title exercise-children exercise-location make-exercise
21
example? example-id example-source-kind example-source example-language example-session example-target
−22
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location make-example
+22
example-expectation-kind example-expected example-expected-stderr example-timeout example-display example-location example-check-kind example-check make-example
23
project-check? project-check-id project-check-root project-check-files project-check-config project-check-target
24
project-check-expected-stdout project-check-timeout project-check-location make-project-check
25
anchor? anchor-id anchor-location make-anchor
@@ -51,7 +51,8 @@
51
(define-struct admonition-record (kind) (title) (children) (location))
52
(define-struct exercise-record (id) (title) (children) (location))
53
(define-struct example-record (id) (source-kind) (source) (language) (session) (target)
−54
(expectation-kind) (expected) (expected-stderr) (timeout) (display) (location))
+54
(expectation-kind) (expected) (expected-stderr) (timeout) (display) (location)
+55
(check-kind) (check))
56
(define-struct project-check-record (id) (root) (files) (config) (target) (expected-stdout) (timeout) (location))
57
(define-struct anchor-record (id) (location))
58
(define-struct reference-record (target) (label) (location))
@@ -115,6 +116,11 @@
116
(define example-timeout example-record-timeout)
117
(define example-display example-record-display)
118
(define (example-location x) (safe-location (example-record-location x)))
+119
;; A check is hidden code the verifier runs after the shown source, in
+120
;; the same program. CHECK-KIND is #f (no check), inline (CHECK is
+121
;; Sigil source) or file (CHECK is a path under the verification root).
+122
(define example-check-kind example-record-check-kind)
+123
(define (example-check x) (copy-string-or-false (example-record-check x)))
124
(define project-check? project-check-record?)
125
(define (project-check-id x) (string-copy (project-check-record-id x)))
126
(define (project-check-root x) (string-copy (project-check-record-root x)))
@@ -397,7 +403,20 @@
403
(define (make-exercise id title children loc)
404
(if (not (valid-id? id)) (fail "make-exercise" "invalid id") (exercise-record id: (string-copy id) title: (copy-list (require-inline "make-exercise" title #f)) children: (copy-list (require-block "make-exercise" children #f)) location: (require-location "make-exercise" loc))))
405
(define (valid-timeout? value) (and (integer? value) (>= value 1) (<= value 60)))
−400
(define (make-example id source-kind source language session target expectation-kind expected expected-stderr timeout display loc)
+406
(define (valid-check? kind check)
+407
(case kind
+408
((#f) (not check))
+409
((inline) (string? check))
+410
((file) (nonempty-string? check))
+411
(else #f)))
+412
(define (make-example id source-kind source language session target expectation-kind expected expected-stderr timeout display loc
+413
(keys: (check-kind #f) (check #f)))
+414
(unless (valid-check? check-kind check)
+415
(fail "make-example" "check kind must be #f, inline with Sigil source, or file with a path"))
+416
;; A check's value is the example's result, so showing the expected
+417
;; result under the shown source would show the check's value instead.
+418
(when (and check-kind (eq? display 'source+expected))
+419
(fail "make-example" "an example with a check can't use display source+expected"))
420
(if (not (valid-id? id)) (fail "make-example" "invalid id")
421
(if (not (memq source-kind '(inline file))) (fail "make-example" "source kind must be inline or file")
422
(if (or (not (string? source)) (and (eq? source-kind 'inline) (not (string=? language "sigil"))))
@@ -413,7 +432,8 @@
432
language: (copy-string-or-false language) session: (copy-string-or-false session) target: target
433
expectation-kind: expectation-kind expected: (string-copy expected)
434
expected-stderr: (string-copy expected-stderr) timeout: timeout display: display
−416
location: (require-location "make-example" loc)))))))))))
+435
location: (require-location "make-example" loc)
+436
check-kind: check-kind check: (copy-string-or-false check)))))))))))
437
(define (make-project-check id root files config target expected-stdout timeout loc)
438
(if (not (valid-id? id)) (fail "make-project-check" "invalid id")
439
(if (or (not (nonempty-string? root)) (not (proper-list? files)) (null? files)
src/sigil/doc/traversal.sglmodified
@@ -59,12 +59,19 @@
59
((glossary? n) (list 'glossary (loc->datum (glossary-location n))))
60
((admonition? n) (list 'admonition (list 'kind (admonition-kind n)) (cons 'title (if (admonition-title n) (map-node (admonition-title n)) '())) (cons 'children (map-node (admonition-children n))) (loc->datum (admonition-location n))))
61
((exercise? n) (list 'exercise (list 'id (exercise-id n)) (cons 'title (map-node (exercise-title n))) (cons 'children (map-node (exercise-children n))) (loc->datum (exercise-location n))))
−62
((example? n) (list 'example (list 'id (example-id n)) (list 'source-kind (example-source-kind n))
−63
(list 'source (example-source n)) (list 'language (example-language n))
−64
(list 'session (example-session n)) (list 'target (example-target n))
−65
(list 'expectation (example-expectation-kind n) (example-expected n))
−66
(list 'expected-stderr (example-expected-stderr n)) (list 'timeout (example-timeout n))
−67
(list 'display (example-display n)) (loc->datum (example-location n))))
+62
;; A check entry appears only when there is a check, so an example
+63
;; without one keeps the datum (and cache key) it always had.
+64
((example? n) (append
+65
(list 'example (list 'id (example-id n)) (list 'source-kind (example-source-kind n))
+66
(list 'source (example-source n)) (list 'language (example-language n))
+67
(list 'session (example-session n)) (list 'target (example-target n))
+68
(list 'expectation (example-expectation-kind n) (example-expected n))
+69
(list 'expected-stderr (example-expected-stderr n)) (list 'timeout (example-timeout n))
+70
(list 'display (example-display n)))
+71
(if (example-check-kind n)
+72
(list (list 'check (example-check-kind n) (example-check n)))
+73
'())
+74
(list (loc->datum (example-location n)))))
75
((project-check? n) (list 'project-check (list 'id (project-check-id n)) (list 'root (project-check-root n))
76
(cons 'files (project-check-files n)) (list 'config (project-check-config n))
77
(list 'target (project-check-target n)) (list 'expected-stdout (project-check-expected-stdout n))
src/sigil/doc/verify.sglmodified
@@ -34,11 +34,16 @@
34
(define (example-specification-contract example)
35
(sha256
36
(format "~s"
−37
(list 'example
−38
(example-id example) (example-source-kind example) (example-source example)
−39
(example-language example) (example-session example) (example-target example)
−40
(example-expectation-kind example) (example-expected example)
−41
(example-expected-stderr example) (example-timeout example) (example-display example)))))
+37
(append
+38
(list 'example
+39
(example-id example) (example-source-kind example) (example-source example)
+40
(example-language example) (example-session example) (example-target example)
+41
(example-expectation-kind example) (example-expected example)
+42
(example-expected-stderr example) (example-timeout example) (example-display example))
+43
;; Present only with a check, so existing contracts don't change.
+44
(if (example-check-kind example)
+45
(list (list 'check (example-check-kind example) (example-check example)))
+46
'())))))
47
(define (project-specification-contract check)
48
(sha256
49
(format "~s"
@@ -115,21 +120,29 @@
120
(define (location-prefix location)
121
(format "~a:~a:~a" (srcloc-file location) (srcloc-line location) (srcloc-column location)))
122
(define (example-fail example detail)
−118
(error (format "~a: sigil/doc verify: @example ~a ~a; subject ~a"
+123
(error (format "~a: sigil/doc verify: @example ~a ~a; subject ~a~a"
124
(location-prefix (example-location example)) (example-id example) detail
125
(if (eq? (example-source-kind example) 'file)
126
(example-source example)
−122
(string-append "inline:" (example-id example))))))
+127
(string-append "inline:" (example-id example)))
+128
(case (example-check-kind example)
+129
((file) (string-append "; check " (example-check example)))
+130
((inline) (string-append "; check inline:" (example-id example)))
+131
(else "")))))
132
(define (project-fail check detail)
133
(error (format "~a: sigil/doc verify: @project-check ~a ~a"
134
(location-prefix (project-check-location check))
135
(project-check-id check) detail)))
136
(define (bounded-output who text)
137
(if (> (string-length text) 1048576) (error (string-append "sigil/doc verify: " who " exceeds 1 MiB")) text))
−129
(define (write-wrapper dir)
+138
;; With a check, the runner loads the shown source, binds its last value
+139
;; as example-result, then loads the check in the same environment; the
+140
;; check's value is the result. Without one the runner is unchanged.
+141
(define (write-wrapper dir with-check?)
142
(write-file-string (path-join dir "runner.sgl")
143
(string-append
144
"(import (sigil core) (sigil fs) (sigil io))\n"
+145
(if with-check? "(define example-result #f)\n" "")
146
"(let ((exception-path (read-line (current-input-port)))\n"
147
" (exception-nonce (read-line (current-input-port))))\n"
148
" (close-input-port (current-input-port))\n"
@@ -137,7 +150,9 @@
150
" (write-file-string exception-path\n"
151
" (format \"~s\" (list exception-nonce (exception-kind exn) (exception-message exn))))\n"
152
" (raise exn)))\n"
−140
" (write-file-string \"result.txt\" (format \"~s\" (load \"subject.sgl\")))))\n")))
+153
(if with-check?
+154
" (write-file-string \"result.txt\" (format \"~s\" (begin (set! example-result (load \"subject.sgl\")) (load \"check.sgl\"))))))\n"
+155
" (write-file-string \"result.txt\" (format \"~s\" (load \"subject.sgl\")))))\n"))))
156
(define (read-captured path who)
157
(if (not (file? path)) ""
158
;; The active drainers cap this file at 1 MiB + one sentinel byte.
@@ -278,7 +293,19 @@
293
(example-location example)
294
(string-append "@example " (example-id example))))
295
(cumulative-example-source examples example)))
−281
(key (cache-key document example source executable))
+296
;; The check runs after the shown source in the same program, so
+297
;; the expectation applies to its result. Results keep SOURCE as
+298
;; their effective source: the check is never presented.
+299
(check-source
+300
(case (example-check-kind example)
+301
((inline) (example-check example))
+302
((file) (read-file-string (contained-file root (example-check example)
+303
(example-location example)
+304
(string-append "@example " (example-id example) " check-file"))))
+305
(else #f)))
+306
(key (cache-key document example
+307
(if check-source (string-append source "\n;; check\n" check-source) source)
+308
executable))
309
(record (cache-path cache-dir key)))
310
(if (and (not no-cache?) (cache-hit? record key))
311
(make-result example source (example-expectation-kind example) #t
@@ -297,7 +324,10 @@
324
(let* ((nonce (make-exception-nonce))
325
(exception-path (path-join channel-dir "result")))
326
(write-file-string (path-join dir "subject.sgl") source)
−300
(write-wrapper dir)
+327
(if check-source
+328
(begin (write-file-string (path-join dir "check.sgl") check-source)
+329
(write-wrapper dir #t))
+330
(write-wrapper dir #f))
331
(let* ((run (run-subject executable dir (example-timeout example) exception-path nonce))
332
(stdout (run-stdout run)) (stderr (run-stderr run))
333
(value (if (file? (path-join dir "result.txt"))
test/fixtures/example-check.sgdadded
@@ -0,0 +1,19 @@
+1
#lang sigil/doc/book
+2
@chapter["Hidden Checks" authors: '("David Wilson")]{
+3
@example["inline-check" expect-value: "#t" timeout: 30 check: '(equal? greeting "hello")]{
+4
@code["sigil"]{(define greeting "hello")}
+5
}
+6
@example["file-check" source-file: "example-check/settings.sgl" check-file: "example-check/settings.check.sgl" expect-value: "(\"dark\" 14)" timeout: 30 display: 'source]{}
+7
@example["result-check" expect-value: "3" timeout: 30 check: '(length example-result)]{
+8
@code["sigil"]{(list 'alpha 'beta 'gamma)}
+9
}
+10
@example["session-base" session: "doubling" expect-value: "#t" timeout: 30 check: '(= base 10)]{
+11
@code["sigil"]{(define base 10)}
+12
}
+13
@example["session-doubled" session: "doubling" expect-value: "20" timeout: 30 check: 'doubled]{
+14
@code["sigil"]{(define doubled (* base 2))}
+15
}
+16
@example["plain" expect-value: "3" timeout: 30]{
+17
@code["sigil"]{(+ 1 2)}
+18
}
+19
}
test/fixtures/example-check/failing.check.sgladded
@@ -0,0 +1,3 @@
+1
;; A check that fails: the shown settings are fine, but this expects
+2
;; a theme they don't have.
+3
(equal? (cdr (assq 'theme settings)) "light")
test/fixtures/example-check/settings.check.sgladded
@@ -0,0 +1,3 @@
+1
;; Hidden: runs after settings.sgl in the same program.
+2
(list (cdr (assq 'theme settings))
+3
(cdr (assq 'font-size settings)))
test/fixtures/example-check/settings.sgladded
@@ -0,0 +1,4 @@
+1
;; What a reader writes: settings only.
+2
(define settings
+3
(list (cons 'theme "dark")
+4
(cons 'font-size 14)))
test/gate-d4a.shmodified
@@ -16,6 +16,13 @@ export SIGIL_REDIRECTS="$redirects"
16
rg -q '37 passed' "$cache_dir/publication.log"
17
echo D4A_GATE_01_D0_D2_D3_PUBLICATION_37_37
18
+19
# Hidden example checks run real subjects, so they need the executable.
+20
SIGIL_BIN="$sigil" "$sigil" -L "$sigil_sxml_root/src" -L src -L test test \
+21
test/test-example-check.sgl --report \
+22
| tee "$cache_dir/example-check.log"
+23
rg -q '10 passed' "$cache_dir/example-check.log"
+24
echo D4A_GATE_01B_EXAMPLE_CHECKS_10_10
+25
26
"$sigil" -L "$sigil_sxml_root/src" -L src -L test eval -f test/d4a-trusted-presentation.sgl \
27
| tee "$cache_dir/trusted.log"
28
rg -q D4A_TRUSTED_PRESENTATION_GREEN "$cache_dir/trusted.log"
test/test-example-check.sgladded
@@ -0,0 +1,164 @@
+1
;; Hidden example checks: check: and check-file: run after the shown source
+2
;; in the same program, and no renderer or extracted source shows them.
+3
(import (sigil test) (sigil fs) (sigil string) (sigil process) (sigil list)
+4
(sigil doc) (sigil doc examples) (sigil doc verify)
+5
(sigil doc presentation) (sigil doc publication) (sigil doc print)
+6
(sigil doc render html) (sigil doc render typst)
+7
(only (sigil doc author) example code))
+8
+9
(load "test/fixtures/example-check.sgd")
+10
(define fixture (eval 'document))
+11
(define examples (document-examples fixture))
+12
(define (example-named id) (find (lambda (e) (string=? (example-id e) id)) examples))
+13
(define loc (document-location fixture))
+14
(define root "test/fixtures")
+15
(define sigil-bin
+16
(or (getenv "SIGIL_D1_COMPILER") (getenv "SIGIL_BIN")
+17
(error "set SIGIL_BIN to the Sigil executable the examples run under")))
+18
+19
(define (raised-message thunk)
+20
(guard (e (else (exception-message e))) (thunk) #f))
+21
+22
(define (verify-one doc id)
+23
(let ((xs (document-examples doc)))
+24
(verify-example doc xs (find (lambda (e) (string=? (example-id e) id)) xs)
+25
root sigil-bin #f #t)))
+26
+27
(define (present doc)
+28
(map (lambda (r)
+29
(let ((spec (find (lambda (e) (string=? (example-id e) (verify-result-id r)))
+30
(document-examples doc))))
+31
(make-verified-presentation spec r (verify-result-effective-source r))))
+32
(verify-document doc root: root sigil-bin: sigil-bin no-cache: #t)))
+33
+34
(test "the author commands record checks, and an example without one is unchanged"
+35
(let ((inline (example-named "inline-check"))
+36
(file (example-named "file-check"))
+37
(plain (example-named "plain")))
+38
(assert-equal 'inline (example-check-kind inline))
+39
(assert-equal "(equal? greeting \"hello\")" (example-check inline))
+40
(assert-equal "(define greeting \"hello\")" (example-source inline))
+41
(assert-equal 'file (example-check-kind file))
+42
(assert-equal "example-check/settings.check.sgl" (example-check file))
+43
;; With a check the default display is the source alone; without one
+44
;; it is unchanged.
+45
(assert-equal 'source (example-display inline))
+46
(assert-equal 'source+expected (example-display plain))
+47
(assert-equal #f (example-check-kind plain))
+48
(assert-equal #f (example-check plain))
+49
;; No check entry in the datum of an example without a check.
+50
(assert-equal #f (assq 'check (cdr (document->datum plain))))
+51
(assert-true (assq 'check (cdr (document->datum inline))))))
+52
+53
(test "a passing inline check, a check-file and example-result all verify"
+54
(let ((inline (verify-one fixture "inline-check"))
+55
(file (verify-one fixture "file-check"))
+56
(plain (verify-one fixture "plain")))
+57
(assert-equal "#t" (verify-result-value inline))
+58
(assert-equal "(define greeting \"hello\")" (verify-result-effective-source inline))
+59
(assert-equal "(\"dark\" 14)" (verify-result-value file))
+60
(assert-false (string-contains? (verify-result-effective-source file) "assq"))
+61
(assert-equal "3" (verify-result-value plain))
+62
;; The check sees the shown program's last value as example-result.
+63
(assert-equal "3" (verify-result-value (verify-one fixture "result-check")))
+64
(assert-equal "(+ 1 2)" (verify-result-effective-source plain))))
+65
+66
(define (failure-of . examples)
+67
(raised-message
+68
(lambda ()
+69
(verify-document (make-document 'chapter "Failing" '() examples loc)
+70
root: root sigil-bin: sigil-bin no-cache: #t))))
+71
+72
(define (inline-with-check id source check expected timeout)
+73
(make-example id 'inline source "sigil" #f 'bytecode 'value expected "" timeout 'source loc
+74
check-kind: 'inline check: check))
+75
+76
(test "a failing check fails verification and names the check"
+77
(let ((message (failure-of (inline-with-check "failing" "(define greeting \"hi\")"
+78
"(equal? greeting \"hello\")" "#t" 30))))
+79
(assert-true (string? message))
+80
(assert-true (string-contains? message "value mismatch"))
+81
(assert-true (string-contains? message "check inline:failing"))))
+82
+83
(test "the check decides even when the shown source alone would pass"
+84
;; The shown program's own value is #t, which matches the expectation;
+85
;; the check's #f is what is compared.
+86
(let ((message (failure-of (inline-with-check "shown-passes" "(define ready #t)\nready"
+87
"(not ready)" "#t" 30))))
+88
(assert-true (string? message))
+89
(assert-true (string-contains? message "got \"#f\""))))
+90
+91
(test "a check that raises or times out fails verification"
+92
(let ((raised (failure-of (inline-with-check "raises" "(define x 1)" "(error \"check broke\")" "#t" 30)))
+93
(looped (failure-of (inline-with-check "loops" "(define x 1)" "(let loop () (loop))" "#t" 2))))
+94
(assert-true (string-contains? raised "raised unexpected Sigil exception"))
+95
(assert-true (string-contains? raised "check broke"))
+96
;; The run is killed at the timeout with no value, so the value
+97
;; comparison is the first to fail.
+98
(assert-true (string? looped))
+99
(assert-true (string-contains? looped "check inline:loops"))))
+100
+101
(test "a failing check-file fails verification and names the file"
+102
(let ((message (failure-of (make-example "failing-file" 'file "example-check/settings.sgl" #f #f 'bytecode
+103
'value "#t" "" 30 'source loc
+104
check-kind: 'file check: "example-check/failing.check.sgl"))))
+105
(assert-true (string-contains? message "value mismatch"))
+106
(assert-true (string-contains? message "check example-check/failing.check.sgl"))))
+107
+108
(test "a session runs earlier examples' shown source but not their checks"
+109
(let ((doubled (verify-one fixture "session-doubled")))
+110
(assert-equal "20" (verify-result-value doubled))
+111
(assert-true (string-contains? (verify-result-effective-source doubled) "(define base 10)"))
+112
(assert-false (string-contains? (verify-result-effective-source doubled) "(= base 10)"))))
+113
+114
(test "constructors and author commands refuse malformed checks"
+115
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc
+116
check-kind: 'inline check: #f))
+117
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc
+118
check-kind: 'file check: ""))
+119
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc
+120
check-kind: 'hidden check: "1"))
+121
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source loc
+122
check: "1"))
+123
;; The expected value would be the check's, so it can't be shown.
+124
(assert-error (make-example "x" 'inline "1" "sigil" #f 'bytecode 'value "1" "" 5 'source+expected loc
+125
check-kind: 'inline check: "1"))
+126
(assert-error (example "x" expect-value: "1" check: "1" display: 'source+expected
+127
(code "sigil" "1")))
+128
(assert-error (example "x" expect-value: "1" check: "1" check-file: "a.sgl"
+129
(code "sigil" "1"))))
+130
+131
(test "a check-file outside the verification root is refused"
+132
(let* ((escaping (make-example "escaping" 'inline "1" "sigil" #f 'bytecode
+133
'value "1" "" 30 'source loc
+134
check-kind: 'file check: "../escape.sgl"))
+135
(doc (make-document 'chapter "Escape" '() (list escaping) loc)))
+136
(assert-true (string? (raised-message
+137
(lambda () (verify-document doc root: root sigil-bin: sigil-bin no-cache: #t)))))))
+138
+139
(test "HTML, plain text and typst show only the shown source, not the check or its value"
+140
(let* ((presentations (present fixture))
+141
(publication (build-publication-index (list (make-publication-document "checks" fixture))))
+142
(context (make-html-render-context publication "checks"
+143
(lambda (target) (string-append "#" (publication-target-id target)))
+144
presentations 2))
+145
(html (document->html-fragment fixture context))
+146
(plain (document->plain fixture))
+147
(typst (print-publication->typst
+148
(make-print-publication "Checks" #f '("David Wilson")
+149
(list (make-publication-document "checks" fixture))
+150
presentations))))
+151
(for-each
+152
(lambda (output)
+153
(assert-true (string-contains? output "greeting"))
+154
(assert-false (string-contains? output "equal?"))
+155
(assert-false (string-contains? output "assq"))
+156
(assert-false (string-contains? output "(= base 10)")))
+157
(list html plain typst))
+158
;; The checks return #t and no shown example's own value is #t, so #t
+159
;; on a page would be a check's value. (Typst markup uses #text, so
+160
;; there the value is looked for as a string.)
+161
(assert-false (string-contains? html "#t"))
+162
(assert-false (string-contains? plain "#t"))
+163
(assert-false (string-contains? typst "\"#t\""))
+164
(assert-true (string-contains? html "font-size"))))