AtlatestRepositorysigil-doc
1
# The passive parse API3
`(sigil doc inspect)` exposes a `#lang sigil/doc` manuscript's **raw command4
tree with exact spans**, obtained **without evaluating any author code**.6
It exists because the only other way out of the parser is7
`parse-document-source` / `parse-document-forms`, which return *lowered*8
generated Sigil syntax: to learn a document's structure through those you must9
evaluate the author forms they produce. A consumer that indexes documents it10
does not trust, or documents that are half-written, cannot do that.12
```scheme13
(import (sigil doc inspect) (sigil fs) (sigil language))15
(define source (read-file-string "notes/topics/example.sgd"))16
(define header (inspect-language-source source "notes/topics/example.sgd"))17
(define tree18
(inspect-document-source source19
"notes/topics/example.sgd"20
(language-header-end header)))21
```23
## What you get25
`inspect-document-source` returns a **document tree**:27
| accessor | meaning |28
| --- | --- |29
| `document-tree-source-name` | the name the manuscript was parsed under |30
| `document-tree-header-end` | the byte offset where the `#lang` header ended |31
| `document-tree-roots` | top-level nodes, in source order |32
| `document-tree-index-steps` | scalars visited building the location index (one pass; equals the manuscript's scalar length) |34
Each node is a **tree node**:36
| accessor | meaning |37
| --- | --- |38
| `tree-node-kind` | `command`, `text`, or `expr` |39
| `tree-node-name` | the command's name as a symbol; `#f` for `text` / `expr` |40
| `tree-node-arguments` | the `[...]` group as a list of **unevaluated** syntax objects; `()` when absent |41
| `tree-node-children` | child nodes, in source order |42
| `tree-node-body-kind` | `brace`, `line`, or `named`; `#f` for `text` / `expr` |43
| `tree-node-text` | folded prose for a `text` node (see the warning below); `#f` otherwise |44
| `tree-node-expression` | the **unevaluated** syntax of an `@(...)` form; `#f` otherwise |45
| `tree-node-start`, `tree-node-end` | **scalar** indices, half-open |46
| `tree-node-location` | an `srcloc`: **UTF-8 byte** position and byte span, plus scalar line and column of the start |47
| `tree-node-end-line`, `tree-node-end-column` | scalar line and column of the end |49
Spans are exact and total. For every node,50
`(substring source (tree-node-start node) (tree-node-end node))` is precisely51
the manuscript text that node was scanned from; children are contained within52
their parent and never overlap each other.54
Both coordinate systems are given because consumers need both: byte offsets55
address the file, scalar indices address the string, and the two diverge as soon56
as a multi-byte character precedes a node.58
### Per-field spans60
A node's **arguments are ordinary Sigil syntax**, so each carries its own61
`srcloc` addressing the original manuscript. A consumer that needs a single62
field's extent — a task's `status` separately from its body — reads it from the63
argument, not from the node:65
```scheme66
(let* ((argument (car (tree-node-arguments node)))67
(loc (syntax-srcloc argument)))68
(substring source (srcloc-position loc)69
(+ (srcloc-position loc) (srcloc-span loc))))70
```72
## Guarantees74
- **No evaluation.** No author expression runs, no author-defined command runs,75
no author macro expands, no `@code` body runs, and the author runtime is never76
loaded. `(sigil doc inspect)` does not import `(sigil doc author)`,77
`(sigil doc model)`, or `(sigil doc private author-runtime)`.78
- **No lowering.** Nothing is rewritten into generated Sigil syntax and no79
semantic model record is constructed.80
- **No root-container policy.** "Exactly one `@chapter`" is a rule the81
*lowering* path enforces, not a fact about the grammar. A manuscript rooted at82
`@note` or `@project`, or with several roots, inspects normally. (The lowering83
path is unchanged and still refuses it; parameterizing the root container is84
SD-P1.)85
- **Reentrant.** All parser state lives in a per-parse value threaded as an86
argument. Nested and interleaved parses cannot observe each other.88
## Explicit non-guarantees90
- **Malformed source raises.** The passive path raises exactly the error the91
ordinary parse path raises, with the identical message and location. There is92
deliberately no diagnostic-collecting mode here: guard the call, or build the93
wrapper in the consumer that knows what a diagnostic should mean. The parser's94
bounds (nesting 64, commands 8192) raise the same way.95
- **`tree-node-text` is folded prose, not the raw slice.** Single newlines96
become spaces and body-edge whitespace is trimmed. **Inside a raw `@code`97
body this is not the content** — there, the authoritative text is the source98
slice named by the node's span, which is also what the lowering path uses.99
When in doubt, slice the source; the spans are exact.100
- **The command vocabulary is not interpreted.** `@chapter`, `@note` and a101
command an author defined five lines earlier are all just commands with names,102
arguments and children. Deciding which are meaningful is the consumer's job,103
and a consumer that only understands a built-in vocabulary should report104
unknown commands as opaque rather than dropping them.105
- **No revision or content identity.** The API is a pure function of the source106
text you hand it; a consumer that needs a revision hashes the same bytes.107
- **Copying is one level deep. Treat syntax objects and locations as108
immutable.** List spines and strings are copied on access; the *elements* of109
an argument list are syntax objects shared with the parser, and110
`tree-node-location` returns a shared `srcloc`. Sigil pairs are mutable, so a111
consumer that mutates an argument's datum corrupts the view of anything else112
holding the same tree. Two separate `inspect-document-source` calls share113
nothing (each re-parses), so this only bites a consumer that caches one tree114
and hands it to several readers. Deep-copying syntax on every access would115
make traversal pay for a hazard no in-tree consumer has.116
- **The argument-group bound does not bound the cost of a single read.**117
`read-arguments` caps how many candidate `]` boundaries it will try118
(65536 — a backstop, deliberately far above any real manuscript, because the119
candidates are counted over the rest of the *file* rather than the group). It120
does **not** limit the work inside one delegated read.122
Concretely, and **this is not a sigil-doc defect**: the standard reader123
exhausts memory on a `}` appearing inside an unclosed `(` or `[`. The minimal124
input is **two bytes, `(}`**, and it fatal-OOMs a plain `.sgl` file through the125
ordinary compiler front end — no `#lang sigil/doc`, no delegation, no126
substitution involved. `(]` by contrast produces a clean127
`mismatched brackets` error, which is why the shape is easy to miss.128
A `.sgd` manuscript reaches it the same way any Sigil source does. Filed129
against `(sigil syntax)`; recorded here only so a consumer indexing untrusted130
manuscripts knows that **no bound at the parser layer prevents it** and that131
process-level limits are the answer.133
## A note on `text` and `expr` as command names135
The parser's internal tree represents prose and `@(...)` forms as `text` and136
`expr` pseudo-nodes. `@text{...}` and `@expr{...}` are *legal author commands*137
whose names collide with those. This API distinguishes them structurally, so an138
author's `@text{...}` is reported as a `command` named `text`, never as prose.140
## Consumers142
folio's passive indexer (phase F2 of the folio/doc plan) is the first consumer:143
it indexes the built-in vocabulary declaratively from this tree and records144
custom author abstractions as opaque spans, rather than evaluating a document to145
learn its graph.