AtlatestRepositorysigil-doc

sigil-doc / tree / docspassive-parse-api.md

Read it rendered

1# The passive parse API
2
3`(sigil doc inspect)` exposes a `#lang sigil/doc` manuscript's **raw command
4tree with exact spans**, obtained **without evaluating any author code**.
5
6It exists because the only other way out of the parser is
7`parse-document-source` / `parse-document-forms`, which return *lowered*
8generated Sigil syntax: to learn a document's structure through those you must
9evaluate the author forms they produce. A consumer that indexes documents it
10does not trust, or documents that are half-written, cannot do that.
12```scheme
13(import (sigil doc inspect) (sigil fs) (sigil language))
15(define source (read-file-string "notes/topics/example.sgd"))
16(define header (inspect-language-source source "notes/topics/example.sgd"))
17(define tree
18 (inspect-document-source source
19 "notes/topics/example.sgd"
20 (language-header-end header)))
21```
23## What you get
25`inspect-document-source` returns a **document tree**:
27| accessor | meaning |
28| --- | --- |
29| `document-tree-source-name` | the name the manuscript was parsed under |
30| `document-tree-header-end` | the byte offset where the `#lang` header ended |
31| `document-tree-roots` | top-level nodes, in source order |
32| `document-tree-index-steps` | scalars visited building the location index (one pass; equals the manuscript's scalar length) |
34Each node is a **tree node**:
36| accessor | meaning |
37| --- | --- |
38| `tree-node-kind` | `command`, `text`, or `expr` |
39| `tree-node-name` | the command's name as a symbol; `#f` for `text` / `expr` |
40| `tree-node-arguments` | the `[...]` group as a list of **unevaluated** syntax objects; `()` when absent |
41| `tree-node-children` | child nodes, in source order |
42| `tree-node-body-kind` | `brace`, `line`, or `named`; `#f` for `text` / `expr` |
43| `tree-node-text` | folded prose for a `text` node (see the warning below); `#f` otherwise |
44| `tree-node-expression` | the **unevaluated** syntax of an `@(...)` form; `#f` otherwise |
45| `tree-node-start`, `tree-node-end` | **scalar** indices, half-open |
46| `tree-node-location` | an `srcloc`: **UTF-8 byte** position and byte span, plus scalar line and column of the start |
47| `tree-node-end-line`, `tree-node-end-column` | scalar line and column of the end |
49Spans are exact and total. For every node,
50`(substring source (tree-node-start node) (tree-node-end node))` is precisely
51the manuscript text that node was scanned from; children are contained within
52their parent and never overlap each other.
54Both coordinate systems are given because consumers need both: byte offsets
55address the file, scalar indices address the string, and the two diverge as soon
56as a multi-byte character precedes a node.
58### Per-field spans
60A node's **arguments are ordinary Sigil syntax**, so each carries its own
61`srcloc` addressing the original manuscript. A consumer that needs a single
62field's extent — a task's `status` separately from its body — reads it from the
63argument, not from the node:
65```scheme
66(let* ((argument (car (tree-node-arguments node)))
67 (loc (syntax-srcloc argument)))
68 (substring source (srcloc-position loc)
69 (+ (srcloc-position loc) (srcloc-span loc))))
70```
72## Guarantees
74- **No evaluation.** No author expression runs, no author-defined command runs,
75 no author macro expands, no `@code` body runs, and the author runtime is never
76 loaded. `(sigil doc inspect)` does not import `(sigil doc author)`,
77 `(sigil doc model)`, or `(sigil doc private author-runtime)`.
78- **No lowering.** Nothing is rewritten into generated Sigil syntax and no
79 semantic model record is constructed.
80- **No root-container policy.** "Exactly one `@chapter`" is a rule the
81 *lowering* path enforces, not a fact about the grammar. A manuscript rooted at
82 `@note` or `@project`, or with several roots, inspects normally. (The lowering
83 path is unchanged and still refuses it; parameterizing the root container is
84 SD-P1.)
85- **Reentrant.** All parser state lives in a per-parse value threaded as an
86 argument. Nested and interleaved parses cannot observe each other.
88## Explicit non-guarantees
90- **Malformed source raises.** The passive path raises exactly the error the
91 ordinary parse path raises, with the identical message and location. There is
92 deliberately no diagnostic-collecting mode here: guard the call, or build the
93 wrapper in the consumer that knows what a diagnostic should mean. The parser's
94 bounds (nesting 64, commands 8192) raise the same way.
95- **`tree-node-text` is folded prose, not the raw slice.** Single newlines
96 become spaces and body-edge whitespace is trimmed. **Inside a raw `@code`
97 body this is not the content** — there, the authoritative text is the source
98 slice named by the node's span, which is also what the lowering path uses.
99 When in doubt, slice the source; the spans are exact.
100- **The command vocabulary is not interpreted.** `@chapter`, `@note` and a
101 command an author defined five lines earlier are all just commands with names,
102 arguments and children. Deciding which are meaningful is the consumer's job,
103 and a consumer that only understands a built-in vocabulary should report
104 unknown commands as opaque rather than dropping them.
105- **No revision or content identity.** The API is a pure function of the source
106 text you hand it; a consumer that needs a revision hashes the same bytes.
107- **Copying is one level deep. Treat syntax objects and locations as
108 immutable.** List spines and strings are copied on access; the *elements* of
109 an argument list are syntax objects shared with the parser, and
110 `tree-node-location` returns a shared `srcloc`. Sigil pairs are mutable, so a
111 consumer that mutates an argument's datum corrupts the view of anything else
112 holding the same tree. Two separate `inspect-document-source` calls share
113 nothing (each re-parses), so this only bites a consumer that caches one tree
114 and hands it to several readers. Deep-copying syntax on every access would
115 make traversal pay for a hazard no in-tree consumer has.
116- **The argument-group bound does not bound the cost of a single read.**
117 `read-arguments` caps how many candidate `]` boundaries it will try
118 (65536 — a backstop, deliberately far above any real manuscript, because the
119 candidates are counted over the rest of the *file* rather than the group). It
120 does **not** limit the work inside one delegated read.
122 Concretely, and **this is not a sigil-doc defect**: the standard reader
123 exhausts memory on a `}` appearing inside an unclosed `(` or `[`. The minimal
124 input is **two bytes, `(}`**, and it fatal-OOMs a plain `.sgl` file through the
125 ordinary compiler front end — no `#lang sigil/doc`, no delegation, no
126 substitution involved. `(]` by contrast produces a clean
127 `mismatched brackets` error, which is why the shape is easy to miss.
128 A `.sgd` manuscript reaches it the same way any Sigil source does. Filed
129 against `(sigil syntax)`; recorded here only so a consumer indexing untrusted
130 manuscripts knows that **no bound at the parser layer prevents it** and that
131 process-level limits are the answer.
133## A note on `text` and `expr` as command names
135The parser's internal tree represents prose and `@(...)` forms as `text` and
136`expr` pseudo-nodes. `@text{...}` and `@expr{...}` are *legal author commands*
137whose names collide with those. This API distinguishes them structurally, so an
138author's `@text{...}` is reported as a `command` named `text`, never as prose.
140## Consumers
142folio's passive indexer (phase F2 of the folio/doc plan) is the first consumer:
143it indexes the built-in vocabulary declaratively from this tree and records
144custom author abstractions as opaque spans, rather than evaluating a document to
145learn its graph.