AtlatestRenderedmarkdown

sigil-doc / tree / docspassive-parse-api.md

Read the source

Readme

The passive parse API

(sigil doc inspect) exposes a #lang sigil/doc manuscript's raw command tree with exact spans, obtained without evaluating any author code.

It exists because the only other way out of the parser is parse-document-source / parse-document-forms, which return lowered generated Sigil syntax: to learn a document's structure through those you must evaluate the author forms they produce. A consumer that indexes documents it does not trust, or documents that are half-written, cannot do that.

(import (sigil doc inspect) (sigil fs) (sigil language))

(define source (read-file-string "notes/topics/example.sgd"))
(define header (inspect-language-source source "notes/topics/example.sgd"))
(define tree
  (inspect-document-source source
                           "notes/topics/example.sgd"
                           (language-header-end header)))

What you get

inspect-document-source returns a document tree:

accessormeaning
document-tree-source-namethe name the manuscript was parsed under
document-tree-header-endthe byte offset where the #lang header ended
document-tree-rootstop-level nodes, in source order
document-tree-index-stepsscalars visited building the location index (one pass; equals the manuscript's scalar length)

Each node is a tree node:

accessormeaning
tree-node-kindcommand, text, or expr
tree-node-namethe command's name as a symbol; #f for text / expr
tree-node-argumentsthe [...] group as a list of unevaluated syntax objects; () when absent
tree-node-childrenchild nodes, in source order
tree-node-body-kindbrace, line, or named; #f for text / expr
tree-node-textfolded prose for a text node (see the warning below); #f otherwise
tree-node-expressionthe unevaluated syntax of an @(...) form; #f otherwise
tree-node-start, tree-node-endscalar indices, half-open
tree-node-locationan srcloc: UTF-8 byte position and byte span, plus scalar line and column of the start
tree-node-end-line, tree-node-end-columnscalar line and column of the end

Spans are exact and total. For every node, (substring source (tree-node-start node) (tree-node-end node)) is precisely the manuscript text that node was scanned from; children are contained within their parent and never overlap each other.

Both coordinate systems are given because consumers need both: byte offsets address the file, scalar indices address the string, and the two diverge as soon as a multi-byte character precedes a node.

Per-field spans

A node's arguments are ordinary Sigil syntax, so each carries its own srcloc addressing the original manuscript. A consumer that needs a single field's extent — a task's status separately from its body — reads it from the argument, not from the node:

(let* ((argument (car (tree-node-arguments node)))
       (loc (syntax-srcloc argument)))
  (substring source (srcloc-position loc)
                    (+ (srcloc-position loc) (srcloc-span loc))))

Guarantees

  • No evaluation. No author expression runs, no author-defined command runs, no author macro expands, no @code body runs, and the author runtime is never loaded. (sigil doc inspect) does not import (sigil doc author), (sigil doc model), or (sigil doc private author-runtime).
  • No lowering. Nothing is rewritten into generated Sigil syntax and no semantic model record is constructed.
  • No root-container policy. "Exactly one @chapter" is a rule the lowering path enforces, not a fact about the grammar. A manuscript rooted at @note or @project, or with several roots, inspects normally. (The lowering path is unchanged and still refuses it; parameterizing the root container is SD-P1.)
  • Reentrant. All parser state lives in a per-parse value threaded as an argument. Nested and interleaved parses cannot observe each other.

Explicit non-guarantees

  • Malformed source raises. The passive path raises exactly the error the ordinary parse path raises, with the identical message and location. There is deliberately no diagnostic-collecting mode here: guard the call, or build the wrapper in the consumer that knows what a diagnostic should mean. The parser's bounds (nesting 64, commands 8192) raise the same way.
  • tree-node-text is folded prose, not the raw slice. Single newlines become spaces and body-edge whitespace is trimmed. Inside a raw @code body this is not the content — there, the authoritative text is the source slice named by the node's span, which is also what the lowering path uses. When in doubt, slice the source; the spans are exact.
  • The command vocabulary is not interpreted. @chapter, @note and a command an author defined five lines earlier are all just commands with names, arguments and children. Deciding which are meaningful is the consumer's job, and a consumer that only understands a built-in vocabulary should report unknown commands as opaque rather than dropping them.
  • No revision or content identity. The API is a pure function of the source text you hand it; a consumer that needs a revision hashes the same bytes.
  • Copying is one level deep. Treat syntax objects and locations as immutable. List spines and strings are copied on access; the elements of an argument list are syntax objects shared with the parser, and tree-node-location returns a shared srcloc. Sigil pairs are mutable, so a consumer that mutates an argument's datum corrupts the view of anything else holding the same tree. Two separate inspect-document-source calls share nothing (each re-parses), so this only bites a consumer that caches one tree and hands it to several readers. Deep-copying syntax on every access would make traversal pay for a hazard no in-tree consumer has.
  • The argument-group bound does not bound the cost of a single read. read-arguments caps how many candidate ] boundaries it will try (65536 — a backstop, deliberately far above any real manuscript, because the candidates are counted over the rest of the file rather than the group). It does not limit the work inside one delegated read.

Concretely, and this is not a sigil-doc defect: the standard reader exhausts memory on a } appearing inside an unclosed ( or [. The minimal input is two bytes, (}, and it fatal-OOMs a plain .sgl file through the ordinary compiler front end — no #lang sigil/doc, no delegation, no substitution involved. (] by contrast produces a clean mismatched brackets error, which is why the shape is easy to miss. A .sgd manuscript reaches it the same way any Sigil source does. Filed against (sigil syntax); recorded here only so a consumer indexing untrusted manuscripts knows that no bound at the parser layer prevents it and that process-level limits are the answer.

A note on text and expr as command names

The parser's internal tree represents prose and @(...) forms as text and expr pseudo-nodes. @text{...} and @expr{...} are legal author commands whose names collide with those. This API distinguishes them structurally, so an author's @text{...} is reported as a command named text, never as prose.

Consumers

folio's passive indexer (phase F2 of the folio/doc plan) is the first consumer: it indexes the built-in vocabulary declaratively from this tree and records custom author abstractions as opaque spans, rather than evaluating a document to learn its graph.