Census: a failed metadata query or a projects/ symlink keeps the full census

Message

sqlite-query returns '() when prepare fails and a partial result when a step fails, so a failed notes_meta or link_sources query looked like "no rows" and let the fast path arm on a census that never saw them. The census now checks sqlite-errmsg after each metadata query and raises, which records a retryable failure.

The projects/ walk reads dotfiles, so a hidden symlink there is followed; only notes/ keeps the exemption for links under a dot component.

The task-notes parity tool now runs with the cache settle window off, so a wrong cache hit on a changed file can show.

Changed

 CHANGELOG.md                      |  2 +-
 src/folio/index.sgl               | 27 ++++++++++++++++++++-------
 test/test-idle-census.sgl         | 27 +++++++++++++++++++++++++++
 tools/check-task-notes-parity.sgl | 13 +++++++++++++
 4 files changed, 61 insertions(+), 8 deletions(-)

Diff

CHANGELOG.mdmodified

@@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [0.3.2] - Unreleased

−10

- Skip the per-file census on idle ticks. Each 15 s tick compares one native metadata listing of `notes/` and `projects/` (size, mtime-ns and ctime-ns of every entry) and SQLite's `data_version` with the last complete census, and runs the full census only when something differs, another connection committed, or a retryable failure is pending. A listing is trusted only once its newest mtime and ctime are 2 s old, so writes inside one coarse timestamp granule cannot hide. A visible symlink, or a listing line that does not parse as 7 fields, disables the skip. On a 4,467-note corpus an idle tick drops from 216-611 ms of CPU to about 24 ms.

+10

- Skip the per-file census on idle ticks. Each 15 s tick compares one native metadata listing of `notes/` and `projects/` (size, mtime-ns and ctime-ns of every entry) and SQLite's `data_version` with the last complete census, and runs the full census only when something differs, another connection committed, or a retryable failure is pending. A listing is trusted only once its newest mtime and ctime are 2 s old, so writes inside one coarse timestamp granule cannot hide. A symlink the walk would follow (any in `projects/`, any not under a dot component in `notes/`), a metadata query that fails, or a listing line that does not parse as 7 fields disables the skip. On a 4,467-note corpus an idle tick drops from 216-611 ms of CPU to about 24 ms.

- Yield every 32 files during a full census, so a large outside change no longer holds a shared HTTP server's only thread while it is reconciled.

- Reuse parsed project files, the parsed inbox and buried-id scans while each file's `stat` signature (dev, inode, size, mtime-ns, ctime-ns) is unchanged. A file is cached only once its mtime and ctime are 2 s old; a recently written file is parsed on every call. `tasks`, `projects` and `status` no longer re-parse every project file on each call.

- Answer `task-notes` from cached note frontmatter, keyed the same way, instead of parsing every note on each call (6.5 s on the corpus above). The answer is still computed from the files at call time.

src/folio/index.sglmodified

@@ -401,10 +401,21 @@

401

(unless (eq? value 'deferred) (forget-failure! idx key)))

402

#t))

403

+404

;; `sqlite-query` returns '() when prepare fails and the rows read so far

+405

;; when a step fails; only the connection's errmsg tells either from a real

+406

;; result. A failed query must raise here, so safe-index-item! records a

+407

;; retryable failure and the fast path cannot arm on a census that never

+408

;; saw the rows.

409

(define (safe-index-query idx label sql)

410

(let ((rows '()))

411

(safe-index-item! idx label

−407

(lambda () (set! rows (sqlite-query (folio-index-db idx) sql))))

+412

(lambda ()

+413

(let* ((db (folio-index-db idx))

+414

(result (sqlite-query db sql))

+415

(message (sqlite-errmsg db)))

+416

(unless (equal? message "not an error")

+417

(error (format "folio index: ~a failed: ~a" label message)))

+418

(set! rows result))))

419

rows))

420

421

(define (failed-content-unchanged? idx path signature)

@@ -556,8 +567,8 @@

567

;; defers until the file's signature changes, which changes the listing.

568

(define (armable? idx state captured-seconds)

569

(let ((cutoff-ns (* (- captured-seconds (census-settle-seconds)) 1000000000)))

−559

(and (listing-settled? (car state) cutoff-ns)

−560

(listing-settled? (cadr state) cutoff-ns)

+570

(and (listing-settled? (car state) cutoff-ns #t)

+571

(listing-settled? (cadr state) cutoff-ns #f)

572

(let ((failures (vector-ref (folio-index-failed-fingerprints idx) 0)))

573

(let loop ((keys (dict-keys failures)))

574

(or (null? keys)

@@ -567,9 +578,11 @@

578

;; Every line must parse as exactly 7 fields, so a name holding a newline

579

;; or a tab cannot hide an entry; no line may be a symlink the store walk

580

;; would follow (the walk follows them, the listing does not show their

−570

;; target's changes; the walk skips every dot component); and every

−571

;; mtime and ctime must be older than cutoff-ns.

−572

(define (listing-settled? listing cutoff-ns)

+581

;; target's changes); and every mtime and ctime must be older than

+582

;; cutoff-ns. hidden-exempt? is true only for notes/, whose walk skips

+583

;; every dot component; the projects/ walk reads dotfiles too, so any

+584

;; symlink there counts.

+585

(define (listing-settled? listing cutoff-ns hidden-exempt?)

586

(let loop ((lines (string-split listing "\n")))

587

(cond

588

((null? lines) #t)

@@ -578,7 +591,7 @@

591

(let ((fields (string-split (car lines) "\t")))

592

(and (= (length fields) 7)

593

(not (and (equal? (cadr fields) "l")

−581

(not (any-hidden-component? (car fields)))))

+594

(not (and hidden-exempt? (any-hidden-component? (car fields))))))

595

(let ((mtime (string->number (list-ref fields 4)))

596

(ctime (string->number (list-ref fields 5))))

597

(and mtime ctime (< mtime cutoff-ns) (< ctime cutoff-ns)))

test/test-idle-census.sglmodified

@@ -220,6 +220,33 @@

220

(assert-equal '(("source" . note)) (index-backlinks idx "target"))

221

(assert-equal 'unchanged (index-census-tick! idx))))))

222

+223

;; `sqlite-query` answers '() when the statement cannot run; the census must

+224

;; not take that for "no rows" and arm on it.

+225

(test "a read query that fails keeps every tick on the full census"

+226

(with-census-store

+227

(lambda (root store idx source)

+228

(index-census-tick! idx)

+229

(assert-equal 'unchanged (index-census-tick! idx))

+230

(let ((other (sqlite-open (path-join root ".folio-index.db"))))

+231

(assert-true (sqlite-exec other "ALTER TABLE notes_meta RENAME TO notes_meta_away"))

+232

(assert-equal 'reconciled (index-census-tick! idx))

+233

(assert-equal 'reconciled (index-census-tick! idx))

+234

(assert-true (sqlite-exec other "ALTER TABLE notes_meta_away RENAME TO notes_meta"))

+235

(sqlite-close other))

+236

(assert-equal 'reconciled (index-census-tick! idx))

+237

(assert-equal 'unchanged (index-census-tick! idx)))))

+238

+239

;; The projects/ walk reads dotfiles, so a hidden link there is followed and

+240

;; its target's edits are invisible to the listing.

+241

(test "any symlink in projects/, hidden or not, disables the fast path"

+242

(with-census-store

+243

(lambda (root store idx source)

+244

(let ((outside (path-join root "plan.md")))

+245

(write-file-string outside "# Plan\n")

+246

(sh "ln" "-s" outside (path-join (store-project-dir store) ".plan.md"))

+247

(assert-equal 'reconciled (index-census-tick! idx))

+248

(assert-equal 'reconciled (index-census-tick! idx))))))

+249

250

(test "an unsettled listing is not trusted until its newest time is old enough"

251

(with-settle 1

252

(lambda ()

tools/check-task-notes-parity.sglmodified

@@ -6,6 +6,7 @@

;;; change, an outside frontmatter edit, an outside note move, a note-rename

;;; through the tool, a task-move through the tool, and a deletion.

(import (sigil fs)

(only (sigil async) sleep)

(sigil list)

(sigil path)

(sigil string)

@@ -18,8 +19,15 @@

(folio task)

(folio query)

(folio index)

+22

(folio filecache)

(folio tools))

+25

;; With the settle window on, a file changed by a round is never a cache hit

+26

;; in that round, so a wrong hit could not show. Run with it off, and wait one

+27

;; timestamp granule before each mutation so the signatures can differ.

+28

(file-cache-settle-seconds 0)

+29

(define (granule) (sleep 0.05))

+30

(define root (getenv "FOLIO_CORPUS"))

(unless root (error "FOLIO_CORPUS must name a disposable corpus copy"))

(when (string-contains? root "/Ops/folio") (error "refusing to run against a live store"))

@@ -105,6 +113,7 @@

113

;; Outside edit: add a task id to a different, unlinked note's frontmatter.

114

(define plain

115

(car (filter (lambda (n) (null? (note-tasks n))) (all-notes store))))

+116

(granule)

117

(write-file-string (folio-note-file-path plain)

118

(string-append "---\ntasks:\n - " linking-task "\n---\n\n" (folio-note-body plain)))

119

(round! "outside-edit")

@@ -112,16 +121,19 @@

121

;; Outside move of the linking note into a new directory.

122

(define moved-path (path-join (store-notes-dir store) "parity-moved" "moved-note.md"))

123

(ensure-directory (path-join (store-notes-dir store) "parity-moved"))

+124

(granule)

125

(rename-file (folio-note-file-path linking) moved-path)

126

(round! "outside-move")

127

128

;; Rename through the tool.

+129

(granule)

130

(let ((reply (tool "folio/note-rename" (dict name: "parity-moved/moved-note"

131

new-name: "parity-moved/renamed-note"))))

132

(unless (string-contains? reply "Renamed") (error "note-rename failed" reply)))

133

(round! "tool-rename")

134

135

;; Move a linked task to another project through the tool (when it is a project task).

+136

(granule)

137

(let* ((task (find (lambda (t) (equal? (folio-task-id t) linking-task)) (all-tasks store)))

138

(projects (store-project-files store))

139

(target (and task

@@ -135,6 +147,7 @@

147

(round! "tool-task-move")

148

149

;; Deletion.

+150

(granule)

151

(delete-file (path-join (store-notes-dir store) "parity-moved" "renamed-note.md"))

152

(round! "delete")

153