Commit31f4d6e9Recorded24 Oct 2026Repositoryfolio
Make an idle folio server nearly free; cache per-call parses (0.3.2)
Message
Census fast path: each 15 s tick takes one native listing of notes/ and projects/ plus PRAGMA data_version and skips the per-file census when both match the state captured before the last complete census. It arms only when every listing line has 7 fields, no symlink the walk follows is present, every mtime and ctime is at least 2 s old (git's racily-clean rule), and no retryable failure is pending. The full census yields every 32 files. An idle tick drops from 216-611 ms to about 24 ms.
Parse cache (filecache.sgl): project files, the inbox, the buried-id integrity scan and task-notes frontmatter are cached per path, keyed on (dev inode size mtime-ns ctime-ns), and stored only when the signature is stable across the compute and the file is 2 s old.
No index or file format change: 0.3.1 and 0.3.2 share a store.
Tests: test/test-idle-census.sgl (19), test/test-census-bound.mjs (an outside edit reaches backlinks within 17 s of an idle server), and tools/check-task-notes-parity.sgl (differential check on a corpus copy).
Changed
CHANGELOG.md | 8 +++
package.sgl | 2 +-
src/folio/filecache.sgl | 89 ++++++++++++++++++++++++++++++
src/folio/inbox.sgl | 21 ++++---
src/folio/index.sgl | 129 ++++++++++++++++++++++++++++++++++++++++++-
src/folio/main.sgl | 4 +-
src/folio/project.sgl | 17 +++++-
src/folio/query.sgl | 2 +-
src/folio/tools.sgl | 94 +++++++++++++++++++++----------
test/test-census-bound.mjs | 117 +++++++++++++++++++++++++++++++++++++++
test/test-idle-census.sgl | 478 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
tools/check-task-notes-parity.sgl | 142 +++++++++++++++++++++++++++++++++++++++++++++++
12 files changed, 1059 insertions(+), 44 deletions(-)Diff
CHANGELOG.mdmodified
@@ -5,6 +5,14 @@ All notable changes to **folio** are documented in this file.
5
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), 6
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7
+8
## [0.3.2] - Unreleased+9
+10
- Skip the per-file census on idle ticks. Each 15 s tick compares one native metadata listing of `notes/` and `projects/` (size, mtime-ns and ctime-ns of every entry) and SQLite's `data_version` with the last complete census, and runs the full census only when something differs, another connection committed, or a retryable failure is pending. A listing is trusted only once its newest mtime and ctime are 2 s old, so writes inside one coarse timestamp granule cannot hide. A visible symlink, or a listing line that does not parse as 7 fields, disables the skip. On a 4,467-note corpus an idle tick drops from 216-611 ms of CPU to about 24 ms.+11
- Yield every 32 files during a full census, so a large outside change no longer holds a shared HTTP server's only thread while it is reconciled.+12
- Reuse parsed project files, the parsed inbox and buried-id scans while each file's `stat` signature (dev, inode, size, mtime-ns, ctime-ns) is unchanged. A file is cached only once its mtime and ctime are 2 s old; a recently written file is parsed on every call. `tasks`, `projects` and `status` no longer re-parse every project file on each call.+13
- Answer `task-notes` from cached note frontmatter, keyed the same way, instead of parsing every note on each call (6.5 s on the corpus above). The answer is still computed from the files at call time.+14
- No index or file format change: 0.3.1 and 0.3.2 processes share one store and index.+15
16
## [0.3.1] - 2026-10-22 17
18
- Recheck file existence inside census removal transactions and forget vanished failure entries. Require offline migration before the first upgraded server start.package.sglmodified
@@ -6,7 +6,7 @@
6
7
(package 8
name: "folio"−9
version: "0.3.1"+9
version: "0.3.2" 10
sigil: "0.22.15" 11
description: "Markdown-native knowledge and project management via MCP" 12
url: "https://codeberg.org/sigil/folio"src/folio/filecache.sgladded
@@ -0,0 +1,89 @@
+1
;;; (folio filecache) - reuse a value computed from a file while the file is unchanged.+2
;;;+3
;;; Read-only tools used to re-parse every project file (and, for task-notes,+4
;;; every note) on every call: 0.6-6.5 s of CPU per call on the 4,467-note+5
;;; corpus, during which a shared HTTP server answers nobody else. A cache+6
;;; entry is keyed on the file's path and reused only while one `stat` still+7
;;; reports the same (dev inode size mtime-ns ctime-ns):+8
;;;+9
;;; - an editor write in the same second moves mtime-ns and ctime-ns;+10
;;; - `touch -d` can restore mtime but never ctime, which every write moves;+11
;;; - a replacement (git checkout, write-file-atomic's rename) is a new inode+12
;;; with a new ctime, even when size and mtime match the old file.+13
;;;+14
;;; File times come from a coarse clock (a few ms on Linux, whole seconds on+15
;;; some filesystems), so two same-size writes inside one granule can share+16
;;; every field. A value is therefore cached only when the file's mtime and+17
;;; ctime are both at least file-cache-settle-seconds old (git's "racily+18
;;; clean" rule): any later write must then move a time. A recently written+19
;;; file is simply parsed again on each call until it settles.+20
;;;+21
;;; The value is cached only when the signature is the same before and after+22
;;; computing it, so a file written while it is being read is never cached+23
;;; with a signature that does not describe the bytes that were parsed.+24
;;; Cached values are shared between callers and must be treated as+25
;;; immutable (folio's structs, dicts and lists are).+26
(define-library (folio filecache)+27
(import (sigil fs)+28
(sigil string)+29
(sigil time))+30
(export make-file-cache+31
file-cache-ref+32
file-cache-size+33
file-cache-prune!+34
file-cache-settle-seconds)+35
(begin+36
+37
;;; How old a file's newest time must be before its value is cached.+38
;;; Tests set it to 0 with (file-cache-settle-seconds 0).+39
(define file-cache-settle-seconds (make-parameter 2))+40
+41
;; signature = (dev inode size mtime-ns ctime-ns)+42
(define (settled? signature)+43
(let ((cutoff-ns (* (- (current-second) (file-cache-settle-seconds)) 1000000000)))+44
(and (< (list-ref signature 3) cutoff-ns)+45
(< (list-ref signature 4) cutoff-ns))))+46
+47
;; A cache is a box holding a dict: path -> (signature . value).+48
(define (make-file-cache)+49
(: -> vector?)+50
(vector (dict)))+51
+52
;;; Return (compute path), reusing the value from an earlier call while+53
;;; the file's stat signature is unchanged. A missing file is never cached.+54
;;; Errors from compute propagate and cache nothing.+55
(define (file-cache-ref cache path compute)+56
(: vector? string? procedure? -> any?)+57
(let ((before (file-stat-fingerprint path)))+58
(if (not before)+59
(compute path)+60
(let ((hit (dict-ref (vector-ref cache 0) path #f)))+61
(if (and hit (equal? (car hit) before))+62
(cdr hit)+63
(let* ((value (compute path))+64
(after (file-stat-fingerprint path)))+65
(when (and (equal? before after) (settled? before))+66
(vector-set! cache 0+67
(dict-set (vector-ref cache 0) path (cons before value))))+68
value))))))+69
+70
(define (file-cache-size cache)+71
(: vector? -> integer?)+72
(length (dict-keys (vector-ref cache 0))))+73
+74
;;; Drop the entries under `dir` whose path is not in `live-paths`, so a+75
;;; deleted or renamed file does not stay in memory. Entries under other+76
;;; directories (another store in the same process) are kept.+77
(define (file-cache-prune! cache dir live-paths)+78
(: vector? string? list? -> void?)+79
(let ((prefix (if (string-ends-with? dir "/") dir (string-append dir "/")))+80
(live (dict))+81
(entries (vector-ref cache 0)))+82
(for-each (lambda (path) (set! live (dict-set live path #t))) live-paths)+83
(for-each+84
(lambda (path)+85
(when (and (string-starts-with? path prefix)+86
(not (dict-contains? live path)))+87
(set! entries (dict-remove entries path))))+88
(dict-keys entries))+89
(vector-set! cache 0 entries)))))src/folio/inbox.sglmodified
@@ -25,7 +25,8 @@
25
(sigil fs) 26
(folio store) 27
(folio task)−28
(folio id))+28
(folio id)+29
(folio filecache)) 30
(export read-inbox 31
inbox-add! 32
inbox-remove-by-id!@@ -46,14 +47,20 @@
47
;;; Read all tasks from the inbox. 48
;;; 49
;;; Returns a list of folio-task structs with source-file set.+50
;;; The parse is reused while inbox.md's stat signature is unchanged; see+51
;;; (folio filecache). 52
(define (read-inbox store) 53
(: folio-store? -> list?)−51
(let ((path (store-inbox-path store)))−52
(if (file-exists? path)−53
(let* ((content (read-file-string path))−54
(lines (string-split content "\n")))−55
(map task-block-task (find-task-blocks lines path)))−56
'())))+54
(file-cache-ref inbox-cache (store-inbox-path store) parse-inbox-file))+55
+56
(define inbox-cache (make-file-cache))+57
+58
(define (parse-inbox-file path)+59
(if (file-exists? path)+60
(let* ((content (read-file-string path))+61
(lines (string-split content "\n")))+62
(map task-block-task (find-task-blocks lines path)))+63
'())) 64
65
;;; Add a task to the inbox. 66
;;;src/folio/index.sglmodified
@@ -14,6 +14,7 @@
14
(sigil time) 15
(sigil log) 16
(sigil sqlite)+17
(only (sigil async) yield in-async-context?) 18
(folio store) 19
(folio note)) 20
(export make-folio-index@@ -21,6 +22,8 @@
22
folio-index-db 23
24
index-reconcile!+25
index-census-tick!+26
census-settle-seconds 27
index-watch! 28
index-rebuild! 29
index-migrate!@@ -42,7 +45,9 @@
45
(store) 46
(fingerprints) 47
(failed-fingerprints)−45
(transaction-mutex))+48
(transaction-mutex)+49
;; Box: what the last complete census saw (see census-state), or #f.+50
(census-state)) 51
52
(define (checked-sql result) 53
(unless result (error "Folio index SQL operation failed")))@@ -141,7 +146,8 @@
146
(folio-index db: db store: store 147
fingerprints: (vector versions) 148
failed-fingerprints: (vector (dict))−144
transaction-mutex: mutex))))+149
transaction-mutex: mutex+150
census-state: (vector #f))))) 151
152
;; ============================================================ 153
;; Write-through operations@@ -419,6 +425,7 @@
425
(store (folio-index-store idx)) 426
(notes-dir (store-notes-dir store)) 427
(files (store-note-files store))+428
(pace (make-census-pacer)) 429
(versions (dict)) 430
(seen (dict)) 431
(note-names (dict))@@ -433,6 +440,7 @@
440
(query "query:link-sources" "SELECT path, fingerprint FROM link_sources")) 441
(for-each 442
(lambda (path)+443
(pace) 444
(let ((name (note-relative-name path notes-dir))) 445
(set! seen (dict-set seen path #t)) 446
(set! note-names (dict-set note-names name #t))@@ -458,6 +466,7 @@
466
(query "query:notes" "SELECT name FROM notes_meta")) 467
(for-each 468
(lambda (path)+469
(pace) 470
(set! seen (dict-set seen path #t)) 471
(set! project-paths (dict-set project-paths path #t)) 472
(safe-index-item! idx path@@ -489,15 +498,129 @@
498
(forget-failure! idx key))) 499
(dict-keys (vector-ref box 0)))))) 500
+501
;; A full census over thousands of files must not hold the server's only+502
;; thread: when it runs as a task, it yields every census-yield-every+503
;; files so queued requests are answered between slices. Outside a+504
;; scheduler (startup, --migrate-index) it never yields.+505
(define census-yield-every 32)+506
+507
(define (make-census-pacer)+508
(let ((n 0))+509
(lambda ()+510
(set! n (+ n 1))+511
(when (>= n census-yield-every)+512
(set! n 0)+513
(when (in-async-context?) (yield))))))+514
+515
;; ============================================================+516
;; Census fast path+517
;; ============================================================+518
+519
;;; Seconds a listing's newest mtime and ctime must lie in the past before+520
;;; the fast path may trust it. File times come from a coarse clock (a+521
;;; few ms on Linux, whole seconds on some filesystems), so a second+522
;;; same-size write, or an atomic replace (the listing has no inode),+523
;;; inside one granule can leave every listed field unchanged. Once every+524
;;; time is this old, any later write must move a time: git's "racily+525
;;; clean" rule. Tests set it to 0 with (census-settle-seconds 0).+526
(define census-settle-seconds (make-parameter 2))+527
+528
;; What a tick compares with the last complete census, or #f when it+529
;; cannot be read (the full census then runs, as it always did):+530
;;+531
;; - the native metadata listing of notes/ and of projects/: size,+532
;; mtime-ns and ctime-ns of every entry, directories included. An+533
;; edit moves the file's times even at the same size; `touch -d` can+534
;; reset mtime but not ctime; an add, remove, rename or replacement+535
;; (git checkout) also moves the parent directory's times. Any+536
;; difference runs the full census, which compares each file's own+537
;; (dev inode size mtime ctime).+538
;; - SQLite's data_version, which moves whenever ANOTHER connection+539
;; commits, so a row written by a different process (a stdio session,+540
;; an older folio) is always reconciled.+541
;;+542
;; file-tree-metadata does not follow symlinks, so an editor's dangling+543
;; lock link (.#note.md) cannot make it fail.+544
(define (census-state idx)+545
(guard (exn (else #f))+546
(let* ((store (folio-index-store idx))+547
(notes (file-tree-metadata (store-notes-dir store)))+548
(projects (file-tree-metadata (store-project-dir store)))+549
(row (sqlite-query-row (folio-index-db idx) "PRAGMA data_version")))+550
(and notes projects row+551
(list notes projects (cdr (assoc 'data_version row)))))))+552
+553
;; May the state captured at `captured-seconds`, before a complete+554
;; census, arm the fast path? Both listings must be settled+555
;; (listing-settled?) and every remaining failure must be one the census+556
;; defers until the file's signature changes, which changes the listing.+557
(define (armable? idx state captured-seconds)+558
(let ((cutoff-ns (* (- captured-seconds (census-settle-seconds)) 1000000000)))+559
(and (listing-settled? (car state) cutoff-ns)+560
(listing-settled? (cadr state) cutoff-ns)+561
(let ((failures (vector-ref (folio-index-failed-fingerprints idx) 0)))+562
(let loop ((keys (dict-keys failures)))+563
(or (null? keys)+564
(and (not (cadr (dict-ref failures (car keys))))+565
(loop (cdr keys)))))))))+566
+567
;; Every line must parse as exactly 7 fields, so a name holding a newline+568
;; or a tab cannot hide an entry; no line may be a symlink the store walk+569
;; would follow (the walk follows them, the listing does not show their+570
;; target's changes; the walk skips every dot component); and every+571
;; mtime and ctime must be older than cutoff-ns.+572
(define (listing-settled? listing cutoff-ns)+573
(let loop ((lines (string-split listing "\n")))+574
(cond+575
((null? lines) #t)+576
((equal? (car lines) "") (loop (cdr lines)))+577
(else+578
(let ((fields (string-split (car lines) "\t")))+579
(and (= (length fields) 7)+580
(not (and (equal? (cadr fields) "l")+581
(not (any-hidden-component? (car fields)))))+582
(let ((mtime (string->number (list-ref fields 4)))+583
(ctime (string->number (list-ref fields 5))))+584
(and mtime ctime (< mtime cutoff-ns) (< ctime cutoff-ns)))+585
(loop (cdr lines))))))))+586
+587
(define (any-hidden-component? relative-path)+588
(let loop ((parts (string-split relative-path "/")))+589
(and (pair? parts)+590
(or (string-starts-with? (car parts) ".")+591
(loop (cdr parts))))))+592
+593
;;; One census tick: skip the per-file census when nothing it reads can+594
;;; have changed since the last complete one, otherwise run it. Returns+595
;;; 'unchanged or 'reconciled.+596
(define (index-census-tick! idx)+597
(: folio-index? -> symbol?)+598
(let* ((captured (current-second))+599
(before (census-state idx))+600
(box (folio-index-census-state idx)))+601
(if (and before (equal? before (vector-ref box 0)))+602
'unchanged+603
(begin+604
(vector-set! box 0 #f)+605
(index-reconcile! idx)+606
;; Keep the state seen BEFORE this census, so a change made+607
;; while it ran differs at the next tick; and only when it is+608
;; armable, so an unsettled time or a retryable failure keeps+609
;; the next tick on the full census.+610
(when (and before (armable? idx before captured))+611
(vector-set! box 0 before))+612
'reconciled))))+613
614
(define (index-watch! idx) 615
(let loop () 616
(sleep 15) 617
(guard (exn 618
(else+619
(vector-set! (folio-index-census-state idx) 0 #f) 620
(log-error "Index refresh failed; retrying next tick" 621
error: (if (error-object? exn) (error-object-message exn) 622
(format "~s" exn)))))−500
(index-reconcile! idx))+623
(log-debug "Census tick" result: (index-census-tick! idx))) 624
(loop))) 625
626
;;; Rebuild by atomic document replacement. Readers keep the previoussrc/folio/main.sglmodified
@@ -279,7 +279,9 @@
279
(log-warn "Initial index refresh deferred; census will retry" 280
error: (if (error-object? exn) (error-object-message exn) 281
(format "~s" exn)))))−282
(index-reconcile! idx))+282
;; A full census (nothing is remembered yet) that also records+283
;; what it saw, so the first tick can already take the fast path.+284
(index-census-tick! idx)) 285
(if http-port 286
(begin 287
(with-asyncsrc/folio/project.sglmodified
@@ -38,7 +38,8 @@
38
(folio store) 39
(folio task) 40
(folio id)−41
(folio frontmatter))+41
(folio frontmatter)+42
(folio filecache)) 43
(export folio-project 44
folio-project? 45
folio-project-name@@ -48,6 +49,7 @@
49
folio-project-log-entries 50
51
read-project+52
read-project-cached 53
project-status 54
project-goal 55
project-area@@ -419,10 +421,21 @@
421
;; Queries 422
;; ============================================================ 423
+424
;; Parsed project files, reused while each file's stat signature is+425
;; unchanged. See (folio filecache) for why that is safe.+426
(define project-cache (make-file-cache))+427
+428
;;; read-project, reusing the parse of an unchanged file.+429
(define (read-project-cached file-path)+430
(: string? -> folio-project?)+431
(file-cache-ref project-cache file-path read-project))+432
433
;;; Load all projects as folio-project structs. 434
(define (all-projects store) 435
(: folio-store? -> list?)−425
(map read-project (store-project-files store)))+436
(let ((files (store-project-files store)))+437
(file-cache-prune! project-cache (store-project-dir store) files)+438
(map read-project-cached files))) 439
440
;;; Find a project file by name (filename without .md). 441
(define (find-project-file store name)src/folio/query.sglmodified
@@ -32,7 +32,7 @@
32
(project-tasks 33
(apply append 34
(map (lambda (f)−35
(folio-project-tasks (read-project f)))+35
(folio-project-tasks (read-project-cached f))) 36
project-files)))) 37
(append inbox-tasks project-tasks))) 38
src/folio/tools.sglmodified
@@ -23,7 +23,8 @@
23
(folio query) 24
(folio integrity) 25
(folio migrate)−26
(folio frontmatter))+26
(folio frontmatter)+27
(folio filecache)) 28
(export register-folio-tools! 29
never-silent 30
update-task-in-store!@@ -44,7 +45,8 @@
45
find-all-tasks-by-id 46
make-tool-task-move-batch 47
move-task-in-store!−47
integrity-warning-for-file)+48
integrity-warning-for-file+49
task-notes-text) 50
(begin 51
52
;; ============================================================@@ -503,6 +505,16 @@
505
;; comparison (a predicate on both sides made the withdrawn attempt's 506
;; check blind by construction). 507
(define (integrity-warning-for-file file-path label)+508
(map (lambda (occ)+509
(format " ~a line ~a: {id: ~a} is on disk but not visible as a task"+510
label (car occ) (cdr occ)))+511
(file-cache-ref integrity-cache file-path buried-occurrences)))+512
+513
;; The scan result depends only on the file's bytes, so it is reused+514
;; while the file's stat signature is unchanged; see (folio filecache).+515
(define integrity-cache (make-file-cache))+516
+517
(define (buried-occurrences file-path) 518
(if (not (file-exists? file-path)) 519
'() 520
(let* ((content (read-file-string file-path))@@ -515,10 +527,7 @@
527
(claimed (map (lambda (b) (cons (+ 1 (task-block-start b)) 528
(+ 1 (task-block-end b)))) 529
blocks)))−518
(map (lambda (occ)−519
(format " ~a line ~a: {id: ~a} is on disk but not visible as a task"−520
label (car occ) (cdr occ)))−521
(buried-task-ids content visible claimed)))))+530
(buried-task-ids content visible claimed)))) 531
532
(define (make-tool-inbox store) 533
(lambda (args)@@ -792,11 +801,12 @@
801
802
;; Integrity lines across everything a mutation can reach. 803
(define (store-integrity-lines store)−795
(apply append−796
(map (lambda (f)−797
(integrity-warning-for-file f (path-basename f)))−798
(cons (store-inbox-path store)−799
(store-project-files store)))))+804
(let ((files (store-project-files store)))+805
(file-cache-prune! integrity-cache (store-project-dir store) files)+806
(apply append+807
(map (lambda (f)+808
(integrity-warning-for-file f (path-basename f)))+809
(cons (store-inbox-path store) files))))) 810
811
(define (make-tool-status store idx) 812
(lambda (args)@@ -1021,28 +1031,54 @@
1031
(write-file-atomic file-path (string-join new-lines "\n")))))))) 1032
1033
;; Find all notes that reference a given task ID.+1034
;; task-notes parsed all 4,467 notes on every call (6.5 s on the real+1035
;; corpus). It needs only each note's name and frontmatter, so those are+1036
;; kept per file and reused while the file's stat signature is unchanged+1037
;; (see (folio filecache)). The answer is still computed from the files+1038
;; on disk at call time: a rename, an outside edit or a moved task is+1039
;; seen at the next call, with no index to keep in step.+1040
(define note-header-cache (make-file-cache))+1041
+1042
;; Only the frontmatter is cached: it depends on the bytes alone, while+1043
;; the note's name depends on which store's notes/ it is read under.+1044
(define (read-note-frontmatter path)+1045
(folio-note-frontmatter (read-note path)))+1046
1047
(define (find-notes-for-task store task-id)−1025
(filter (lambda (note)−1026
(member task-id (note-tasks note)))−1027
(all-notes store)))+1048
(let ((notes-dir (store-notes-dir store))+1049
(files (store-note-files store)))+1050
(file-cache-prune! note-header-cache notes-dir files)+1051
(filter (lambda (note)+1052
(member task-id (note-tasks note)))+1053
(map (lambda (path)+1054
(folio-note name: (note-relative-name path notes-dir)+1055
file-path: path+1056
frontmatter: (file-cache-ref note-header-cache path+1057
read-note-frontmatter)+1058
body: ""))+1059
files)))) 1060
1061
(define (make-tool-task-notes store) 1062
(lambda (args)−1031
(let* ((task-id (require-arg args id: "id"))−1032
(notes (find-notes-for-task store task-id)))−1033
(if (null? notes)−1034
(format "No notes linked to task ~a." task-id)−1035
(string-append−1036
(format "Notes linked to ~a (~a):\n\n" task-id (length notes))−1037
(string-join−1038
(map (lambda (n)−1039
(let ((tags (note-tags n)))−1040
(string-append−1041
(format " ~a" (folio-note-name n))−1042
(if (null? tags) ""−1043
(format " [~a]" (format-string-list tags ", "))))))−1044
notes)−1045
"\n"))))))+1063
(task-notes-text store (require-arg args id: "id"))))+1064
+1065
;;; The folio/task-notes reply for one task id. Exported so the parity+1066
;;; check can compare it with a full parse of every note.+1067
(define (task-notes-text store task-id)+1068
(let ((notes (find-notes-for-task store task-id)))+1069
(if (null? notes)+1070
(format "No notes linked to task ~a." task-id)+1071
(string-append+1072
(format "Notes linked to ~a (~a):\n\n" task-id (length notes))+1073
(string-join+1074
(map (lambda (n)+1075
(let ((tags (note-tags n)))+1076
(string-append+1077
(format " ~a" (folio-note-name n))+1078
(if (null? tags) ""+1079
(format " [~a]" (format-string-list tags ", "))))))+1080
notes)+1081
"\n"))))) 1082
1083
(define (make-tool-note-create store idx) 1084
(lambda (args)test/test-census-bound.mjsadded
@@ -0,0 +1,117 @@
+1
// The census fast path must not delay outside edits: once a server is idle+2
// (the fast path armed), an outside change must reach backlinks within one+3
// 15 s tick plus a 2 s margin. Each edit below keeps the file's size, and two+4
// of them also keep its mtime, so only ctime, inode or a directory time moves.+5
import assert from 'node:assert/strict';+6
import { spawn, execFileSync } from 'node:child_process';+7
import { mkdtemp, mkdir, writeFile, rm, stat, utimes, rename } from 'node:fs/promises';+8
import { tmpdir } from 'node:os';+9
import { join, resolve } from 'node:path';+10
import { once } from 'node:events';+11
import test from 'node:test';+12
+13
const binary = resolve(process.env.FOLIO_TEST_BINARY || 'build/dev/bin/folio');+14
const pause = ms => new Promise(r => setTimeout(r, ms));+15
const BOUND_MS = 17000;+16
+17
function client(root) {+18
const child = spawn(binary, ['serve', '--stdio', '--log-level', 'debug'], {+19
env: { ...process.env, FOLIO_ROOT: root }, stdio: ['pipe', 'pipe', 'pipe'],+20
});+21
let sequence = 0, buffer = '', diagnostic = '';+22
const pending = new Map();+23
child.stderr.on('data', d => { diagnostic = (diagnostic + d).slice(-4000); });+24
child.stdout.on('data', d => {+25
buffer += d;+26
for (let end; (end = buffer.indexOf('\n')) >= 0;) {+27
const line = buffer.slice(0, end); buffer = buffer.slice(end + 1);+28
let m; try { m = JSON.parse(line); } catch { continue; }+29
const p = pending.get(m.id); if (!p) continue;+30
pending.delete(m.id);+31
if (m.error || m.result?.isError) p.reject(new Error(JSON.stringify(m))); else p.resolve(m.result);+32
}+33
});+34
return {+35
child,+36
call(name, args = {}) {+37
const id = ++sequence;+38
return new Promise((res, rej) => {+39
pending.set(id, { resolve: res, reject: rej });+40
child.stdin.write(JSON.stringify({ jsonrpc: '2.0', id, method: 'tools/call',+41
params: { name: `folio/${name}`, arguments: args } }) + '\n');+42
}).then(r => r.content.map(c => c.text || '').join('\n'));+43
},+44
async close() {+45
const exited = once(child, 'exit');+46
child.stdin.end();+47
const timer = setTimeout(() => child.kill('SIGKILL'), 3000);+48
await exited; clearTimeout(timer);+49
},+50
diagnostic: () => diagnostic,+51
};+52
}+53
+54
// Poll note-links until `predicate` holds; return the elapsed ms or fail at the bound.+55
async function within(c, name, predicate, label) {+56
const start = performance.now();+57
for (;;) {+58
if (predicate(await c.call('note-links', { name }))) return performance.now() - start;+59
const elapsed = performance.now() - start;+60
assert.ok(elapsed < BOUND_MS, `${label}: not visible after ${Math.round(elapsed)} ms`);+61
await pause(200);+62
}+63
}+64
+65
const linked = '# Source\n[[target]]\n';+66
const unlinked = '# Source\n[[zzzzzz]]\n';+67
+68
test('outside edits reach backlinks within one census tick of an idle server', { timeout: 120000 }, async t => {+69
const root = await mkdtemp(join(tmpdir(), 'folio-census-bound-'));+70
t.after(() => rm(root, { recursive: true, force: true }));+71
await mkdir(join(root, 'notes')); await mkdir(join(root, 'projects'));+72
await writeFile(join(root, 'notes/target.md'), '# Target\n');+73
const source = join(root, 'notes/source.md');+74
await writeFile(source, unlinked);+75
const git = (...args) => execFileSync('git', ['-C', root, '-c', 'user.name=t', '-c', 'user.email=t@t', ...args]);+76
git('init', '-q'); git('add', '-A'); git('commit', '-qm', 'unlinked');+77
+78
// Let the files settle (the 2 s racily-clean window) so the startup census+79
// arms the fast path and the first 15 s tick can skip.+80
await pause(2500);+81
const c = client(root);+82
t.after(() => c.close());+83
assert.doesNotMatch(await c.call('note-links', { name: 'target' }), /← source/);+84
// Let at least one idle tick pass, so the fast path is what sees the edits.+85
await pause(16000);+86
assert.match(c.diagnostic(), /Census tick.*unchanged/, 'the fast path must be armed before the edits');+87
const has = s => /← source \(note\)/.test(s);+88
const results = {};+89
+90
// 1. Same size, same second, mtime restored: only ctime moves.+91
const before = await stat(source);+92
await writeFile(source, linked);+93
await utimes(source, before.atime, before.mtime);+94
results.restoredMtime = await within(c, 'target', has, 'same-size edit with restored mtime');+95
+96
// 2. git checkout of the equal-size committed revision (a replacement).+97
await pause(1000);+98
git('checkout', '-q', '--', 'notes/source.md');+99
results.gitCheckout = await within(c, 'target', s => !has(s), 'git checkout');+100
+101
// 3. Rename over the file with a same-size, same-mtime copy (new inode).+102
await pause(1000);+103
const staged = join(root, 'staged.md');+104
const now = await stat(source);+105
await writeFile(staged, linked);+106
await utimes(staged, now.atime, now.mtime);+107
await rename(staged, source);+108
results.renameOver = await within(c, 'target', has, 'rename-over replacement');+109
+110
// 4. A new linking note in a new directory.+111
await mkdir(join(root, 'notes/deep'));+112
await writeFile(join(root, 'notes/deep/added.md'), '# Added\n[[target]]\n');+113
results.added = await within(c, 'target', s => /← deep\/added \(note\)/.test(s), 'added note');+114
+115
console.log(JSON.stringify({ step: 'census-bound', boundMs: BOUND_MS,+116
visibleMs: Object.fromEntries(Object.entries(results).map(([k, v]) => [k, Math.round(v)])) }));+117
});test/test-idle-census.sgladded
@@ -0,0 +1,478 @@
+1
;;; The 0.3.2 idle work: the census fast path, its yields, the parse cache+2
;;; behind read-only tools, and task-notes answered from cached frontmatter.+3
;;;+4
;;; Every "outside edit" case writes a file the way a non-folio writer would+5
;;; and checks the result against an independent full parse, so a fast path+6
;;; that skips a real change fails here.+7
;;;+8
;;; Most tests set both settle windows to 0, so a file written a moment ago+9
;;; can be trusted at once, and wait `granule` before each outside edit:+10
;;; Linux stamps files from a coarse clock, and two writes inside one tick of+11
;;; it can be indistinguishable (that is what the settle window guards; its+12
;;; own tests run with it on).+13
(import (sigil test)+14
(sigil async)+15
(sigil fs)+16
(sigil path)+17
(sigil sqlite)+18
(sigil string)+19
(sigil process)+20
(sigil list)+21
(sigil mcp server)+22
(sigil mcp protocol)+23
(folio store)+24
(folio note)+25
(folio links)+26
(folio project)+27
(folio task)+28
(folio query)+29
(folio index)+30
(folio filecache)+31
(folio tools))+32
+33
(census-settle-seconds 0)+34
(file-cache-settle-seconds 0)+35
+36
(define (sh . args)+37
(assert-equal 0 (apply process-run args)))+38
+39
(define (granule) (sleep 0.05))+40
+41
(define (call-tool server name args)+42
(response->json+43
(mcp-server-handle-message server+44
(jsonrpc-request id: 1 method: "tools/call"+45
params: (dict name: name arguments: args)))))+46
+47
(define (note-text body) (string-append "---\ntags: []\n---\n\n" body "\n"))+48
+49
;; Two bodies of equal length: rewriting one with the other keeps the size.+50
(define linked "# Source\n[[target]]")+51
(define unlinked "# Source\n[[zzzzzz]]")+52
+53
(define (backlinks-match? store idx name)+54
(equal? (find-all-backlinks store name) (index-backlinks idx name)))+55
+56
(define (with-census-store thunk)+57
(call-with-temp-directory+58
(lambda (root)+59
(let* ((store (make-folio-store root))+60
(_ (store-ensure-dirs! store))+61
(notes (store-notes-dir store))+62
(source (path-join notes "source.md")))+63
(write-file-string (path-join notes "target.md") (note-text "# Target"))+64
(write-file-string source (note-text unlinked))+65
(let ((idx (make-folio-index store)))+66
(thunk root store idx source)+67
(sqlite-close (folio-index-db idx)))))))+68
+69
(define (with-settle seconds thunk)+70
(census-settle-seconds seconds)+71
(file-cache-settle-seconds seconds)+72
(guard (e (else (census-settle-seconds 0) (file-cache-settle-seconds 0) (raise e)))+73
(thunk)+74
(census-settle-seconds 0)+75
(file-cache-settle-seconds 0)))+76
+77
(test-group "census fast path"+78
(test "an unchanged corpus skips the per-file census"+79
(with-census-store+80
(lambda (root store idx source)+81
(assert-equal 'reconciled (index-census-tick! idx))+82
(assert-equal 'unchanged (index-census-tick! idx))+83
(assert-equal 'unchanged (index-census-tick! idx)))))+84
+85
(test "a same-second, same-size outside edit is reconciled"+86
(with-census-store+87
(lambda (root store idx source)+88
(index-census-tick! idx)+89
(assert-equal 'unchanged (index-census-tick! idx))+90
(assert-equal '() (index-backlinks idx "target"))+91
(granule)+92
(write-file-string source (note-text linked))+93
(assert-equal 'reconciled (index-census-tick! idx))+94
(assert-equal '(("source" . note)) (index-backlinks idx "target"))+95
(assert-true (backlinks-match? store idx "target")))))+96
+97
(test "an edit whose mtime is restored with touch is reconciled (ctime moves)"+98
(with-census-store+99
(lambda (root store idx source)+100
(let ((stamp (path-join root "stamp")))+101
(index-census-tick! idx)+102
(assert-equal 'unchanged (index-census-tick! idx))+103
(write-file-string stamp "")+104
(sh "touch" "-r" source stamp)+105
(granule)+106
(write-file-string source (note-text linked))+107
(sh "touch" "-r" stamp source)+108
(assert-equal 'reconciled (index-census-tick! idx))+109
(assert-equal '(("source" . note)) (index-backlinks idx "target"))))))+110
+111
(test "a replacement with the same size and mtime is reconciled (rename over)"+112
(with-census-store+113
(lambda (root store idx source)+114
(let ((staged (path-join root "staged.md")))+115
(index-census-tick! idx)+116
(assert-equal 'unchanged (index-census-tick! idx))+117
(granule)+118
(write-file-string staged (note-text linked))+119
(sh "touch" "-r" source staged)+120
(sh "mv" staged source)+121
(assert-equal 'reconciled (index-census-tick! idx))+122
(assert-equal '(("source" . note)) (index-backlinks idx "target"))))))+123
+124
(test "a git checkout of an equal-size revision is reconciled"+125
(with-census-store+126
(lambda (root store idx source)+127
(let ((git (lambda args+128
(apply sh "git" "-C" root "-c" "user.name=t" "-c" "user.email=t@t"+129
args))))+130
(git "init" "-q")+131
(write-file-string source (note-text linked))+132
(git "add" "-A")+133
(git "commit" "-qm" "linked")+134
(write-file-string source (note-text unlinked))+135
(git "commit" "-qam" "unlinked")+136
(index-census-tick! idx)+137
(assert-equal 'unchanged (index-census-tick! idx))+138
(assert-equal '() (index-backlinks idx "target"))+139
(granule)+140
(git "checkout" "-q" "HEAD~1" "--" "notes/source.md")+141
(assert-equal 'reconciled (index-census-tick! idx))+142
(assert-equal '(("source" . note)) (index-backlinks idx "target"))))))+143
+144
(test "rows another connection deletes are restored by the census it forces"+145
(with-census-store+146
(lambda (root store idx source)+147
(write-file-string source (note-text linked))+148
(index-census-tick! idx)+149
(assert-equal 'unchanged (index-census-tick! idx))+150
(assert-equal '(("source" . note)) (index-backlinks idx "target"))+151
(let ((other (sqlite-open (path-join root ".folio-index.db"))))+152
(assert-true (sqlite-run other "DELETE FROM note_links WHERE source = ?" source))+153
(assert-true (sqlite-run other "DELETE FROM link_sources WHERE path = ?" source))+154
(sqlite-close other))+155
(assert-equal '() (index-backlinks idx "target"))+156
(assert-equal 'reconciled (index-census-tick! idx))+157
(assert-equal '(("source" . note)) (index-backlinks idx "target"))+158
(assert-equal 'unchanged (index-census-tick! idx)))))+159
+160
(test "an editor's dangling lock link keeps the fast path; a note symlink disables it"+161
(with-census-store+162
(lambda (root store idx source)+163
(let ((notes (store-notes-dir store))+164
(outside (path-join root "outside.md")))+165
(sh "ln" "-s" "user@host.1234:1700000000" (path-join notes ".#source.md"))+166
(index-census-tick! idx)+167
(assert-equal 'unchanged (index-census-tick! idx))+168
(write-file-string outside (note-text unlinked))+169
(sh "ln" "-s" outside (path-join notes "linked.md"))+170
(assert-equal 'reconciled (index-census-tick! idx))+171
(assert-equal 'reconciled (index-census-tick! idx))+172
;; The walk follows the link, so its target's edit must be seen.+173
(granule)+174
(write-file-string outside (note-text linked))+175
(assert-equal 'reconciled (index-census-tick! idx))+176
(assert-equal '(("linked" . note)) (index-backlinks idx "target"))))))+177
+178
(test "a symlink whose name holds a newline cannot pass as a hidden entry"+179
(with-census-store+180
(lambda (root store idx source)+181
(let ((outside (path-join root "outside.md")))+182
(write-file-string outside (note-text unlinked))+183
(sh "ln" "-s" outside (path-join (store-notes-dir store) "x\n.y.md"))+184
(assert-equal 'reconciled (index-census-tick! idx))+185
(assert-equal 'reconciled (index-census-tick! idx))))))+186
+187
(test "a failure deferred until the file changes still arms the fast path"+188
(with-census-store+189
(lambda (root store idx source)+190
(index-census-tick! idx)+191
(granule)+192
(write-file-string source (note-text linked))+193
(sh "chmod" "000" source)+194
(assert-equal 'reconciled (index-census-tick! idx))+195
(assert-equal 'unchanged (index-census-tick! idx))+196
(granule)+197
(sh "chmod" "600" source)+198
(assert-equal 'reconciled (index-census-tick! idx))+199
(assert-equal '(("source" . note)) (index-backlinks idx "target"))+200
(assert-equal 'unchanged (index-census-tick! idx)))))+201
+202
(test "a retryable failure (database busy) keeps every tick on the full census"+203
(with-census-store+204
(lambda (root store idx source)+205
(index-census-tick! idx)+206
(assert-equal 'unchanged (index-census-tick! idx))+207
(let ((budget (sqlite-busy-budget))+208
(other (sqlite-open (path-join root ".folio-index.db"))))+209
(sqlite-set-busy-budget! 0.2)+210
(granule)+211
(write-file-string source (note-text linked))+212
(assert-true (sqlite-exec other "BEGIN IMMEDIATE"))+213
(assert-equal 'reconciled (index-census-tick! idx))+214
(assert-equal '() (index-backlinks idx "target"))+215
(assert-equal 'reconciled (index-census-tick! idx))+216
(assert-true (sqlite-exec other "ROLLBACK"))+217
(sqlite-close other)+218
(sqlite-set-busy-budget! budget)+219
(assert-equal 'reconciled (index-census-tick! idx))+220
(assert-equal '(("source" . note)) (index-backlinks idx "target"))+221
(assert-equal 'unchanged (index-census-tick! idx))))))+222
+223
(test "an unsettled listing is not trusted until its newest time is old enough"+224
(with-settle 1+225
(lambda ()+226
(with-census-store+227
(lambda (root store idx source)+228
(assert-equal 'reconciled (index-census-tick! idx))+229
(assert-equal 'reconciled (index-census-tick! idx))+230
(sleep 1.2)+231
(assert-equal 'reconciled (index-census-tick! idx))+232
(assert-equal 'unchanged (index-census-tick! idx)))))))+233
+234
(test "a full census with no per-file work still yields to other tasks"+235
(call-with-temp-directory+236
(lambda (root)+237
(let* ((store (make-folio-store root))+238
(_ (store-ensure-dirs! store))+239
(idx (make-folio-index store))+240
(done #f)+241
(turns 0))+242
(let loop ((i 0))+243
(when (< i 200)+244
(write-file-string (path-join (store-notes-dir store) (format "n~a.md" i))+245
(note-text (format "# N~a\n[[n0]]" i)))+246
(loop (+ i 1))))+247
(index-census-tick! idx)+248
(let ((second (index-census-tick! idx)))+249
(display (format "yield test: second tick=~a\n" second))+250
(assert-equal 'unchanged second))+251
;; Another connection commits: the next tick runs the full census,+252
;; but every file is current, so the pacer's yields are the only ones.+253
(let ((other (sqlite-open (path-join root ".folio-index.db"))))+254
(assert-true (sqlite-run other "UPDATE notes_meta SET indexed_at = indexed_at + 1"))+255
(sqlite-close other))+256
(with-async+257
(go (set! done (index-census-tick! idx)))+258
(go (let loop ()+259
(unless done+260
(set! turns (+ turns 1))+261
(yield)+262
(loop)))))+263
(display (format "yield test: tick=~a turns=~a\n" done turns))+264
(assert-equal 'reconciled done)+265
;; 200 files with a yield every 32: at least 6 turns for the other task.+266
(assert-true (>= turns 6))+267
(assert-true (backlinks-match? store idx "n0"))+268
(sqlite-close (folio-index-db idx)))))))+269
+270
(test-group "parse cache"+271
(test "an unchanged file is computed once and the value reused"+272
(call-with-temp-directory+273
(lambda (root)+274
(let ((cache (make-file-cache))+275
(path (path-join root "f.md"))+276
(calls 0))+277
(write-file-string path "one")+278
(let* ((compute (lambda (p) (set! calls (+ calls 1)) (read-file-string p)))+279
(a (file-cache-ref cache path compute))+280
(b (file-cache-ref cache path compute)))+281
(assert-equal "one" a)+282
(assert-true (eq? a b))+283
(assert-equal 1 calls)+284
(assert-equal 1 (file-cache-size cache)))))))+285
+286
(test "same-second same-size write, restored mtime, rename-over and git checkout all recompute"+287
(call-with-temp-directory+288
(lambda (root)+289
(let* ((cache (make-file-cache))+290
(path (path-join root "f.md"))+291
(stamp (path-join root "stamp"))+292
(staged (path-join root "staged"))+293
(read (lambda () (file-cache-ref cache path read-file-string)))+294
(git (lambda args+295
(apply sh "git" "-C" root "-c" "user.name=t" "-c" "user.email=t@t"+296
args))))+297
(write-file-string path "aaaa")+298
(assert-equal "aaaa" (read))+299
;; Same second, same size, in place.+300
(granule)+301
(write-file-string path "bbbb")+302
(assert-equal "bbbb" (read))+303
;; Same size, mtime put back: ctime still moves.+304
(write-file-string stamp "")+305
(sh "touch" "-r" path stamp)+306
(granule)+307
(write-file-string path "cccc")+308
(sh "touch" "-r" stamp path)+309
(assert-equal "cccc" (read))+310
;; Replaced by a file with the same size and mtime: a new inode.+311
(granule)+312
(write-file-string staged "dddd")+313
(sh "touch" "-r" path staged)+314
(sh "mv" staged path)+315
(assert-equal "dddd" (read))+316
;; git checkout of an equal-size revision.+317
(git "init" "-q")+318
(git "add" "f.md")+319
(git "commit" "-qm" "d")+320
(granule)+321
(write-file-string path "eeee")+322
(assert-equal "eeee" (read))+323
(granule)+324
(git "checkout" "-q" "--" "f.md")+325
(assert-equal "dddd" (read))))))+326
+327
(test "a recently written file is not cached until it settles"+328
(with-settle 1+329
(lambda ()+330
(call-with-temp-directory+331
(lambda (root)+332
(let* ((cache (make-file-cache))+333
(path (path-join root "f.md"))+334
(calls 0)+335
(read (lambda () (file-cache-ref cache path+336
(lambda (p) (set! calls (+ calls 1)) (read-file-string p))))))+337
(write-file-string path "fresh")+338
(read) (read)+339
(assert-equal 2 calls)+340
(assert-equal 0 (file-cache-size cache))+341
(sleep 1.2)+342
(read) (read)+343
(assert-equal 3 calls)+344
(assert-equal 1 (file-cache-size cache))))))))+345
+346
(test "a compute that rewrites the file or raises caches nothing; prune drops vanished paths"+347
(call-with-temp-directory+348
(lambda (root)+349
(let ((cache (make-file-cache))+350
(path (path-join root "f.md"))+351
(other (path-join root "g.md")))+352
(write-file-string path "1111")+353
(granule)+354
(assert-equal "1111"+355
(file-cache-ref cache path+356
(lambda (p) (let ((v (read-file-string p))) (write-file-string p "2222") v))))+357
(assert-equal 0 (file-cache-size cache))+358
(assert-equal "2222" (file-cache-ref cache path read-file-string))+359
(assert-true (guard (e (else #t))+360
(file-cache-ref cache other (lambda (p) (error "boom")))+361
#f))+362
(write-file-string other "g")+363
(assert-equal "g" (file-cache-ref cache other read-file-string))+364
(assert-equal 2 (file-cache-size cache))+365
(file-cache-prune! cache root (list path))+366
(assert-equal 1 (file-cache-size cache))))))+367
+368
(test "tasks, projects and status follow equal-size outside edits of a project file"+369
(call-with-temp-directory+370
(lambda (root)+371
(let* ((store (make-folio-store root))+372
(_ (store-ensure-dirs! store))+373
(project (path-join (store-project-dir store) "p.md"))+374
(body (lambda (word status)+375
(format "---\nstatus: ~a\n---\n\n# P\n\n## Tasks\n\n- [ ] ~a {id: t-1111}\n"+376
status word)))+377
(texts (lambda () (map folio-task-text (all-tasks store)))))+378
(write-file-string project (body "alpha" "active"))+379
(assert-equal '("alpha") (texts))+380
(granule)+381
(write-file-string project (body "bravo" "active"))+382
(assert-equal '("bravo") (texts))+383
(assert-equal '("active") (map project-status (all-projects store)))+384
(granule)+385
(write-file-string project (body "bravo" "paused"))+386
(assert-equal '("paused") (map project-status (all-projects store)))+387
(assert-equal 0 (dict-ref (folio-status store) active-project-count:)))))))+388
+389
;; The reply task-notes gave before 0.3.2: a full parse of every note.+390
(define (reference-task-notes store task-id)+391
(let ((notes (filter (lambda (n) (member task-id (note-tasks n))) (all-notes store))))+392
(if (null? notes)+393
(format "No notes linked to task ~a." task-id)+394
(string-append+395
(format "Notes linked to ~a (~a):\n\n" task-id (length notes))+396
(string-join+397
(map (lambda (n)+398
(let ((tags (note-tags n)))+399
(string-append (format " ~a" (folio-note-name n))+400
(if (null? tags) ""+401
(format " [~a]" (string-join tags ", "))))))+402
notes)+403
"\n")))))+404
+405
(define (assert-task-notes-parity store ids)+406
(for-each+407
(lambda (id) (assert-equal (reference-task-notes store id) (task-notes-text store id)))+408
ids))+409
+410
(test-group "task-notes from cached frontmatter"+411
(test "matches a full parse across create, outside edits, rename, task move and delete"+412
(call-with-temp-directory+413
(lambda (root)+414
(let* ((store (make-folio-store root))+415
(_ (store-ensure-dirs! store))+416
(idx (make-folio-index store))+417
(server (mcp-server name: "task-notes-test"))+418
(ids '("t-aaaa" "t-bbbb" "t-cccc" "t-none" "42"))+419
(notes-dir (store-notes-dir store))+420
(project (path-join (store-project-dir store) "p.md")))+421
(register-folio-tools! server store idx)+422
(write-file-string project+423
"---\nstatus: active\n---\n\n# P\n\n## Tasks\n\n- [ ] A {id: t-aaaa}\n- [ ] B {id: t-bbbb}\n")+424
(write-file-string (path-join (store-project-dir store) "q.md")+425
"---\nstatus: active\n---\n\n# Q\n\n## Tasks\n\n")+426
(note-create! store "brief-a" '("brief") "About A." '("t-aaaa"))+427
(note-create! store "nested/both" '("brief" "x") "Both." '("t-aaaa" "t-bbbb"))+428
(note-create! store "plain" '() "No tasks.")+429
;; Malformed tasks: values, handled the same way by both answers.+430
(write-file-string (path-join notes-dir "scalar.md") "---\ntasks: t-aaaa\n---\n\nScalar.\n")+431
(write-file-string (path-join notes-dir "number.md") "---\ntasks: 42\n---\n\nNumber.\n")+432
(write-file-string (path-join notes-dir "nested.md")+433
"---\ntasks:\n - [t-bbbb]\n - t-cccc\n---\n\nNested.\n")+434
(write-file-string (path-join notes-dir "empty.md") "---\ntasks:\n---\n\nEmpty.\n")+435
(write-file-string (path-join notes-dir "odd-tags.md")+436
"---\ntags: [7, x]\ntasks: [t-aaaa]\n---\n\nOdd tags.\n")+437
(assert-task-notes-parity store ids)+438
;; Outside edit: same-size swap of a task id in the frontmatter.+439
(let ((path (path-join notes-dir "brief-a.md")))+440
(granule)+441
(write-file-string path "---\ntags: [brief]\ntasks: [t-cccc]\n---\n\nAbout A.\n")+442
(assert-task-notes-parity store ids)+443
(granule)+444
(write-file-string path "---\ntags: [brief]\ntasks: [t-bbbb]\n---\n\nAbout A.\n")+445
(assert-task-notes-parity store ids))+446
;; Rename through the tool, and an outside rename.+447
(assert-true (string-contains?+448
(call-tool server "folio/note-rename"+449
(dict name: "nested/both" new-name: "renamed/both"))+450
"Renamed"))+451
(assert-task-notes-parity store ids)+452
(sh "mv" (path-join notes-dir "plain.md") (path-join notes-dir "plain2.md"))+453
(assert-task-notes-parity store ids)+454
;; Moving a task between projects keeps its id and its notes.+455
(assert-true (string-contains?+456
(call-tool server "folio/task-move" (dict id: "t-aaaa" project: "q"))+457
"Moved task"))+458
(assert-task-notes-parity store ids)+459
(assert-true (string-contains? (task-notes-text store "t-aaaa") "renamed/both"))+460
;; Deleting a note drops it.+461
(delete-file (path-join notes-dir "brief-a.md"))+462
(assert-task-notes-parity store ids)+463
(sqlite-close (folio-index-db idx))))))+464
+465
(test "the same file is named by the store that reads it"+466
(call-with-temp-directory+467
(lambda (root)+468
(let* ((outer (make-folio-store root))+469
(_ (store-ensure-dirs! outer))+470
(inner-root (path-join (store-notes-dir outer) "inner"))+471
(_ (ensure-directory inner-root))+472
(inner (make-folio-store inner-root)))+473
(store-ensure-dirs! inner)+474
(write-file-string (path-join (store-notes-dir inner) "n.md")+475
"---\ntasks: [t-dddd]\n---\n\nInner.\n")+476
(assert-true (string-contains? (task-notes-text outer "t-dddd") " inner/notes/n"))+477
(assert-true (string-contains? (task-notes-text inner "t-dddd") " n"))+478
(assert-false (string-contains? (task-notes-text inner "t-dddd") "inner/notes/n")))))))tools/check-task-notes-parity.sgladded
@@ -0,0 +1,142 @@
+1
;;; Differential check of folio/task-notes on a DISPOSABLE copy of a real corpus.+2
;;;+3
;;; The reference is the pre-0.3.2 algorithm: parse every note, keep those whose+4
;;; frontmatter `tasks:` holds the id. Every id any note mentions, a sample of+5
;;; project task ids and some ids nobody uses are compared after each of: no+6
;;; change, an outside frontmatter edit, an outside note move, a note-rename+7
;;; through the tool, a task-move through the tool, and a deletion.+8
(import (sigil fs)+9
(sigil list)+10
(sigil path)+11
(sigil string)+12
(sigil time)+13
(sigil sqlite)+14
(sigil mcp server)+15
(sigil mcp protocol)+16
(folio store)+17
(folio note)+18
(folio task)+19
(folio query)+20
(folio index)+21
(folio tools))+22
+23
(define root (getenv "FOLIO_CORPUS"))+24
(unless root (error "FOLIO_CORPUS must name a disposable corpus copy"))+25
(when (string-contains? root "/Ops/folio") (error "refusing to run against a live store"))+26
(define store (make-folio-store root))+27
(define idx (make-folio-index store))+28
(define server (mcp-server name: "task-notes-parity"))+29
(register-folio-tools! server store idx)+30
+31
(define (wall-ms) (* 1000.0 (/ (current-jiffy) (jiffies-per-second))))+32
(define (tool name args)+33
(response->json+34
(mcp-server-handle-message server+35
(jsonrpc-request id: 1 method: "tools/call" params: (dict name: name arguments: args)))))+36
+37
(define (reference-text notes task-id)+38
(let ((hits (filter (lambda (n) (member task-id (note-tasks n))) notes)))+39
(if (null? hits)+40
(format "No notes linked to task ~a." task-id)+41
(string-append+42
(format "Notes linked to ~a (~a):\n\n" task-id (length hits))+43
(string-join+44
(map (lambda (n)+45
(let ((tags (note-tags n)))+46
(string-append (format " ~a" (folio-note-name n))+47
(if (null? tags) ""+48
(format " [~a]" (string-join tags ", "))))))+49
hits)+50
"\n")))))+51
+52
(define (unique strings)+53
(let loop ((rest strings) (seen (dict)) (out '()))+54
(cond ((null? rest) (reverse out))+55
((dict-contains? seen (car rest)) (loop (cdr rest) seen out))+56
(else (loop (cdr rest) (dict-set seen (car rest) #t) (cons (car rest) out))))))+57
+58
(define (take-up-to lst n)+59
(if (or (null? lst) (= n 0)) '() (cons (car lst) (take-up-to (cdr lst) (- n 1)))))+60
+61
(define (ids-now notes)+62
(unique+63
(append (apply append (map note-tasks notes))+64
(take-up-to (filter (lambda (x) x) (map folio-task-id (all-tasks store))) 60)+65
'("t-0000" "t-ffffff" "nonexistent" ""))))+66
+67
(define (round! label)+68
(let* ((t0 (wall-ms))+69
(notes (all-notes store))+70
(ref-ms (- (wall-ms) t0))+71
(ids (ids-now notes))+72
(linked 0))+73
(for-each+74
(lambda (id)+75
(let ((expected (reference-text notes id))+76
(actual (task-notes-text store id)))+77
(unless (equal? expected actual)+78
(error "task-notes differs from the full parse" label id expected actual))+79
(unless (string-starts-with? actual "No notes") (set! linked (+ linked 1)))))+80
ids)+81
(let* ((t1 (wall-ms))+82
(_ (task-notes-text store (car ids)))+83
(one-ms (- (wall-ms) t1)))+84
(display (format "PARITY ~a ids=~a linked_ids=~a reference_parse_ms=~a cached_call_ms=~a\n"+85
label (length ids) linked (round ref-ms) (round one-ms))))+86
ids))+87
+88
;; First call from cold (fills the cache) against the old cost.+89
(let* ((t0 (wall-ms)) (_ (task-notes-text store "t-0000")))+90
(display (format "COLD first_call_ms=~a\n" (round (- (wall-ms) t0)))))+91
+92
(round! "initial")+93
+94
;; A note that links tasks, for the mutations.+95
(define linking+96
(let ((notes (filter (lambda (n) (and (pair? (note-tasks n))+97
(string-starts-with? (car (note-tasks n)) "t-")))+98
(all-notes store))))+99
(when (null? notes) (error "corpus has no task-linked notes"))+100
(car notes)))+101
(define linking-name (folio-note-name linking))+102
(define linking-task (car (note-tasks linking)))+103
(display (format "MUTATE note=~a task=~a\n" linking-name linking-task))+104
+105
;; Outside edit: add a task id to a different, unlinked note's frontmatter.+106
(define plain+107
(car (filter (lambda (n) (null? (note-tasks n))) (all-notes store))))+108
(write-file-string (folio-note-file-path plain)+109
(string-append "---\ntasks:\n - " linking-task "\n---\n\n" (folio-note-body plain)))+110
(round! "outside-edit")+111
+112
;; Outside move of the linking note into a new directory.+113
(define moved-path (path-join (store-notes-dir store) "parity-moved" "moved-note.md"))+114
(ensure-directory (path-join (store-notes-dir store) "parity-moved"))+115
(rename-file (folio-note-file-path linking) moved-path)+116
(round! "outside-move")+117
+118
;; Rename through the tool.+119
(let ((reply (tool "folio/note-rename" (dict name: "parity-moved/moved-note"+120
new-name: "parity-moved/renamed-note"))))+121
(unless (string-contains? reply "Renamed") (error "note-rename failed" reply)))+122
(round! "tool-rename")+123
+124
;; Move a linked task to another project through the tool (when it is a project task).+125
(let* ((task (find (lambda (t) (equal? (folio-task-id t) linking-task)) (all-tasks store)))+126
(projects (store-project-files store))+127
(target (and task+128
(find (lambda (p) (not (equal? p (folio-task-source-file task)))) projects))))+129
(if (and task target)+130
(let ((reply (tool "folio/task-move"+131
(dict id: linking-task project: (path-stem (path-basename target))))))+132
(unless (string-contains? reply "Moved task") (error "task-move failed" reply))+133
(display (format "MOVED ~a to ~a\n" linking-task (path-basename target))))+134
(display (format "MOVED skipped: ~a is not a project task here\n" linking-task))))+135
(round! "tool-task-move")+136
+137
;; Deletion.+138
(delete-file (path-join (store-notes-dir store) "parity-moved" "renamed-note.md"))+139
(round! "delete")+140
+141
(sqlite-close (folio-index-db idx))+142
(display "TASK-NOTES PARITY PASSED\n")