Remove unused FTS work from searches and existing-note writes

Changed

 CHANGELOG.md                  |  2 ++
 docs/shared-http.md           |  4 ++++
 src/folio/index.sgl           | 50 +++++++++++++++++++++++++++++++++++++-------------
 src/folio/tools.sgl           |  2 +-
 test/test-fts-replacement.sgl | 23 +++++++++++++++++++++++
 5 files changed, 67 insertions(+), 14 deletions(-)

Diff

CHANGELOG.mdmodified

@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Dispatch validated HTTP tool calls in separate cooperative tasks, so a writer waiting on a store lock does not hold up independent reads. Mutating handlers remain serialized through their index flush; existing cross-process file locking is preserved.

- Use a separate HTTP index connection for reads, keeping uncommitted census/write rows private. Concurrent reads can finish before an in-flight write and see its previous committed state. Reads still have per-query SQLite consistency and atomic Markdown files, without a global multi-file snapshot. The 15-second outside-edit census is unchanged.

- Reduce JSON response encoding CPU with sigil-json's native string escaping. Require sigil-json 0.16.2 and sigil-mcp 0.18.4. HTTP tool replies remain JSON-RPC JSON documents and use chunked responses; stdio is unchanged.

+13

- Skip unused snippets in note-list searches while retaining default search-index snippets, ranking and filters. Locate existing full-text rows by indexed literal name for updates, falling back to the previous scan for tokenless/missing names or lookup failure. Keep the existing schema and transactions.

+14

- Allow socket polling between HTTP tool execution and response encoding with a one-millisecond cooperative pause; individual CPU-bound phases remain cooperative.

## [0.3.3] - 2026-10-24

docs/shared-http.mdmodified

@@ -29,6 +29,10 @@ for another process's store lock leaves independent reads free to complete.

CPU-bound calls still share one thread; they are not preempted. Mutating tool

handlers are serialized through their index flush, including inbox reads that

may backfill ids. Other read tools do not enter that writer queue.

+32

The transport allows socket polling between tool execution and JSON encoding

+33

with a one-millisecond cooperative pause. Either phase can still delay peers

+34

while it uses the thread. Note listings skip unused full-text snippets, and

+35

updates locate existing full-text rows through their indexed names.

Note and project mutations use the cross-process SQLite store lock. Index

maintenance never takes that lock: each indexed document is replaced in its

src/folio/index.sglmodified

@@ -259,11 +259,31 @@

259

(remove-note-from-index! idx old-name)

260

(index-note-by-name! idx new-name))

261

−262

;; Delete the FTS entry for a note by name.

−263

;; For a standalone FTS5 table, delete by matching the name column.

+262

;; Existing names use the FTS index instead of scanning every document.

+263

;; Keep exact-name filtering and all matching rowids: older writers may

+264

;; have left duplicate rows, and metadata rowids are unrelated to FTS.

+265

;; A missing/tokenless name or lookup failure retains the old scan path.

266

(define (delete-fts-entry! db name)

−265

(checked-sql

−266

(sqlite-run db "DELETE FROM notes_fts WHERE name = ?" name)))

+267

(let ((rows

+268

(guard (exn (else #f))

+269

(let ((found

+270

(sqlite-query db

+271

"SELECT rowid AS rid FROM notes_fts

+272

WHERE notes_fts MATCH ? AND name = ?"

+273

(string-append "name : \"" (escape-fts-quotes name) "\"")

+274

name)))

+275

(and (equal? (sqlite-errmsg db) "not an error")

+276

(pair? found)

+277

found)))))

+278

(if rows

+279

(for-each

+280

(lambda (row)

+281

(checked-sql

+282

(sqlite-run db "DELETE FROM notes_fts WHERE rowid = ?"

+283

(cdr (assoc 'rid row)))))

+284

rows)

+285

(checked-sql

+286

(sqlite-run db "DELETE FROM notes_fts WHERE name = ?" name)))))

287

288

;; ============================================================

289

;; Startup reconciliation

@@ -737,8 +757,9 @@

757

;;; Search the FTS5 index.

758

;;;

759

;;; Returns a list of alists with keys: name, title, tags (and snippet

−740

;;; when a text query is provided).

−741

(define (search-index idx query tag since)

+760

;;; when a text query is provided). Metadata-only callers can opt out of

+761

;;; snippets without changing ranking or filters.

+762

(define (search-index idx query tag since (keys: (snippets? #t)))

763

(: folio-index? (maybe string?) (maybe string?) (maybe string?) -> list?)

764

(let ((db (folio-index-db idx)))

765

(cond

@@ -747,13 +768,16 @@

768

(let* ((fts-query (sanitize-fts-query query))

769

(results

770

(sqlite-query db

−750

"SELECT f.name, f.title, f.tags,

−751

snippet(notes_fts, 3, '', '', '...', 48) AS snippet,

−752

m.created, m.last_updated

−753

FROM notes_fts f

−754

JOIN notes_meta m ON m.name = f.name

−755

WHERE notes_fts MATCH ?

−756

ORDER BY rank"

+771

(string-append

+772

"SELECT f.name, f.title, f.tags, "

+773

(if snippets?

+774

"snippet(notes_fts, 3, '', '', '...', 48) AS snippet, "

+775

"")

+776

"m.created, m.last_updated

+777

FROM notes_fts f

+778

JOIN notes_meta m ON m.name = f.name

+779

WHERE notes_fts MATCH ?

+780

ORDER BY rank")

781

fts-query)))

782

(filter-results results tag since)))

783

;; No text query: list/filter from metadata only

src/folio/tools.sglmodified

@@ -934,7 +934,7 @@

934

(let* ((query (dict-ref args query: #f))

935

(tag (dict-ref args tag: #f))

936

(since (dict-ref args since: #f))

−937

(results (search-index idx query tag since)))

+937

(results (search-index idx query tag since snippets?: #f)))

938

(if (null? results)

939

"No notes found."

940

(string-join

test/test-fts-replacement.sglmodified

@@ -44,4 +44,27 @@

(assert-equal 1

(length (sqlite-query db "SELECT rowid FROM notes_fts WHERE name = ?" sibling)))))

'("normal" "a\"quoted" "中文" "---" "😀" "Case-Fold" "a.b" "with space"))

+47

(sqlite-close db)))))

+48

+49

(test "a failed indexed lookup falls back to literal-name replacement"

+50

(call-with-temp-directory

+51

(lambda (root)

+52

(let* ((store (make-folio-store root))

+53

(idx (make-folio-index store))

+54

(db (folio-index-db idx))

+55

(path (path-join (store-notes-dir store) "one.md")))

+56

;; This table accepts the old literal DELETE but cannot run MATCH.

+57

;; It injects a lookup failure independently of note tokenization.

+58

(assert-true (sqlite-exec db "DROP TABLE notes_fts"))

+59

(assert-true

+60

(sqlite-exec db

+61

"CREATE TABLE notes_fts (name TEXT, title TEXT, tags TEXT, content TEXT)"))

+62

(assert-true

+63

(sqlite-run db

+64

"INSERT INTO notes_fts VALUES ('one', 'Old', '', 'staleword')"))

+65

(write-file-string path "# One\nafterword\n")

+66

(index-note! idx path)

+67

(let ((rows (sqlite-query db "SELECT content FROM notes_fts WHERE name = 'one'")))

+68

(assert-equal 1 (length rows))

+69

(assert-equal "# One\nafterword\n" (cdr (assoc 'content (car rows)))))

(sqlite-close db))))))