Status cache: key on the transaction count, so a rollback retires reads

Message

A status read inside this connection's own transaction sees its uncommitted rows, and ROLLBACK moves neither data_version nor total_changes(), so those rows could stay cached. with-index-transaction now counts each start and end, and the count is part of the key. with-index-transaction is exported for the test that aborts one.

Changed

 CHANGELOG.md               |  2 +-
 src/folio/index.sgl        | 25 +++++++++++++++++++------
 test/test-status-cache.sgl | 20 ++++++++++++++++++++
 3 files changed, 40 insertions(+), 7 deletions(-)

Diff

CHANGELOG.mdmodified

@@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [0.3.3] - Unreleased

−10

- `status` reuses its index work while nothing it reads has changed. The note summaries and orphan names, with the sorted recent notes and the orphan list derived from them, are kept while the native `notes/` listing, SQLite's `data_version` (another connection's commit) and `total_changes()` (this connection's own writes) are unchanged, so a note deleted outside folio still leaves `status` at once. A `notes/` symlink the walk follows, or a failed query, turns the reuse off.

+10

- `status` reuses its index work while nothing it reads has changed. The note summaries and orphan names, with the sorted recent notes and the orphan list derived from them, are kept while the native `notes/` listing, SQLite's `data_version` (another connection's commit) and `total_changes()` (this connection's own writes) are unchanged, so a note deleted outside folio still leaves `status` at once. The key also holds the connection's transaction count, so rows read inside a transaction that rolls back are not reused. A `notes/` symlink the walk follows, or a failed query, turns the reuse off.

- No index or file format change: 0.3.1, 0.3.2 and 0.3.3 processes share one store and index.

## [0.3.2] - Unreleased

src/folio/index.sglmodified

@@ -22,6 +22,7 @@

folio-index-db

index-reconcile!

+25

with-index-transaction

index-census-tick!

census-settle-seconds

index-watch!

@@ -49,7 +50,10 @@

;; Box: what the last complete census saw (see census-state), or #f.

(census-state)

;; Box: (key . #(summaries orphan-names)) for status, or #f.

−52

(summary-cache))

+53

(summary-cache)

+54

;; Box: bumped when with-index-transaction takes and releases the

+55

;; connection, so it is odd while a transaction is open.

+56

(transaction-count))

(define (checked-sql result)

(unless result (error "Folio index SQL operation failed")))

@@ -58,20 +62,24 @@

;; active transaction as a nested call: a busy SQLite operation can yield.

(define (with-index-transaction idx thunk)

(let ((db (folio-index-db idx))

−61

(mutex (folio-index-transaction-mutex idx)))

+65

(mutex (folio-index-transaction-mutex idx))

+66

(count (folio-index-transaction-count idx)))

(channel-receive mutex)

+68

(vector-set! count 0 (+ 1 (vector-ref count 0)))

(let ((result

(guard (exn

(else

;; Preserve the original failure if rollback fails.

(guard (rollback-error (else #f))

(sqlite-exec db "ROLLBACK"))

+75

(vector-set! count 0 (+ 1 (vector-ref count 0)))

(channel-send mutex #t)

(raise exn)))

(checked-sql (sqlite-exec db "BEGIN IMMEDIATE"))

(let ((value (thunk)))

(checked-sql (sqlite-exec db "COMMIT"))

value))))

+82

(vector-set! count 0 (+ 1 (vector-ref count 0)))

(channel-send mutex #t)

result)))

@@ -150,7 +158,8 @@

158

failed-fingerprints: (vector (dict))

159

transaction-mutex: mutex

160

census-state: (vector #f)

−153

summary-cache: (vector #f)))))

+161

summary-cache: (vector #f)

+162

transaction-count: (vector 0)))))

163

164

;; ============================================================

165

;; Write-through operations

@@ -350,16 +359,20 @@

359

;; reused while the notes/ listing, data_version (another connection's

360

;; commit) and total_changes() (this connection's own writes) are all

361

;; unchanged. A listing with a symlink the walk follows cannot vouch for

−353

;; what is behind it, so then nothing is cached. Returns the key or #f.

+362

;; what is behind it, so then nothing is cached. The key also holds this

+363

;; connection's transaction count: a read inside its own transaction sees

+364

;; uncommitted rows, and a ROLLBACK moves neither data_version nor

+365

;; total_changes, but it does end the transaction. Returns the key or #f.

366

(define (summary-cache-key idx)

367

(guard (exn (else #f))

−356

(let* ((listing (file-tree-metadata (store-notes-dir (folio-index-store idx))))

+368

(let* ((transactions (vector-ref (folio-index-transaction-count idx) 0))

+369

(listing (file-tree-metadata (store-notes-dir (folio-index-store idx))))

370

(db (folio-index-db idx))

371

(dv (sqlite-query-row db "PRAGMA data_version"))

372

(tc (sqlite-query-row db "SELECT total_changes() AS n")))

373

(and listing dv tc

374

(listing-settled? listing #f #t)

−362

(list listing (cdr (assoc 'data_version dv)) (cdr (assoc 'n tc)))))))

+375

(list listing (cdr (assoc 'data_version dv)) (cdr (assoc 'n tc)) transactions)))))

376

377

;; compute returns (cacheable? . value). The key is read first, so a cached

378

;; value is never older than its key.

test/test-status-cache.sglmodified

@@ -130,6 +130,26 @@

130

(assert-equal 3 (length (index-note-summaries idx)))

131

(assert-matches-fresh store idx))))

132

+133

;; A read inside this connection's transaction sees its uncommitted rows,

+134

;; and a ROLLBACK moves neither data_version nor total_changes.

+135

(test "a read inside a transaction that rolls back is not cached"

+136

(with-status-store

+137

(lambda (root store idx server)

+138

(index-note-summaries idx)

+139

(let ((inside #f))

+140

(guard (e ((eq? e 'abort) #f))

+141

(with-index-transaction idx

+142

(lambda ()

+143

(assert-true (sqlite-run (folio-index-db idx)

+144

"UPDATE notes_meta SET last_updated = ? WHERE name = ?"

+145

"2099-12-31" "a"))

+146

(set! inside (folio-note-name

+147

(car (dict-ref (folio-status store idx) recent-notes:))))

+148

(raise 'abort))))

+149

(assert-equal "a" inside))

+150

(assert-equal "c" (folio-note-name (car (dict-ref (folio-status store idx) recent-notes:))))

+151

(assert-matches-fresh store idx))))

+152

153

(test "a note deleted outside folio leaves status before any census"

154

(with-status-store

155

(lambda (root store idx server)