Commit569ef106Recorded2 Sep 2026Repositorysigil-sqlite

docs: the busy budget is now a latency knob, and release notes for 0.16.3

Message

The parameter did not change but its job did, and the next person to tune it will be reasoning about the old failure mode.

BEFORE, exhausting sqlite-busy-budget-seconds was catastrophic: it raised, the raise leaked the statement, and the leak pinned the connection for the life of the process. 5 seconds was a number chosen to make exhaustion RARE.

NOW exhaustion is merely a raise the caller sees, and every statement is finalized on the way out. What the budget sets instead is the WORST-CASE STALL every other database user on that connection pays, because they queue behind the holder. Different jobs; the number was chosen for the first one.

Deliberately not changed here. Tuning it is a separate decision that should be made against the new meaning rather than inherited from the old.

Also adds RELEASE-NOTES-v0.16.3.md, which leads with the behaviour change rather than the defect: a consumer reading "0.16.3" learns nothing, and what they need to know is that database calls on one connection now serialise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuSfnnNgQ81hjRygkkqWM6

Changed
 RELEASE-NOTES-v0.16.3.md | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 docs/concurrency.md      |  23 +++++++++++++++++++++++
 2 files changed, 149 insertions(+)
Diff
RELEASE-NOTES-v0.16.3.mdadded
@@ -0,0 +1,126 @@
+1
# sigil-sqlite 0.16.3
+2
+3
**A connection is now serialised across each high-level call. This changes
+4
concurrency semantics: database calls on ONE connection queue behind each
+5
other. If you want them to proceed concurrently, give each fiber its own
+6
connection.**
+7
+8
Patch release, no API break. Three defects, all reachable in normal use, all in
+9
this library. Consumers need no code change; they may need to reconsider how
+10
many connections they open.
+11
+12
## What was wrong
+13
+14
A SQLite connection is a serial resource. SQLite ends a connection's implicit
+15
read transaction only when *no statement on it is active*. So if fiber A has a
+16
statement open on a connection -- parked in the cooperative busy-retry, or
+17
part-way through a result set -- and fiber B runs a query on the *same*
+18
connection, B's read transaction cannot be released when B finalizes: A is still
+19
using the btree. The connection is pinned to B's snapshot until A completes, and
+20
every later read on it silently misses anything committed in between.
+21
+22
Then it got worse on its own. Once pinned, the connection's own parked write can
+23
never complete, so it exhausts the busy budget and RAISES -- and `sqlite-query`
+24
finalized only on its `else` branch, so the raise leaked the statement. A leaked
+25
statement holds the read transaction open **for the life of the process**.
+26
+27
Observed in kiln, whose controller shares one connection between its HTTP app
+28
and its run loop: `GET /api/groups` answered **HTTP 200 with a pre-commit view
+29
of its own database** -- 55 rows where an independent connection to the same
+30
file saw 755 -- and stayed that way until the process was restarted. No error,
+31
no log line, no way for a caller to tell.
+32
+33
## What changed
+34
+35
**1. The connection is held across the whole primitive.** `sqlite-query`,
+36
`sqlite-query-row`, `sqlite-run` and `sqlite-exec` take a per-connection lock
+37
for their entire duration, yields included.
+38
+39
Guarding only the busy window would not have been enough, and that is measured
+40
rather than argued: a fiber yields inside an *uncontended* query too,
+41
allocation-driven, roughly **once per 450 rows**. A witness fiber counts ~44
+42
yields across a 20000-row query with no contention anywhere.
+43
+44
Scope is deliberately the primitive. Held any coarser it would deadlock the
+45
moment one of a consumer's database helpers called another. At primitive scope
+46
reentrancy is impossible by construction: no user code runs between prepare and
+47
finalize.
+48
+49
**2. Statements are finalized on every exit path**, raises included.
+50
+51
**3. `PRAGMA busy_timeout=N` is honoured instead of silently swallowed.** It
+52
cannot be passed to SQLite -- a blocking step freezes the whole cooperative
+53
scheduler, measured at 0 witness ticks in two seconds against 4701 for the
+54
retrying path -- so it now sets the cooperative retry budget, which is the same
+55
intent expressed where this library can act on it. `sqlite-busy-budget` and
+56
`sqlite-set-busy-budget!` are the explicit form. The budget is process-wide.
+57
+58
## What it costs you, measured
+59
+60
An external writer held the write lock for 500ms while two fibers used one
+61
shared connection:
+62
+63
| | |
+64
|---|---|
+65
| read on the **shared** connection | **452 ms** -- it queued behind the parked write |
+66
| read on **its own** connection | **1.1 ms** |
+67
+68
At rest the lock costs nothing; the existing cooperative-busy-retry test's
+69
timings are unchanged. Under contention a fiber waits for however long the
+70
connection's holder is parked, **bounded by the busy budget -- 5 seconds by
+71
default**. That is the number to weigh. The remedy is a connection per fiber,
+72
and WAL readers are cheap.
+73
+74
A fiber that waits more than three times the busy budget for a connection
+75
**raises** rather than waiting forever, so two fibers deadlocking over two
+76
connections is a loud error instead of a silent stop.
+77
+78
+79
## THE BUSY BUDGET NOW MEANS SOMETHING DIFFERENT
+80
+81
Read this before tuning `*sqlite-busy-budget-seconds*` (5.0s), because the
+82
parameter did not change but **its job did**.
+83
+84
**Before:** exhausting the budget was catastrophic. It raised, the raise leaked
+85
the statement, and the leaked statement pinned the connection for the life of
+86
the process. So the budget was a number chosen to make exhaustion *rare* — long
+87
enough that ordinary contention never reached it.
+88
+89
**Now:** exhaustion is merely a raise the caller sees and can handle, and every
+90
statement is finalized on the way out. What the budget sets instead is the
+91
**worst-case stall every other database user on that connection pays**, because
+92
they queue behind the holder.
+93
+94
Those are different jobs, and 5 seconds was chosen for the first one. A shorter
+95
budget now means faster failure and a shorter worst-case stall; a longer one
+96
means more patience under contention and a longer stall. Neither reading is
+97
available from the old number.
+98
+99
It has deliberately not been changed here. Tuning it is a separate decision that
+100
should be made against the new meaning rather than inherited from the old.
+101
+102
## Not covered, deliberately
+103
+104
- **The low-level API.** `sqlite-prepare` / `sqlite-step` / `sqlite-finalize`
+105
are not serialised and cannot be: the library cannot know when your statement
+106
is finished. Holding a statement across steps on a connection another fiber
+107
uses keeps the hazard above.
+108
- **Multi-statement transactions.** `BEGIN` … `COMMIT` spans several
+109
primitives, so another fiber's query can still interleave inside your
+110
transaction. Give a transaction its own connection.
+111
+112
Both are in `docs/concurrency.md`.
+113
+114
## New
+115
+116
- `sqlite-busy-budget`, `sqlite-set-busy-budget!` -- the retry budget, explicitly.
+117
- `sqlite-open-statements` -- statements open through the high-level API. 0 at
+118
rest; non-zero at rest is a leaked statement.
+119
- `sqlite-connection-locked?` -- diagnostic.
+120
- `docs/concurrency.md`.
+121
- `test/test-sqlite-connection-pin.sgl` -- the regression test. Four arms, the
+122
first two differing by exactly one read, which is the variable that decides
+123
the bug.
+124
- `sigil-test` / `sigil-test-runner` as dev-dependencies: `sigil test` refuses
+125
to build a native package without them, so the README's `sigil test --report`
+126
could not have worked.
docs/concurrency.mdmodified
@@ -79,3 +79,26 @@ statement and has no connection to look a per-connection value up on. Prefer
79
`sqlite-connection-locked?` answers whether a fiber currently holds a
80
connection. It is a diagnostic, not a synchronisation primitive: by the time you
81
act on the answer it may have changed.
+82
+83
## THE BUSY BUDGET NOW MEANS SOMETHING DIFFERENT
+84
+85
Read this before tuning `*sqlite-busy-budget-seconds*` (5.0s), because the
+86
parameter did not change but **its job did**.
+87
+88
**Before:** exhausting the budget was catastrophic. It raised, the raise leaked
+89
the statement, and the leaked statement pinned the connection for the life of
+90
the process. So the budget was a number chosen to make exhaustion *rare* — long
+91
enough that ordinary contention never reached it.
+92
+93
**Now:** exhaustion is merely a raise the caller sees and can handle, and every
+94
statement is finalized on the way out. What the budget sets instead is the
+95
**worst-case stall every other database user on that connection pays**, because
+96
they queue behind the holder.
+97
+98
Those are different jobs, and 5 seconds was chosen for the first one. A shorter
+99
budget now means faster failure and a shorter worst-case stall; a longer one
+100
means more patience under contention and a longer stall. Neither reading is
+101
available from the old number.
+102
+103
It has deliberately not been changed here. Tuning it is a separate decision that
+104
should be made against the new meaning rather than inherited from the old.