v0.16.3

A connection is now serialised across each high-level call. This changes concurrency semantics: database calls on ONE connection queue behind each other. If you want them to proceed concurrently, give each fiber its own connection.

Patch release, no API break. Three defects, all reachable in normal use, all in this library. Consumers need no code change; they may need to reconsider how many connections they open.

What was wrong

A SQLite connection is a serial resource. SQLite ends a connection's implicit read transaction only when no statement on it is active. So if fiber A has a statement open on a connection -- parked in the cooperative busy-retry, or part-way through a result set -- and fiber B runs a query on the same connection, B's read transaction cannot be released when B finalizes: A is still using the btree. The connection is pinned to B's snapshot until A completes, and every later read on it silently misses anything committed in between.

Then it got worse on its own. Once pinned, the connection's own parked write can never complete, so it exhausts the busy budget and RAISES -- and sqlite-query finalized only on its else branch, so the raise leaked the statement. A leaked statement holds the read transaction open for the life of the process.

Observed in kiln, whose controller shares one connection between its HTTP app and its run loop: GET /api/groups answered HTTP 200 with a pre-commit view of its own database -- 55 rows where an independent connection to the same file saw 755 -- and stayed that way until the process was restarted. No error, no log line, no way for a caller to tell.

What changed

1. The connection is held across the whole primitive. sqlite-query, sqlite-query-row, sqlite-run and sqlite-exec take a per-connection lock for their entire duration, yields included.

Guarding only the busy window would not have been enough, and that is measured rather than argued: a fiber yields inside an uncontended query too, allocation-driven, roughly once per 450 rows. A witness fiber counts ~44 yields across a 20000-row query with no contention anywhere.

Scope is deliberately the primitive. Held any coarser it would deadlock the moment one of a consumer's database helpers called another. At primitive scope reentrancy is impossible by construction: no user code runs between prepare and finalize.

2. Statements are finalized on every exit path, raises included.

3. PRAGMA busy_timeout=N is honoured instead of silently swallowed. It cannot be passed to SQLite -- a blocking step freezes the whole cooperative scheduler, measured at 0 witness ticks in two seconds against 4701 for the retrying path -- so it now sets the cooperative retry budget, which is the same intent expressed where this library can act on it. sqlite-busy-budget and sqlite-set-busy-budget! are the explicit form. The budget is process-wide.

What it costs you, measured

An external writer held the write lock for 500ms while two fibers used one shared connection:

| | | |---|---| | read on the shared connection | 452 ms -- it queued behind the parked write | | read on its own connection | 1.1 ms |

At rest the lock costs nothing; the existing cooperative-busy-retry test's timings are unchanged. Under contention a fiber waits for however long the connection's holder is parked, bounded by the busy budget -- 5 seconds by default. That is the number to weigh. The remedy is a connection per fiber, and WAL readers are cheap.

A fiber that waits more than three times the busy budget for a connection raises rather than waiting forever, so two fibers deadlocking over two connections is a loud error instead of a silent stop.

THE BUSY BUDGET NOW MEANS SOMETHING DIFFERENT

Read this before tuning *sqlite-busy-budget-seconds* (5.0s), because the parameter did not change but its job did.

Before: exhausting the budget was catastrophic. It raised, the raise leaked the statement, and the leaked statement pinned the connection for the life of the process. So the budget was a number chosen to make exhaustion rare — long enough that ordinary contention never reached it.

Now: exhaustion is merely a raise the caller sees and can handle, and every statement is finalized on the way out. What the budget sets instead is the worst-case stall every other database user on that connection pays, because they queue behind the holder.

Those are different jobs, and 5 seconds was chosen for the first one. A shorter budget now means faster failure and a shorter worst-case stall; a longer one means more patience under contention and a longer stall. Neither reading is available from the old number.

It has deliberately not been changed here. Tuning it is a separate decision that should be made against the new meaning rather than inherited from the old.

Not covered, deliberately

• The low-level API. sqlite-prepare / sqlite-step / sqlite-finalize are not serialised and cannot be: the library cannot know when your statement is finished. Holding a statement across steps on a connection another fiber uses keeps the hazard above.
• Multi-statement transactions. BEGIN … COMMIT spans several primitives, so another fiber's query can still interleave inside your transaction. Give a transaction its own connection.

Both are in docs/concurrency.md.

New

• sqlite-busy-budget, sqlite-set-busy-budget! -- the retry budget, explicitly.
• sqlite-open-statements -- statements open through the high-level API. 0 at rest; non-zero at rest is a leaked statement.
• sqlite-connection-locked? -- diagnostic.
• docs/concurrency.md.
• test/test-sqlite-connection-pin.sgl -- the regression test. Four arms, the first two differing by exactly one read, which is the variable that decides the bug.
• sigil-test / sigil-test-runner as dev-dependencies: sigil test refuses to build a native package without them, so the README's sigil test --report could not have worked.