AtlatestRepositoryfolio

folio / treeCHANGELOG.md

Read it rendered

1# Changelog
2
3All notable changes to **folio** are documented in this file.
4
5The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
8## [0.3.4] - 2026-10-25
9
10- Dispatch validated HTTP tool calls in separate cooperative tasks, so a writer waiting on a store lock does not hold up independent reads. Mutating handlers remain serialized through their index flush; existing cross-process file locking is preserved.
11- Use a separate HTTP index connection for reads, keeping uncommitted census/write rows private. Concurrent reads can finish before an in-flight write and see its previous committed state. Reads still have per-query SQLite consistency and atomic Markdown files, without a global multi-file snapshot. The 15-second outside-edit census is unchanged.
12- Reduce JSON response encoding CPU with sigil-json's native string escaping. Require sigil-json 0.16.2 and sigil-mcp 0.18.4. HTTP tool replies remain JSON-RPC JSON documents and use chunked responses; stdio is unchanged.
13- Skip unused snippets in note-list searches while retaining default search-index snippets, ranking and filters. Locate existing full-text rows by indexed literal name for updates, falling back to the previous scan for tokenless/missing names or lookup failure. Keep the existing schema and transactions.
14- HTTP tools taking over ten milliseconds allow socket polling before response encoding with a one-millisecond cooperative pause; fast tools can reply immediately, and individual CPU-bound phases remain cooperative.
15- Add cooperative checkpoints during cold file-cache fills and task-block parsing, allowing independent HTTP reads while these tools populate their caches. Warm file-cache hits keep their direct path. Recheck file fingerprints after a pause before publishing a cached value.
17## [0.3.3] - 2026-10-24
19- `status` reuses its index work while nothing it reads has changed. The note summaries and orphan names, with the sorted recent notes and the orphan list derived from them, are kept while the native `notes/` listing, SQLite's `data_version` (another connection's commit) and `total_changes()` (this connection's own writes) are unchanged, so a note deleted outside folio still leaves `status` at once. The key also holds the connection's transaction count, so rows read inside a transaction that rolls back are not reused. A `notes/` symlink the walk follows, or a failed query, turns the reuse off.
20- No index or file format change: 0.3.1, 0.3.2 and 0.3.3 processes share one store and index.
22## [0.3.2] - 2026-10-24
24- Skip the per-file census on idle ticks. Each 15 s tick compares one native metadata listing of `notes/` and `projects/` (size, mtime-ns and ctime-ns of every entry) and SQLite's `data_version` with the last complete census, and runs the full census only when something differs, another connection committed, or a retryable failure is pending. A listing is trusted only once its newest mtime and ctime are 2 s old, so writes inside one coarse timestamp granule cannot hide. A symlink the walk would follow (any in `projects/`, any not under a dot component in `notes/`), a metadata query that fails, or a listing line that does not parse as 7 fields disables the skip. On a 4,467-note corpus an idle tick drops from 216-611 ms of CPU to about 24 ms.
25- Yield every 32 files during a full census, so a large outside change no longer holds a shared HTTP server's only thread while it is reconciled.
26- Reuse parsed project files, the parsed inbox and buried-id scans while each file's `stat` signature (dev, inode, size, mtime-ns, ctime-ns) is unchanged. A file is cached only once its mtime and ctime are 2 s old; a recently written file is parsed on every call. `tasks`, `projects` and `status` no longer re-parse every project file on each call.
27- Answer `task-notes` from cached note frontmatter, keyed the same way, instead of parsing every note on each call (6.5 s on the corpus above). The answer is still computed from the files at call time.
28- No index or file format change: 0.3.1 and 0.3.2 processes share one store and index.
30## [0.3.1] - 2026-10-22
32- Recheck file existence inside census removal transactions and forget vanished failure entries. Require offline migration before the first upgraded server start.
33- Isolate startup/census failures per item, skip work another process already indexed, and rebuild documents without clearing the shared index. Add offline `--migrate-index` for staged upgrades.
34- Refresh derived indexes after releasing the file-write lock and preserve the original handler error if index maintenance fails.
35- Persist backlinks for notes and projects in the existing SQLite index. Refresh affected entries after writes, renames and project archival while preserving literal link matching and source order.
36- Build status note summaries and orphan detection from indexed metadata instead of loading every note body.
37- Detect external changes periodically in stdio mode as well as HTTP mode. Finish the initial index reconciliation before answering requests, and stop the stdio census when input closes.
39## [0.3.0] - 2026-10-16
41- Add shared MCP HTTP mode with a token-file bearer credential and an IPv4 loopback listener. Preserve stdio.
42- Deprecate the local note viewer: accept --web/--web-port with one stderr warning and ignore them.
43- Detect external edits incrementally without holding the note/project write lock during index maintenance. Refuse full index rebuilds over HTTP; keep stdio maintenance available.
44- Treat invalid crosslink targets as broken links and reject project names that escape their directory.
45- Serialize complete index transactions per connection, and report invalid batch-move entries without aborting later moves.
47## [0.2.14] - 2026-10-16
49### Fixed
51- Update sigil-json to 0.16.1 so encoding long Unicode strings takes linear time, preventing CPU stalls while serializing note responses.
53## [0.2.13] - 2026-10-01
55### Fixed
57- **Concurrent writers no longer lose or tear notes.** Two folio processes on one
58 root (several MCP clients, or a restart overlapping the old process) could
59 each read a note, change it and write it back, so one write silently replaced
60 the other, and a reader could see a half-written file. Writes now take a
61 cross-process lock (`BEGIN IMMEDIATE` on `.folio-write-lock.db` in the root)
62 and replace files atomically through a temporary file and a rename. Measured:
63 2, 4 and 8 processes doing 20 appends each lost 1, 5 and 10 before, and 0
64 after; 6 writers plus a reader damaged 4 of 10 trials before, and 0 after.
65 `FOLIO_LOCK_TIMEOUT` sets how long a writer waits.
67- **A fresh root no longer kills processes that start together.** Several
68 processes creating the root's directories at once could fail with "already
69 exists" and exit. Creating a directory another process just made is now
70 fine. 160 concurrent starts lost 38 before and 0 after
71 (`test/test-startup-race.sh`).
73- **A tool refuses arguments it doesn't declare.** Every tool schema is closed
74 (30 of 30). An undeclared argument used to be ignored, which is how a
75 misnamed `mode` or `operation` plus `content` on `note-edit` replaced a whole
76 245KB note with a few bytes while reporting success. It's now an error, and
77 the note is untouched. `inbox-add` and `task-add` declare `priority`,
78 `source` and `ref`.
80- **`note-edit` keeps a note's title.** `note-create` writes the title as the
81 body's first heading, and replacing the body with `content` that had no
82 heading dropped it. The existing heading now stays unless the new content
83 starts with its own.
85### Changed
87- **Built on sigil 0.22.12.** `sigil-lib` and `sigil-stdlib` move from 0.21.0
88 to 0.22.12 for its build-cache fixes; the other dependencies take patch
89 updates (`sigil-mcp` 0.18.2).
91## [0.2.11] - 2026-08-11
93### Fixed
95- **An unknown argument is refused instead of ignored.** `folio --typo` used to
96 start the MCP server and exit 0, so a mistyped flag looked exactly like a
97 successful run. Unknown arguments now exit 1 with a usage message naming what
98 is actually valid, and `--help` / `-h` work with no environment configured at
99 all (previously `--help` required `FOLIO_ROOT`).
101- **`--version` and `--help` no longer bypass validation.** They were scanned
102 for across the whole argument list before validation ran, so
103 `folio migrate-buried --version` printed a version and exited 0 — the same
104 silent success this release exists to remove, reachable through the fix for
105 it. The check is a property of what can reach it, not of the check itself.
107- **The validator no longer disagrees with the code that reads the arguments.**
108 `folio --log --web` was accepted as "`--log` with the value `--web`", after
109 which the port finder started an HTTP server nobody asked for and the logger
110 wrote a file literally named `--web`, all at exit 0. A validator that blesses
111 an invocation the program then reads differently is worse than no validator.
113- **`--web-port` no longer fails at runtime.** The port check called `exact?`,
114 which is unbound in sigil: it compiled cleanly and raised on every real
115 `--web-port` invocation. Caught by a test arm that runs the actual deployment
116 command rather than a synthetic one.
118### Added
120- `--stdio` is accepted and ignored, as a documented compatibility no-op. It
121 names the transport folio already uses, and it is a common MCP client
122 convention.
124- `test/test-cli-contract.sh`, 23 arms, each verified individually falsifiable
125 against a stubbed CLI. Some arms deliberately run live deployment invocations
126 rather than defect cases; they are marked as such because that is what caught
127 the `exact?` runtime failure, and they are the opposite of redundant.
129### Known
131- `--log-level=trace` is refused with a correction pointing at
132 `--log-level trace`. The consumers match flag names exactly and never split
133 on `=`, so accepting the joined spelling would start folio and silently
134 ignore the flag. Accepting both spellings honestly requires folding the three
135 argument parsers into one pass, which is tracked separately.
137- `serve` remains accepted. It was never a subcommand: there was no dispatch,
138 and the old loop ignored unrecognised arguments, so `folio serve` has always
139 worked by accident. Live deployments depend on it.
141## [0.2.10] - 2026-08-06
143### Fixed
145- **A multi-line capture round-trips: an item's extent is SYNTACTIC, and the
146 burial class is closed.** A task is its checkbox header line — metadata at
147 the end of that line — plus the run of following lines indented at least
148 two columns past it (one blank line is allowed between body lines when
149 bracketed by indented ones). Extent is decided by indentation alone, never
150 by content: no heading, fence or hashtag heuristics anywhere, so a removal
151 can only ever take lines indented under the item's own header, and
152 unindented prose, headings and config fragments are untouchable by task
153 operations *by construction*.
155 This ends the mechanism that buried at least 22 items between April and
156 August 2026: the old writer appended `{id: ...}` after the whole text, a
157 newline put it on the capture's LAST line, the line-oriented parser saw
158 only the headless first line, and `backfill-ids!` then minted a phantom id
159 onto it — one capture became a phantom head plus an invisible tail, while
160 `folio/status` reported the inbox empty. Writers now put metadata on the
161 header and indent the body; every mutation (complete, reopen, update, move,
162 triage, sweep, clean) operates on whole blocks by their recorded line
163 extents. Moves also now preserve completion — a completed task used to
164 arrive reopened while still carrying its `completed:` date.
166- **Buried items are reported loudly, and id assignment freezes around
167 them.** A new independent scanner — sharing no code with the parser, by
168 design and by lesson — walks raw file text for `t-` ids and compares
169 against what the parser actually produced. Anything written on disk but
170 unreachable by every query is named, with its line, in `folio/inbox` and
171 `folio/status`. `backfill-ids!` refuses to touch a file holding a buried
172 id, so the phantom-mint path is closed from both ends. Ids inside
173 terminated column-0 HTML comments are annotation and are not flagged; ids
174 quoted inside a visible task's own text are that task's content, not a
175 burial.
177- **Rewrites that would destroy bytes now refuse instead.** A header whose
178 parse is lossy (stray braces around the metadata) loses part of the line
179 on every re-serialize — completing one such task destroyed its tail, and a
180 plain LISTING destroyed `{braces} in text}` via backfill (a defect that
181 predates this release). Every in-place rewrite, move and triage now checks
182 the header survives parse→serialize byte-identically and refuses by name
183 when it does not; backfill skips such lines.
185- **Text containing tool-call markup is refused at the door.** `</text>`,
186 `<parameter name=` and `</parameter>` can never be legitimate captured
187 content — their presence means the client's tool call fell apart and its
188 argument list leaked into a value (this buried two items live on
189 2026-08-06). All string arguments of `inbox-add`, `task-add` and
190 `task-update` are checked; nothing is stored on refusal.
192### Added
194- **`folio/migrate-buried`** — explicit recovery of legacy buried items.
195 Dry-run by default: prints per-id actions and a unified diff, writes
196 nothing. It converts the two understood burial shapes (headless checkbox
197 head with a trailing-metadata body; a bare prose block ending in an id
198 line), REFUSES what it cannot convert without guessing — a bare-id phantom
199 head above the tail (recovery is a human MERGE of two tasks), and
200 tool-fragment shapes whose metadata does not close on its own line — and
201 verifies every conversion with the independent scanner before accepting
202 it. A conversion deletes nothing: lines are reindented and one metadata
203 substring relocated, line count preserved exactly.
205### Compatibility
207- Files written by 0.2.10 remain readable by older binaries: headers are
208 unchanged as task lines, and indented bodies read as inert prose. Older
209 binaries cannot SEE bodies (they list the header line only) and must not
210 be used to mutate a store containing multi-line items — their line-based
211 removal would strand bodies. Upgrade the binary before writing.
213## [0.2.9] - 2026-08-05
215### Fixed
217- **The note walk no longer has a size ceiling, and folio can start again.**
218 `find-md-files-recursive` folded each directory's entries with `fold-right`,
219 which is not tail recursive — it burned one interpreter stack frame per
220 file. The maximum size of a folio was therefore set by the interpreter's
221 stack rather than by anything folio chose, and past roughly **1011 entries in
222 a single directory the walk blew the stack outright**.
224 Because every tool that enumerates notes starts at that walk, the blast
225 radius was the whole server. A fresh `folio serve` could not finish starting
226 — it died during index reconciliation, after logging `Opened search index`
227 and before `MCP server ready`. In an already-running server, startup had
228 succeeded back when the corpus was smaller, so the process stayed healthy
229 while individual calls began to fail: `note-links`, `status` and
230 `note-rename` stopped answering, and everything that addresses a note by path
231 (`note-read`, `note-create`, `note-patch`, the task tools) kept working. That
232 partition is what the defect looks like from outside.
234 The walk is now iterative, with an explicit worklist. Stack depth is the
235 directory *nesting* depth — single digits — instead of the file count.
236 Ordering is unchanged: `directory-list` order, depth-first, subdirectories
237 expanded in place. Verified against a real corpus with all 2460 notes in one
238 flat directory, 2.4x past the old ceiling, at no measurable cost (cold
239 reindex 14.0s before, 14.2s after).
241 Measured, on a copy of a real vault: 1010 entries fine, 1012 fatal. Six empty
242 notes containing no links at all were enough to cross it, and the same
243 threshold held whichever six were added — it was the file count, not the
244 content and not the link graph.
246- **A tool that dies mid-call now says so instead of going silent.** When a
247 handler raised through the internal CPS frames of a higher-order stdlib
248 procedure, the native backend mis-unwound it: the dispatcher's guard never
249 ran, no JSON-RPC response was written at all, and the caller blocked until
250 its idle timeout while the server stayed up and answered later calls.
252 Silence is the worst failure a caller can be handed — it is indistinguishable
253 from *slow* and from *succeeded*, so the reasonable response is to retry, and
254 a retry of a mutation whose disk write already landed double-applies. This
255 was not hypothetical: a `note-rename` that never answered had already renamed
256 the file and updated none of its crosslinks.
258 Every tool handler is now wrapped. Ordinary failures are re-raised unchanged,
259 so they still arrive as a proper JSON-RPC error envelope; stack exhaustion —
260 the one condition measured to be unserialisable — is returned as an explicit
261 report that also warns the change may be partially applied. It is a
262 known-list rather than a catch-all: an unmeasured condition keeps existing
263 behaviour instead of getting a guess.
265### Known issues
267- **`note-rename` is not atomic.** It renames the file first, then rewrites
268 crosslinks across referring files with no transaction and no rollback, so a
269 failure partway through leaves the note renamed and some or all crosslinks
270 stale — with no error if the failure was the silent kind described above.
271 The walk fix removes the cause seen so far, but not the exposure. Tracked
272 separately.
274## [0.2.7] - 2026-08-02
276### Fixed
278- **Moving a task no longer renumbers it.** Ids are preserved in every
279 direction — inbox → project, project → inbox, project → project, and
280 `folio/inbox-triage`. An id you hold before a move is still valid after it,
281 so `task-move` → `task-update <same-id>` chains correctly and a triage plan
282 built from an inbox dump keeps working once the items are routed.
284 This was never a design constraint. Ids are globally unique random hex with
285 no per-project id space, so nothing required a new one on move.
286 `project-add-task!` simply called `generate-task-id` unconditionally and
287 ignored any id handed to it, while `move-task-in-store!` stripped the id out
288 on the way in. The asymmetry gave it away as an oversight: project → inbox
289 passed metadata through untouched, and only inbox → project destroyed the id.
291- **A task can no longer end up carrying two `id` entries.** Moving a task from
292 a project back to the inbox prepended a fresh id without removing the
293 existing one, leaving both on the line; lookups silently resolved the first.
295### Added
297- **`folio/task-move-batch`** — move many tasks in one call, either
298 `{ids: [...], project: "x"}` for a shared target or
299 `{moves: [{id, project}, ...]}` to fan a single call out across many
300 projects. An entry with no `project` moves that task to the inbox.
302 Returns a result line per id so partial failures are visible rather than
303 swallowed, and the header counts successes rather than attempts. Because ids
304 are now stable, the reported id is the one the task landed under.
306 Triaging 227 inbox items on 2026-08-02 cost 226 individual round-trips, which
307 is a large part of why that inbox went untriaged for four months.
309### Changed
311- Toolchain moved to sigil 0.18 (`sigil ^0.18`, `sigil-run ^0.18.0`,
312 `sigil-http ^0.18.0`, `sigil-sxml ^0.15.0`), with runtime core resolving to
313 0.18.1. `sigil-test` / `sigil-test-runner` remain pinned at `^0.17` (0.17.16):
314 bumping them aborts the install because sigil-http, sigil-tls, sigil-crypto
315 and sigil-web-styles each declare a dev-dependency on `sigil-test ^0.17` and
316 the resolver unifies across the graph. That affects the test harness only,
317 not the shipped runtime, and clears once those four repos move their dev-dep.
319### Known issues
321- **A capture containing newlines is still stored in a shape folio cannot read
322 back.** `folio/inbox-add` accepts multi-line text, but the `{id: ...}`
323 metadata is appended to the END of the text, so on a multi-line capture it
324 lands on the item's last line. The parser sees only the first line, which has
325 no id, and `backfill-ids!` then mints a fresh id onto it — the real item and
326 its metadata become unreachable and a phantom one-line duplicate appears in
327 its place. Six items were hidden this way between April and August 2026.
329 This is not fixed in 0.2.7. A block-parsing fix was written and withdrawn: it
330 cured the burial but introduced worse defects, because it inferred where an
331 item ends and then let destructive operations act on that inference. The
332 redesign makes an item's extent syntactic rather than inferred.
334 Until it lands, **capture multi-line content as a note and reference it from
335 a single-line inbox item**, rather than putting the body in the item itself.