Commit4e088cc2Recorded5 Aug 2026Repositoryapiary
Stop reporting delivery we cannot evidence, and stop letting our own writes hide a dead link
Message
Two defects in the Enclave bridge, measured 2026-08-04 against an independent observer on a healthy connection. They compose, so they are fixed together: the second supplies the signal the first needs.
BLOCKER 2 -- the idle clock was reset by OUTBOUND traffic.
enclave-conn-touch-activity! was called from the enclave-post* helpers as well as from the inbound handlers, and its docstring described that as the design: "any traffic on the wire -- in either direction -- resets the idle clock". A write succeeds into the local kernel send buffer and proves nothing whatever about the peer, so a bot posting more often than the 60s idle threshold kept resetting its own timer with writes that went nowhere, never PINGed, and never noticed the link was dead. The busier the bot, the blinder it was. Matched pair, same proxy, frozen at the same instant, differing only in whether they posted: the silent one hit its PONG timeout at t+96s and recovered, the chatty one never fired at all.
The clock is now inbound-only, and renamed to say so -- last-inbound-ts, enclave-conn-note-inbound! -- so that a future outbound caller reads as obviously wrong. The old name described the mechanism while its docstring described the bug as though it were intended. Seeding the clock at connect and after a successful reconnect stays: completing registration means the server sent us 001, which is genuine inbound evidence.
BLOCKER 1 -- send-channel reported success for messages that vanished.
With a socket frozen, 14 consecutive posts each returned "Broadcast to <channel> (1 line)" and none arrived. That is worse than a wedge: a wedge is visibly stuck, this lies, and it lies in the transport proposed for carrying worker status.
A post is now followed by a PING carrying a fresh random token, and the result string depends on whether that exact token comes back. TCP is ordered, so a PONG for a token sent AFTER the message means the peer's parser has already consumed the message. That is evidence, not a freshness heuristic -- a reading of the idle clock taken BEFORE the write says nothing about the write.
confirmed "Broadcast to #x (1 line)" (unchanged)
otherwise "Wrote to #x (1 line) - UNCONFIRMED: ..."Deliberately not the word "queued". Nothing is queued: no buffer holds the message and no retry will send it. A caller told its message was queued would wait instead of re-sending -- a fresh false claim in the opposite direction, which is the defect being removed, one layer up.
What this proves is receipt and parsing by the server, NOT fan-out to the channel or to any member. That is a large improvement over "we wrote to a socket" and it is still short of a delivery receipt; the code says so where somebody might over-read it.
Two structural guards, because the obvious implementation of blocker 1 reintroduces blocker 2 inside itself:
- Outstanding PINGs are a LIST of (token . sent-ts), and confirmation can only ADD. There is deliberately no "update the pending ping" operation. The PONG timeout is measured from the OLDEST entry, so a caller posting frequently can never postpone its own dead-link deadline. With a single re-stamped slot it could -- which is exactly blocker 2 moved one function over.
- A PING already outstanding past the budget settles the question, so confirmation answers immediately rather than sending another and spending the budget again. Only the first post in an outage waits.
Useful side effect: a post-time PING seeds the keepalive, so a busy bot now reaches its PONG timeout from the moment it posted into a dead link instead of never reaching it at all.
Tests: the inbound-only guard is aimed at the product, not at a mirror of it. enclave-post-multiline with no lines returns 'sent-empty without touching the irc connection, and the old code refreshed the clock above that check -- so the test fails against the pre-fix source and passes after.
Changed
src/apiary/enclave.sgl | 423 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------
src/apiary/tools.sgl | 59 ++++++++++++++++++++----
test/test-enclave.sgl | 162 +++++++++++++++++++++++++++++++++++++++++++++++++----------------
test/test-tools.sgl | 25 ++++++++++
4 files changed, 534 insertions(+), 135 deletions(-)Diff
src/apiary/enclave.sglmodified
@@ -82,20 +82,26 @@
82
set-enclave-conn-caps-acked! 83
enclave-conn-last-batch-fail 84
set-enclave-conn-last-batch-fail!−85
enclave-conn-last-activity-ts−86
set-enclave-conn-last-activity-ts!−87
enclave-conn-pending-ping-token−88
set-enclave-conn-pending-ping-token!−89
enclave-conn-pending-ping-sent-ts−90
set-enclave-conn-pending-ping-sent-ts!+85
enclave-conn-last-inbound-ts+86
set-enclave-conn-last-inbound-ts!+87
enclave-conn-pending-pings+88
set-enclave-conn-pending-pings!+89
enclave-conn-ping-add!+90
enclave-conn-ping-ack!+91
enclave-conn-ping-waiting?+92
enclave-conn-pings-clear!+93
enclave-conn-ping-outstanding?+94
enclave-conn-oldest-ping-ts 95
enclave-conn-reconnecting? 96
set-enclave-conn-reconnecting?! 97
enclave-conn-reconnect-attempt 98
set-enclave-conn-reconnect-attempt! 99
enclave-conn-shutdown? 100
set-enclave-conn-shutdown?!−97
enclave-conn-touch-activity!+101
enclave-conn-note-inbound! 102
enclave-conn-keepalive-tick!+103
enclave-confirm-delivery!+104
+post-confirm-timeout+ 105
await-registration! 106
enclave-connect 107
enclave-disconnect@@ -218,19 +224,39 @@
224
(last-batch-fail default: #f mutable: #t) 225
;; Client-side PING/PONG keepalive bookkeeping, paired with 226
;; the enclave-server side in [[topics/enclave-deployment]].−221
;; `last-activity-ts` is the epoch second of the most recent−222
;; outbound or inbound IRC traffic — refreshed on every send−223
;; and every receive. The keepalive tick uses−224
;; `(now - last-activity-ts) < idle-threshold` to decide−225
;; whether to PING.−226
(last-activity-ts default: #f mutable: #t)−227
;; Pending client→server PING bookkeeping. `token` is what we−228
;; sent in the trailing param; the PONG handler clears both−229
;; slots on a token match. When `(now - sent-ts) > pong-−230
;; timeout` and the slots are still set, the connection is−231
;; presumed dead and the reconnect path fires.−232
(pending-ping-token default: #f mutable: #t)−233
(pending-ping-sent-ts default: #f mutable: #t)+227
;;+228
;; `last-inbound-ts` is the epoch second of the most recent+229
;; INBOUND IRC traffic. INBOUND ONLY — this is load-bearing+230
;; and it used to be wrong.+231
;;+232
;; An outbound write succeeds into the local kernel send+233
;; buffer and proves NOTHING about the peer: on a blackholed+234
;; route every send still "succeeds" while nothing arrives.+235
;; When outbound traffic refreshed this clock, a bot that+236
;; posted more often than `+keepalive-idle-threshold+` kept+237
;; resetting its own idle timer with writes that went nowhere+238
;; and so never PINGed and never detected the dead link. The+239
;; busier the bot, the blinder it was. Measured 2026-08-04 as+240
;; a matched pair: a silent bot hit its PONG timeout at t+96s+241
;; and recovered; a bot posting every 20s never fired at all.+242
;;+243
;; Only inbound bytes are evidence that the peer is alive.+244
(last-inbound-ts default: #f mutable: #t)+245
;; Outstanding client→server PINGs as a list of+246
;; `(token . sent-ts)`, most recent first. The PONG handler+247
;; drops the entry whose token matches; the keepalive tick+248
;; measures its timeout from the OLDEST outstanding entry.+249
;;+250
;; A LIST rather than a single slot, deliberately: post-time+251
;; delivery confirmation (`enclave-confirm-delivery!`) also+252
;; emits PINGs, and if it could re-stamp an already-pending+253
;; one, a bot posting faster than `+keepalive-pong-timeout+`+254
;; would keep pushing its own deadline out — reintroducing+255
;; exactly the defect above in a new place. Entries can only+256
;; be ADDED, each keeps its own sent-ts, and the deadline is+257
;; always measured from the oldest, so that is structurally+258
;; impossible rather than merely avoided by care.+259
(pending-pings default: '() mutable: #t) 260
;; Throttle the keepalive sweep — the event loop ticks every 261
;; 50ms but the keepalive logic only needs to run every few 262
;; seconds, so we record when we last ran the sweep and skip@@ -291,8 +317,34 @@
317
;; connection is reaped + reconnected before the next send- 318
;; channel call sails into the bit-bucket. 319
(define +keepalive-poll-interval+ 5.0) ;; sweep cadence (seconds)−294
(define +keepalive-idle-threshold+ 60.0) ;; ping if no traffic in last N seconds+320
(define +keepalive-idle-threshold+ 60.0) ;; ping if no INBOUND traffic in last N seconds 321
(define +keepalive-pong-timeout+ 30.0) ;; reconnect if PONG hasn't arrived after N seconds+322
+323
;; How long `enclave-confirm-delivery!` waits for the PONG that+324
;; proves the peer consumed the bytes we just wrote.+325
;;+326
;; MEASURED, not chosen. Against a local enclave-server the+327
;; observed confirmation latency is in `test/harness/` (see+328
;; `exp_confirm_latency.py`, which prints p50/p95/max over N+329
;; posts). The budget is set far above the measured worst case+330
;; on purpose: a false UNCONFIRMED on a merely slow link is+331
;; worse than a slow post, because the first person who sees a+332
;; spurious one starts ignoring the warning, and then the+333
;; signal this whole change exists to create is gone.+334
;;+335
;; It also stays well under `+keepalive-pong-timeout+` (30 s),+336
;; so a confirmation wait can never outlive the window in which+337
;; the keepalive would declare the same link dead.+338
(define +post-confirm-timeout+ 3.0)+339
;; Poll step while waiting. Sleeping yields to the cooperative+340
;; scheduler so the event-loop goroutine can read the socket and+341
;; clear our token — VERIFIED by running it, not assumed. If+342
;; nothing else ran during this sleep, every post would cost the+343
;; full budget and confirm nothing.+344
(define +post-confirm-poll+ 0.02)+345
;; Hard cap on outstanding PINGs so a burst of posts inside one+346
;; confirmation budget cannot grow the list without bound.+347
(define +max-outstanding-pings+ 4) 348
(define +reconnect-backoff-min+ 1.0) 349
(define +reconnect-backoff-max+ 60.0) 350
;; sigil-irc completes its handshake (TCP connect → CAP LS →@@ -901,13 +953,84 @@
953
(: enclave-conn? -> boolean?) 954
(irc-connected? (enclave-conn-irc conn))) 955
−904
;;; Refresh the keepalive activity timestamp. Called from the−905
;;; PRIVMSG / BATCH / CAP / FAIL inbound handlers and from the−906
;;; outbound `enclave-post*` helpers, so any traffic on the−907
;;; wire — in either direction — resets the idle clock.−908
(define (enclave-conn-touch-activity! conn)+956
;;; Record that INBOUND traffic just arrived, resetting the+957
;;; keepalive idle clock.+958
;;;+959
;;; Call this ONLY from inbound handlers, and only when bytes+960
;;; genuinely arrived from the peer. It must never be called+961
;;; from an outbound path: a successful write proves the local+962
;;; kernel accepted the bytes, not that anybody received them,+963
;;; and treating it as liveness evidence is what let a busy bot+964
;;; suppress its own dead-connection detection. The one other+965
;;; legitimate caller is a freshly established connection,+966
;;; where completing registration IS inbound evidence.+967
;;;+968
;;; Named for what it means rather than for what it touches, so+969
;;; that a future outbound caller reads as obviously wrong. The+970
;;; previous name (`touch-activity!`) described the mechanism+971
;;; and its docstring described the bug as though it were the+972
;;; design.+973
(define (enclave-conn-note-inbound! conn)+974
(: enclave-conn? -> any?)+975
(set-enclave-conn-last-inbound-ts! conn (current-second)))+976
+977
;; ---- outstanding PING bookkeeping ---------------------------+978
;;+979
;; Entries are `(token . sent-ts)`, most recent first. See the+980
;; `pending-pings` field comment for why this is a list.+981
+982
;;; Record a PING we just put on the wire. ADD-only by design —+983
;;; there is deliberately no "update the pending ping" operation,+984
;;; because that is precisely how a frequent caller would push+985
;;; its own dead-link deadline out indefinitely.+986
(define (enclave-conn-ping-add! conn token sent-ts)+987
(: enclave-conn? string? number? -> any?)+988
(set-enclave-conn-pending-pings!+989
conn (cons (cons token sent-ts)+990
(enclave-conn-pending-pings conn))))+991
+992
;;; #t when at least one PING is still unanswered.+993
(define (enclave-conn-ping-outstanding? conn)+994
(: enclave-conn? -> boolean?)+995
(pair? (enclave-conn-pending-pings conn)))+996
+997
;;; #t when THIS token specifically is still unanswered.+998
(define (enclave-conn-ping-waiting? conn token)+999
(: enclave-conn? string? -> boolean?)+1000
(and (assoc token (enclave-conn-pending-pings conn)) #t))+1001
+1002
;;; Epoch second of the OLDEST unanswered PING, or #f if none.+1003
;;; The keepalive measures its PONG timeout from this, so a+1004
;;; newer PING can never postpone an older one's deadline.+1005
(define (enclave-conn-oldest-ping-ts conn) 1006
(: enclave-conn? -> any?)−910
(set-enclave-conn-last-activity-ts! conn (current-second)))+1007
(let loop ((entries (enclave-conn-pending-pings conn))+1008
(oldest #f))+1009
(cond+1010
((null? entries) oldest)+1011
((or (not oldest) (< (cdar entries) oldest))+1012
(loop (cdr entries) (cdar entries)))+1013
(else (loop (cdr entries) oldest)))))+1014
+1015
;;; Drop the entry matching `token`. Returns #t if one was+1016
;;; removed, #f if the token was unknown (a stale or forged+1017
;;; PONG, which must not clear anybody else's entry).+1018
(define (enclave-conn-ping-ack! conn token)+1019
(: enclave-conn? string? -> boolean?)+1020
(let ((entries (enclave-conn-pending-pings conn)))+1021
(cond+1022
((not (assoc token entries)) #f)+1023
(else+1024
(set-enclave-conn-pending-pings!+1025
conn (filter (lambda (e) (not (equal? (car e) token)))+1026
entries))+1027
#t))))+1028
+1029
;;; Forget every outstanding PING — used when the socket the+1030
;;; PINGs were written to is being abandoned.+1031
(define (enclave-conn-pings-clear! conn)+1032
(: enclave-conn? -> any?)+1033
(set-enclave-conn-pending-pings! conn '())) 1034
1035
;; ============================================================ 1036
;; Internal: PRIVMSG dispatcher + watch tables@@ -993,7 +1116,7 @@
1116
entry)))) 1117
1118
(define (handle-incoming-batch conn msg)−996
(enclave-conn-touch-activity! conn)+1119
(enclave-conn-note-inbound! conn) 1120
(let* ((params (irc-message-params msg)) 1121
(first (and (pair? params) (car params))) 1122
(rest (if (pair? params) (cdr params) '())))@@ -1059,9 +1182,8 @@
1182
(define (handle-incoming-privmsg conn msg) 1183
;; Any inbound PRIVMSG (chat, services reply, react) means 1184
;; the connection is alive — refresh the keepalive idle−1062
;; clock before dispatch so a chatty conn never PINGs even−1063
;; if the apiary itself isn't sending.−1064
(enclave-conn-touch-activity! conn)+1185
;; clock before dispatch.+1186
(enclave-conn-note-inbound! conn) 1187
(let ((sender (or (irc-message-nick msg) "?")) 1188
(target (or (irc-message-target msg) "?")) 1189
(text (or (irc-message-text msg) ""))@@ -1101,7 +1223,7 @@
1223
;; ============================================================ 1224
1225
(define (handle-cap-message conn msg)−1104
(enclave-conn-touch-activity! conn)+1226
(enclave-conn-note-inbound! conn) 1227
(let* ((params (irc-message-params msg)) 1228
;; `:server CAP <client> <subcmd> :<caps>` 1229
(subcmd (and (pair? params) (pair? (cdr params))@@ -1128,7 +1250,7 @@
1250
;; fall back to per-line PRIVMSG. Other FAIL kinds (PRIVMSG, 1251
;; etc.) just log; the handler is intentionally narrow. 1252
(define (handle-fail-message conn msg)−1131
(enclave-conn-touch-activity! conn)+1253
(enclave-conn-note-inbound! conn) 1254
(let* ((params (irc-message-params msg)) 1255
(command (and (pair? params) (car params))) 1256
(code (and (pair? params) (pair? (cdr params))@@ -1182,7 +1304,7 @@
1304
;;; trailing param back as `PONG :<token>`. The enclave-side 1305
;;; keepalive treats this as proof the client is alive. 1306
(define (handle-server-ping conn msg)−1185
(enclave-conn-touch-activity! conn)+1307
(enclave-conn-note-inbound! conn) 1308
(let* ((params (irc-message-params msg)) 1309
(token (or (irc-message-trailing msg) 1310
(and (pair? params) (car params))@@ -1196,13 +1318,18 @@
1318
(log-debug "Apiary responded to server PING"))) 1319
1320
−1199
;;; Inbound `PONG :<token>` clearing our pending client-side−1200
;;; PING. Token mismatch is logged and ignored — the activity−1201
;;; touch above still happens (any inbound line counts as−1202
;;; "still alive"), so a mismatch isn't fatal — it just means−1203
;;; the prior PING is left pending until the next sweep.+1321
;;; Inbound `PONG :<token>` clearing the matching outstanding+1322
;;; PING. Token mismatch is logged and ignored — the inbound+1323
;;; note above still happens (any inbound line counts as "still+1324
;;; alive"), so a mismatch isn't fatal — it just means that PING+1325
;;; stays outstanding until the next sweep.+1326
;;;+1327
;;; The match must stay EXACT: `enclave-confirm-delivery!` reads+1328
;;; "my specific token was acked" as proof that the peer consumed+1329
;;; the bytes written before it. A PONG that cleared some other+1330
;;; token would turn that into a claim about a different write. 1331
(define (handle-server-pong conn msg)−1205
(enclave-conn-touch-activity! conn)+1332
(enclave-conn-note-inbound! conn) 1333
(let* ((params (irc-message-params msg)) 1334
(token (or (irc-message-trailing msg) 1335
;; Some servers reply `PONG <server> :<token>`@@ -1215,12 +1342,9 @@
1342
(cond 1343
((null? (cdr params)) (car params)) 1344
(else (cadr params))))−1218
""))−1219
(pending (enclave-conn-pending-ping-token conn)))+1345
""))) 1346
(cond−1221
((and pending (string? token) (string=? pending token))−1222
(set-enclave-conn-pending-ping-token! conn #f)−1223
(set-enclave-conn-pending-ping-sent-ts! conn #f)+1347
((and (string? token) (enclave-conn-ping-ack! conn token)) 1348
(log-debug "Apiary keepalive PONG ack")) 1349
(else 1350
(log-debug "Apiary keepalive PONG stale-or-mismatched")))))@@ -1305,56 +1429,80 @@
1429
1430
;;; Single keepalive sweep over a single connection. Pulled 1431
;;; out of the throttle wrapper so tests can drive it−1308
;;; synchronously by setting `last-activity-ts` /−1309
;;; `pending-ping-sent-ts` and calling this directly.+1432
;;; synchronously by setting `last-inbound-ts` / adding a+1433
;;; pending PING and calling this directly. 1434
(define (enclave-conn-keepalive-tick! conn now) 1435
(: enclave-conn? number? -> any?) 1436
(cond 1437
((enclave-conn-shutdown? conn) #f) 1438
((enclave-conn-reconnecting? conn) #f) 1439
(else−1316
(let ((pending (enclave-conn-pending-ping-token conn))−1317
(pending-ts (enclave-conn-pending-ping-sent-ts conn))−1318
(last-act (enclave-conn-last-activity-ts conn)))+1440
(let ((oldest-ping (enclave-conn-oldest-ping-ts conn))+1441
(last-in (enclave-conn-last-inbound-ts conn))) 1442
(cond−1320
;; PING in flight and the PONG window has expired.+1443
;; A PING has been outstanding past the PONG window.+1444
;; Measured from the OLDEST outstanding entry, so a+1445
;; later PING (from a post-time delivery confirmation,+1446
;; say) can never push this deadline out.+1447
;; 1448
;; Loud log + reconnect — the brief insists on INFO- 1449
;; visible failure so an operator reading apiary's 1450
;; log can correlate "send-channel didn't arrive" to 1451
;; "the keepalive timed out N seconds before."−1325
((and pending pending-ts−1326
(> (- now pending-ts) +keepalive-pong-timeout+))+1452
((and oldest-ping+1453
(> (- now oldest-ping) +keepalive-pong-timeout+)) 1454
(log-warn "Apiary keepalive PONG timeout — connection appears dead, reconnecting"−1328
sent-sec-ago: (number->string (- now pending-ts)))+1455
sent-sec-ago: (number->string (- now oldest-ping))) 1456
(trigger-reconnect! conn))−1330
;; No PING in flight and the activity clock has−1331
;; crossed the idle threshold — emit a fresh PING.−1332
((and (not pending)−1333
last-act−1334
(> (- now last-act) +keepalive-idle-threshold+))+1457
;; Nothing in flight and no INBOUND traffic for longer+1458
;; than the idle threshold — emit a fresh PING. Note+1459
;; this is deliberately blind to how much WE have sent:+1460
;; our own writes say nothing about the peer.+1461
((and (not oldest-ping)+1462
last-in+1463
(> (- now last-in) +keepalive-idle-threshold+)) 1464
(emit-keepalive-ping! conn now)) 1465
(else #f)))))) 1466
1467
−1339
;;; Send a client→server PING, stash the pending token, and−1340
;;; refresh the activity clock so the next sweep doesn't−1341
;;; re-PING immediately.−1342
(define (emit-keepalive-ping! conn now)+1468
;;; Put a client→server PING on the wire and record it as+1469
;;; outstanding. Returns the token on success, #f if the write+1470
;;; itself failed (in which case a reconnect has been triggered).+1471
;;;+1472
;;; Deliberately does NOT refresh the inbound clock: we just+1473
;;; WROTE, which is not evidence of anything. The reason the+1474
;;; keepalive does not immediately re-PING is that the entry is+1475
;;; now outstanding, and the idle branch requires an empty list.+1476
(define (send-ping! conn now)+1477
(: enclave-conn? number? -> any?) 1478
(let ((token (generate-keepalive-token))−1344
(irc (enclave-conn-irc conn)))−1345
(set-enclave-conn-pending-ping-token! conn token)−1346
(set-enclave-conn-pending-ping-sent-ts! conn now)+1479
(irc (enclave-conn-irc conn))+1480
(sent? #f))+1481
;; Record BEFORE the write. If the write half-succeeds and+1482
;; then raises, the peer may still answer, and an entry we+1483
;; never recorded could not be acked — leaving a PONG that+1484
;; matches nothing and, worse, no deadline on the link.+1485
(enclave-conn-ping-add! conn token now) 1486
(guard (e (else 1487
(log-warn "Apiary keepalive PING send failed — triggering reconnect" 1488
error: (format "~a" e))+1489
(enclave-conn-ping-ack! conn token) 1490
(trigger-reconnect! conn))) 1491
(irc-send-raw irc 1492
(string-append "PING :" token "\r\n"))−1353
(enclave-conn-touch-activity! conn))+1493
(set! sent? #t)) 1494
;; Token never logged at INFO — defense-in-depth so a 1495
;; misbehaved peer reading log tail can't forge a PONG. 1496
(log-debug "Apiary keepalive PING out"−1357
token-prefix: (safe-token-prefix token))))+1497
token-prefix: (safe-token-prefix token)+1498
outstanding: (number->string+1499
(length (enclave-conn-pending-pings conn))))+1500
(and sent? token)))+1501
+1502
;;; Keepalive's own PING. Thin wrapper so the sweep reads as+1503
;;; intent rather than mechanism.+1504
(define (emit-keepalive-ping! conn now)+1505
(send-ping! conn now)) 1506
1507
1508
;;; Trigger the reconnect path. Idempotent on consecutive@@ -1371,8 +1519,7 @@
1519
;; ONLY on success; preserve it across multiple kicks of 1520
;; this function within a single outage so the backoff 1521
;; correctly compounds.−1374
(set-enclave-conn-pending-ping-token! conn #f)−1375
(set-enclave-conn-pending-ping-sent-ts! conn #f)+1522
(enclave-conn-pings-clear! conn) 1523
;; Best-effort close of the dead irc. If sigil-irc's 1524
;; disconnect path raises (the socket is already half- 1525
;; gone), swallow the exception — we don't want it to@@ -1576,9 +1723,13 @@
1723
channel: ch) 1724
(irc-join new-irc ch))) 1725
channels))−1579
;; Refresh the activity clock so the keepalive doesn't−1580
;; immediately re-PING the freshly-reconnected socket.−1581
(enclave-conn-touch-activity! conn)+1726
;; Seed the inbound clock. This is NOT an outbound+1727
;; touch in disguise: completing registration means+1728
;; the server sent us 001, so inbound evidence+1729
;; genuinely just arrived. Seeding it also stops the+1730
;; keepalive re-PINGing a socket that is one second+1731
;; old.+1732
(enclave-conn-note-inbound! conn) 1733
(connect-outcome kind: 'ok)))))))) 1734
1735
@@ -1651,7 +1802,7 @@
1802
;; immediately PING (the 60s idle threshold runs 1803
;; from "now" not from "1970"). 1804
(conn (enclave-conn irc: irc config: config−1654
last-activity-ts: (current-second))))+1805
last-inbound-ts: (current-second)))) 1806
(install-handlers! conn) 1807
;; Switch TLS to non-blocking so `tls-read` returns "" 1808
;; immediately when no data is available — the polling@@ -1780,8 +1931,7 @@
1931
(define (enclave-disconnect conn . quit-message) 1932
(: enclave-conn? string? ... -> void?) 1933
(set-enclave-conn-shutdown?! conn #t)−1783
(set-enclave-conn-pending-ping-token! conn #f)−1784
(set-enclave-conn-pending-ping-sent-ts! conn #f)+1934
(enclave-conn-pings-clear! conn) 1935
(let ((irc (enclave-conn-irc conn))) 1936
(when (and irc (irc-connection-socket irc)) 1937
(apply irc-disconnect irc quit-message))))@@ -1790,6 +1940,108 @@
1940
;; Public: channel posting + watchers 1941
;; ============================================================ 1942
+1943
;;; Ask the peer to prove it consumed what we just wrote.+1944
;;;+1945
;;; Returns `'confirmed`, `'unconfirmed`, or `'not-ready`.+1946
;;;+1947
;;; WHY THIS WORKS, and what it does and does not prove.+1948
;;;+1949
;;; A successful `irc-send-raw` means the local kernel accepted+1950
;;; the bytes. On a blackholed route every write "succeeds"+1951
;;; forever while nothing arrives, which is how `send-channel`+1952
;;; came to report `Broadcast to <channel> (1 line)` for 14+1953
;;; consecutive posts that were never delivered (measured+1954
;;; 2026-08-04, against an independent observer on a healthy+1955
;;; connection).+1956
;;;+1957
;;; So: write the message, THEN send a PING carrying a fresh+1958
;;; random token, and wait for that exact token to come back.+1959
;;; TCP is ordered, so a PONG for a token sent AFTER the message+1960
;;; means the peer's parser has already consumed the message.+1961
;;; The ordering is what makes this evidence rather than a+1962
;;; freshness heuristic — a reading of the idle clock taken+1963
;;; BEFORE the write says nothing whatsoever about the write.+1964
;;;+1965
;;; What it does NOT prove: that the server DELIVERED the message+1966
;;; to the channel or to any member. It proves receipt and+1967
;;; parsing, not fan-out. That is a large improvement over "we+1968
;;; wrote to a socket" and it is still short of a delivery+1969
;;; receipt; callers must not describe it as one.+1970
;;;+1971
;;; Two structural guards, both load-bearing:+1972
;;;+1973
;;; 1. This only ever ADDS to the outstanding-PING list. If it+1974
;;; could re-stamp an existing entry, a caller posting more+1975
;;; often than `+keepalive-pong-timeout+` would postpone its+1976
;;; own dead-link deadline indefinitely — the exact defect+1977
;;; this change removes from the idle clock, moved one+1978
;;; function over.+1979
;;; 2. If a PING has ALREADY been outstanding longer than the+1980
;;; budget, the answer is known: there is no liveness+1981
;;; evidence. Say so immediately rather than sending another+1982
;;; PING and spending the budget again. That also bounds the+1983
;;; list during a long outage — only the first post pays the+1984
;;; wait.+1985
;;;+1986
;;; Useful side effect: a PING sent at post time seeds theShowing the first 500 of 601 diff lines for this file. This diff is INCOMPLETE; read the file or clone the repository for the rest.
src/apiary/tools.sglmodified
@@ -72,6 +72,7 @@
72
set-enclave-bridge-state-stop?! 73
set-link-status! 74
bridge-unavailable-message+75
qualify-post-result 76
77
enclave-bridge-start! 78
enclave-bridge-run!@@ -860,15 +861,53 @@
861
(let* ((lines (apply-mention-to-first-line 862
mention (split-lines text))) 863
(outcome (enclave-post-multiline conn primary lines))−863
(style (post-outcome-label outcome (length lines))))+864
(style (post-outcome-label outcome (length lines)))+865
;; Ask the peer to prove it took those bytes BEFORE we+866
;; describe them as posted. See+867
;; `enclave-confirm-delivery!`.+868
(confirmed (enclave-confirm-delivery! conn))) 869
(cond 870
((and (string? mention) (> (string-length mention) 0))−866
(string-append "Posted to " primary−867
" mentioning " mention−868
" (" style ")"))+871
(qualify-post-result+872
confirmed+873
(string-append "Posted to " primary+874
" mentioning " mention+875
" (" style ")")+876
(string-append "Wrote to " primary+877
" mentioning " mention+878
" (" style ")"))) 879
(else−870
(string-append "Broadcast to " primary−871
" (" style ")")))))+880
(qualify-post-result+881
confirmed+882
(string-append "Broadcast to " primary " (" style ")")+883
(string-append "Wrote to " primary " (" style ")"))))))+884
+885
;;; Choose between the confirmed and the unconfirmed rendering of+886
;;; a post result.+887
;;;+888
;;; The verbs are chosen with care and are not interchangeable.+889
;;; "Broadcast"/"Posted"/"DM to" are delivery claims and are only+890
;;; used when a PONG proved the peer consumed the bytes. Anything+891
;;; else says "Wrote", which is the whole of what is known.+892
;;;+893
;;; It deliberately does NOT say "queued". Nothing is queued:+894
;;; there is no buffer holding the message and no retry that will+895
;;; send it again. A caller told its message was queued would+896
;;; reasonably wait instead of repeating itself — a new false+897
;;; claim in the opposite direction, which is the same defect+898
;;; this change exists to remove.+899
(define (qualify-post-result confirmed ok-text unconfirmed-text)+900
(: symbol? string? string? -> string?)+901
(case confirmed+902
((confirmed) ok-text)+903
(else+904
(string-append+905
unconfirmed-text+906
" — UNCONFIRMED: the server did not answer within "+907
(number->string +post-confirm-timeout+)+908
"s, so the link may be dead and this message may not have "+909
"been delivered. Re-send once the bridge reports "+910
"connected; nothing is holding a copy.")))) 911
912
;; Translate enclave-post-multiline's outcome symbol into a 913
;; tool-call status string. The string ships back to the agent@@ -921,8 +960,12 @@
960
(markdown->irc normalized))) 961
(lines (split-lines formatted)) 962
(outcome (enclave-post-multiline conn nick lines))−924
(style (post-outcome-label outcome (length lines))))−925
(string-append "DM to " nick ": " style))))))+963
(style (post-outcome-label outcome (length lines)))+964
(confirmed (enclave-confirm-delivery! conn)))+965
(qualify-post-result+966
confirmed+967
(string-append "DM to " nick ": " style)+968
(string-append "Wrote DM to " nick ": " style))))))) 969
970
;;; Send a `+draft/react` reaction PRIVMSG to a target msgid. 971
;;; Returns a human-readable status string for the MCP caller.test/test-enclave.sglmodified
@@ -151,45 +151,111 @@
151
152
(test "fresh enclave-conn defaults" 153
(let ((c (enclave-conn irc: #f config: (enclave-config))))−154
(assert-false (enclave-conn-last-activity-ts c))−155
(assert-false (enclave-conn-pending-ping-token c))−156
(assert-false (enclave-conn-pending-ping-sent-ts c))+154
(assert-false (enclave-conn-last-inbound-ts c))+155
(assert-false (enclave-conn-ping-outstanding? c))+156
(assert-false (enclave-conn-oldest-ping-ts c)) 157
(assert-false (enclave-conn-reconnecting? c)) 158
(assert-false (enclave-conn-shutdown? c)) 159
(assert-equal 0 (enclave-conn-reconnect-attempt c)))) 160
−161
(test "enclave-conn-touch-activity! advances last-activity-ts"+161
(test "enclave-conn-note-inbound! advances last-inbound-ts" 162
(let ((c (enclave-conn irc: #f config: (enclave-config))))−163
(set-enclave-conn-last-activity-ts! c 100)−164
(enclave-conn-touch-activity! c)−165
(assert-true (> (enclave-conn-last-activity-ts c) 100))))+163
(set-enclave-conn-last-inbound-ts! c 100)+164
(enclave-conn-note-inbound! c)+165
(assert-true (> (enclave-conn-last-inbound-ts c) 100))))+166
+167
;; ---- the idle clock is INBOUND-ONLY -------------------------+168
;;+169
;; This is the blocker-2 regression guard and it is deliberately+170
;; aimed at the PRODUCT, not at a mirror of it.+171
;;+172
;; `enclave-post-multiline` with no lines returns 'sent-empty+173
;; without touching the irc connection at all, so it runs fine on+174
;; a bare conn. The old code refreshed the activity clock at the+175
;; TOP of that procedure, before the empty check — so this test+176
;; FAILS against the pre-fix source and passes after. A test that+177
;; could not distinguish the two would be worth nothing here.+178
(test "an outbound post does NOT refresh the idle clock"+179
(let ((c (enclave-conn irc: #f config: (enclave-config))))+180
(set-enclave-conn-last-inbound-ts! c 100)+181
(assert-equal 'sent-empty (enclave-post-multiline c "#chan" '()))+182
(assert-equal 100 (enclave-conn-last-inbound-ts c)))) 183
184
(test "keepalive-tick: idle conn under threshold does NOT PING"−168
;; No pending PING, but last activity is only 30s ago — the−169
;; idle threshold is 60s, so we should sit tight.+185
;; No outstanding PING, but inbound traffic was only 30s ago —+186
;; the idle threshold is 60s, so we should sit tight. 187
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 188
(let* ((now 1000)−172
(last-act (- now 30)))−173
(set-enclave-conn-last-activity-ts! c last-act)+189
(last-in (- now 30)))+190
(set-enclave-conn-last-inbound-ts! c last-in) 191
(enclave-conn-keepalive-tick! c now) 192
;; No PING was emitted (we have no irc, so an attempted−176
;; emit would have raised — the lack of a pending token+193
;; emit would have raised — the empty outstanding list 194
;; confirms emit was skipped).−178
(assert-false (enclave-conn-pending-ping-token c))−179
(assert-false (enclave-conn-pending-ping-sent-ts c)))))+195
(assert-false (enclave-conn-ping-outstanding? c)))))+196
+197
(test "keepalive-tick: an OUTSTANDING ping suppresses a new one"+198
;; Idle well past the threshold, but a PING is already in+199
;; flight and inside its PONG window. Emitting another would be+200
;; pointless; more importantly the tick must fall through to+201
;; neither branch rather than piling on.+202
(let ((c (enclave-conn irc: #f config: (enclave-config))))+203
(set-enclave-conn-last-inbound-ts! c (- 1000 500))+204
(enclave-conn-ping-add! c "apk-inflight" (- 1000 5))+205
(enclave-conn-keepalive-tick! c 1000)+206
(assert-equal 1 (length (enclave-conn-pending-pings c)))+207
(assert-false (enclave-conn-reconnecting? c)))) 208
209
(test "keepalive-tick: shutdown? short-circuits" 210
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 211
(set-enclave-conn-shutdown?! c #t) 212
;; No irc, but shutdown? short-circuits before touching it. 213
(enclave-conn-keepalive-tick! c 1000)−186
(assert-false (enclave-conn-pending-ping-token c))))+214
(assert-false (enclave-conn-ping-outstanding? c)))) 215
216
(test "keepalive-tick: reconnecting? short-circuits" 217
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 218
(set-enclave-conn-reconnecting?! c #t) 219
(enclave-conn-keepalive-tick! c 1000)−192
(assert-false (enclave-conn-pending-ping-token c))))+220
(assert-false (enclave-conn-ping-outstanding? c))))+221
+222
;; ---- outstanding-PING bookkeeping ---------------------------+223
+224
(test "ping-ack! clears only the matching token"+225
(let ((c (enclave-conn irc: #f config: (enclave-config))))+226
(enclave-conn-ping-add! c "apk-a" 100)+227
(enclave-conn-ping-add! c "apk-b" 110)+228
(assert-false (enclave-conn-ping-ack! c "apk-unknown"))+229
(assert-equal 2 (length (enclave-conn-pending-pings c)))+230
(assert-true (enclave-conn-ping-ack! c "apk-b"))+231
(assert-equal 1 (length (enclave-conn-pending-pings c)))+232
(assert-equal 100 (enclave-conn-oldest-ping-ts c))))+233
+234
(test "oldest-ping-ts tracks the OLDEST, not the newest"+235
;; The guard that makes blocker 2 structurally unable to+236
;; reappear inside the delivery-confirmation path: a later PING+237
;; must never postpone an earlier one's deadline.+238
(let ((c (enclave-conn irc: #f config: (enclave-config))))+239
(enclave-conn-ping-add! c "apk-old" 100)+240
(enclave-conn-ping-add! c "apk-new" 900)+241
(assert-equal 100 (enclave-conn-oldest-ping-ts c))))+242
+243
(test "a NEWER ping cannot postpone an older ping's PONG timeout"+244
;; Same property, scored where it matters: through the tick.+245
;; The old entry is 31s old (past the 30s timeout); the new one+246
;; is 1s old. If the timeout were measured from the newest —+247
;; which is exactly what re-stamping a single slot would do —+248
;; this conn would look healthy forever under a steady stream+249
;; of posts.+250
(with-async+251
(let ((c (enclave-conn irc: #f config: (enclave-config))))+252
(set-enclave-conn-last-inbound-ts! c (- 1000 100))+253
(enclave-conn-ping-add! c "apk-old" (- 1000 31))+254
(enclave-conn-ping-add! c "apk-new" (- 1000 1))+255
(enclave-conn-keepalive-tick! c 1000)+256
(assert-true (enclave-conn-reconnecting? c))+257
(assert-false (enclave-conn-ping-outstanding? c))+258
(set-enclave-conn-shutdown?! c #t)))) 259
260
(test "keepalive-tick: pending-ping past timeout flips reconnecting?" 261
;; Simulate a PING sent 31s ago (>30s timeout). Shutdown? is@@ -207,41 +273,23 @@
273
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 274
(let* ((now 1000) 275
(sent-ts (- now 31)))−210
(set-enclave-conn-last-activity-ts! c (- now 100))−211
(set-enclave-conn-pending-ping-token! c "apk-test-token")−212
(set-enclave-conn-pending-ping-sent-ts! c sent-ts)+276
(set-enclave-conn-last-inbound-ts! c (- now 100))+277
(enclave-conn-ping-add! c "apk-test-token" sent-ts) 278
(enclave-conn-keepalive-tick! c now) 279
(assert-true (enclave-conn-reconnecting? c))−215
(assert-false (enclave-conn-pending-ping-token c))−216
(assert-false (enclave-conn-pending-ping-sent-ts c))+280
(assert-false (enclave-conn-ping-outstanding? c)) 281
;; Flag the conn as shutdown so the spawned reconnect 282
;; goroutine bails on its next sleep wakeup, otherwise 283
;; the with-async block waits forever for it. 284
(set-enclave-conn-shutdown?! c #t))))) 285
−222
(test "generate-keepalive-token returns a non-empty unique string"−223
;; Two consecutive calls should produce different tokens —−224
;; jiffy + random combined gives well over 32 bits of entropy.−225
(let ((t1 (begin (set-enclave-conn-pending-ping-token!−226
(enclave-conn irc: #f config: (enclave-config))−227
"ignored")−228
"first"))−229
(a (number->string (current-jiffy))))−230
;; Indirect: the public API doesn't expose generate-token,−231
;; but the post-tick token must include the "apk-" prefix.−232
;; Hard to test without an irc-connection in scope. Skip−233
;; this — covered by the smoke harness.−234
(assert-true (string? a))−235
(assert-true (> (string-length a) 0))))−236
286
(test "shutdown? prevents reconnecting when set first" 287
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 288
(set-enclave-conn-shutdown?! c #t) 289
;; Force a "PING timed out" condition into the conn but 290
;; with shutdown? already set — keepalive-tick should 291
;; observe shutdown? and skip the trigger entirely.−243
(set-enclave-conn-pending-ping-token! c "stale-token")−244
(set-enclave-conn-pending-ping-sent-ts! c (- 1000 100))+292
(enclave-conn-ping-add! c "stale-token" (- 1000 100)) 293
(enclave-conn-keepalive-tick! c 1000) 294
(assert-false (enclave-conn-reconnecting? c)))) 295
@@ -251,9 +299,43 @@
299
;; Indirectly assert via the threshold behavior. 300
(let ((c (enclave-conn irc: #f config: (enclave-config)))) 301
;; 59s of idle should NOT trigger a PING.−254
(set-enclave-conn-last-activity-ts! c (- 1000 59))+302
(set-enclave-conn-last-inbound-ts! c (- 1000 59)) 303
(enclave-conn-keepalive-tick! c 1000)−256
(assert-false (enclave-conn-pending-ping-token c)))))+304
(assert-false (enclave-conn-ping-outstanding? c))))+305
+306
;; ---- delivery confirmation ---------------------------------+307
+308
(test "confirm-delivery! on a shut-down conn is 'not-ready"+309
(let ((c (enclave-conn irc: #f config: (enclave-config))))+310
(set-enclave-conn-shutdown?! c #t)+311
(assert-equal 'not-ready (enclave-confirm-delivery! c))))+312
+313
(test "confirm-delivery! while reconnecting is 'unconfirmed"+314
(let ((c (enclave-conn irc: #f config: (enclave-config))))+315
(set-enclave-conn-reconnecting?! c #t)+316
(assert-equal 'unconfirmed (enclave-confirm-delivery! c))))+317
+318
(test "confirm-delivery! answers immediately when a ping is already overdue"+319
;; Guard 2, scored through the real entry point.+320
;;+321
;; An outstanding PING older than the budget already settles the+322
;; question, so the call must return 'unconfirmed WITHOUT+323
;; sending another PING and without spending the budget again.+324
;;+325
;; This conn has irc: #f, which is what makes the test sharp:+326
;; if the guard were removed the call would fall through to the+327
;; ready? check and then to `send-ping!`, both of which reach+328
;; into a #f connection. The test would fail rather than pass+329
;; quietly. It also asserts the list did not grow, so a guard+330
;; that returned the right symbol after emitting a PING anyway+331
;; would still be caught.+332
(let ((c (enclave-conn irc: #f config: (enclave-config))))+333
(enclave-conn-ping-add! c "apk-overdue"+334
(- (current-second)+335
(+ +post-confirm-timeout+ 1)))+336
(assert-equal 'unconfirmed (enclave-confirm-delivery! c))+337
(assert-equal 1 (length (enclave-conn-pending-pings c))))))+338
339
340
341
;; ============================================================test/test-tools.sglmodified
@@ -619,3 +619,28 @@
619
'leader #f #f))) 620
(assert-false result) 621
(assert-equal 'terminal (enclave-bridge-state-link-state s))))))+622
+623
+624
;; ============================================================+625
;; Post-result vocabulary+626
;; ============================================================+627
+628
(test-group "post-result vocabulary"+629
+630
(test "qualify-post-result says nothing about delivery when unconfirmed"+631
;; The tool-facing vocabulary is part of the contract. The+632
;; confirmed rendering may claim delivery; the unconfirmed one+633
;; must not, and specifically must not say "queued" — nothing+634
;; holds a copy, so a caller told that would wait instead of+635
;; re-sending.+636
(assert-equal "Broadcast to #chan (1 line)"+637
(qualify-post-result 'confirmed+638
"Broadcast to #chan (1 line)"+639
"Wrote to #chan (1 line)"))+640
(let ((out (qualify-post-result 'unconfirmed+641
"Broadcast to #chan (1 line)"+642
"Wrote to #chan (1 line)")))+643
(assert-true (string-contains? out "UNCONFIRMED"))+644
(assert-true (string-contains? out "Wrote to #chan"))+645
(assert-false (string-contains? out "Broadcast"))+646
(assert-false (string-contains? out "ueued")))))