Commita486bda9Recorded5 Aug 2026Repositoryapiary

Bound the Enclave connect, retry at boot, and classify the failure

Message

A blocking connect(2) on apiary's single cooperative scheduler took the kernel's full ~135s SYN-retransmit budget, and for that whole time NO MCP request could be answered -- including ones that touch no network at all. Measured against tools/list over a 240s outage: 53/120 answered, with one contiguous ~134s dead window. Healthy control in the same session: 20/20.

WHY THIS IS ONE COMMIT AND NOT TWO.

Bounding the connect and adding boot-time retry look independent. They are not, and the trap is asymmetric. Before this change a failed boot connect was terminal, so dispatch RECOVERED once the process had given up -- the wedge ended precisely when apiary became permanently useless. Adding retry alone would convert a 134s wedge followed by a dead-but-responsive process into one wedged for the entire outage. Bound first, then retry, is safe in that order only. Two changes are independently shippable only if EVERY ordering leaves the system no worse; this pair fails that test, so they ship together.

WHAT CHANGED.

Apiary now opens the transport itself and hands the ready socket to sigil-irc, rather than calling irc-connect, which performs an unbounded connect with no way to pass a deadline.

  • TCP connect is bounded and CLASSIFIED via %tcp-connect-ip-timeout-status, which distinguishes refused / timed out / host unreachable / network unreachable. The whole call shares one budget across every resolved address, so N addresses cannot multiply into N x the bound.
  • The TLS handshake is bounded SEPARATELY via tls-upgrade/status. Once a peer accepts the TCP connection the connect phase is over, so the connect bound cannot reach a peer that then goes silent -- that read had no bound at all. tls-connect/status was rejected because it collapses every connect-phase failure into one "tcp-connect-failed" status, which cannot tell refused from blackholed and so defeats the point.
  • Certificate verification is unchanged: tls-upgrade and tls-connect configure VERIFY_REQUIRED against the system CA chain identically. Read in sigil-tls's native source rather than assumed, because a silent downgrade here would trade an outage for an MITM window.
  • enclave-bridge-run! retries the boot connect with the same backoff the established-connection path uses. Previously a process that booted during an outage never connected again for its whole life while still advertising every tool.
  • Tool results now distinguish "not connected yet, retrying (attempt N; last failure X; next attempt in Ms)" from "will NOT connect (terminal)". Both used to be "Error: enclave bridge not connected", so a process that would never recover read exactly like one two seconds from succeeding.

The connect bound (5s) sits deliberately below the 30s PONG timeout. The comment records what that does and does NOT buy: overlapping reconnects are actually prevented by the reconnecting? flag, and the ordering is defence in depth on top of it.

MEASURED, baseline vs this build on the same harness:

  blackhole dispatch, 240s   53/120 answered  ->  102/120
  longest single stall       ~134s            ->  ~4s
  boot during outage         never recovers   ->  connected 4.5s after the
                                                  endpoint returns
  error discrimination       1/5              ->  4/5, log and tool level
  clean network switch       89.1s / attempt 1 at +9.3s / zero stall
                          -> 84.4s / attempt 1 at +9.3s / zero stall

The clean-switch difference is not an improvement and is not claimed as one -- both sit inside the 60s-idle plus up-to-30s-PONG window.

This also removes a pre-existing leak: the old enclave-connect spawned the event loop BEFORE the registration wait and raised on timeout, leaking a live socket and a goroutine on every failed boot.

Relocks sigil-tls 0.16.3 -> 0.16.5 for tls-upgrade/status. sigil-irc moves 0.16.1 -> 0.16.2 within its caret range; its src trees are byte-identical, so the bump is inert for the registration internals this depends on.

Changed
 sigil.lock             |  20 ++--
 src/apiary/enclave.sgl | 869 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------
 src/apiary/main.sgl    |  26 +++++-
 src/apiary/tools.sgl   | 305 +++++++++++++++++++++++++++++++++++++++++++++++++++++------
 test/test-enclave.sgl  | 231 +++++++++++++++++++++++++++++++++++++++++++++
 test/test-tools.sgl    | 172 ++++++++++++++++++++++++++++++++++
 6 files changed, 1434 insertions(+), 189 deletions(-)
Diff
sigil.lockmodified
@@ -35,20 +35,20 @@
35
(package name: "sigil-irc"
36
url: "codeberg:sigil/sigil-irc"
37
ref: "^0.16"
−38
sha: "bffc5e791261bac27b7fd61eb95c6843f5c570e9"
−39
version: "0.16.1")
+38
sha: "62721752e87415b62372a663d1bf0dd7a93bbda8"
+39
version: "0.16.2")
40
(package name: "sigil-test"
41
url: "codeberg:sigil/sigil"
42
ref: "^0.17"
−43
sha: "bd2a47b44bff2051073044046e000fd7f0e587ca"
+43
sha: "070f133a3524142b4febfd7c287385186b616252"
44
package-selector: "sigil-test"
−45
version: "0.17.12")
+45
version: "0.17.20")
46
(package name: "sigil-test-runner"
47
url: "codeberg:sigil/sigil"
48
ref: "^0.17"
−49
sha: "bd2a47b44bff2051073044046e000fd7f0e587ca"
+49
sha: "070f133a3524142b4febfd7c287385186b616252"
50
package-selector: "sigil-test-runner"
−51
version: "0.17.12")
+51
version: "0.17.20")
52
(package name: "sigil-version"
53
url: "codeberg:sigil/sigil-version"
54
ref: "^0.16"
@@ -62,13 +62,13 @@
62
(package name: "sigil-tls"
63
url: "codeberg:sigil/sigil-tls"
64
ref: "^0.16"
−65
sha: "30e8c2a84345dc4d484f5c085daa5bd16fc60fe1"
−66
version: "0.16.3")
+65
sha: "fe4b756f2f46eacf5e3a4eb5d17f2681d88c5ee8"
+66
version: "0.16.5")
67
(package name: "sigil-crypto"
68
url: "codeberg:sigil/sigil-crypto"
69
ref: "^0.16"
−70
sha: "a391c4ed73891ac5dc8a6da98b80a340c03f801f"
−71
version: "0.16.3")
+70
sha: "964eec2fad375834e2b49da454209fabc059ca30"
+71
version: "0.16.4")
72
(package name: "sigil-ansi"
73
url: "codeberg:sigil/sigil-ansi"
74
ref: "^0.16.0"
src/apiary/enclave.sglmodified
@@ -41,6 +41,7 @@
41
(sigil random)
42
(sigil time)
43
(sigil log)
+44
(sigil socket)
45
(sigil irc connection)
46
(sigil irc message)
47
(sigil tls))
@@ -54,9 +55,24 @@
55
enclave-config-token
56
enclave-config-channel
57
enclave-config-services-timeout
+58
enclave-config-connect-timeout
+59
enclave-config-tls-handshake-timeout
60
load-enclave-config
61
enclave-config-ready?
62
+63
connect-outcome
+64
connect-outcome?
+65
connect-outcome-kind
+66
connect-outcome-detail
+67
connect-outcome-socket
+68
connect-outcome-ok?
+69
connect-outcome-message
+70
classify-connect-status
+71
open-enclave-transport
+72
irc-connection-on-socket
+73
enclave-connect/detail
+74
reconnect-delay-for
+75
76
enclave-conn
77
enclave-conn?
78
enclave-conn-irc
@@ -98,6 +114,11 @@
114
enclave-watch-channel
115
enclave-watch-dms
116
+117
enclave-conn-status-hook
+118
set-enclave-conn-status-hook!
+119
report-link-status!
+120
report-link-connected!
+121
122
safe-token-prefix)
123
(begin
124
@@ -117,7 +138,49 @@
138
;; The single coordination channel apiary joins on connect.
139
(channel default: "#hive")
140
;; Timeout (seconds) for a single EnclaveServ request/reply.
−120
(services-timeout default: 5.0))
+141
(services-timeout default: 5.0)
+142
;; Upper bound (seconds) on the TCP connect phase. See
+143
;; `+connect-timeout-default+` for why this number matters far
+144
;; more than it looks like it should.
+145
;;
+146
;; KEEP IN SYNC with `+connect-timeout-default+`. This literal
+147
;; cannot reference that constant, because this file defines
+148
;; types before constants — so the same default is written
+149
;; twice, and code building a config directly reads THIS one
+150
;; while `load-enclave-config` reads the other. A test
+151
;; ("the struct default and the loaded default cannot drift")
+152
;; asserts they are equal; it exists because a sabotage run
+153
;; changed one of them and nothing turned red.
+154
(connect-timeout default: 5.0)
+155
;; Upper bound (seconds) on the TLS handshake phase. Separate
+156
;; from `connect-timeout` on purpose — they bound different
+157
;; failures. See `+tls-handshake-timeout-default+`, and the
+158
;; keep-in-sync note above, which applies here identically.
+159
(tls-handshake-timeout default: 10.0))
+160
+161
;; The result of one attempt to open the transport. Carrying a
+162
;; `kind` symbol rather than only a message is what lets callers
+163
;; ACT on the difference between "refused" and "timed out" — and
+164
;; lets an operator read a diagnosis instead of an echo of the
+165
;; host:port they already configured.
+166
;;
+167
;; kind is one of:
+168
;; 'ok transport open; `socket` holds it
+169
;; 'dns the hostname did not resolve
+170
;; 'refused the port answered with RST — something is
+171
;; listening on the host but not on that port
+172
;; 'timeout no answer within the connect bound — the
+173
;; blackhole case (route gone, tunnel down)
+174
;; 'unreachable the kernel had no route / ICMP unreachable
+175
;; 'tls TCP connected, TLS handshake failed or timed out
+176
;; 'registration transport open, but no 001 within the bound
+177
;; (also where a SASL rejection surfaces)
+178
;; 'config incomplete configuration — will NOT self-heal
+179
;; 'other a connect error outside the classified set
+180
(define-struct connect-outcome
+181
(kind)
+182
(detail default: "")
+183
(socket default: #f))
184
185
;; Wraps the underlying irc-connection plus services-call
186
;; coordination state. The mailbox is a single-slot rendezvous:
@@ -187,7 +250,18 @@
250
;; goroutine bail out instead of looping. `enclave-disconnect`
251
;; flips this so a graceful shutdown doesn't trigger the
252
;; reconnect path on the way out.
−190
(shutdown? default: #f mutable: #t))
+253
(shutdown? default: #f mutable: #t)
+254
;; Optional callback the reconnect path uses to report link
+255
;; status upward, so a layer that owns a user-facing surface
+256
;; (tool results, a status line) can say "retrying, attempt 4,
+257
;; last failure X" instead of a flat "not connected".
+258
;;
+259
;; Called as (hook state-symbol detail attempt next-delay) where
+260
;; state-symbol is 'connecting / 'connected / 'failed, detail is
+261
;; a human string, attempt an integer, and next-delay seconds or
+262
;; #f. A #f hook means nobody is listening; this module never
+263
;; depends on it having been set.
+264
(status-hook default: #f mutable: #t))
265
266
;; ============================================================
267
;; Constants
@@ -232,6 +306,73 @@
306
;; enclave was already up and accepting registrations.
307
(define +reconnect-handshake-timeout+ 10.0)
308
+309
;; ---- Bounded transport knobs -------------------------------
+310
;;
+311
;; Apiary runs the MCP stdin fiber and the Enclave link on ONE
+312
;; cooperative scheduler. A blocking syscall does not merely delay
+313
;; the thing making it — for its whole duration NO MCP request can
+314
;; be answered. So every phase of opening the transport has to be
+315
;; bounded, and the bounds are chosen against measured numbers
+316
;; rather than picked for tidiness.
+317
;;
+318
;; CONNECT bound. The kernel's own SYN-retransmit budget is ~135 s,
+319
;; and an unreachable endpoint therefore wedged dispatch essentially
+320
;; continuously: each attempt blocked ~135 s while the backoff
+321
;; between attempts was seconds.
+322
;;
+323
;; 5 s, because:
+324
;; * a healthy connect over a working link is single-digit
+325
;; milliseconds, so this leaves ~3 orders of magnitude of
+326
;; headroom for a slow or lossy path;
+327
;; * it is ~27x below the kernel's ~135 s; and
+328
;; * — the interaction that is easy to miss — it sits deliberately
+329
;; BELOW `+keepalive-pong-timeout+` (30 s), so a connect attempt
+330
;; cannot outlive the window in which the keepalive sweep would
+331
;; declare the link dead.
+332
;;
+333
;; BE PRECISE ABOUT WHAT THAT ORDERING DOES AND DOES NOT BUY, because
+334
;; an earlier version of this comment credited the wrong mechanism.
+335
;; What actually prevents two overlapping reconnects is the
+336
;; `reconnecting?` flag: `enclave-conn-keepalive-tick!` bails while
+337
;; it is set, and `trigger-reconnect!` is idempotent. The timeout
+338
;; ordering is DEFENCE IN DEPTH on top of that, not the guarantee.
+339
;;
+340
;; The ordering only holds per CALL, not per address, so
+341
;; `open-enclave-transport` shares one budget across every address a
+342
;; hostname resolves to. Without that, six blackholed addresses at
+343
;; 5 s each would put a single call at 30 s and break the property.
+344
(define +connect-timeout-default+ 5.0)
+345
+346
;; HANDSHAKE bound, deliberately a SEPARATE number rather than one
+347
;; tidy timeout, because it bounds a different failure. Once a peer
+348
;; has ACCEPTED the TCP connection the connect phase is over, so the
+349
;; connect bound cannot reach a peer that then goes silent — that
+350
;; read had no bound at all.
+351
;;
+352
;; 10 s costs nothing in either real case: a healthy handshake is
+353
;; ~100 ms, and an expired-certificate failure completes the connect
+354
;; and fails the handshake in ~5 s on its own. It exists only for
+355
;; the pathological "accepted, then silence" case. Matching
+356
;; `+reconnect-handshake-timeout+` keeps connect + handshake +
+357
;; registration under ~25 s for a single attempt.
+358
(define +tls-handshake-timeout-default+ 10.0)
+359
+360
;; Bounded TCP connect, reaching past sigil-socket's Scheme surface
+361
;; to the native the VM already registers in the `(sigil socket)`
+362
;; module. `tcp-connect` is unbounded and sigil-irc's `irc-connect`
+363
;; offers no way to pass a deadline, so apiary opens the transport
+364
;; itself. Declaring the native this way is how sigil-socket reaches
+365
;; its own natives; sigil-socket simply does not re-export this one.
+366
;;
+367
;; Returns `(status . socket-or-#f)`. Statuses observed in practice:
+368
;; "connected", "connection refused", "connection timed out",
+369
;; "host unreachable", "network unreachable", or strerror(errno).
+370
;; That status string is the whole reason for using this native over
+371
;; a plain bounded connect: symptom 3 of the defect this fixes is
+372
;; that every failure shape produced one indistinguishable message.
+373
(define-native (%tcp-connect-ip-timeout-status ip port timeout-ms)
+374
(: string? integer? integer? -> pair?))
+375
376
;; ============================================================
377
;; Pure helpers (no upward references)
378
;; ============================================================
@@ -331,6 +472,92 @@
472
(string-append (substring token 0 4) "…("
473
(number->string (string-length token)) ")"))))
474
+475
;;; True when the transport opened.
+476
(define (connect-outcome-ok? outcome)
+477
(: connect-outcome? -> boolean?)
+478
(eq? (connect-outcome-kind outcome) 'ok))
+479
+480
;;; Map a `%tcp-connect-ip-timeout-status` status string onto a
+481
;;; `connect-outcome` kind.
+482
;;;
+483
;;; Note what this deliberately does NOT do: it never folds an
+484
;;; unrecognised status into 'timeout or 'refused. An unknown
+485
;;; failure is 'other and says so. Guessing here would recreate
+486
;;; exactly the defect being fixed — a message that reports a
+487
;;; diagnosis it does not actually have.
+488
(define (classify-connect-status status)
+489
(: any? -> symbol?)
+490
(cond
+491
((not (string? status)) 'other)
+492
((string=? status "connected") 'ok)
+493
((string=? status "connection refused") 'refused)
+494
((string=? status "connection timed out") 'timeout)
+495
((string=? status "host unreachable") 'unreachable)
+496
((string=? status "network unreachable") 'unreachable)
+497
(else 'other)))
+498
+499
;; Short, operator-facing label per kind. The kind names the
+500
;; DIAGNOSIS; `detail` carries the specifics.
+501
(define (connect-kind-label kind)
+502
(case kind
+503
((ok) "connected")
+504
((dns) "DNS failure")
+505
((refused) "connection refused")
+506
((timeout) "connect timed out")
+507
((unreachable) "network unreachable")
+508
((tls) "TLS failure")
+509
((registration) "registration failed")
+510
((config) "configuration incomplete")
+511
(else "connect failed")))
+512
+513
;;; One-line message for logs and tool results. Leads with the
+514
;;; diagnosis, not with the host:port the caller already knows.
+515
(define (connect-outcome-message outcome)
+516
(: connect-outcome? -> string?)
+517
(let ((label (connect-kind-label (connect-outcome-kind outcome)))
+518
(detail (connect-outcome-detail outcome)))
+519
(if (and (string? detail) (> (string-length detail) 0))
+520
(string-append label ": " detail)
+521
label)))
+522
+523
(define (seconds->ms secs)
+524
(: number? -> integer?)
+525
(let ((ms (exact (round (* 1000 (max 0 secs))))))
+526
(if (< ms 1) 1 ms)))
+527
+528
(define (endpoint-label host port)
+529
(string-append host ":" (number->string port)))
+530
+531
;;; Parse a positive-number env override, falling back to `fallback`
+532
;;; when the value is absent, unparseable, or NOT POSITIVE.
+533
;;;
+534
;;; That last clause is load-bearing rather than defensive tidiness.
+535
;;; The underlying native treats a NEGATIVE timeout as "no bound"
+536
;;; and reverts to a plain blocking connect, so a stray
+537
;;; `APIARY_CONNECT_TIMEOUT=-1` would silently restore the exact
+538
;;; ~135 s wedge this module exists to prevent, while looking like
+539
;;; a deliberate configuration choice.
+540
;;;
+541
;;; Precisely: only `< 0` disables the bound; `0` polls with a zero
+542
;;; timeout and fails instantly, which is useless rather than
+543
;;; dangerous. `seconds->ms` also clamps to at least 1 ms, so a
+544
;;; zero can never reach the native anyway. Both are refused here
+545
;;; regardless — a bound of "instantly fail every connect" is not a
+546
;;; configuration anyone means — but the DANGEROUS case is the
+547
;;; negative one, and the earlier version of this comment was wrong
+548
;;; about which.
+549
(define (positive-or raw fallback)
+550
(: any? number? -> number?)
+551
(let ((n (and (string? raw) (string->number raw))))
+552
(cond
+553
((not raw) fallback)
+554
((and (number? n) (> n 0)) n)
+555
(else
+556
(log-warn "Apiary ignoring non-positive timeout override — a zero or negative bound would disable it entirely"
+557
value: raw
+558
using: (number->string fallback))
+559
fallback))))
+560
561
;; Pull result.* tags off a message into an alist of
562
;; (key-string . value-string-or-flag-symbol). Stripping the
563
;; "+enclave/result." prefix so callers see "status", "code",
@@ -390,6 +617,14 @@
617
;;; APIARY_USER bot nick to SASL as
618
;;; APIARY_TOKEN SASL PLAIN password
619
;;; APIARY_CHANNEL coordination channel (default #hive)
+620
;;; APIARY_CONNECT_TIMEOUT seconds; bound on the TCP connect
+621
;;; APIARY_TLS_HANDSHAKE_TIMEOUT
+622
;;; seconds; bound on the TLS handshake
+623
;;;
+624
;;; The two timeout knobs exist so a deployment on a slow or lossy
+625
;;; link can raise them. Raising them trades MCP responsiveness for
+626
;;; connect tolerance, which is the right trade only when a healthy
+627
;;; connect genuinely takes longer than the default allows.
628
(define (load-enclave-config)
629
(: -> enclave-config?)
630
(let* ((host (env-or-false "APIARY_ENCLAVE_HOST"))
@@ -401,7 +636,14 @@
636
(channel (or (env-or-false "APIARY_CHANNEL") "#hive"))
637
(timeout-str (env-or-false "APIARY_SERVICES_TIMEOUT"))
638
(timeout (or (and timeout-str (string->number timeout-str))
−404
5.0)))
+639
5.0))
+640
(connect-str (env-or-false "APIARY_CONNECT_TIMEOUT"))
+641
(connect-timeout (positive-or connect-str
+642
+connect-timeout-default+))
+643
(handshake-str (env-or-false "APIARY_TLS_HANDSHAKE_TIMEOUT"))
+644
(handshake-timeout
+645
(positive-or handshake-str
+646
+tls-handshake-timeout-default+)))
647
(enclave-config
648
host: host
649
port: port
@@ -409,7 +651,9 @@
651
user: user
652
token: token
653
channel: channel
−412
services-timeout: timeout)))
+654
services-timeout: timeout
+655
connect-timeout: connect-timeout
+656
tls-handshake-timeout: handshake-timeout)))
657
658
;;; True when the config has the minimum needed to attempt a connect.
659
(define (enclave-config-ready? config)
@@ -419,6 +663,230 @@
663
(enclave-config-token config)
664
#t))
665
+666
;; ============================================================
+667
;; Internal: bounded transport
+668
;; ============================================================
+669
;;
+670
;; Why apiary opens the socket itself instead of calling
+671
;; sigil-irc's `irc-connect`:
+672
;;
+673
;; `irc-connect` performs the TCP connect (and, for TLS, the
+674
;; handshake) internally with no way to pass a deadline. An
+675
;; unbounded connect on this process's single cooperative
+676
;; scheduler is the entire defect — one blackholed endpoint
+677
;; starved MCP dispatch for ~132 s at a stretch.
+678
;;
+679
;; So apiary opens a BOUNDED, CLASSIFIED transport here and hands
+680
;; the ready socket to sigil-irc. The alternative considered and
+681
;; rejected was a pre-flight probe followed by `irc-connect`: that
+682
;; leaves a window between the probe and the real connect in which
+683
;; the route can disappear, which would reintroduce the ~135 s
+684
;; hang at low probability. A low-probability recurrence of the
+685
;; exact bug is not a fix.
+686
+687
;;; Resolve `host` to a list of numeric addresses.
+688
;;;
+689
;;; sigil-socket's `resolve-hostname` is already scheduler-aware —
+690
;;; under a scheduler it starts an async resolve and yields — so
+691
;;; DNS is NOT part of the blocking problem and needs no bound of
+692
;;; its own here. That was verified rather than assumed: a
+693
;;; non-existent host produced its verdict promptly while a
+694
;;; blackholed IP hung, which is only possible if the two phases
+695
;;; are separate.
+696
(define (resolve-enclave-host host)
+697
(guard (e (else
+698
(log-debug "Apiary host resolve raised"
+699
host: host error: (format "~a" e))
+700
#f))
+701
(resolve-hostname host)))
+702
+703
;;; Bounded TLS handshake over an already-connected socket.
+704
;;;
+705
;;; `tls-upgrade/status` is used rather than `tls-connect/status`
+706
;;; for two reasons. First, it lets the CONNECT phase run through
+707
;;; the classified bounded connect above; `tls-connect/status`
+708
;;; collapses every connect-phase failure into one
+709
;;; "tcp-connect-failed" status, which cannot tell refused from
+710
;;; blackholed and so defeats the point. Second, it bounds the
+711
;;; handshake separately from the connect.
+712
;;;
+713
;;; Certificate verification posture is IDENTICAL between the two
+714
;;; entry points — both configure VERIFY_REQUIRED against the
+715
;;; system CA chain and set the SNI hostname; this was checked in
+716
;;; sigil-tls's native source, not assumed, because a silent
+717
;;; downgrade here would trade an outage for an MITM window.
+718
(define (upgrade-to-tls config sock host)
+719
(let* ((hs-ms (seconds->ms
+720
(enclave-config-tls-handshake-timeout config)))
+721
;; `tls-upgrade/status` RAISES rather than returning a
+722
;; pair when handed a socket it considers already closed.
+723
;; Without this guard that escapes `open-enclave-transport`
+724
;; entirely and is caught several frames up as an
+725
;; unclassified 'other, losing the diagnosis.
+726
(result (guard (e (else (cons (format "~a" e) #f)))
+727
(tls-upgrade/status sock host hs-ms)))
+728
(status (car result))
+729
(tls-conn (cdr result)))
+730
(cond
+731
(tls-conn (connect-outcome kind: 'ok socket: tls-conn))
+732
(else
+733
;; Do NOT close `sock` here. `tls-upgrade` marks the
+734
;; original socket object closed BEFORE handshaking and
+735
;; from that point owns the descriptor, closing it itself
+736
;; on its failure paths. Closing again would be a
+737
;; double-close on a number the process may have reused.
+738
;;
+739
;; Read the source before strengthening this: ownership is
+740
;; taken partway INTO `tls_upgrade_core`, so its earliest
+741
;; returns (a TLS-init failure, and the already-closed
+742
;; raise handled above) happen while apiary still nominally
+743
;; owns the fd. Those paths drop the last Scheme reference
+744
;; instead, so the descriptor lives until GC finalises the
+745
;; socket rather than leaking permanently. Measured across
+746
;; 300 s and 10 handshake failures on the ownership-transfer
+747
;; path: fd count 4 -> 4, no per-attempt leak. The earlier
+748
;; comment here claimed closure on "every failure path",
+749
;; which is stronger than the source supports.
+750
(connect-outcome
+751
kind: 'tls
+752
detail: (string-append (if (string? status) status "unknown")
+753
" with "
+754
(endpoint-label
+755
host
+756
(enclave-config-port config))))))))
+757
+758
;;; Open the transport to the configured endpoint with a bounded
+759
;;; connect, returning a `connect-outcome` rather than a bare #f.
+760
;;;
+761
;;; Every address the hostname resolves to is tried, and a TLS
+762
;;; handshake failure falls through to the next address exactly as
+763
;;; a connect failure does. That matters for a dual-stack endpoint
+764
;;; whose v6 path has a PMTU black hole: the TCP connect succeeds
+765
;;; and the handshake stalls, and stopping there would mean never
+766
;;; reaching a working A record while reporting a TLS fault that
+767
;;; sends an operator after a certificate problem that does not
+768
;;; exist.
+769
;;;
+770
;;; The LAST failure's classification is what surfaces — what an
+771
;;; operator wants is why the final attempt failed, not the first.
+772
;;;
+773
;;; THE WHOLE CALL SHARES ONE BUDGET. Per-address bounds alone are
+774
;;; not a bound on this procedure: six blackholed addresses at 5 s
+775
;;; each is 30 s of dead scheduler in a single call, which would
+776
;;; break the "stay under the PONG window" property the connect
+777
;;; timeout is chosen for. So addresses are tried against a shared
+778
;;; deadline and the remainder is abandoned once it is spent.
+779
(define (open-enclave-transport config)
+780
(: enclave-config? -> connect-outcome?)
+781
(let* ((host (enclave-config-host config))
+782
(port (enclave-config-port config))
+783
(budget (enclave-config-connect-timeout config))
+784
(deadline (+ (current-second) budget))
+785
(ips (resolve-enclave-host host)))
+786
(cond
+787
((or (not ips) (null? ips))
+788
(connect-outcome
+789
kind: 'dns
+790
detail: (string-append "no address for host " host)))
+791
(else
+792
(let try ((remaining ips)
+793
(last-kind 'other)
+794
(last-detail (string-append "no address tried for "
+795
host)))
+796
(let ((left (- deadline (current-second))))
+797
(cond
+798
((null? remaining)
+799
(connect-outcome kind: last-kind detail: last-detail))
+800
;; Budget spent with addresses still untried. Report
+801
;; the last real classification rather than inventing
+802
;; one for the addresses never attempted.
+803
((<= left 0)
+804
(log-debug "Apiary connect budget exhausted"
+805
host: host
+806
untried: (number->string (length remaining)))
+807
(connect-outcome kind: last-kind detail: last-detail))
+808
(else
+809
(let* ((ip (car remaining))
+810
(result (%tcp-connect-ip-timeout-status
+811
ip port (seconds->ms left)))
+812
(status (car result))
+813
(sock (cdr result)))
+814
(cond
+815
((not sock)
+816
(log-debug "Apiary bounded connect failed"
+817
address: (endpoint-label ip port)
+818
status: (if (string? status) status "?"))
+819
(try (cdr remaining)
+820
(classify-connect-status status)
+821
;; The KIND already carries the diagnosis, so
+822
;; the detail is just the endpoint — repeating
+823
;; the status would render as "connection
+824
;; refused: connection refused at host:port".
+825
;; An UNCLASSIFIED status is the exception:
+826
;; there the raw text is the only diagnosis
+827
;; available and must not be dropped.
+828
(if (eq? (classify-connect-status status) 'other)
+829
(string-append
+830
(if (string? status) status "unknown")
+831
" at " (endpoint-label ip port))
+832
(endpoint-label ip port))))
+833
((not (enclave-config-tls? config))
+834
(connect-outcome kind: 'ok socket: sock))
+835
(else
+836
(let ((upgraded (upgrade-to-tls config sock host)))

Showing the first 500 of 1010 diff lines for this file. This diff is INCOMPLETE; read the file or clone the repository for the rest.

src/apiary/main.sglmodified
@@ -195,11 +195,27 @@
195
;; ready to receive `channel-notify!` events. The tools
196
;; are already registered above; the connect just makes
197
;; their backing IRC connection live.
−198
(when (enclave-config-ready? cfg)
−199
(go (begin
−200
(channel-wait-initialized! server)
−201
(enclave-bridge-start!
−202
state server cfg mode owner-nick reports-to))))
+198
;;
+199
;; `enclave-bridge-run!` rather than `enclave-bridge-start!`:
+200
;; a single attempt made a boot-time failure TERMINAL for
+201
;; the life of the process, so an apiary that started
+202
;; during an outage never connected again even after the
+203
;; endpoint came back. It kept advertising every tool the
+204
;; whole time.
+205
;;
+206
;; This is safe to loop only because the connect underneath
+207
;; is now bounded — see `enclave-bridge-run!`'s comment for
+208
;; why retrying an unbounded connect would be worse than
+209
;; not retrying at all.
+210
;;
+211
;; When the config is not ready the runner marks the link
+212
;; TERMINAL and returns rather than spinning, so tools
+213
;; report a failure that will not resolve instead of
+214
;; implying a retry that is not happening.
+215
(go (begin
+216
(channel-wait-initialized! server)
+217
(enclave-bridge-run!
+218
state server cfg mode owner-nick reports-to)))
219
220
;; Run the MCP server loop (stdin → tools → stdout).
221
(mcp-server-run server load-env?: #f)))))
src/apiary/tools.sglmodified
@@ -60,7 +60,21 @@
60
env-or-false
61
or-empty-env
62
+63
enclave-bridge-state-link-state
+64
enclave-bridge-state-link-detail
+65
enclave-bridge-state-link-attempt
+66
enclave-bridge-state-link-next-delay
+67
set-enclave-bridge-state-link-state!
+68
set-enclave-bridge-state-link-detail!
+69
set-enclave-bridge-state-link-attempt!
+70
set-enclave-bridge-state-link-next-delay!
+71
enclave-bridge-state-stop?
+72
set-enclave-bridge-state-stop?!
+73
set-link-status!
+74
bridge-unavailable-message
+75
76
enclave-bridge-start!
+77
enclave-bridge-run!
78
enclave-bridge-send-channel!
79
enclave-bridge-send-dm!
80
enclave-bridge-send-react!
@@ -83,7 +97,30 @@
97
(owner-nick default: #f mutable: #t)
98
(reports-to default: #f mutable: #t)
99
(subordinates default: '() mutable: #t) ;; list of nicks (case preserved)
−86
(listened-peers default: '() mutable: #t)) ;; list of nicks (case preserved)
+100
(listened-peers default: '() mutable: #t) ;; list of nicks (case preserved)
+101
;; ---- link status, for the tool-facing error surface --------
+102
;;
+103
;; A tool call that cannot reach the channel has to tell the
+104
;; caller WHICH of two very different situations it is in:
+105
;; something that will fix itself, or something that never will.
+106
;; Before this, both produced "Error: enclave bridge not
+107
;; connected", and a process that would never connect for the
+108
;; rest of its life said exactly what a process mid-retry said.
+109
;;
+110
;; link-state is one of:
+111
;; 'idle nothing has been attempted yet
+112
;; 'connecting an attempt is in flight or scheduled
+113
;; 'connected the bridge is up
+114
;; 'terminal it will NOT connect without operator action
+115
(link-state default: 'idle mutable: #t)
+116
(link-detail default: "" mutable: #t) ;; last classified failure
+117
(link-attempt default: 0 mutable: #t) ;; attempts made so far
+118
(link-next-delay default: #f mutable: #t) ;; seconds until next try
+119
;; Set to #t to stop the start-up retry loop. The established-
+120
;; connection reconnect path has `enclave-conn-shutdown?` for
+121
;; this, but during start-up there is no conn to carry the flag,
+122
;; so it lives here.
+123
(stop? default: #f mutable: #t))
124
125
(define (make-enclave-bridge-state)
126
(: -> enclave-bridge-state?)
@@ -529,9 +566,88 @@
566
;; Public: bridge lifecycle
567
;; ============================================================
568
+569
;; Record the current link status on the bridge state. This is the
+570
;; single place the tool-facing surface reads from, so a caller's
+571
;; view can never drift from what the connect machinery last did.
+572
;;
+573
;; A `detail` of #f means "no NEW diagnosis — keep the existing
+574
;; one". Coercing #f to "" here instead would erase the classified
+575
;; failure every time an attempt is merely SCHEDULED, and since the
+576
;; backoff sleep dominates the attempt by an order of magnitude,
+577
;; almost every tool call during an outage would then report an
+578
;; empty reason. Only a real outcome overwrites a real outcome.
+579
(define (set-link-status! state link-state detail attempt next-delay)
+580
(set-enclave-bridge-state-link-state! state link-state)
+581
(when detail
+582
(set-enclave-bridge-state-link-detail! state detail))
+583
(set-enclave-bridge-state-link-attempt! state attempt)
+584
(set-enclave-bridge-state-link-next-delay! state next-delay))
+585
+586
;;; The message a tool returns when it cannot reach the channel.
+587
;;;
+588
;;; The whole point is that a caller can tell TRANSIENT from
+589
;;; TERMINAL. "Not connected" said both, and a process that would
+590
;;; never connect again looked identical to one two seconds from
+591
;;; succeeding — which is why an outage read as a transient blip
+592
;;; for a day and a half.
+593
(define (bridge-unavailable-message state)
+594
(: enclave-bridge-state? -> string?)
+595
(let ((link (enclave-bridge-state-link-state state))
+596
(detail (enclave-bridge-state-link-detail state))
+597
(attempt (enclave-bridge-state-link-attempt state))
+598
(next (enclave-bridge-state-link-next-delay state)))
+599
(case link
+600
((terminal)
+601
(string-append
+602
"Error: enclave bridge will NOT connect (terminal): "
+603
detail
+604
". This will not resolve on its own — it needs operator "
+605
"action and a restart."))
+606
((connecting idle)
+607
(string-append
+608
"Error: enclave bridge not connected yet — retrying"
+609
(if (> attempt 0)
+610
(string-append " (attempt " (number->string attempt) ")")
+611
"")
+612
(if (and (string? detail) (> (string-length detail) 0))
+613
(string-append "; last failure: " detail)
+614
"")
+615
(if next
+616
(string-append "; next attempt in "
+617
(number->string next) "s")
+618
"")
+619
"."))
+620
(else
+621
;; link-state says 'connected but somebody asked for an
+622
;; unavailable message, so the IRC session underneath has
+623
;; dropped and the reconnect path has not reported in yet.
+624
;;
+625
;; DO NOT call this window narrow — an earlier version did
+626
;; and it was wrong. A reconnect is only triggered once
+627
;; sigil-irc observes EOF, and for a HALF-OPEN connection
+628
;; (the NAT/conntrack case the keepalive constants were
+629
;; sized for) nothing observes anything until the keepalive
+630
;; notices: 60 s idle threshold plus up to 30 s waiting for
+631
;; a PONG. So this state can persist for a minute and a half
+632
;; and the message has to be honest about that rather than
+633
;; implying recovery is seconds away.
+634
(string-append
+635
"Error: enclave session is not registered — detection of a "
+636
"dead link takes up to 90s (60s idle + 30s PONG), so a "
+637
"reconnect may not have started yet"
+638
(if (and (string? detail) (> (string-length detail) 0))
+639
(string-append "; last failure: " detail)
+640
"")
+641
".")))))
+642
643
;;; Start the Enclave bridge if config is ready. Idempotent: a
644
;;; second call while connected is a no-op. Returns #t if the
−534
;;; bridge is now connected, #f if config wasn't ready.
+645
;;; bridge is now connected, #f otherwise. Either way the link
+646
;;; status on `state` is updated, so a failed attempt leaves a
+647
;;; classified reason behind rather than just a #f.
+648
;;;
+649
;;; This makes ONE attempt. `enclave-bridge-run!` is what a
+650
;;; long-lived process should call.
651
;;;
652
;;; Inputs:
653
;;; server mcp-channel-server
@@ -550,6 +666,10 @@
666
#t)
667
((not (enclave-config-ready? config))
668
(log-debug "Apiary bridge skipped (config not ready)")
+669
(set-link-status!
+670
state 'terminal
+671
"incomplete configuration (need APIARY_ENCLAVE_HOST, APIARY_USER, APIARY_TOKEN)"
+672
0 #f)
673
#f)
674
(else
675
(set-enclave-bridge-state-config! state config)
@@ -557,23 +677,150 @@
677
(set-enclave-bridge-state-owner-nick! state owner-nick)
678
(set-enclave-bridge-state-reports-to!
679
state (or reports-to owner-nick))
−560
(guard (e (else
−561
(log-error "Apiary bridge failed to start"
−562
error: (format "~a" e))
−563
#f))
−564
(let ((conn (enclave-connect config)))
−565
(set-enclave-bridge-state-conn! state conn)
−566
(install-bridge-handlers! state server)
−567
(log-info "Apiary bridge connected"
−568
mode: (symbol->string mode)
−569
host: (enclave-config-host config)
−570
nick: (enclave-conn-current-nick conn)
−571
owner: (or owner-nick "<none>")
−572
reports-to: (or reports-to owner-nick "<none>"))
−573
;; Reconnect-presence broadcast — uniform across leader
−574
;; and worker. Trusted-set filter handles privacy.
−575
(post-presence! state)
−576
#t)))))
+680
;; THE GUARD COVERS THE WHOLE SEQUENCE, NOT JUST THE CONNECT.
+681
;; An intermediate version narrowed it to `enclave-connect/detail`
+682
;; alone, which was a regression of exactly the kind this change
+683
;; exists to remove: a raise from `install-bridge-handlers!` would
+684
;; escape, kill the boot goroutine, and leave `conn` SET with no
+685
;; handlers installed and link-state still 'idle — so every tool
+686
;; reported "not connected yet, retrying" forever while nothing
+687
;; was retrying and no inbound traffic could ever arrive. A lie
+688
;; that outlives the process is the worst version of the bug.
+689
(let* ((result
+690
(guard (e (else
+691
(cons (connect-outcome
+692
kind: 'other
+693
detail: (format "~a" e))
+694
#f)))
+695
(let* ((r (enclave-connect/detail config))
+696
(conn (cdr r)))
+697
(cond
+698
((not conn) r)
+699
(else
+700
(install-bridge-handlers! state server)
+701
;; Let the reconnect machinery report through
+702
;; to the same link status the tools read, so an
+703
;; established connection that later drops does
+704
;; not fall back to an undifferentiated message.
+705
(set-enclave-conn-status-hook!
+706
conn
+707
(lambda (link-state detail attempt next-delay)
+708
(set-link-status!
+709
state
+710
(if (eq? link-state 'connected)
+711
'connected
+712
'connecting)
+713
detail attempt next-delay)))
+714
;; Publish 'connected BEFORE the conn is visible
+715
;; and before the hook can fire, so a drop
+716
;; detected in this window is not clobbered back
+717
;; to 'connected by a later unconditional write.
+718
(set-link-status! state 'connected "" 0 #f)
+719
(set-enclave-bridge-state-conn! state conn)
+720
r)))))
+721
(outcome (car result))
+722
(conn (cdr result)))
+723
(cond
+724
((not conn)
+725
(log-error "Apiary bridge failed to start"
+726
reason: (connect-outcome-message outcome))
+727
;; Set the STATE as well as the detail. A detail without a
+728
;; state leaves whatever was there before — which after a
+729
;; 'terminal verdict would report a retryable failure as
+730
;; permanent.
+731
(set-link-status! state 'connecting
+732
(connect-outcome-message outcome)
+733
(enclave-bridge-state-link-attempt state)
+734
(enclave-bridge-state-link-next-delay state))
+735
#f)
+736
(else
+737
(log-info "Apiary bridge connected"
+738
mode: (symbol->string mode)
+739
host: (enclave-config-host config)
+740
nick: (enclave-conn-current-nick conn)
+741
owner: (or owner-nick "<none>")
+742
reports-to: (or reports-to owner-nick "<none>"))
+743
;; Reconnect-presence broadcast — uniform across leader
+744
;; and worker. Trusted-set filter handles privacy.
+745
(post-presence! state)
+746
#t))))))
+747
+748
;;; Bring the bridge up and KEEP TRYING until it does, with the
+749
;;; same backoff schedule the established-connection reconnect
+750
;;; path uses.
+751
;;;
+752
;;; Why this exists: `enclave-bridge-start!` attempts once, and
+753
;;; the reconnect machinery in (apiary enclave) only guards a
+754
;;; connection that was already established. So an apiary that
+755
;;; booted during an outage never connected for the rest of its
+756
;;; life — while advertising every tool, and while every tool call
+757
;;; returned a message that could not distinguish "not yet" from
+758
;;; "never".
+759
;;;
+760
;;; THE PAIRING WITH THE BOUNDED CONNECT IS NOT OPTIONAL, and this
+761
;;; is the part worth remembering if anyone ever splits them:
+762
;;; retrying an UNBOUNDED connect is strictly WORSE than not
+763
;;; retrying at all. Measured, one unbounded attempt starved MCP
+764
;;; dispatch for ~134 s, and the only reason the process became
+765
;;; responsive again was that it had given up permanently. Looping
+766
;;; that would wedge dispatch for as long as the outage lasted.
+767
;;; The bound has to be in place before the retry is.
+768
(define (enclave-bridge-run! state server config mode
+769
owner-nick reports-to)
+770
(: enclave-bridge-state? any? enclave-config? symbol?
+771
any? any? -> boolean?)
+772
(cond
+773
((not (enclave-config-ready? config))
+774
;; Terminal by construction: no configuration is arriving
+775
;; later in this process's life. Saying "retrying" here would
+776
;; be a lie, and saying "not connected" is the lie by
+777
;; omission this change exists to remove.
+778
(set-link-status!
+779
state 'terminal
+780
"incomplete configuration (need APIARY_ENCLAVE_HOST, APIARY_USER, APIARY_TOKEN)"
+781
0 #f)
+782
(log-warn "Apiary bridge not configured — every tool call will report a TERMINAL failure")
+783
#f)
+784
(else
+785
(let loop ((attempt 1))
+786
(cond
+787
;; Honour a shutdown requested while the loop was mid-
+788
;; outage. Without this the loop's only exit is success,
+789
;; so a process asked to stop during an outage would keep
+790
;; retrying and could establish a connection nobody wanted
+791
;; after the request to stop.
+792
((enclave-bridge-state-stop? state)
+793
(log-info "Apiary bridge start-up loop stopping on request"
+794
attempt: (number->string attempt))
+795
(set-link-status! state 'terminal
+796
"start-up cancelled"
+797
attempt #f)
+798
#f)
+799
;; The single-attempt call is itself fully guarded, but
+800
;; guard the loop body too: an escape from here kills the
+801
;; boot goroutine outright and nothing restarts it, so the
+802
;; process would sit advertising tools that can never work.
+803
((guard (e (else
+804
(log-error "Apiary bridge attempt raised"
+805
attempt: (number->string attempt)
+806
error: (format "~a" e))
+807
(set-enclave-bridge-state-link-detail!
+808
state (format "~a" e))
+809
#f))
+810
(enclave-bridge-start! state server config mode
+811
owner-nick reports-to))
+812
#t)
+813
(else
+814
(let ((delay (reconnect-delay-for attempt)))
+815
;; #f detail: keep the classified reason the attempt
+816
;; just recorded rather than blanking it for the sleep.
+817
(set-link-status! state 'connecting #f attempt delay)
+818
(log-info "Apiary bridge retrying"
+819
attempt: (number->string attempt)
+820
delay-sec: (number->string delay)
+821
last-failure: (enclave-bridge-state-link-detail state))
+822
(sleep delay)
+823
(loop (+ attempt 1)))))))))
824
825
;; ============================================================
826
;; Public: outbound message helpers
@@ -591,9 +838,9 @@
838
(let ((conn (enclave-bridge-state-conn state))
839
(config (enclave-bridge-state-config state)))
840
(cond
−594
((not conn) "Error: enclave bridge not connected")
+841
((not conn) (bridge-unavailable-message state))
842
((not (enclave-conn-ready? conn))
−596
"Error: enclave connection not ready")
+843
(bridge-unavailable-message state))
844
(else
845
(let ((channels (split-channels
846
(enclave-config-channel config))))
@@ -664,9 +911,9 @@
911
(format: any?) -> string?)
912
(let ((conn (enclave-bridge-state-conn state)))
913
(cond
−667
((not conn) "Error: enclave bridge not connected")
+914
((not conn) (bridge-unavailable-message state))
915
((not (enclave-conn-ready? conn))
−669
"Error: enclave connection not ready")
+916
(bridge-unavailable-message state))
917
(else
918
(let* ((normalized (normalize-literal-newlines text))
919
(formatted (if (eq? format 'plain)
@@ -683,9 +930,9 @@
930
(: enclave-bridge-state? string? string? string? -> string?)
931
(let ((conn (enclave-bridge-state-conn state)))
932
(cond
−686
((not conn) "Error: enclave bridge not connected")
+933
((not conn) (bridge-unavailable-message state))
934
((not (enclave-conn-ready? conn))
−688
"Error: enclave connection not ready")
+935
(bridge-unavailable-message state))
936
((or (not (string? target)) (string=? target ""))
937
"Error: target (channel or nick) required")
938
((or (not (string? msgid)) (string=? msgid ""))
@@ -777,7 +1024,7 @@
1024
(reports-to (or reports-to-arg default-reports-to)))
1025
(cond
1026
((not nick) "Error: missing required argument 'nick'")
−780
((not conn) "Error: enclave bridge not connected")
+1027
((not conn) (bridge-unavailable-message state))
1028
(else
1029
(let* ((result (enclave-register-bot conn nick
1030
auto-join: auto-join
@@ -803,7 +1050,7 @@
1050
(conn (enclave-bridge-state-conn state)))
1051
(cond
1052
((not nick) "Error: missing required argument 'nick'")
−806
((not conn) "Error: enclave bridge not connected")
+1053
((not conn) (bridge-unavailable-message state))
1054
(else
1055
(let ((result (enclave-revoke-bot conn nick)))
1056
(when (string=? (or (assoc-or "status" result "") "") "ok")
@@ -825,7 +1072,7 @@
1072
(nick (or (dict-ref args nick: #f)
1073
(and conn (enclave-conn-current-nick conn)))))
1074
(cond
−828
((not conn) "Error: enclave bridge not connected")
+1075
((not conn) (bridge-unavailable-message state))
1076
((not nick) "Error: no bot nick available")
1077
(else
1078
(let* ((result (enclave-rotate-token conn nick))
@@ -861,7 +1108,7 @@
1108
(let* ((conn (enclave-bridge-state-conn state))
1109
(self-nick (and conn (enclave-conn-current-nick conn))))
1110
(cond
−864
((not conn) "Error: enclave bridge not connected")
+1111
((not conn) (bridge-unavailable-message state))
1112
((not self-nick) "Error: own nick not yet known")
1113
(else
1114
(let* ((cmd (string-append "bot list-reports-to " self-nick))
test/test-enclave.sglmodified
@@ -280,6 +280,237 @@
280
(assert-true (procedure? await-registration!))))
281
282
+283
;; ============================================================
+284
;; Bounded connect
+285
;;
+286
;; The defect these guard: a blocking connect(2) on this process's
+287
;; single cooperative scheduler took the kernel's full ~135s
+288
;; SYN-retransmit budget, during which NO MCP request could be
+289
;; answered — measured at 53/120 requests answered with a ~134s
+290
;; contiguous dead window. These tests pin the two properties that
+291
;; prevent it: the connect is BOUNDED, and its failure is CLASSIFIED.
+292
;;
+293
;; Note what is deliberately real here rather than mocked: the
+294
;; blackhole case actually opens a socket against TEST-NET-1
+295
;; (192.0.2.1, RFC 5737, guaranteed unrouted) and asserts on the
+296
;; wall-clock. A mocked bound would pass just as happily against an
+297
;; unbounded connect, which is the whole thing being prevented.
+298
;; ============================================================
+299
+300
(test-group "bounded connect"
+301
+302
(test "connect bound is below the PONG timeout"
+303
;; Not a style preference. A connect attempt allowed to outlive
+304
;; +keepalive-pong-timeout+ (30s) would let the keepalive sweep
+305
;; declare the link dead and kick a SECOND reconnect while the
+306
;; first is still in flight. This assertion is what keeps that
+307
;; ordering true if someone raises the default later.
+308
;;
+309
;; ASSERTED VIA `load-enclave-config`, DELIBERATELY. An earlier
+310
;; version of this test read the struct's field default instead,
+311
;; and a sabotage run proved it VACUOUS: raising the constant that
+312
;; `load-enclave-config` actually uses from 5.0 to 45.0 left this
+313
;; test green, because the struct default is a separate literal.
+314
;; The bound that governs a real connection is the loaded one, so
+315
;; that is the one under test.
+316
(setenv! "APIARY_CONNECT_TIMEOUT" "")
+317
(setenv! "APIARY_TLS_HANDSHAKE_TIMEOUT" "")
+318
(let ((cfg (load-enclave-config)))
+319
(assert-true (< (enclave-config-connect-timeout cfg) 30.0))
+320
(assert-true (> (enclave-config-connect-timeout cfg) 0))))
+321
+322
(test "the struct default and the loaded default cannot drift"
+323
;; The same default is written in two places — the `enclave-config`
+324
;; field default and `+connect-timeout-default+`, which the struct
+325
;; cannot reference because this file defines types before
+326
;; constants. Code that builds a config directly reads the first;
+327
;; code that loads from the environment reads the second. They must
+328
;; agree, and nothing in the compiler makes them.
+329
;;
+330
;; This test exists because the duplication was only discovered
+331
;; when a sabotage of one of them failed to turn anything red.
+332
(setenv! "APIARY_CONNECT_TIMEOUT" "")
+333
(setenv! "APIARY_TLS_HANDSHAKE_TIMEOUT" "")
+334
(let ((loaded (load-enclave-config))
+335
(fresh (enclave-config)))
+336
(assert-equal (enclave-config-connect-timeout loaded)
+337
(enclave-config-connect-timeout fresh))
+338
(assert-equal (enclave-config-tls-handshake-timeout loaded)
+339
(enclave-config-tls-handshake-timeout fresh))))
+340
+341
(test "handshake bound is separate from the connect bound"
+342
;; They bound different failures: the connect bound cannot reach
+343
;; a peer that ACCEPTS and then goes silent, because at that
+344
;; point the connect phase is over.
+345
(setenv! "APIARY_CONNECT_TIMEOUT" "")
+346
(setenv! "APIARY_TLS_HANDSHAKE_TIMEOUT" "")
+347
(let ((cfg (load-enclave-config)))
+348
(assert-true (> (enclave-config-tls-handshake-timeout cfg)
+349
(enclave-config-connect-timeout cfg)))))
+350
+351
(test "a non-positive timeout override is REFUSED, not honoured"
+352
;; The underlying native reads a non-positive timeout as "no
+353
;; bound" and reverts to a plain blocking connect. So honouring
+354
;; APIARY_CONNECT_TIMEOUT=0 would silently restore the exact
+355
;; ~135s wedge, while looking like a deliberate configuration
+356
;; choice. It must fall back to the default instead.
+357
(setenv! "APIARY_CONNECT_TIMEOUT" "0")
+358
(let ((cfg (load-enclave-config)))
+359
(assert-true (> (enclave-config-connect-timeout cfg) 0)))
+360
(setenv! "APIARY_CONNECT_TIMEOUT" "-5")
+361
(let ((cfg (load-enclave-config)))
+362
(assert-true (> (enclave-config-connect-timeout cfg) 0)))
+363
(setenv! "APIARY_CONNECT_TIMEOUT" "not-a-number")
+364
(let ((cfg (load-enclave-config)))
+365
(assert-true (> (enclave-config-connect-timeout cfg) 0)))
+366
;; ...and a legitimate override IS honoured, so the test above
+367
;; cannot pass merely because the override is ignored wholesale.
+368
;; (This is the positive control: without it, a `positive-or` that
+369
;; returned the fallback unconditionally would pass every
+370
;; assertion above.)
+371
(setenv! "APIARY_CONNECT_TIMEOUT" "2.5")
+372
(let ((cfg (load-enclave-config)))
+373
(assert-equal 2.5 (enclave-config-connect-timeout cfg)))
+374
;; Cleared LAST so a failure above cannot leak 2.5 into every
+375
;; later test in this process. Any assertion that fires early
+376
;; leaves the env set, which is why the clears also run at the
+377
;; TOP of each test that depends on the defaults.
+378
(setenv! "APIARY_CONNECT_TIMEOUT" ""))
+379
+380
(test "classify-connect-status maps statuses to kinds"
+381
(assert-equal 'ok (classify-connect-status "connected"))
+382
(assert-equal 'refused (classify-connect-status "connection refused"))
+383
(assert-equal 'timeout (classify-connect-status "connection timed out"))
+384
(assert-equal 'unreachable (classify-connect-status "host unreachable"))
+385
(assert-equal 'unreachable
+386
(classify-connect-status "network unreachable"))
+387
;; An UNKNOWN status must NOT be folded into a specific kind.
+388
;; Guessing here would recreate the defect: a message asserting a
+389
;; diagnosis it does not have.
+390
(assert-equal 'other (classify-connect-status "some new errno text"))
+391
(assert-equal 'other (classify-connect-status #f)))
+392
+393
(test "a refused endpoint is classified 'refused, fast"
+394
;; POSITIVE CONTROL for the blackhole test below: this proves the
+395
;; instrument can SEE a fast, correctly-labelled failure. Without
+396
;; it, "the blackhole case returned in under the bound" is also
+397
;; what a connect that never really ran would report.
+398
(let* ((cfg (enclave-config host: "127.0.0.1" port: 1
+399
user: "n" token: "t"
+400
connect-timeout: 5.0))
+401
(t0 (current-second))
+402
(outcome (open-enclave-transport cfg))
+403
(elapsed (- (current-second) t0)))
+404
(assert-equal 'refused (connect-outcome-kind outcome))
+405
(assert-false (connect-outcome-socket outcome))
+406
;; Fast, i.e. it did not sit out the bound.
+407
(assert-true (< elapsed 2.0))))
+408
+409
(test "a blackholed endpoint fails WITHIN the bound"
+410
;; The measured defect, inverted into an assertion. 192.0.2.1 is
+411
;; TEST-NET-1 (RFC 5737) and is guaranteed not to be routed, so
+412
;; this is the same condition that took ~135s before.
+413
;;
+414
;; THE BOUND IS THE ASSERTION. The KIND is deliberately not pinned
+415
;; to 'timeout: a host with no default route, or a restricted CI
+416
;; network, gets ENETUNREACH immediately and correctly reports
+417
;; 'unreachable. Demanding 'timeout there would fail the suite for
+418
;; a difference in the machine's routing table rather than a
+419
;; difference in this code. Both are non-'ok, both are classified,
+420
;; and both are bounded — which is what this test is about.
+421
(let* ((cfg (enclave-config host: "192.0.2.1" port: 6697
+422
user: "n" token: "t"
+423
connect-timeout: 2.0))
+424
(t0 (current-second))
+425
(outcome (open-enclave-transport cfg))
+426
(elapsed (- (current-second) t0))
+427
(kind (connect-outcome-kind outcome)))
+428
(assert-true (or (eq? kind 'timeout) (eq? kind 'unreachable)))
+429
(assert-false (connect-outcome-socket outcome))
+430
;; The bound fires well before the kernel's ~135s budget.
+431
(assert-true (< elapsed 8.0))))
+432
+433
(test "an unresolvable host is classified 'dns"
+434
(let ((outcome (open-enclave-transport
+435
(enclave-config
+436
host: "no-such-host.invalid" port: 6697
+437
user: "n" token: "t"
+438
connect-timeout: 2.0))))
+439
(assert-equal 'dns (connect-outcome-kind outcome))
+440
(assert-false (connect-outcome-socket outcome))))
+441
+442
(test "the four failure kinds produce DISTINCT messages"
+443
;; Symptom 3 of the defect was that connection-refused, NXDOMAIN,
+444
;; a TLS mismatch and a timeout all produced ONE message whose
+445
;; only varying part was the host:port echoed back — the input,
+446
;; not a diagnosis. Distinctness is the property, so assert it
+447
;; directly rather than eyeballing the strings.
+448
(let ((msgs (map (lambda (kind)
+449
(connect-outcome-message
+450
(connect-outcome kind: kind detail: "same-detail")))
+451
'(refused dns timeout tls registration config))))
+452
(let loop ((rest msgs))
+453
(cond
+454
((null? rest) #t)
+455
(else
+456
(for-each
+457
(lambda (other)
+458
(assert-false (string=? (car rest) other)))
+459
(cdr rest))
+460
(loop (cdr rest)))))))
+461
+462
(test "enclave-connect/detail reports 'config for an incomplete config"
+463
;; TERMINAL rather than retryable: no configuration is arriving
+464
;; later in this process's life, so a caller must be able to tell
+465
;; this apart from "not connected yet".
+466
(let* ((result (enclave-connect/detail (enclave-config)))
+467
(outcome (car result)))
+468
(assert-equal 'config (connect-outcome-kind outcome))
+469
(assert-false (cdr result))))
+470
+471
(test "a successful connect reports NO failure text and a reset count"
+472
;; Pins the VALUES the success path reports, not a formatter's
+473
;; tolerance of values a test supplied itself. `detail` reaches the
+474
;; operator only as "last failure: <detail>", so anything
+475
;; success-flavoured here renders as "last failure: connected" the
+476
;; next time the session drops — which is what an adversarial
+477
;; review found in an earlier version of this code.
+478
;;
+479
;; A recording hook is the whole point: it captures what the
+480
;; PRODUCTION call site passes, so sabotaging that call site turns
+481
;; this red. A test that called `report-link-status!` with its own
+482
;; arguments would assert nothing about the code under test.
+483
(let* ((seen '())
+484
(c (enclave-conn irc: #f config: (enclave-config))))
+485
(set-enclave-conn-status-hook!
+486
c (lambda (state detail attempt next-delay)
+487
(set! seen (list state detail attempt next-delay))))
+488
(report-link-connected! c)
+489
(assert-equal 'connected (car seen))
+490
(assert-equal "" (cadr seen))
+491
(assert-equal 0 (caddr seen))
+492
(assert-false (cadddr seen))))
+493
+494
(test "a status hook that raises cannot break the reconnect loop"
+495
;; The hook is supplied by another module. If a raise from it could
+496
;; escape, a reporting bug in the consumer would take down the
+497
;; reconnect machinery — trading a cosmetic defect for an outage.
+498
(let ((c (enclave-conn irc: #f config: (enclave-config))))
+499
(set-enclave-conn-status-hook!
+500
c (lambda (state detail attempt next-delay)
+501
(error "hook exploded")))
+502
;; Must return normally rather than propagating.
+503
(report-link-connected! c)
+504
(report-link-status! c 'connecting #f 3 8)
+505
(assert-true #t)))
+506
+507
(test "backoff schedule grows and is capped"
+508
(assert-equal 1.0 (reconnect-delay-for 1))
+509
(assert-equal 2.0 (reconnect-delay-for 2))
+510
(assert-equal 4.0 (reconnect-delay-for 3))
+511
(assert-equal 60.0 (reconnect-delay-for 20))))
+512
+513
514
(define (string-prefix? prefix s)
515
(and (>= (string-length s) (string-length prefix))
516
(string=? prefix (substring s 0 (string-length prefix)))))
test/test-tools.sglmodified
@@ -5,6 +5,8 @@
5
(sigil mcp server)
6
(sigil mcp protocol)
7
(apiary markdown-irc)
+8
;; for `enclave-config`, which `enclave-bridge-run!` takes
+9
(apiary enclave)
10
(apiary tools))
11
12
;; A literal backslash-n as it actually arrives from an LLM caller:
@@ -447,3 +449,173 @@
449
(let ((s (make-enclave-bridge-state)))
450
(assert-false (trusted-sender? s "anyone"))
451
(assert-false (trusted-sender? s "owner")))))
+452
+453
+454
;; ============================================================
+455
;; Tool-facing link status
+456
;;
+457
;; The defect: every tool call that could not reach the channel
+458
;; returned the string "Error: enclave bridge not connected",
+459
;; whether the bridge was two seconds from succeeding or would
+460
;; never connect again for the life of the process. Those are
+461
;; different facts and a caller has to be able to act on the
+462
;; difference — a process that will never recover reading as a
+463
;; transient blip is what let an outage look survivable for a day
+464
;; and a half.
+465
;; ============================================================
+466
+467
(define (contains? haystack needle)
+468
(and (string? haystack)
+469
(>= (string-length haystack) (string-length needle))
+470
(let loop ((i 0))
+471
(cond
+472
((> (+ i (string-length needle)) (string-length haystack)) #f)
+473
((string=? needle (substring haystack i
+474
(+ i (string-length needle))))
+475
#t)
+476
(else (loop (+ i 1)))))))
+477
+478
(test-group "bridge-unavailable-message"
+479
+480
(test "terminal says it will NOT connect and asks for operator action"
+481
(let ((s (make-enclave-bridge-state)))
+482
(set-enclave-bridge-state-link-state! s 'terminal)
+483
(set-enclave-bridge-state-link-detail! s "incomplete configuration")
+484
(let ((msg (bridge-unavailable-message s)))
+485
(assert-true (contains? msg "will NOT connect"))
+486
(assert-true (contains? msg "incomplete configuration"))
+487
;; It must NOT imply a retry that is not happening.
+488
(assert-false (contains? msg "retrying")))))
+489
+490
(test "retrying names the attempt, the last failure and the next try"
+491
(let ((s (make-enclave-bridge-state)))
+492
(set-enclave-bridge-state-link-state! s 'connecting)
+493
(set-enclave-bridge-state-link-detail!
+494
s "connect timed out: connection timed out at 10.0.0.1:6697")
+495
(set-enclave-bridge-state-link-attempt! s 4)
+496
(set-enclave-bridge-state-link-next-delay! s 8)
+497
(let ((msg (bridge-unavailable-message s)))
+498
(assert-true (contains? msg "retrying"))
+499
(assert-true (contains? msg "attempt 4"))
+500
(assert-true (contains? msg "connect timed out"))
+501
(assert-true (contains? msg "next attempt in 8s"))
+502
;; And it must NOT claim the situation is hopeless.
+503
(assert-false (contains? msg "will NOT connect")))))
+504
+505
(test "terminal and retrying are DISTINGUISHABLE"
+506
;; The single property the whole change exists to provide. Asserted
+507
;; directly so it cannot regress into two differently-worded
+508
;; strings that a caller still cannot act on differently.
+509
(let ((terminal (make-enclave-bridge-state))
+510
(retrying (make-enclave-bridge-state)))
+511
(set-enclave-bridge-state-link-state! terminal 'terminal)
+512
(set-enclave-bridge-state-link-detail! terminal "same detail")
+513
(set-enclave-bridge-state-link-state! retrying 'connecting)
+514
(set-enclave-bridge-state-link-detail! retrying "same detail")
+515
(assert-false (string=? (bridge-unavailable-message terminal)
+516
(bridge-unavailable-message retrying)))))
+517
+518
;; ----------------------------------------------------------------
+519
;; The tests above exercise the FORMATTER only: they write
+520
;; link-detail through a setter and then assert the string renders
+521
;; it. Every one of them passed while the real plumbing was
+522
;; overwriting the classified failure with the literal word
+523
;; "reconnecting" before each backoff sleep — a bug an adversarial
+524
;; reviewer found and these tests structurally could not.
+525
;;
+526
;; A test that writes a value through a setter and then asserts the
+527
;; formatter rendered it is testing the formatter. The tests below
+528
;; drive `set-link-status!` in the SEQUENCE the connect machinery
+529
;; drives it, which is where the defect lived.
+530
;; ----------------------------------------------------------------
+531
+532
(test "a scheduled attempt does NOT erase the last classified failure"
+533
;; The exact regression. Sequence per retry iteration is:
+534
;; report 'connecting (attempt scheduled, no new diagnosis)
+535
;; ... sleep, up to 60s ...
+536
;; attempt
+537
;; report the classified outcome
+538
;; A tool call landing in the sleep — which is most of them, since
+539
;; the sleep dominates the attempt by an order of magnitude — must
+540
;; still see the real reason.
+541
(let ((s (make-enclave-bridge-state)))
+542
;; A real failure lands.
+543
(set-link-status! s 'connecting
+544
"connect timed out: 10.0.0.1:6697" 3 8)
+545
;; Next iteration schedules an attempt with NO new diagnosis.
+546
(set-link-status! s 'connecting #f 4 16)
+547
(let ((msg (bridge-unavailable-message s)))
+548
(assert-true (contains? msg "connect timed out"))
+549
(assert-false (contains? msg "last failure: reconnecting"))
+550
;; The freshly-scheduled attempt/delay DO update.
+551
(assert-true (contains? msg "attempt 4"))
+552
(assert-true (contains? msg "next attempt in 16s")))))
+553
+554
(test "a successful connect leaves no failure text behind"
+555
;; link-detail is only ever rendered as "last failure: <detail>",
+556
;; so writing a success string into it produces "last failure:
+557
;; connected" the next time the session drops.
+558
(let ((s (make-enclave-bridge-state)))
+559
(set-link-status! s 'connecting "connect timed out: host:6697" 3 8)
+560
(set-link-status! s 'connected "" 0 #f)
+561
;; Session drops; nothing has reported in yet.
+562
(let ((msg (bridge-unavailable-message s)))
+563
(assert-false (contains? msg "last failure: connected"))
+564
(assert-false (contains? msg "connect timed out"))
+565
;; And it must be honest about how long detection can take.
+566
(assert-true (contains? msg "90s")))))
+567
+568
(test "a fresh state defaults to idle, and idle reads as retrying"
+569
;; 'idle means the connect goroutine has not reported yet, which
+570
;; is a "not yet" rather than a "never" — reporting it as terminal
+571
;; would be the same lie in the other direction.
+572
(let ((s (make-enclave-bridge-state)))
+573
(assert-equal 'idle (enclave-bridge-state-link-state s))
+574
(assert-true (contains? (bridge-unavailable-message s)
+575
"retrying")))))
+576
+577
+578
;; ============================================================
+579
;; The start-up retry loop
+580
;;
+581
;; This was the largest new control-flow addition in the change and
+582
;; had NO coverage at all — not in this suite and not in the smoke
+583
;; harness, which calls `enclave-bridge-start!` rather than
+584
;; `enclave-bridge-run!`. Found by adversarial review, not by me.
+585
;;
+586
;; A loop whose only exit is success cannot be tested by letting it
+587
;; run, so both terminating paths are what get asserted here.
+588
;; ============================================================
+589
+590
(test-group "enclave-bridge-run!"
+591
+592
(test "an unconfigured bridge is TERMINAL, and does not spin"
+593
;; No configuration is arriving later in this process's life, so
+594
;; retrying would burn the scheduler forever on something that
+595
;; cannot succeed AND would report "retrying" while doing it.
+596
(let* ((s (make-enclave-bridge-state))
+597
(result (enclave-bridge-run! s #f (enclave-config)
+598
'leader #f #f)))
+599
(assert-false result)
+600
(assert-equal 'terminal (enclave-bridge-state-link-state s))
+601
(let ((msg (bridge-unavailable-message s)))
+602
(assert-true (contains? msg "will NOT connect"))
+603
(assert-false (contains? msg "retrying")))))
+604
+605
(test "a stop request ends the loop instead of connecting anyway"
+606
;; Without this the loop's only exit is success, so a process asked
+607
;; to shut down mid-outage would keep retrying and could establish
+608
;; a connection after the request to stop. The config here is
+609
;; deliberately COMPLETE, so 'terminal cannot be reached by the
+610
;; unconfigured path above — the stop flag is the only way out, and
+611
;; a regression that ignored it would hang this test rather than
+612
;; failing it quietly.
+613
(let ((s (make-enclave-bridge-state)))
+614
(set-enclave-bridge-state-stop?! s #t)
+615
(let ((result (enclave-bridge-run!
+616
s #f
+617
(enclave-config host: "127.0.0.1" port: 1
+618
user: "n" token: "t")
+619
'leader #f #f)))
+620
(assert-false result)
+621
(assert-equal 'terminal (enclave-bridge-state-link-state s))))))