Add COURIER_TELEGRAM_POLL_FORCE, an escape hatch with a stand-down handshake
David's ruling: "Telegram polling is only for the leader and there should only ever be one running by default." That confirms the semantics already built -- leader-mode eligibility, oldest-live-wins, no new configuration required for the default -- so nothing about the default path changes here. An opt-in marker was explicitly NOT added: it would make the poller default to OFF, and a leader session that forgot to set it would silently stop receiving Telegram, which is the go-dark class this branch removed.
What is added is a way OUT of the one case the re-acquire cannot handle. The claim decides which session receives David's INBOUND messages, not merely which one avoids 409s. The 60s re-acquire recovers from a DEAD holder; a live-but-abandoned session holds a perfectly valid claim while his "task: ..." messages land where nobody is watching, and the only remedy was killing processes by hand. COURIER_TELEGRAM_POLL_FORCE=1 makes that one env var and an /mcp.
The hard part is not taking the claim, it is that the displaced holder must stop. Two pollers arriving through the repair path would be the original bug re-entering through its own fix, so:
- the holder re-confirms ownership every 10s while polling, and on loss kills its child, abandons the claim WITHOUT unlinking it (the file names the new owner; deleting would leave nobody polling), logs at info, and falls back to being a normal non-polling instance that keeps re-checking;
- a FORCED acquire returns a distinct verdict and waits 14s -- strictly more than the holder's check interval -- before it starts polling, so there is no window with two consumers rather than merely a short one;
- force applies to the FIRST attempt only. Two forcing instances would otherwise steal from each other every interval forever.
break-stale-claim! generalised to take-claim-from!: taking a claim from a dead owner and from a live one are the same mechanics, and the difference is only whether we were entitled to, which the caller has already decided. The conditional-unlink TOCTOU discipline applies to both.
Harness case 9 proves it end to end: A polls, B forces, B polls, A stops, A stays healthy and serving, A leaves B's claim intact, and ZERO 409s across the whole handover. Sabotage-tested by setting the stand-down wait to 0 -- conflicts 1, max_inflight 2, i.e. the assertion detects the overlap it exists to forbid.
The unit-level force test is deliberately NOT the obvious one. The force branch only runs when the owner reads 'alive, which needs a live pid with our comm -- a second courier, which does not exist in a unit test. A claim owned by pid 1 takes the ordinary stale path whether or not force is set, so the first draft asserted "forcing works" while never executing the force branch. It is replaced by tests that force does NOT change any other outcome, with the scope limit stated in the file.
Default path demonstrated unchanged, not asserted: the full harness run against the 95515da binary and against this one give identical verdicts for cases 1-8, and differ only in case 9, which the older binary cannot pass.
Suite 116/0. Harness 54/0 with the pre-fix control still red. Storm repro 1 send -> 1 delivery with its positive control at 4. Sentinel-zero green.
repro/single_poller_harness.py | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
src/courier/claim.sgl | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------
src/courier/config.sgl | 16 ++++++++++++++++
src/courier/poller.sgl | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
test/test-claim.sgl | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
5 files changed, 326 insertions(+), 38 deletions(-)repro/single_poller_harness.pymodified
pointed at the local mock, which is the only endpoint the send AND poll pathsuse once that variable is set.The cases are NUMBERED BY TOPIC, and run in the order 1, 3, 4, 7, 8, 5, 6, 2 --The cases are NUMBERED BY TOPIC, and run in the order 1, 3, 4, 7, 9, 8, 5, 6, 2 --the ordering is dictated by shared state (cases 1, 3, 4 and 7 all operate on thesame pair of live instances, and the pre-fix control in case 2 runs last so its409s can never be confused with the fixed binary's). What each proves: 8 LOG PRUNING instance logs are capped, a LIVE instance's log is never pruned however old it looks, and nothing that is not an instance log is touched. 9 FORCED TAKEOVER COURIER_TELEGRAM_POLL_FORCE takes the claim from a LIVE holder; the displaced instance stops polling, stays healthy, and does not delete the new owner's claim -- with zero 409s across the handover.Usage: single_poller_harness.py --bin <courier> [--control-bin <pre-fix courier>] log = b.log_text() check("the takeover is logged, once, at INFO", len([l for l in log.splitlines() if "previous holder is gone" in l]) == 1, if "claim acquired -- the previous holder is gone" in l and "[INFO]" in l]) == 1, repr([l for l in log.splitlines() if "claim" in l])) # And the skip line was NOT repeated every recheck interval. check("the periodic re-check does not spam the log", claim_owner(state_dir) == str(c.pid), f"owner={claim_owner(state_dir)} C={c.pid}") log = c.log_text() check("the break is logged", "Breaking stale" in log, repr([l for l in log.splitlines() if "stale" in l][:2])) check("the break is logged", any("Taking the Telegram poller claim" in l and "previous holder is gone" in l for l in log.splitlines()), repr([l for l in log.splitlines() if "Taking" in l][:2])) return cdef case_forced_takeover(binp, run_dir): """COURIER_TELEGRAM_POLL_FORCE: a specific session takes the claim from a LIVE holder, and the displaced one stands down. This is the case the unit tests cannot reach: the force branch only runs when the owner reads as 'alive, which needs a second real courier. The property that matters is not just "B ends up polling" -- it is that there is never a moment with TWO pollers on the token. A takeover that produced even ten seconds of overlap would be the original 409 bug arriving through the escape hatch, so conflicts are asserted at zero across the whole handover. """ print("=== 9 FORCED TAKEOVER (COURIER_TELEGRAM_POLL_FORCE) ===") d = os.path.join(run_dir, "force") os.makedirs(d, exist_ok=True) post("/_seed", {"updates": []}) post("/_dwell", {"seconds": DWELL}) a = Courier(binp, d, "force-a") a.initialize() got_a = wait_for(lambda: len(poller_children(a.pid)) == 1, timeout=40) check("holder A is polling before the takeover", got_a, f"A children={poller_children(a.pid)}") b = Courier(binp, d, "force-b", extra_env={"COURIER_TELEGRAM_POLL_FORCE": "1"}) b.initialize() # B waits out the stand-down window before it starts, so allow for it. took = wait_for(lambda: len(poller_children(b.pid)) == 1, timeout=90) check("the forcing instance takes over and polls", took, f"B children={poller_children(b.pid)} owner={claim_owner(d)}") check("the claim records the forcing instance", claim_owner(d) == str(b.pid), f"owner={claim_owner(d)} B={b.pid}") stood_down = wait_for(lambda: len(poller_children(a.pid)) == 0, timeout=45) check("the displaced holder STOPS polling", stood_down, f"A children={poller_children(a.pid)}") alog = a.log_text() check("the displaced holder logs that it lost the claim", "claim taken by another instance" in alog, repr([l for l in alog.splitlines() if "claim" in l][-2:])) check("the displaced holder is still alive and serving", a.alive() and a.call("list-relays", {}) is not None) blog = b.log_text() check("the forced takeover is logged at INFO", len([l for l in blog.splitlines() if "COURIER_TELEGRAM_POLL_FORCE" in l and "[INFO]" in l]) == 1, repr([l for l in blog.splitlines() if "claim" in l][:3])) # Let both run a while so any overlap would show up at the mock. time.sleep(OBSERVE_WINDOW) s = stats() check("the handover produced NO 409s at any point", s["conflicts"] == 0, f"conflicts={s['conflicts']} getupdates={s['getupdates']} " f"max_inflight={s['max_inflight']}") check("the mock never saw two concurrent pollers during the handover", s["max_inflight"] <= 1, f"max_inflight={s['max_inflight']}") # And the displaced instance must NOT have deleted the new owner's claim. check("the displaced holder left the new owner's claim intact", claim_owner(d) == str(b.pid), f"owner={claim_owner(d)}") for inst in (a, b): inst.close_stdin() time.sleep(1) for inst in (a, b): inst.kill()def case_log_pruning(binp, run_dir): """Per-instance logs must not accumulate forever, and must not take anything else with them. inst.kill() instances = [] case_forced_takeover(args.bin, run_dir) case_log_pruning(args.bin, run_dir) dry_dir, text = case_dry_run(args.bin, run_dir)src/courier/claim.sglmodified
(courier relay)) (export acquire-poller-claim! release-poller-claim! poller-claim-held? abandon-poller-claim! poller-claim-path claim-token-hash read-claim-owner (make-symlink me path) (if (equal? (read-claim-owner path) me) 'acquired 'lost))) ;;; Does this process still hold the claim for `token`? ;;; ;;; The poller calls this while running, because holding the claim once ;;; is not the same as holding it now: COURIER_TELEGRAM_POLL_FORCE lets ;;; another session take it. A displaced holder that kept polling would ;;; put two consumers on the token again, which is the whole bug. (define (poller-claim-held? token) (: string? -> boolean?) (equal? (read-claim-owner (poller-claim-path token)) (number->string (process-id)))) ;;; Try to become the single poller for `token`. ;;; ;;; `force?` (COURIER_TELEGRAM_POLL_FORCE) takes the claim from a LIVE ;;; holder too. Default off. It is applied only on the FIRST attempt, not ;;; on the periodic re-checks -- two forcing instances would otherwise ;;; steal from each other every interval forever. ;;; ;;; Returns: ;;; 'acquired -- this process polls, and must call ;;; release-poller-claim! on clean shutdown ;;; 'acquired-by-force -- as 'acquired, but taken from a LIVE holder, so ;;; the caller must let that holder stand down first ;;; 'held-by-other -- another live courier polls. NORMAL. Log at info. ;;; 'undetermined -- liveness could not be established; do not poll, ;;; and do not break the existing claim. (define (acquire-poller-claim! token) (: string? -> symbol?) (define (acquire-poller-claim! token force?) (: string? any? -> symbol?) (let* ((path (poller-claim-path token)) (me (number->string (process-id)))) (guard (e (else 'acquired) (else (case (pid-liveness owner) ((alive) 'held-by-other) ((alive) (if force? (forced (take-claim-from! path owner me "forced by COURIER_TELEGRAM_POLL_FORCE (previous holder is alive)")) 'held-by-other)) ((undetermined) (log-warn "Poller claim owner liveness undetermined" owner-pid: owner) 'undetermined) (else (break-stale-claim! path owner me)))))))))) (if force? (forced (take-claim-from! path owner me "forced by COURIER_TELEGRAM_POLL_FORCE (owner liveness undetermined)")) (begin (log-warn "Poller claim owner liveness undetermined" owner-pid: owner) 'undetermined))) (else (take-claim-from! path owner me "previous holder is gone")))))))))) ;; Distinguish a FORCED win from an ordinary one. The caller has to wait ;; for the displaced holder to notice before it starts polling, or there ;; would be a window with two consumers on the token -- the original bug, ;; arriving through the escape hatch. A stale-claim break needs no such ;; wait, because its previous owner is already gone. (define (forced result) (if (eq? result 'acquired) 'acquired-by-force result)) ;; Owner is gone: remove the claim and try once to take it. Deliberately ;; ONE retry -- a loop here would be the retry-against-a-dead-peer shape ;; that caused the send storm. (define (break-stale-claim! path owner me) (log-info "Breaking stale Telegram poller claim (owner gone)" owner-pid: owner) ;; Take the claim from `owner`. Used both for a stale claim (owner dead) ;; and for a forced takeover (owner alive) -- the mechanics are identical ;; and the difference is only whether we were entitled to, which the ;; caller has already decided. Deliberately ONE attempt: a loop here ;; would be the retry-against-a-peer shape that caused the send storm. (define (take-claim-from! path owner me reason) (log-info "Taking the Telegram poller claim" owner-pid: owner reason: reason) ;; Unlink ONLY while the claim still names the dead pid we just judged. ;; An unconditional delete here is a TOCTOU big enough to drive two ;; pollers through: deciding `owner` is dead involves a proc-comm port (begin (set-car! *held-claim* path) 'acquired) 'held-by-other)))) ;;; Forget a claim without deleting it: another instance forced it away ;;; from us and now owns the file. Deleting here would remove THEIR live ;;; claim and leave nobody polling. (define (abandon-poller-claim!) (: -> void?) (set-car! *held-claim* #f)) ;;; Release the claim held by this process, if any. Safe to call when no ;;; claim is held. Only removes a claim still recording OUR pid, so a ;;; late shutdown cannot delete a claim another instance has since taken.src/courier/config.sglmodified
courier-config-telegram-api-url courier-config-telegram-send-disabled courier-config-telegram-send-delay courier-config-telegram-poll-force courier-config-allowed-senders load-courier-config) ;; Test hook (COURIER_TELEGRAM_SEND_DELAY=<sec>): sleep before the ;; send (latency simulation). 0 = inert (default). (telegram-send-delay default: 0) ;; COURIER_TELEGRAM_POLL_FORCE=1: this session takes the single-poller ;; claim even from a LIVE holder, instead of deferring to it. Default ;; off, so it changes nothing unless deliberately reached for. ;; ;; It exists because the claim decides which session receives David's ;; INBOUND Telegram messages, not merely which one avoids 409s. The ;; periodic re-acquire recovers from a DEAD holder; it can do nothing ;; about a live-but-abandoned session, whose claim is perfectly valid ;; while his "task: ..." messages land where nobody is watching. This ;; makes the answer to that one env var and an /mcp, rather than ;; hunting processes by hand. (telegram-poll-force default: #f) (allowed-senders default: '())) ;; ============================================================ (and v (not (string=? v "")) (not (string=? v "0"))))) (send-delay (let ((v (getenv "COURIER_TELEGRAM_SEND_DELAY"))) (or (and v (string->number v)) 0))) (poll-force (let ((v (getenv "COURIER_TELEGRAM_POLL_FORCE"))) (and v (not (string=? v "")) (not (string=? v "0"))))) (yaml (load-config-file))) (courier-config telegram-token: token telegram-api-url: api-url telegram-send-disabled: send-disabled telegram-send-delay: send-delay telegram-poll-force: poll-force allowed-senders: (parse-allowed-senders yaml)))) ;; ============================================================src/courier/poller.sglmodified
;; case of another instance holding it. (define *claim-recheck-interval* 60) ;; How often the INSTANCE THAT IS POLLING re-confirms it still owns the ;; claim. Holding it once is not the same as holding it now: ;; COURIER_TELEGRAM_POLL_FORCE lets another session take it, and a ;; displaced holder that kept polling would put two consumers back on the ;; token -- the original bug, arriving through the escape hatch. Faster ;; than the re-check above because the cost of being wrong is a live 409 ;; storm, not a delayed takeover. (define *claim-ownership-check-interval* 10) ;; How long a FORCED takeover waits before it starts polling, so the ;; instance it displaced has seen the change and killed its child first. ;; Strictly greater than the check interval above. (define *claim-standdown-wait* 14) ;; Supervisor: wedge detection and restart backoff (define *poller-stale-threshold* 30) ;; seconds of child silence (define *watchdog-interval* 10) ;; seconds between checks ;; so relying on the child alone can leak a blocked orphan. (define *active-poller* (list #f)) ;; Set while this instance is polling; cleared when it stops. (define *polling* (list #f)) ;; Set when an ownership check finds the claim is no longer ours, so the ;; supervise loop stops respawning instead of treating the killed child ;; as a wedge to recover from. (define *claim-lost* (list #f)) ;;; Kill the active poller child, if any. Called by the leader ;;; when the MCP session ends. (define (telegram-poller-shutdown!) (: -> void?) (set-car! *polling* #f) (let ((st (car *active-poller*))) (when st (set-poller-state-running?! st #f) ;; Give up the single-poller claim so the next session can poll ;; without waiting for a staleness check. Called unconditionally: an ;; instance that claimed and then failed to spawn still owns the claim. ;; release-poller-claim! re-checks ownership before unlinking, so this ;; cannot delete a claim another instance has since forced away. (release-poller-claim!)) ;; ============================================================ (: mcp-server? courier-config? -> void?) (channel-wait-initialized! server) (let ((token (courier-config-telegram-token config))) (let loop ((announced #f)) (let ((result (acquire-poller-claim! token))) ;; force? applies to the FIRST attempt only. Two forcing instances ;; would otherwise steal the claim from each other every re-check, ;; forever. "This session takes the claim on startup" is the whole ;; intent of the flag. (let loop ((announced #f) (force? (courier-config-telegram-poll-force config))) (let ((result (acquire-poller-claim! token force?))) (case result ((acquired) ((acquired acquired-by-force) (when (eq? result 'acquired-by-force) ;; The displaced holder checks its ownership every ;; *claim-ownership-check-interval*; give it more than one ;; full interval to see the change and kill its child. Two ;; pollers for ten seconds is a small 409 storm, and the ;; whole point of this module is that there is never one. (log-info "Waiting for the displaced poller to stand down" seconds: *claim-standdown-wait*) (sleep *claim-standdown-wait*)) (when announced (log-info "Telegram poller claim acquired -- the previous holder is gone")) ;; Never returns: supervises and respawns the child forever. (poller-supervise-loop server)) ;; Returns only if another instance forces the claim away. (poller-supervise-loop server token)) (else (unless (eq? announced result) (if (eq? result 'held-by-other) (log-warn "Telegram polling skipped -- could not establish poller claim ownership" recheck-seconds: *claim-recheck-interval*))) (sleep *claim-recheck-interval*) (loop result))))))) (define (poller-supervise-loop server) (loop result #f))))))) ;; Watch the claim while we poll. If it is no longer ours, stop: another ;; session forced it and is polling now, so continuing would put two ;; consumers on the token. Killing the child unblocks poller-event-loop, ;; and *claim-lost* tells the supervise loop this was a handover rather ;; than a wedge to recover from. (define (poller-claim-guard token) (let loop () (sleep *claim-ownership-check-interval*) (when (car *polling*) (if (poller-claim-held? token) (loop) (begin (set-car! *claim-lost* #t) ;; Do NOT release: the file now names the instance that took ;; it, and unlinking would leave nobody polling at all. (abandon-poller-claim!) (log-info "Telegram poller claim taken by another instance -- stopping this poller") (let ((st (car *active-poller*))) (when st (set-poller-state-running?! st #f) (poller-kill! st)))))))) (define (poller-supervise-loop server token) (log-info "Telegram polling started (isolated poller process)") (set-car! *claim-lost* #f) (set-car! *polling* #t) (go (poller-claim-guard token)) (let respawn ((delay *restart-delay-initial*)) (let ((started (current-second)) (proc (spawn-poller-child!))) (poller-kill! st) (guard (e (else #f)) (process-wait proc)) (let* ((uptime (- (current-second) started)) (next (poller-next-restart-delay delay uptime))) (log-warn "Telegram poller exited -- restarting" uptime: (format-elapsed uptime) delay: (format-elapsed next)) (sleep next) (respawn next))))))))) (if (car *claim-lost*) ;; Handover, not a failure: fall back to being a normal ;; non-polling instance, which will re-check on the usual ;; interval in case the new holder goes away. (begin (set-car! *polling* #f) (log-info "Telegram polling stopped -- another instance holds the claim now")) (let* ((uptime (- (current-second) started)) (next (poller-next-restart-delay delay uptime))) (log-warn "Telegram poller exited -- restarting" uptime: (format-elapsed uptime) delay: (format-elapsed next)) (sleep next) (respawn next))))))))))test/test-claim.sglmodified
(courier dedup) (courier claim));; Every existing test predates the force? argument and means "the normal,;; unforced path", so bind that spelling once rather than threading #f through;; twenty call sites.(define (acquire-poller-claim/1 token) (acquire-poller-claim! token #f))(define (fresh-dir) (let ((d (make-temp-directory))) (setenv! "COURIER_RELAY_DIR" (path-join d "relays"))(test-group "acquire-poller-claim!" (test "a first caller acquires" (fresh-dir) (assert-equal 'acquired (acquire-poller-claim! "tok-a")) (assert-equal 'acquired (acquire-poller-claim/1 "tok-a")) (release-poller-claim!)) (test "records OUR pid in the claim" (fresh-dir) (acquire-poller-claim! "tok-b") (acquire-poller-claim/1 "tok-b") (assert-equal (number->string (process-id)) (read-claim-owner (poller-claim-path "tok-b"))) (release-poller-claim!)) (test "release removes the claim" (fresh-dir) (acquire-poller-claim! "tok-c") (acquire-poller-claim/1 "tok-c") (release-poller-claim!) (assert-equal #f (read-claim-owner (poller-claim-path "tok-c")))) (ensure-directory (path-dirname path)) (make-symlink "999999" path) (assert-equal "999999" (read-claim-owner path)) (assert-equal 'acquired (acquire-poller-claim! "tok-stale")) (assert-equal 'acquired (acquire-poller-claim/1 "tok-stale")) (assert-equal (number->string (process-id)) (read-claim-owner path)) (release-poller-claim!)))) (let ((path (poller-claim-path "tok-recycled"))) (ensure-directory (path-dirname path)) (make-symlink "1" path) ;; pid 1 is alive and is definitely not us (assert-equal 'acquired (acquire-poller-claim! "tok-recycled")) (assert-equal 'acquired (acquire-poller-claim/1 "tok-recycled")) (assert-equal (number->string (process-id)) (read-claim-owner path)) (release-poller-claim!)))) (test "does not release a claim that now belongs to someone else" (let ((d (fresh-dir))) (let ((path (poller-claim-path "tok-steal"))) (acquire-poller-claim! "tok-steal") (acquire-poller-claim/1 "tok-steal") ;; Someone else takes over (as would happen if we were SIGKILLed and ;; a later instance broke our stale claim). (delete-file path) (test "is deterministic and differs per input" (assert-equal (fnv1a-hex "abc") (fnv1a-hex "abc")) (assert-true (not (string=? (fnv1a-hex "abc") (fnv1a-hex "abd"))))));; ============================================================;; COURIER_TELEGRAM_POLL_FORCE -- the escape hatch;; ============================================================;;;; The claim decides which session receives inbound Telegram messages, not;; merely which one avoids 409s. The periodic re-acquire recovers from a DEAD;; holder; it can do nothing about a live-but-abandoned session whose claim is;; perfectly valid. Forcing is how a specific session says "me, now".(test-group "forced takeover" ;; SCOPE, stated plainly because the obvious test here is vacuous: the ;; force branch is only reached when the owner reads as 'alive, and 'alive ;; requires a live pid whose comm matches OURS -- i.e. a second courier, ;; which does not exist inside a unit test. A claim owned by pid 1 looks ;; like a recycled pid and takes the ORDINARY stale path whether or not ;; force? is set, so asserting "forcing works" against it would pass ;; without ever executing the force branch. (It did, on the first draft.) ;; ;; The live-owner takeover is therefore proved end-to-end in ;; repro/repro-single-poller.sh case 9, with two real couriers. What is ;; worth pinning HERE is that force? does not change any other outcome. (test "force? does not change the stale-claim path (still plain 'acquired)" (let ((d (fresh-dir))) (let ((path (poller-claim-path "tok-forced-stale"))) (ensure-directory (path-dirname path)) (make-symlink "999999" path) (assert-equal 'acquired (acquire-poller-claim! "tok-forced-stale" #t)) (assert-equal (number->string (process-id)) (read-claim-owner path)) (release-poller-claim!)))) (test "force? does not change an uncontended acquire" (let ((d (fresh-dir))) (assert-equal 'acquired (acquire-poller-claim! "tok-forced-free" #t)) (release-poller-claim!))) ;; The displaced holder must be able to SEE that it lost the claim -- ;; without this it would keep polling and there would be two consumers on ;; the token again, which is the original bug arriving through the hatch. (test "poller-claim-held? reports true only while we own the claim" (let ((d (fresh-dir))) (assert-equal 'acquired (acquire-poller-claim/1 "tok-held")) (assert-true (poller-claim-held? "tok-held")) ;; Someone forces it away. (let ((path (poller-claim-path "tok-held"))) (delete-file path) (make-symlink "1" path)) (assert-true (not (poller-claim-held? "tok-held"))) (abandon-poller-claim!))) ;; Abandoning must not delete the new owner's claim, or the forced takeover ;; would leave NOBODY polling. (test "abandoning leaves the new owner's claim intact" (let ((d (fresh-dir))) (acquire-poller-claim/1 "tok-abandon") (let ((path (poller-claim-path "tok-abandon"))) (delete-file path) (make-symlink "1" path) (abandon-poller-claim!) (release-poller-claim!) ;; must be a no-op now (assert-equal "1" (read-claim-owner path))))) (test "poller-claim-held? is false when there is no claim at all" (let ((d (fresh-dir))) (assert-true (not (poller-claim-held? "tok-absent"))))))