Commit903687c7Recorded4 Aug 2026Repositorycourier

Single poller per Telegram token, and an honest return on a dropped send

Message

Three defects in the same neighbourhood, all of the same kind: courier reporting that something happened which did not.

  1. Every leader-mode courier started its own Telegram poller. Telegram permits exactly ONE getUpdates consumer per bot token, so N concurrent sessions meant N pollers and a continuous 409 stream (three measured concurrently on 2026-08-04, one parented by a session alive since Jul 23).
   New (courier claim): a filesystem claim scoped to a hash of the token and
   to the relay dir. The claim is a SYMLINK whose target is the owner's pid,
   so it is created atomically with its owner already recorded -- a
   mkdir-plus-pidfile lock has a window where a SIGKILLed holder leaves an
   ownerless claim nobody can ever attribute or safely break. sigil's
   make-symlink swallows EEXIST (measured), so the win/lose verdict is read
   back from the link rather than taken from the call.
   Stale claims are breakable: pid-liveness answers 'alive, 'dead or
   'undetermined, never conflating "the owner is gone" with "I could not
   ask". Each tier carries its own positive control, and a live pid whose
   comm does not match ours is treated as a recycled pid, not a live owner.
   Losing the claim is normal and logged at info. The instance runs on as a
   full MCP server: tools, relays, sends, notifications. No retry, no exit.
  1. The dry-run path returned "Message sent." byte-for-byte. Every gagged send from 2026-07-06 to 2026-08-04 reported delivery it had not earned, and mimicked the silent-drop bug it was masking so exactly that production observation of that bug was impossible while the gag was set.
   It now returns a string that cannot be read as success, and does not
   raise (an MCP error invites the retry loop behind the send storm). It
   also RELEASES the dedup key, because a dry-run is a definite
   non-delivery: without that, the next real send of the same text hit the
   suppress path and reported success while delivering nothing. Measured on
   the pre-fix binary: gag on, gag off, same text -> 0 deliveries,
   "Message sent.". Fixed: 1 delivery.
   The suppress path now says which of the two things happened rather than
   reusing the plain success string.
  1. Concurrent leader instances shared one courier.log. Append is emulated (read, truncate, write back), so two instances zero-filled each other's file; the resulting NUL bytes made grep report zero matches on a pattern tail showed plainly, mid-incident. Leader mode now writes courier-<pid>.log, keeping the ten newest. The --log path is deliberately untouched: claude-ops passes an already-unique per-task path there and reads it back by name.

Verification is behavioural throughout. repro/repro-single-poller.sh runs two real couriers against the mock; mock_telegram.py now models Telegram's single-consumer 409, because the previous mock could not produce the error the fix removes and "zero 409s" would have passed for free. The pre-fix binary produces 4 conflicts and 2 poller children in the same window where the fixed binary produces 0 and 1. Run against the pre-fix binary the harness turns 17 assertions red.

The storm repro keeps 1 send -> 1 delivery and gains an always-on positive control: the same binary with only the dedup WINDOW neutered delivers 4, proving the counter can see duplicates. A pre-dedup control binary cannot be built on the current toolchain (SGB v10 bundle, v9 runtime) and reported 0 deliveries because it never started -- which would have read as a result.

Suite 105 passed / 0 failed (baseline 82/0).

Changed
 repro/mock_telegram.py          | 106 ++++++++++++++++++++++++---
 repro/repro-send-duplication.sh |  23 ++++++
 repro/repro-single-poller.sh    |  30 ++++++++
 repro/single_poller_harness.py  | 630 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/courier/claim.sgl           | 278 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/courier/dedup.sgl           |  22 ++++--
 src/courier/main.sgl            |  47 +++++++++++-
 src/courier/poller.sgl          |  22 +++++-
 src/courier/telegram.sgl        |  42 +++++++++--
 test/test-claim.sgl             | 185 +++++++++++++++++++++++++++++++++++++++++++++++
 10 files changed, 1361 insertions(+), 24 deletions(-)
Diff
repro/mock_telegram.pymodified
@@ -17,7 +17,37 @@ let the driver seed and inject updates:
17
18
POST /_seed body: {"updates":[{...},...]} replace the backlog
19
POST /_add body: {"text":"...","sender_id":"..","chat_id":".."} append one update
−20
GET /_stats {"delivered":[ids], "sends":N}
+20
POST /_dwell body: {"seconds":N} long-poll dwell (see below)
+21
GET /_stats {"delivered":[ids], "sends":N,
+22
"conflicts":N, ...}
+23
+24
The SINGLE-CONSUMER rule (409 Conflict) is modelled too, because the
+25
"exactly one poller per token" fix cannot be verified against a mock that
+26
is incapable of producing the error it is supposed to eliminate -- "zero
+27
409s" would then pass for free, on a mock, with the bug fully present.
+28
+29
Telegram permits ONE getUpdates consumer per bot token: a getUpdates that
+30
arrives while another is already in flight is answered
+31
409 {"ok":false,"error_code":409,
+32
"description":"Conflict: terminated by other getUpdates request..."}
+33
and that is the error stream observed live on 2026-08-04 with three
+34
concurrent couriers.
+35
+36
`dwell` is what makes this observable off-device. Real Telegram HOLDS an
+37
empty getUpdates open for the caller's `timeout` seconds, which is what
+38
makes two independent pollers overlap almost continuously. courier polls
+39
with timeout=0, so a mock that answers instantly leaves an in-flight
+40
window of microseconds and two pollers would collide only by luck. Setting
+41
a dwell (e.g. 2.5s, longer than courier's 2s poll interval) restores the
+42
overlap deterministically:
+43
+44
* TWO pollers -> the second one's call always lands inside the first
+45
one's dwell -> conflicts climb.
+46
* ONE poller -> its calls are strictly sequential (poll, wait, poll),
+47
so it can never overlap ITSELF -> conflicts stay 0.
+48
+49
Dwell defaults to 0 (env GETUPDATES_DWELL), so every pre-existing repro
+50
keeps its original timing and behaviour.
51
52
The sentinel-zero (2026-07-02) storm case is exercised by
53
`repro-sentinel-zero.sh`: `_seed []` gives an EMPTY backlog so a cold start
@@ -71,6 +101,11 @@ STATE = {
101
"sends": 0, # count of sendMessage calls
102
"delivered_ids": [], # update_ids the mock RETURNED to a poller (per call)
103
"send_mode": SEND_MODE_DEFAULT,
+104
# --- single-consumer (409) modelling; see module docstring ---
+105
"inflight": 0, # getUpdates calls currently being served
+106
"conflicts": 0, # getUpdates calls rejected with 409
+107
"max_inflight": 0, # high-water mark, i.e. observed concurrent pollers
+108
"dwell": float(os.environ.get("GETUPDATES_DWELL", "0")),
109
}
110
111
@@ -125,6 +160,9 @@ class Handler(BaseHTTPRequestHandler):
160
self._reply({"delivered": STATE["delivered_ids"],
161
"sends": STATE["sends"],
162
"getupdates": STATE["getupdates"],
+163
"conflicts": STATE["conflicts"],
+164
"max_inflight": STATE["max_inflight"],
+165
"dwell": STATE["dwell"],
166
"pending": [u["update_id"] for u in STATE["updates"]]})
167
return
168
self._reply({"ok": False, "description": "not found"}, 404)
@@ -142,8 +180,17 @@ class Handler(BaseHTTPRequestHandler):
180
STATE["getupdates"] = 0
181
STATE["sends"] = 0
182
STATE["delivered_ids"] = []
+183
STATE["conflicts"] = 0
+184
STATE["max_inflight"] = 0
185
self._reply({"ok": True})
186
return
+187
if path == "/_dwell":
+188
with LOCK:
+189
STATE["dwell"] = float(body.get("seconds", 0))
+190
d = STATE["dwell"]
+191
logline(REQ_LOG, f"--- getUpdates dwell set to {d}s ---")
+192
self._reply({"ok": True, "dwell": d})
+193
return
194
if path == "/_add":
195
with LOCK:
196
uid = STATE["next_id"]
@@ -169,16 +216,53 @@ class Handler(BaseHTTPRequestHandler):
216
offset = body.get("offset", 0) or 0
217
with LOCK:
218
STATE["getupdates"] += 1
−172
if offset > 0:
−173
# Confirm: drop everything below the offset.
−174
STATE["updates"] = [u for u in STATE["updates"]
−175
if u["update_id"] >= offset]
−176
result = list(STATE["updates"])
−177
ids = [u["update_id"] for u in result]
−178
STATE["delivered_ids"].append({"offset": offset, "returned": ids})
−179
logline(REQ_LOG,
−180
f"getUpdates offset={offset} -> returned={ids}")
−181
self._reply({"ok": True, "result": result})
+219
# Single-consumer rule: a getUpdates arriving while another
+220
# is in flight is a second poller on this token. Real
+221
# Telegram terminates one of them with 409; we reject the
+222
# newcomer, which is equivalent for counting purposes.
+223
if STATE["inflight"] > 0:
+224
STATE["conflicts"] += 1
+225
STATE["max_inflight"] = max(STATE["max_inflight"],
+226
STATE["inflight"] + 1)
+227
n = STATE["conflicts"]
+228
conflict = True
+229
else:
+230
STATE["inflight"] += 1
+231
STATE["max_inflight"] = max(STATE["max_inflight"],
+232
STATE["inflight"])
+233
conflict = False
+234
if conflict:
+235
logline(REQ_LOG, f"getUpdates offset={offset} -> 409 CONFLICT (#{n})")
+236
self._reply({"ok": False, "error_code": 409,
+237
"description": "Conflict: terminated by other "
+238
"getUpdates request; make sure that "
+239
"only one bot instance is running"},
+240
code=409)
+241
return
+242
try:
+243
with LOCK:
+244
if offset > 0:
+245
# Confirm: drop everything below the offset.
+246
STATE["updates"] = [u for u in STATE["updates"]
+247
if u["update_id"] >= offset]
+248
result = list(STATE["updates"])
+249
ids = [u["update_id"] for u in result]
+250
STATE["delivered_ids"].append({"offset": offset,
+251
"returned": ids})
+252
dwell = STATE["dwell"]
+253
# Long-poll emulation: hold an EMPTY response open, exactly
+254
# as Telegram holds the caller's `timeout`. This is what makes
+255
# two independent pollers overlap observably; see the module
+256
# docstring. A non-empty result returns immediately, as
+257
# Telegram does.
+258
if dwell > 0 and not result:
+259
time.sleep(dwell)
+260
logline(REQ_LOG,
+261
f"getUpdates offset={offset} -> returned={ids}")
+262
self._reply({"ok": True, "result": result})
+263
finally:
+264
with LOCK:
+265
STATE["inflight"] -= 1
266
return
267
268
if method == "sendMessage":
repro/repro-send-duplication.shmodified
@@ -44,9 +44,12 @@ start_mock() {
44
stop_mock() { kill "$MOCK" 2>/dev/null; wait "$MOCK" 2>/dev/null; sleep 0.2; }
45
46
# run_case <label> <bin> <expected-deliveries> <driver-args...>
+47
# Honours DEDUP_WINDOW: when set, exports COURIER_SEND_DEDUP_WINDOW for the
+48
# driver's couriers. Used by the positive control below.
49
run_case() {
50
local label="$1" bin="$2" expect="$3"; shift 3
51
start_mock
+52
COURIER_SEND_DEDUP_WINDOW="${DEDUP_WINDOW:-}" \
53
"$PY" "$HERE/send_driver.py" --bin "$bin" \
54
--api-url "http://127.0.0.1:$PORT" \
55
--stats-url "http://127.0.0.1:$PORT/_stats" \
@@ -69,6 +72,26 @@ run_case "reset (ack lost) -> 1" "$BIN" 1 --mode reset --tool-timeout 30
72
run_case "STORM x3 -> 1 (exactly-once)" "$BIN" 1 --mode delay:8 --tool-timeout 3 --retries 3 --text storm3
73
run_case "STORM x5 -> 1 (exactly-once)" "$BIN" 1 --mode delay:8 --tool-timeout 2 --retries 5 --text storm5
74
+75
# ---------------------------------------------------------------------------
+76
# POSITIVE CONTROL (always runs).
+77
#
+78
# The four cases above are all assertions that something did NOT happen --
+79
# no duplicate delivery -- and an absence proves nothing until the instrument
+80
# has been shown to detect the thing when it IS present. A count of 1 is
+81
# returned just as cheerfully by a courier that never started as by a working
+82
# one. (Measured 2026-08-04: a control binary built from the pre-dedup commit
+83
# reported 0 deliveries because it could not run at all -- SGB v10 bundle
+84
# against a v9 runtime -- and 0 would have read as a very convincing result.)
+85
#
+86
# So neuter an INPUT and nothing else: keep the same binary, the same driver,
+87
# the same scoring, and shrink only the dedup WINDOW so the recorded key
+88
# expires between storm attempts. The suppression should disappear and the
+89
# same run should deliver once per attempt.
+90
# ---------------------------------------------------------------------------
+91
echo "=== POSITIVE CONTROL: same binary, dedup window neutered to 1s ==="
+92
DEDUP_WINDOW=1 run_case "window=1s STORM x3 -> 4 (counter CAN see duplicates)" \
+93
"$BIN" 4 --mode delay:8 --tool-timeout 3 --retries 3 --text storm-window-control
+94
95
if [ -n "${COURIER_CONTROL_BIN:-}" ] && [ -x "${COURIER_CONTROL_BIN}" ]; then
96
echo "=== CONTROL (pre-fix) binary: $COURIER_CONTROL_BIN ==="
97
# The pre-fix binary re-delivers once per attempt: 1 logical send + 3
repro/repro-single-poller.shadded
@@ -0,0 +1,30 @@
+1
#!/usr/bin/env bash
+2
# Entry point for the single-poller + honest-send harness.
+3
#
+4
# Everything runs against repro/mock_telegram.py. NOTHING reaches real
+5
# Telegram: every courier started here has COURIER_TELEGRAM_API_URL pointed
+6
# at the local mock, which both the send and the poll path honour.
+7
#
+8
# The pre-fix CONTROL is not optional in spirit: case 1 asserts ZERO 409s,
+9
# and an absence assertion is worth nothing until the instrument has been
+10
# shown to detect the thing when it IS present. Build one with:
+11
#
+12
# git worktree add /tmp/courier-control <pre-fix-sha>
+13
# (cd /tmp/courier-control && sigil deps install && sigil build)
+14
# COURIER_CONTROL_BIN=/tmp/courier-control/build/dev/bin/courier \
+15
# repro/repro-single-poller.sh
+16
#
+17
# Without it the harness reports the control as FAILED rather than quietly
+18
# passing, because "we did not check" must not read as "it is fine".
+19
set -u
+20
+21
HERE="$(cd "$(dirname "$0")" && pwd)"
+22
BIN="${COURIER_BIN:-$HERE/../build/dev/bin/courier}"
+23
PY="${PYTHON:-python3}"
+24
+25
if [ ! -x "$BIN" ]; then
+26
echo "FATAL: courier binary not found at $BIN (run 'sigil build' first)"
+27
exit 2
+28
fi
+29
+30
exec "$PY" "$HERE/single_poller_harness.py" --bin "$BIN" "$@"
repro/single_poller_harness.pyadded
@@ -0,0 +1,630 @@
+1
#!/usr/bin/env python3
+2
"""Off-device harness for the SINGLE-POLLER claim and the HONEST-SEND return.
+3
+4
Everything here runs against repro/mock_telegram.py. NOTHING can reach real
+5
Telegram: every courier instance is started with COURIER_TELEGRAM_API_URL
+6
pointed at the local mock, which is the only endpoint the send AND poll paths
+7
use once that variable is set.
+8
+9
What it proves, in the order the cases run:
+10
+11
1 ONE POLLER two leader-mode couriers on one token -> exactly one
+12
poller child process, and zero 409s at the mock.
+13
2 409 DETECTOR WORKS the same two-instance run with the PRE-FIX binary
+14
(COURIER_CONTROL_BIN) -> 409s appear. This is the
+15
positive control: without it, case 1's "zero 409s"
+16
is a claim of absence made by an instrument never
+17
shown capable of detecting presence.
+18
3 NON-POLLER IS FINE the instance that lost the claim still serves its
+19
MCP tools -- create-relay / list-relays / close-relay.
+20
4 SIGKILL RECOVERY SIGKILL (not a graceful stop) the claim holder, start
+21
a fresh instance: it detects the dead owner, breaks
+22
the stale claim, and polls.
+23
5 DRY-RUN IS HONEST with COURIER_DISABLE_TELEGRAM_SEND=1 the tool returns
+24
a string that cannot be read as success, does not
+25
raise, and delivers nothing at the mock.
+26
6 GAG DOESN'T POISON a dry-run must not leave a dedup key behind that
+27
would suppress a LATER real send of the same text.
+28
Dry-run, then flag off, then the same message must
+29
actually deliver.
+30
7 LOG HYGIENE two concurrent leader instances write two distinct
+31
log files, and grep works on both (the shared-log
+32
interleaving produced NUL bytes, which made grep
+33
silently report zero matches during an incident).
+34
+35
Usage:
+36
single_poller_harness.py --bin <courier> [--control-bin <pre-fix courier>]
+37
"""
+38
import argparse
+39
import json
+40
import os
+41
import re
+42
import shutil
+43
import signal
+44
import subprocess
+45
import sys
+46
import threading
+47
import time
+48
import urllib.error
+49
import urllib.request
+50
+51
CHAT_ID = "331005009"
+52
PORT = int(os.environ.get("PORT", "8613"))
+53
BASE = f"http://127.0.0.1:{PORT}"
+54
+55
# Longer than courier's 2s poll interval, so two independent pollers overlap
+56
# deterministically rather than by luck. See mock_telegram.py's docstring.
+57
DWELL = 2.5
+58
+59
# How long to watch a two-instance run. Must be long enough that the pre-fix
+60
# control reliably produces 409s inside it, or "zero 409s" from the fixed
+61
# binary would only mean "we did not watch for long enough".
+62
OBSERVE_WINDOW = 20
+63
+64
FAILED = []
+65
PASSED = []
+66
+67
+68
def check(label, ok, detail=""):
+69
(PASSED if ok else FAILED).append(label)
+70
print(f" {'PASS' if ok else 'FAIL'} {label}" + (f": {detail}" if detail else ""))
+71
return ok
+72
+73
+74
# ----------------------------------------------------------------- mock ----
+75
+76
def post(path, obj):
+77
data = json.dumps(obj).encode()
+78
req = urllib.request.Request(BASE + path, data=data,
+79
headers={"Content-Type": "application/json"})
+80
with urllib.request.urlopen(req, timeout=10) as r:
+81
return json.load(r)
+82
+83
+84
def stats():
+85
with urllib.request.urlopen(BASE + "/_stats", timeout=10) as r:
+86
return json.load(r)
+87
+88
+89
class Mock:
+90
def __init__(self, logdir):
+91
self.logdir = logdir
+92
here = os.path.dirname(os.path.abspath(__file__))
+93
os.makedirs(logdir, exist_ok=True)
+94
self.proc = subprocess.Popen(
+95
[sys.executable, os.path.join(here, "mock_telegram.py"),
+96
str(PORT), os.path.join(logdir, "mocklog")],
+97
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+98
for _ in range(50):
+99
try:
+100
stats()
+101
return
+102
except Exception:
+103
time.sleep(0.1)
+104
raise RuntimeError("mock did not come up")
+105
+106
def stop(self):
+107
self.proc.terminate()
+108
try:
+109
self.proc.wait(timeout=5)
+110
except Exception:
+111
self.proc.kill()
+112
+113
+114
# -------------------------------------------------------------- courier ----
+115
+116
class Courier:
+117
"""One courier subprocess speaking MCP over stdio."""
+118
+119
def __init__(self, bin_path, run_dir, name, disable_send=False,
+120
extra_env=None, argv=("serve",)):
+121
self.name = name
+122
self.run_dir = run_dir
+123
self.notifications = []
+124
os.makedirs(run_dir, exist_ok=True)
+125
env = dict(os.environ)
+126
env["COURIER_TELEGRAM_TOKEN"] = "TESTTOKEN-single-poller"
+127
env["COURIER_TELEGRAM_CHAT_ID"] = CHAT_ID
+128
env["COURIER_TELEGRAM_API_URL"] = BASE
+129
# Shared state dir: this is what makes the instances contend, exactly
+130
# as concurrent leader sessions share ~/.courier.
+131
env["COURIER_RELAY_DIR"] = os.path.join(run_dir, "relays")
+132
if disable_send:
+133
env["COURIER_DISABLE_TELEGRAM_SEND"] = "1"
+134
else:
+135
env.pop("COURIER_DISABLE_TELEGRAM_SEND", None)
+136
if extra_env:
+137
env.update(extra_env)
+138
self.errf = open(os.path.join(run_dir, f"{name}.stderr"), "wb")
+139
# NOTE: no --log. That is deliberate -- it exercises the leader-mode
+140
# persistent-log path (configure-persistent-log!), which is the one
+141
# that used to collide on a single shared courier.log.
+142
self.proc = subprocess.Popen(
+143
[bin_path, *argv],
+144
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=self.errf,
+145
env=env, bufsize=0)
+146
self._id = 0
+147
self._pending = {}
+148
self._lock = threading.Lock()
+149
self._reader = threading.Thread(target=self._read_loop, daemon=True)
+150
self._reader.start()
+151
+152
# --- MCP plumbing ---
+153
def _read_loop(self):
+154
for raw in self.proc.stdout:
+155
try:
+156
msg = json.loads(raw)
+157
except Exception:
+158
continue
+159
rid = msg.get("id")
+160
if rid is not None:
+161
with self._lock:
+162
ev = self._pending.get(rid)
+163
if ev:
+164
ev[1] = msg
+165
ev[0].set()
+166
else:
+167
# Server-initiated notification (channel events: relay and
+168
# Telegram messages arrive this way).
+169
with self._lock:
+170
self.notifications.append(msg)
+171
+172
def _send(self, obj):
+173
self.proc.stdin.write((json.dumps(obj) + "\n").encode())
+174
self.proc.stdin.flush()
+175
+176
def rpc(self, method, params, timeout=20):
+177
with self._lock:
+178
self._id += 1
+179
rid = self._id
+180
ev = [threading.Event(), None]
+181
self._pending[rid] = ev
+182
self._send({"jsonrpc": "2.0", "id": rid, "method": method,
+183
"params": params})
+184
if not ev[0].wait(timeout):
+185
raise TimeoutError(f"{self.name}: no response to {method}")
+186
return ev[1]
+187
+188
def initialize(self):
+189
res = self.rpc("initialize", {
+190
"protocolVersion": "2024-11-05", "capabilities": {},
+191
"clientInfo": {"name": "single-poller-harness", "version": "0"}})
+192
self._send({"jsonrpc": "2.0", "method": "notifications/initialized",
+193
"params": {}})
+194
return res
+195
+196
def call(self, tool, arguments, timeout=30):
+197
res = self.rpc("tools/call", {"name": tool, "arguments": arguments},
+198
timeout=timeout)
+199
if "result" in res:
+200
try:
+201
return res["result"]["content"][0]["text"]
+202
except Exception:
+203
return json.dumps(res["result"])
+204
return "ERROR:" + json.dumps(res.get("error", res))
+205
+206
# --- lifecycle ---
+207
@property
+208
def pid(self):
+209
return self.proc.pid
+210
+211
def alive(self):
+212
return self.proc.poll() is None
+213
+214
def sigkill(self):
+215
"""SIGKILL, as a /mcp teardown does. No chance to release a claim."""
+216
os.kill(self.proc.pid, signal.SIGKILL)
+217
self.proc.wait(timeout=10)
+218
+219
def close_stdin(self):
+220
"""Graceful shutdown: stdin EOF is how a session ends."""
+221
try:
+222
self.proc.stdin.close()
+223
except Exception:
+224
pass
+225
+226
def kill(self):
+227
try:
+228
if self.alive():
+229
self.proc.kill()
+230
self.proc.wait(timeout=5)
+231
except Exception:
+232
pass
+233
try:
+234
self.errf.close()
+235
except Exception:
+236
pass
+237
+238
def stderr_text(self):
+239
try:
+240
self.errf.flush()
+241
except Exception:
+242
pass
+243
with open(os.path.join(self.run_dir, f"{self.name}.stderr"),
+244
"rb") as f:
+245
return f.read().decode("utf-8", "replace")
+246
+247
def log_path(self):
+248
"""This instance's own log file.
+249
+250
Leader mode with no --log logs to <state-dir>/courier-<pid>.log, NOT
+251
to stderr, so assertions about log lines must read this.
+252
"""
+253
return os.path.join(self.run_dir, f"courier-{self.pid}.log")
+254
+255
def log_text(self):
+256
try:
+257
with open(self.log_path(), "rb") as f:
+258
return f.read().decode("utf-8", "replace")
+259
except FileNotFoundError:
+260
return ""
+261
+262
+263
# ------------------------------------------------------- process helpers ----
+264
+265
def child_pids(pid):
+266
"""Direct children of `pid`, read straight from /proc.
+267
+268
Deliberately not pgrep/pkill: a name-matching pattern here could match
+269
this harness or an unrelated courier on the host, and a missing tool
+270
would return an empty list that reads exactly like 'no children'.
+271
"""
+272
out = []
+273
for entry in os.listdir("/proc"):
+274
if not entry.isdigit():
+275
continue
+276
try:
+277
with open(f"/proc/{entry}/status") as f:
+278
ppid = None
+279
for line in f:
+280
if line.startswith("PPid:"):
+281
ppid = int(line.split()[1])
+282
break
+283
if ppid == pid:
+284
out.append(int(entry))
+285
except (FileNotFoundError, ProcessLookupError, PermissionError):
+286
continue
+287
return out
+288
+289
+290
def poller_children(pid):
+291
"""Children of `pid` that are courier poller processes."""
+292
found = []
+293
for c in child_pids(pid):
+294
try:
+295
with open(f"/proc/{c}/cmdline", "rb") as f:
+296
cmd = f.read().decode("utf-8", "replace")
+297
except (FileNotFoundError, ProcessLookupError, PermissionError):
+298
continue
+299
if "--telegram-poller" in cmd:
+300
found.append(c)
+301
return found
+302
+303
+304
def claim_files(state_dir):
+305
if not os.path.isdir(state_dir):
+306
return []
+307
return sorted(n for n in os.listdir(state_dir)
+308
if n.startswith("telegram-poller-") and n.endswith(".claim"))
+309
+310
+311
def claim_owner(state_dir):
+312
files = claim_files(state_dir)
+313
if not files:
+314
return None
+315
return os.readlink(os.path.join(state_dir, files[0]))
+316
+317
+318
def wait_for(predicate, timeout=30, interval=0.25):
+319
deadline = time.time() + timeout
+320
while time.time() < deadline:
+321
if predicate():
+322
return True
+323
time.sleep(interval)
+324
return False
+325
+326
+327
# ----------------------------------------------------------------- cases ----
+328
+329
def case_one_poller(binp, run_dir, label="1 ONE POLLER"):
+330
"""Two leader-mode couriers, one token. Returns (a, b, state_dir)."""
+331
print(f"=== {label} ===")
+332
post("/_seed", {"updates": []})
+333
post("/_dwell", {"seconds": DWELL})
+334
state_dir = run_dir
+335
+336
a = Courier(binp, run_dir, "inst-a")
+337
a.initialize()
+338
got_a = wait_for(lambda: len(poller_children(a.pid)) == 1, timeout=40)
+339
# POSITIVE CONTROL for the child-detector: if this is False, every later
+340
# "0 pollers" reading below is meaningless.
+341
check("instance A starts a poller child (detector positive control)",
+342
got_a, f"children={poller_children(a.pid)}")
+343
+344
b = Courier(binp, run_dir, "inst-b")
+345
b.initialize()
+346
# Give B every chance to (wrongly) start one, and give the mock a window
+347
# comfortably longer than the one in which the pre-fix control produces
+348
# its first 409.
+349
time.sleep(OBSERVE_WINDOW)
+350
+351
pa, pb = poller_children(a.pid), poller_children(b.pid)
+352
check("exactly one poller child across both instances",
+353
len(pa) + len(pb) == 1, f"A={pa} B={pb}")
+354
check("instance B did NOT start a poller", len(pb) == 0, f"B={pb}")
+355
+356
owner = claim_owner(state_dir)
+357
check("claim file records the polling instance's pid",
+358
owner == str(a.pid), f"claim owner={owner} A={a.pid} B={b.pid}")
+359
+360
log = b.log_text()
+361
skip_lines = [l for l in log.splitlines() if "poller claim" in l]
+362
check("instance B logs the skip, at INFO and not WARN/ERROR",
+363
(len(skip_lines) == 1
+364
and "[INFO]" in skip_lines[0]
+365
and "another courier instance holds" in skip_lines[0]),
+366
repr(skip_lines))
+367
+368
s = stats()
+369
# An absence is only meaningful if the thing had a chance to happen:
+370
# zero 409s across zero polls would pass for free. Assert the window
+371
# actually contained polling before believing the zero.
+372
check("the observation window contained real polling (denominator)",
+373
s["getupdates"] >= 3,
+374
f"getupdates={s['getupdates']} over {OBSERVE_WINDOW}s")
+375
check("zero 409 conflicts at the mock", s["conflicts"] == 0,
+376
f"conflicts={s['conflicts']} getupdates={s['getupdates']} "
+377
f"max_inflight={s['max_inflight']}")
+378
check("the mock never saw two concurrent pollers", s["max_inflight"] <= 1,
+379
f"max_inflight={s['max_inflight']}")
+380
return a, b, state_dir
+381
+382
+383
def case_control(control_bin, run_dir):
+384
"""Positive control: the pre-fix binary MUST produce 409s."""
+385
print("=== 2 409 DETECTOR WORKS (pre-fix control) ===")
+386
if not control_bin:
+387
print(" SKIP no COURIER_CONTROL_BIN given -- case 1's 'zero 409s' "
+388
"is UNCONTROLLED and proves nothing on its own")
+389
FAILED.append("409 positive control not run")
+390
return
+391
post("/_seed", {"updates": []})
+392
post("/_dwell", {"seconds": DWELL})
+393
+394
# Confirm the control really IS the pre-fix code before drawing any
+395
# conclusion from it. A control that silently pointed at the fixed binary
+396
# would report "no 409s" and look like a disproof of the whole bug.
+397
probe = Courier(control_bin, os.path.join(run_dir, "probe"), "ctl-probe",
+398
disable_send=True)
+399
probe.initialize()
+400
old = probe.call("send-message", {"text": "control identity probe",
+401
"to": CHAT_ID})
+402
check("control binary is genuinely PRE-FIX (dry-run still lies)",
+403
old == "Message sent.", repr(old))
+404
probe.kill()
+405
+406
a = Courier(control_bin, run_dir, "ctl-a")
+407
a.initialize()
+408
b = Courier(control_bin, run_dir, "ctl-b")
+409
b.initialize()
+410
t0 = time.time()
+411
ok = wait_for(lambda: stats()["conflicts"] > 0, timeout=60)
+412
first = time.time() - t0
+413
# Keep watching for the SAME window the fixed run got, so the two numbers
+414
# are comparable rather than one being a shorter look than the other.
+415
time.sleep(OBSERVE_WINDOW)
+416
s = stats()
+417
check("pre-fix binary: two instances DO produce 409s", ok,
+418
f"conflicts={s['conflicts']} in {OBSERVE_WINDOW}s "
+419
f"(first after {first:.1f}s) getupdates={s['getupdates']} "
+420
f"max_inflight={s['max_inflight']}")
+421
check("pre-fix binary: two poller children exist",
+422
len(poller_children(a.pid)) + len(poller_children(b.pid)) == 2,
+423
f"A={poller_children(a.pid)} B={poller_children(b.pid)}")
+424
for c in (a, b):
+425
c.kill()
+426
time.sleep(1)
+427
+428
+429
def case_relay_tools(binp, b, run_dir):
+430
"""The instance that LOST the claim must be a fully working leader.
+431
+432
Create a relay, connect a real worker-mode courier to it, send through it,
+433
and close it. This also re-checks that worker mode is untouched: three
+434
live worker relays depend on it.
+435
"""
+436
print("=== 3 NON-POLLER IS FINE (relay tools on the skipped instance) ===")
+437
r = b.call("create-relay", {"name": "harness-relay"})
+438
check("non-polling instance: create-relay works",
+439
r == "Relay 'harness-relay' created and listening.", repr(r))
+440
+441
w = Courier(binp, run_dir, "worker", argv=("--relay", "harness-relay"))
+442
try:
+443
w.initialize()
+444
text = "leader-to-worker over the relay"
+445
connected = wait_for(
+446
lambda: "harness-relay" in b.call("list-relays", {})
+447
and "connected" in b.call("list-relays", {}).lower(), timeout=20)
+448
check("non-polling instance: list-relays shows the connected worker",
+449
connected, repr(b.call("list-relays", {})))
+450
+451
r = b.call("send-message", {"text": text, "to": "harness-relay"})
+452
check("non-polling instance: send-message over the relay works",
+453
r == "Message sent.", repr(r))
+454
got = wait_for(lambda: any(text in json.dumps(n)
+455
for n in list(w.notifications)), timeout=20)
+456
check("the worker actually RECEIVED the relayed message", got,
+457
repr(w.notifications[-1] if w.notifications else None))
+458
+459
back = w.call("send-message", {"text": "worker-to-leader ack"})
+460
check("worker mode: send-message back to the leader works",
+461
back == "Message sent to leader.", repr(back))
+462
finally:
+463
w.kill()
+464
+465
r = b.call("close-relay", {"name": "harness-relay"})
+466
check("non-polling instance: close-relay works",
+467
r == "Relay 'harness-relay' closed.", repr(r))
+468
+469
+470
def case_sigkill(binp, run_dir, a, state_dir):
+471
print("=== 4 SIGKILL RECOVERY ===")
+472
dead_pid = a.pid
+473
a.sigkill()
+474
time.sleep(1)
+475
owner = claim_owner(state_dir)
+476
check("claim SURVIVES the SIGKILL (nothing released it)",
+477
owner == str(dead_pid),
+478
f"owner={owner} killed={dead_pid}")
+479
+480
c = Courier(binp, run_dir, "inst-c")
+481
c.initialize()
+482
got = wait_for(lambda: len(poller_children(c.pid)) == 1, timeout=45)
+483
check("a fresh instance breaks the stale claim and polls", got,
+484
f"children={poller_children(c.pid)}")
+485
check("claim now records the new instance",
+486
claim_owner(state_dir) == str(c.pid),
+487
f"owner={claim_owner(state_dir)} C={c.pid}")
+488
log = c.log_text()
+489
check("the break is logged", "Breaking stale" in log,
+490
repr([l for l in log.splitlines() if "stale" in l][:2]))
+491
return c
+492
+493
+494
EXPECTED_DRY_RUN = ("Message NOT sent: COURIER_DISABLE_TELEGRAM_SEND is set "
+495
"(dry-run, nothing delivered).")
+496
+497
+498
def case_dry_run(binp, run_dir):
+499
print("=== 5 DRY-RUN IS HONEST ===")

Showing the first 500 of 631 diff lines for this file. This diff is INCOMPLETE; read the file or clone the repository for the rest.

src/courier/claim.sgladded
@@ -0,0 +1,278 @@
+1
;;; (courier claim) - Single-poller claim for the Telegram getUpdates stream.
+2
;;;
+3
;;; Telegram allows exactly ONE getUpdates consumer per bot token. Every
+4
;;; leader-mode courier used to start its own poller child, so N concurrent
+5
;;; Claude sessions meant N pollers fighting over one token and a continuous
+6
;;; stream of `Conflict: terminated by other getUpdates request (409)`.
+7
;;; Measured 2026-08-04: three pollers at once, one parented by a session
+8
;;; alive since Jul 23.
+9
;;;
+10
;;; This module hands out a cooperative, filesystem-backed claim so exactly
+11
;;; one instance polls. Everything else about a non-polling instance keeps
+12
;;; working: MCP tools, relays, sends, notifications. LOSING THE CLAIM IS THE
+13
;;; NORMAL CASE for the 2nd..Nth session and is logged at info, never warn.
+14
;;;
+15
;;; Why a lockfile rather than configuration: config drifts, gets copied
+16
;;; between machines, and its failure mode is silent -- two machines both
+17
;;; believing they are the designated poller looks exactly like no config at
+18
;;; all. The filesystem claim is derived from the running state instead.
+19
;;;
+20
;;; ---------------------------------------------------------------------
+21
;;; Why a SYMLINK is the claim
+22
;;; ---------------------------------------------------------------------
+23
;;; symlink(2) is atomic against a concurrent creator AND carries a payload
+24
;;; chosen at creation time. Putting the owner's pid in the link TARGET means
+25
;;; there is never a moment where the claim exists but its owner is unknown.
+26
;;; The mkdir-plus-pidfile alternative has exactly that window: a holder
+27
;;; SIGKILLed between the mkdir and the pid write leaves an ownerless claim
+28
;;; that no later instance can ever attribute or safely break -- a permanent
+29
;;; estate-wide deadlock, which is the specific outcome this design is
+30
;;; required to avoid.
+31
;;;
+32
;;; MEASURED QUIRK, load-bearing: sigil's `make-symlink` SWALLOWS EEXIST. It
+33
;;; neither raises nor returns a failure indicator when the path is already
+34
;;; taken (probed 2026-08-04: second call over an existing link returns
+35
;;; normally, and the link still points at the first owner). So the win/lose
+36
;;; decision must NEVER be read from the call itself -- it comes from reading
+37
;;; the link back and checking whose pid is in it. A `guard` around
+38
;;; make-symlink would look like a check and would be inert.
+39
;;;
+40
;;; ---------------------------------------------------------------------
+41
;;; Stale claims
+42
;;; ---------------------------------------------------------------------
+43
;;; A courier killed with SIGKILL cannot release its claim, and SIGKILL is
+44
;;; routine here (an MCP reconnect tears the server down that way). So a claim
+45
;;; whose owner is gone must be breakable, or the first crash would silence
+46
;;; Telegram for the whole estate forever. `pid-liveness` answers that, and it
+47
;;; is careful to distinguish three states, not two:
+48
;;;
+49
;;; 'alive -- the owner is running; do not touch the claim
+50
;;; 'dead -- the owner is gone; the claim may be broken
+51
;;; 'undetermined -- WE COULD NOT ASK. Not the same fact as 'dead, and
+52
;;; conflating them is how a probe silently degrades.
+53
;;;
+54
;;; Each tier carries its own positive control, so a broken instrument
+55
;;; reports 'undetermined instead of a confident wrong answer.
+56
+57
(define-library (courier claim)
+58
(import (sigil core)
+59
(sigil string)
+60
(sigil io)
+61
(sigil fs)
+62
(sigil path)
+63
(sigil time)
+64
(sigil process)
+65
(sigil log)
+66
(courier dedup)
+67
(courier relay))
+68
(export acquire-poller-claim!
+69
release-poller-claim!
+70
poller-claim-path
+71
claim-token-hash
+72
read-claim-owner
+73
pid-liveness
+74
pid-liveness/comm
+75
proc-comm)
+76
(begin
+77
+78
;; After breaking a stale claim, two instances can both have decided the
+79
;; same dead pid was stale and both re-created the link. The link ends up
+80
;; holding exactly one pid; re-reading after a short settle lets the loser
+81
;; notice. Small enough to be invisible at session start, long enough to
+82
;; cover the unlink/symlink pair on any real filesystem.
+83
(define *claim-settle* 0.5)
+84
+85
;; ============================================================
+86
;; Claim path
+87
;; ============================================================
+88
+89
;;; Short, non-reversible tag for a bot token. The token itself is NEVER
+90
;;; written anywhere -- not to the claim, not to a log. 48 bits of a
+91
;;; non-cryptographic FNV-1a hash is deliberately lossy: it is enough to
+92
;;; keep two different bots on one machine from colliding, and far too
+93
;;; little to reconstruct a token from.
+94
(define (claim-token-hash token)
+95
(: string? -> string?)
+96
(let* ((h (fnv1a-hex token))
+97
(n (string-length h))
+98
(padded (if (< n 16)
+99
(string-append (make-string (- 16 n) #\0) h)
+100
h)))
+101
(substring padded 0 12)))
+102
+103
;;; Where the claim for `token` lives. Scoped to the token (two bots on
+104
;;; one machine do not collide) AND to the relay dir (so a test courier
+105
;;; with its own COURIER_RELAY_DIR never contends with the live one).
+106
(define (poller-claim-path token)
+107
(: string? -> string?)
+108
(path-join (path-dirname (default-relay-dir))
+109
(string-append "telegram-poller-"
+110
(claim-token-hash token)
+111
".claim")))
+112
+113
;; ============================================================
+114
;; Liveness
+115
;; ============================================================
+116
+117
;;; First line of /proc/<pid>/comm, or #f if it cannot be read.
+118
;;;
+119
;;; MEASURED: procfs files report st_size 0, so `read-file-string` (which
+120
;;; reads stat's size) returns "" for every one of them -- a confident,
+121
;;; wrong empty answer. A PORT read is size-independent and returns the
+122
;;; real content.
+123
(define (proc-comm pid-str)
+124
(: string? -> any?)
+125
(guard (e (else #f))
+126
(let ((line (call-with-input-file
+127
(string-append "/proc/" pid-str "/comm")
+128
(lambda (port) (read-line port)))))
+129
(and (string? line) (string-trim line)))))
+130
+131
;;; Is the process holding a claim still running?
+132
;;;
+133
;;; Returns 'alive, 'dead, or 'undetermined. See the header:
+134
;;; 'undetermined means the instrument could not answer, which is a
+135
;;; different fact from "the process is gone" and must not be treated as
+136
;;; permission to break the claim.
+137
(define (pid-liveness pid-str)
+138
(: string? -> symbol?)
+139
(pid-liveness/comm pid-str (proc-comm (number->string (process-id)))))
+140
+141
;;; pid-liveness with the expected command name passed in explicitly.
+142
;;;
+143
;;; Split out so the recycled-pid discriminator can be tested from both
+144
;;; sides: a live process whose comm MATCHES must read 'alive, and the
+145
;;; same live process with a non-matching comm must read 'dead. Testing
+146
;;; only the second would be satisfied by an implementation that always
+147
;;; answers 'dead -- which is the pre-fix behaviour wearing a lock's
+148
;;; clothes, and would break every claim it ever saw.
+149
(define (pid-liveness/comm pid-str expected-comm)
+150
(: string? any? -> symbol?)
+151
(let ((self (number->string (process-id))))
+152
(cond
+153
;; Tier 1: /proc. POSITIVE CONTROL -- our own pid must be visible
+154
;; through the instrument before its answer about another pid means
+155
;; anything. Without this, a host with no /proc reports every pid
+156
;; dead and every claim breakable.
+157
((file-exists? (string-append "/proc/" self))
+158
(if (not (file-exists? (string-append "/proc/" pid-str)))
+159
'dead
+160
;; The pid exists -- but pids are recycled, and a recycled pid
+161
;; would make a dead courier's claim look alive forever. Compare
+162
;; the command name against OUR OWN (passed in, read at runtime
+163
;; rather than hardcoded, so it cannot drift if the binary is
+164
;; renamed). If either name is unreadable we cannot rule out a
+165
;; live courier, so we say 'alive and leave the claim alone --
+166
;; the conservative direction.
+167
(let ((theirs (proc-comm pid-str)))
+168
(cond
+169
((or (not expected-comm) (not theirs)) 'alive)
+170
((string=? expected-comm theirs) 'alive)
+171
(else 'dead)))))
+172
;; Tier 2: kill -0, for a host with no /proc. Same shape: prove the
+173
;; instrument works on a pid known to be alive (ours) before
+174
;; believing what it says about another. MEASURED exit codes:
+175
;; 0 = alive, 1 = no such process, 127 = `kill` not found. Reading
+176
;; 127 as "dead" is precisely the always-degrade failure this guards.
+177
((and (command-exists? "kill")
+178
(eqv? 0 (guard (e (else #f)) (process-run "kill" "-0" self))))
+179
(let ((rc (guard (e (else #f)) (process-run "kill" "-0" pid-str))))
+180
(cond
+181
((eqv? rc 0) 'alive)
+182
((eqv? rc 1) 'dead)
+183
(else 'undetermined))))
+184
(else 'undetermined))))
+185
+186
;; ============================================================
+187
;; Acquire / release
+188
;; ============================================================
+189
+190
;; Path of the claim this process currently owns, or #f.
+191
(define *held-claim* (list #f))
+192
+193
;;; Owner pid recorded in the claim, as a string, or #f when the claim is
+194
;;; absent or unreadable. Uses `symlink?` (lstat) rather than
+195
;;; `file-exists?` (stat): a claim link never has a target that exists as
+196
;;; a file, so `file-exists?` reports #f for a perfectly valid claim.
+197
(define (read-claim-owner path)
+198
(: string? -> any?)
+199
(guard (e (else #f))
+200
(and (symlink? path)
+201
(let ((target (read-symlink path)))
+202
(and (string? target) (string-trim target))))))
+203
+204
;; Attempt the atomic create. Returns 'acquired or 'lost. The verdict
+205
;; comes from reading the link back -- see the header note on
+206
;; make-symlink swallowing EEXIST.
+207
(define (try-claim! path me)
+208
(guard (e (else 'lost))
+209
(make-symlink me path)
+210
(if (equal? (read-claim-owner path) me) 'acquired 'lost)))
+211
+212
;;; Try to become the single poller for `token`.
+213
;;;
+214
;;; Returns:
+215
;;; 'acquired -- this process polls, and must call
+216
;;; release-poller-claim! on clean shutdown
+217
;;; 'held-by-other -- another live courier polls. NORMAL. Log at info.
+218
;;; 'undetermined -- liveness could not be established; do not poll,
+219
;;; and do not break the existing claim.
+220
(define (acquire-poller-claim! token)
+221
(: string? -> symbol?)
+222
(let* ((path (poller-claim-path token))
+223
(me (number->string (process-id))))
+224
(guard (e (else
+225
(log-warn "Poller claim failed" error: (format "~a" e))
+226
'undetermined))
+227
(ensure-directory (path-dirname path))
+228
(if (eq? (try-claim! path me) 'acquired)
+229
(begin (set-car! *held-claim* path) 'acquired)
+230
(let ((owner (read-claim-owner path)))
+231
(cond
+232
;; Present but unreadable: leave it alone rather than break
+233
;; something we cannot attribute.
+234
((not owner) 'held-by-other)
+235
;; Our own pid already in the link (a re-entry): keep it.
+236
((string=? owner me)
+237
(set-car! *held-claim* path)
+238
'acquired)
+239
(else
+240
(case (pid-liveness owner)
+241
((alive) 'held-by-other)
+242
((undetermined)
+243
(log-warn "Poller claim owner liveness undetermined"
+244
owner-pid: owner)
+245
'undetermined)
+246
(else (break-stale-claim! path owner me))))))))))
+247
+248
;; Owner is gone: remove the claim and try once to take it. Deliberately
+249
;; ONE retry -- a loop here would be the retry-against-a-dead-peer shape
+250
;; that caused the send storm.
+251
(define (break-stale-claim! path owner me)
+252
(log-info "Breaking stale Telegram poller claim (owner gone)"
+253
owner-pid: owner)
+254
(guard (e (else #f)) (delete-file path))
+255
(if (not (eq? (try-claim! path me) 'acquired))
+256
'held-by-other
+257
(begin
+258
;; Two instances can break the SAME stale claim at the same
+259
;; instant and both create the link; the file then holds exactly
+260
;; one pid. Re-read after a settle so the loser finds out.
+261
(sleep *claim-settle*)
+262
(if (equal? (read-claim-owner path) me)
+263
(begin (set-car! *held-claim* path) 'acquired)
+264
'held-by-other))))
+265
+266
;;; Release the claim held by this process, if any. Safe to call when no
+267
;;; claim is held. Only removes a claim still recording OUR pid, so a
+268
;;; late shutdown cannot delete a claim another instance has since taken.
+269
(define (release-poller-claim!)
+270
(: -> void?)
+271
(let ((path (car *held-claim*)))
+272
(when path
+273
(set-car! *held-claim* #f)
+274
(guard (e (else #f))
+275
(when (equal? (read-claim-owner path)
+276
(number->string (process-id)))
+277
(delete-file path)
+278
(log-info "Released Telegram poller claim"))))))))
src/courier/dedup.sglmodified
@@ -35,6 +35,7 @@
35
send-dedup-window
36
send-dedup-path
37
send-dedup-key
+38
fnv1a-hex
39
dedup-check-and-record!
40
dedup-unrecord!
41
;; exported for tests
@@ -72,13 +73,13 @@
73
(define *fnv-prime* 1099511628211)
74
(define *fnv-mod* 18446744073709551616) ;; 2^64
75
−75
(define (send-dedup-key chat-id text)
−76
(: (any-of integer? string?) string? -> string?)
−77
(let* ((s (string-append (if (string? chat-id)
−78
chat-id
−79
(number->string chat-id))
−80
":" text))
−81
(n (string-length s)))
+76
;;; FNV-1a 64-bit hash of a string, as lowercase hex (no leading-zero
+77
;;; padding). Shared with (courier claim), which tags the poller claim
+78
;;; file with a hash of the bot token so the token itself is never
+79
;;; written to disk.
+80
(define (fnv1a-hex s)
+81
(: string? -> string?)
+82
(let ((n (string-length s)))
83
(let loop ((i 0) (h *fnv-offset*))
84
(if (>= i n)
85
(number->string h 16)
@@ -87,6 +88,13 @@
88
*fnv-prime*)
89
*fnv-mod*))))))
90
+91
(define (send-dedup-key chat-id text)
+92
(: (any-of integer? string?) string? -> string?)
+93
(fnv1a-hex (string-append (if (string? chat-id)
+94
chat-id
+95
(number->string chat-id))
+96
":" text)))
+97
98
;; ---- File-backed store: list of (key-string . epoch-seconds) ----
99
100
;; Parse one "epoch hash" line into (hash . epoch), or #f if malformed.
src/courier/main.sglmodified
@@ -87,6 +87,32 @@
87
(flush-output-port port))
88
port)))
89
+90
;; How many courier-<pid>.log files to keep in the state dir.
+91
(define *log-keep-generations* 10)
+92
+93
;; Remove all but the newest *log-keep-generations* courier-<pid>.log
+94
;; files. Best-effort and never fatal: a failure here costs disk, not
+95
;; correctness. A live instance's file has a recent mtime and is
+96
;; therefore kept; even if one were removed, Unix keeps the open fd
+97
;; valid, so the writer cannot be broken by this.
+98
(define (prune-instance-logs! log-dir)
+99
(guard (e (else #f))
+100
(let* ((entries (or (directory-list log-dir) '()))
+101
(logs (filter (lambda (n)
+102
(and (string-prefix? "courier-" n)
+103
(string-suffix? ".log" n)))
+104
entries))
+105
(paths (map (lambda (n) (path-join log-dir n)) logs))
+106
(sorted (sort paths
+107
(lambda (a b)
+108
(> (or (file-mtime a) 0)
+109
(or (file-mtime b) 0))))))
+110
(let loop ((rest sorted) (i 0))
+111
(unless (null? rest)
+112
(when (>= i *log-keep-generations*)
+113
(guard (e (else #f)) (delete-file (car rest))))
+114
(loop (cdr rest) (+ i 1)))))))
+115
116
;; Configure persistent logging for leader mode in APPEND mode, so
117
;; a hung instance's evidence survives across leader restarts. A
118
;; /mcp reconnect SIGKILLs the leader and spawns a fresh one; the
@@ -94,6 +120,21 @@
120
;; generation, so a hang -> /mcp -> still-hung -> /mcp loop lost
121
;; the original (load-bearing) evidence. Appending into one file
122
;; keeps the whole history (capped by *log-carryover-max*).
+123
;;
+124
;; PER-INSTANCE FILE (courier-<pid>.log, was the shared courier.log):
+125
;; append here is EMULATED -- read the old contents, reopen (which
+126
;; truncates), write them back, then log from the end. With two leader
+127
;; couriers running that emulation is mutually destructive: one truncates
+128
;; while the other's port sits at a high offset, so the next write lands
+129
;; past end-of-file and the kernel zero-fills the gap. That is where the
+130
;; NUL bytes in the 2026-08-04 log came from, and NUL bytes make grep
+131
;; treat the file as binary and report ZERO matches for a pattern `tail`
+132
;; shows plainly -- an instrument failing silently, mid-incident.
+133
;; sigil has no O_APPEND open, so the emulation cannot be made atomic;
+134
;; giving each instance its own file removes the sharing instead.
+135
;; The `--log <path>` form is deliberately NOT rewritten: claude-ops
+136
;; passes an already-unique per-task path there and reads it back by
+137
;; name, so inserting a pid would break that contract for no gain.
138
;; Part 2 (sync logging): the target is a plain file port
139
;; (open-output-file → async-port? = #f), so log writes never go
140
;; through the scheduler's await-port-writable / abort-to-prompt path.
@@ -105,8 +146,12 @@
146
;; was given), so production never logs to an async port.
147
(define (configure-persistent-log!)
148
(let* ((log-dir (path-dirname (default-relay-dir)))
−108
(log-path (path-join log-dir "courier.log")))
+149
(log-path (path-join log-dir
+150
(string-append "courier-"
+151
(number->string (process-id))
+152
".log"))))
153
(ensure-directory log-dir)
+154
(prune-instance-logs! log-dir)
155
(log-configure! target: (open-log-append! log-path))))
156
157
;; Configure logging from CLI args, opening any --log <path> target
src/courier/poller.sglmodified
@@ -46,6 +46,7 @@
46
(sigil mcp channel)
47
(sigil log)
48
(courier config)
+49
(courier claim)
50
(courier relay))
51
(export telegram-poller-supervisor
52
telegram-poller-shutdown!
@@ -409,7 +410,11 @@
410
(let ((st (car *active-poller*)))
411
(when st
412
(set-poller-state-running?! st #f)
−412
(poller-kill! st))))
+413
(poller-kill! st)))
+414
;; Give up the single-poller claim so the next session can poll
+415
;; without waiting for a staleness check. Called unconditionally: an
+416
;; instance that claimed and then failed to spawn still owns the claim.
+417
(release-poller-claim!))
418
419
;; ============================================================
420
;; Supervisor: Child Lifecycle
@@ -533,9 +538,24 @@
538
;;; detect wedges, respawn. Runs as a goroutine in the leader.
539
;;; Never returns; never raises past its own guards -- a Telegram
540
;;; failure of any kind cannot wedge MCP request servicing.
+541
;;;
+542
;;; Telegram allows ONE getUpdates consumer per bot token, so this first
+543
;;; takes the single-poller claim (see (courier claim)). Not getting it is
+544
;;; the NORMAL outcome for the 2nd..Nth concurrent session: this returns
+545
;;; quietly and the instance runs on as a full MCP server -- tools,
+546
;;; relays, sends and notifications all unaffected. There is deliberately
+547
;;; no retry loop and no exit.
548
(define (telegram-poller-supervisor server config)
549
(: mcp-server? courier-config? -> void?)
550
(channel-wait-initialized! server)
+551
(case (acquire-poller-claim! (courier-config-telegram-token config))
+552
((acquired) (poller-supervise-loop server))
+553
((held-by-other)
+554
(log-info "Telegram polling skipped -- another courier instance holds the poller claim (normal; this instance is fully functional)"))
+555
(else
+556
(log-warn "Telegram polling skipped -- could not establish poller claim ownership"))))
+557
+558
(define (poller-supervise-loop server)
559
(log-info "Telegram polling started (isolated poller process)")
560
(let respawn ((delay *restart-delay-initial*))
561
(let ((started (current-second))
src/courier/telegram.sglmodified
@@ -24,7 +24,10 @@
24
register-media-upload-handler!
25
path-media-type
26
resolve-media-type
−27
format-size)
+27
format-size
+28
;; exported so tests assert on the exact returned strings
+29
*dry-run-not-sent-message*
+30
*duplicate-suppressed-message*)
31
(begin
32
33
;; HTTP read timeout (seconds) for leader-side Telegram sends. A
@@ -41,6 +44,20 @@
44
;; read, so the request timeout alone can't bound it).
45
(define *send-connect-timeout* 10)
46
+47
;; Returned when COURIER_DISABLE_TELEGRAM_SEND gagged the send. It must
+48
;; be impossible to read as delivery: it does not begin with, and does not
+49
;; contain, a bare "Message sent". Named rather than inlined so the test
+50
;; suite asserts on the exact string the tool returns.
+51
(define *dry-run-not-sent-message*
+52
"Message NOT sent: COURIER_DISABLE_TELEGRAM_SEND is set (dry-run, nothing delivered).")
+53
+54
;; Returned when the persistent dedup window suppressed a repeat. Under
+55
;; record-before-send this means "an identical message to this chat was
+56
;; already accepted for delivery", NOT "this call delivered something" --
+57
;; so it says which happened rather than reusing the plain success string.
+58
(define *duplicate-suppressed-message*
+59
"Message already sent (duplicate suppressed: an identical message to this chat was accepted for delivery within the dedup window; nothing was sent again).")
+60
61
;; Idempotency/dedup backstop for Telegram sends lives in
62
;; (courier dedup): a PERSISTENT, restart-surviving (chat-id,text)
63
;; window. It is the load-bearing defence against the send-path
@@ -117,17 +134,34 @@
134
((suppress)
135
(log-info "Duplicate send-message suppressed"
136
chat-id: chat-id)
−120
"Message sent.")
+137
*duplicate-suppressed-message*)
138
(else
139
;; Test hook: latency simulation (inert by default).
140
(when (and (number? send-delay) (> send-delay 0))
141
(sleep send-delay))
142
(if send-disabled
143
;; Test hook: dry-run — no real delivery.
+144
;; This used to return the real success
+145
;; string, so every gagged send from
+146
;; 2026-07-06 to 2026-08-04 reported
+147
;; delivery it had not earned — and, worse,
+148
;; was a perfect mimic of the silent-drop
+149
;; bug it was masking, making production
+150
;; observation of that bug impossible.
+151
;; A dry-run is a DEFINITE non-delivery, so
+152
;; it also releases the dedup key: without
+153
;; that, the very next identical send hits
+154
;; the 'suppress path and reports "Message
+155
;; sent." — reintroducing the same lie one
+156
;; call later. Returning (not raising) is
+157
;; deliberate: an MCP error invites a client
+158
;; retry, and retry-against-a-killed-courier
+159
;; is the mechanism behind the send storm.
160
(begin
−128
(log-info "Telegram send disabled (dry-run)"
+161
(dedup-unrecord! dedup-path key)
+162
(log-info "Telegram send disabled (dry-run) -- NOT delivered"
163
chat-id: chat-id)
−130
"Message sent.")
+164
*dry-run-not-sent-message*)
165
;; Deliver exactly once. The key is already
166
;; recorded. A transport error must NEVER
167
;; escape this handler (an MCP error invites
test/test-claim.sgladded
@@ -0,0 +1,185 @@
+1
;; Unit tests for (courier claim) -- the single-poller claim.
+2
;;
+3
;; Telegram allows ONE getUpdates consumer per bot token. Every leader-mode
+4
;; courier used to start its own poller, so N concurrent sessions produced N
+5
;; pollers and a continuous 409 stream (measured 2026-08-04: three at once).
+6
;; These tests pin the claim's contract; the end-to-end behaviour -- two real
+7
;; couriers, one poller child, zero 409s at a mock that CAN produce them -- is
+8
;; proved by repro/repro-single-poller.sh.
+9
;;
+10
;; Note on liveness: under the test runner the "courier" process is the sigil
+11
;; runtime, so pid-liveness compares /proc/<pid>/comm against OUR OWN comm
+12
;; rather than a hardcoded name. That is deliberate in the implementation and
+13
;; is what makes these tests meaningful here at all.
+14
+15
(import (sigil test)
+16
(sigil core)
+17
(sigil string)
+18
(sigil fs)
+19
(sigil path)
+20
(sigil process)
+21
(courier dedup)
+22
(courier claim))
+23
+24
(define (fresh-dir)
+25
(let ((d (make-temp-directory)))
+26
(setenv! "COURIER_RELAY_DIR" (path-join d "relays"))
+27
d))
+28
+29
;; ============================================================
+30
;; Token hashing
+31
;; ============================================================
+32
+33
(test-group "claim-token-hash"
+34
(test "is deterministic"
+35
(assert-equal (claim-token-hash "123456:AAbbCC")
+36
(claim-token-hash "123456:AAbbCC")))
+37
+38
(test "differs for different tokens"
+39
(assert-true (not (string=? (claim-token-hash "123456:AAbbCC")
+40
(claim-token-hash "123456:AAbbCD")))))
+41
+42
(test "is a fixed 12 hex characters regardless of the token"
+43
(assert-equal 12 (string-length (claim-token-hash "a")))
+44
(assert-equal 12 (string-length (claim-token-hash "")))
+45
(assert-equal 12 (string-length
+46
(claim-token-hash
+47
"9999999999:AAEEFFbbccddeeff00112233445566778899"))))
+48
+49
;; The token must never be recoverable from anything written to disk.
+50
(test "never embeds the token itself"
+51
(let ((token "123456:SECRETSECRETSECRET"))
+52
(assert-true (not (string-contains? (claim-token-hash token)
+53
"SECRET")))
+54
(assert-true (not (string-contains? (poller-claim-path token)
+55
"SECRET"))))))
+56
+57
(test-group "poller-claim-path"
+58
(test "is scoped to the token"
+59
(let ((d (fresh-dir)))
+60
(assert-true (not (string=? (poller-claim-path "bot-one")
+61
(poller-claim-path "bot-two"))))))
+62
+63
(test "is scoped to the relay dir, so tests never contend with the live one"
+64
(let ((a (begin (fresh-dir) (poller-claim-path "same-token")))
+65
(b (begin (fresh-dir) (poller-claim-path "same-token"))))
+66
(assert-true (not (string=? a b))))))
+67
+68
;; ============================================================
+69
;; Liveness -- three states, not two
+70
;; ============================================================
+71
+72
(test-group "pid-liveness"
+73
;; POSITIVE CONTROL. If this fails, every 'dead below is meaningless:
+74
;; an instrument that answers 'dead for everything would pass those tests
+75
;; and would silently break every claim it ever saw.
+76
(test "reports our own pid alive"
+77
(assert-equal 'alive (pid-liveness (number->string (process-id)))))
+78
+79
(test "reports an impossible pid dead"
+80
(assert-equal 'dead (pid-liveness "999999")))
+81
+82
(test "reports garbage dead rather than raising"
+83
(assert-equal 'dead (pid-liveness "not-a-pid")))
+84
+85
(test "can read our own comm (the recycled-pid discriminator works)"
+86
(assert-true (string? (proc-comm (number->string (process-id)))))
+87
(assert-true (> (string-length (proc-comm (number->string (process-id))))
+88
0)))
+89
+90
(test "reads nothing for a pid that does not exist"
+91
(assert-equal #f (proc-comm "999999")))
+92
+93
;; The recycled-pid discriminator, tested from BOTH sides against the same
+94
;; live process (pid 1 always exists). Only testing the 'dead side would be
+95
;; satisfied by an implementation that always answers 'dead, which would
+96
;; break every claim it ever saw.
+97
(test "a live pid with a MATCHING comm is alive"
+98
(assert-equal 'alive (pid-liveness/comm "1" (proc-comm "1"))))
+99
+100
(test "a live pid with a DIFFERENT comm is treated as dead (recycled pid)"
+101
(assert-equal 'dead (pid-liveness/comm "1" "not-the-same-comm")))
+102
+103
(test "an unreadable comm falls back to 'alive, never to 'dead"
+104
(assert-equal 'alive (pid-liveness/comm "1" #f))))
+105
+106
;; ============================================================
+107
;; Acquire / release
+108
;; ============================================================
+109
+110
(test-group "acquire-poller-claim!"
+111
(test "a first caller acquires"
+112
(fresh-dir)
+113
(assert-equal 'acquired (acquire-poller-claim! "tok-a"))
+114
(release-poller-claim!))
+115
+116
(test "records OUR pid in the claim"
+117
(fresh-dir)
+118
(acquire-poller-claim! "tok-b")
+119
(assert-equal (number->string (process-id))
+120
(read-claim-owner (poller-claim-path "tok-b")))
+121
(release-poller-claim!))
+122
+123
(test "release removes the claim"
+124
(fresh-dir)
+125
(acquire-poller-claim! "tok-c")
+126
(release-poller-claim!)
+127
(assert-equal #f (read-claim-owner (poller-claim-path "tok-c"))))
+128
+129
(test "release is safe when nothing is held"
+130
(fresh-dir)
+131
(release-poller-claim!)
+132
(release-poller-claim!)
+133
(assert-true #t))
+134
+135
;; A claim left by a process that is GONE must not block the estate: that
+136
;; is the SIGKILL case, and SIGKILL is routine (an /mcp reconnect is one).
+137
(test "breaks a stale claim whose owner is gone"
+138
(let ((d (fresh-dir)))
+139
(let ((path (poller-claim-path "tok-stale")))
+140
(ensure-directory (path-dirname path))
+141
(make-symlink "999999" path)
+142
(assert-equal "999999" (read-claim-owner path))
+143
(assert-equal 'acquired (acquire-poller-claim! "tok-stale"))
+144
(assert-equal (number->string (process-id)) (read-claim-owner path))
+145
(release-poller-claim!))))
+146
+147
;; A claim whose owner pid is ALIVE but is not a courier can only be a
+148
;; recycled pid, so it is stale and must be broken -- otherwise one
+149
;; unlucky pid reuse would silence Telegram until that unrelated process
+150
;; exited. (The complementary case, a claim held by a genuinely live
+151
;; courier being left strictly alone, needs a second courier process and
+152
;; is proved end-to-end by repro/repro-single-poller.sh case 1, where
+153
;; instance B neither steals nor rewrites instance A's claim.)
+154
(test "treats a live-but-unrelated owner pid as a recycled, stale claim"
+155
(let ((d (fresh-dir)))
+156
(let ((path (poller-claim-path "tok-recycled")))
+157
(ensure-directory (path-dirname path))
+158
(make-symlink "1" path) ;; pid 1 is alive and is definitely not us
+159
(assert-equal 'acquired (acquire-poller-claim! "tok-recycled"))
+160
(assert-equal (number->string (process-id)) (read-claim-owner path))
+161
(release-poller-claim!))))
+162
+163
(test "does not release a claim that now belongs to someone else"
+164
(let ((d (fresh-dir)))
+165
(let ((path (poller-claim-path "tok-steal")))
+166
(acquire-poller-claim! "tok-steal")
+167
;; Someone else takes over (as would happen if we were SIGKILLed and
+168
;; a later instance broke our stale claim).
+169
(delete-file path)
+170
(make-symlink "1" path)
+171
(release-poller-claim!)
+172
(assert-equal "1" (read-claim-owner path))))))
+173
+174
;; ============================================================
+175
;; Shared hash core
+176
;; ============================================================
+177
+178
(test-group "fnv1a-hex"
+179
(test "send-dedup-key is unchanged by sharing the hash core"
+180
(assert-equal (fnv1a-hex "331005009:hello")
+181
(send-dedup-key 331005009 "hello")))
+182
+183
(test "is deterministic and differs per input"
+184
(assert-equal (fnv1a-hex "abc") (fnv1a-hex "abc"))
+185
(assert-true (not (string=? (fnv1a-hex "abc") (fnv1a-hex "abd"))))))