7.5 KiB
GPU launcher / probe SIGINT teardown — framing
Revision 1. Status: awaiting approval. No implementation.
1. The problem, stated as what is observed
gpu_invocation_acceptance::crdt::ctrl_c_on_launcher_group_does_not_reach_spawned_daemon
fails inside the sweep-crdt gate stage with
child did not exit within 5s (tests/gpu_invocation_acceptance.rs:180, called from :1115)
The test spawns pmacs --gpu --socket <s> in its own process group
(process_group(0), :1107), waits for the probe to report
phase=ready, sends SIGINT to the group, and requires the
launcher to exit within five seconds.
This is pre-existing on main. It is not caused by any feature
branch: the identical build-crdt && sweep-crdt pair run at 72da24a
in a clean worktree with its own target directory fails the same test —
119 test binaries green, one red.
2. Why this blocks more than one lane
sweep-crdt is stage 15 of the sixteen-stage --protocol gate. While it
reds, no branch can present a green gate, main included. The
panel-mapping-generation (§5b) lane is complete with its own fifteen
stages green and is held behind this lane by explicit instruction.
3. Ground truth (cited, not recalled)
- The launcher does not handle signals.
run_gpu(src/main.rs:324) builds aCommandfor the GPU binary and blocks incommand.status()(:363) — a plainwaitpid. It installs no handler, soSIGINT's default action should terminate it outright. pmacs-gpudoes not handle signals either. GreppingSIGINT|signal_hook|sigaction|ctrlc|set_handler|pthread_sigmask|sigprocmaskacrosspmacs-gpu/srcreturns nothing. The probe should also die on the default action.- The probe never blocks indefinitely.
run_headless_managed_probe(pmacs-gpu/src/main.rs:1065) loops onevent_rx.recv_timeout(Duration::from_millis(50)), so it wakes twenty times a second regardless of traffic. - The daemon does handle signals, and deliberately:
src/daemon.rs:629-641registersSIGTERM/SIGINTthroughsignal_hook::flag. The daemon is the process the test asserts must survive; it is detached from the launcher's group, which is the property under test. - Only one test in the suite uses
process_group(0)— this one.
Taken together the ground truth deepens the puzzle rather than
explaining it: two processes with default SIGINT disposition, one of
them polling at 50 ms, should both die immediately.
4. What has been ruled out, each by measurement
Recorded so this lane does not re-run them.
| hypothesis | how it was refuted |
|---|---|
| machine load | red on a quiet box, load 2.77 at launch |
| tmpfs starving RAM | /tmp 21 G → 1.2 G, available 27 G → 45 G; still red |
| leaked test daemons | peak 58, only +8 per sweep; green runs already sat at 46–60 |
| inotify exhaustion | 47 instances in use of 1024 |
--workspace feature unification |
same two targets under --workspace are green |
| a specific preceding test | all 37 preceding targets + the suite run green |
| later targets / other packages | they run at log lines 4848+, after the failure at 3066 |
The last row is the strange one and is the real shape of this problem: the same binaries, in the same order, with the same tests before it, pass as a subset and fail as part of the whole 119-target sweep. Reproduction is 5/5 in the full sweep across two trees, and 0/N in every reduction attempted.
5. One retracted claim, kept as a warning
An earlier attempt reported the mechanism as "the launcher blocks in
do_wait on a probe child stuck in futex_do_wait". That was
retracted on its own evidence. The sampler behind it caught 394
launchers and the longest-lived was 5 s total; the failing instance
must outlive its SIGINT by 5 s, so its lifetime would be 8 s or more.
What was described is a healthy teardown from another test in the
suite. The pair is worth keeping only as the normal shape.
The lesson binds this lane's first step: an instrument that samples every launcher and hopes to catch the failing one is not good enough.
6. Bets
- The failure is a real teardown defect reachable by a user pressing Ctrl-C, not a test artifact — because the test asserts an ordinary product property and the two binaries involved carry no signal handling at all.
- It is not a timing margin. A green run finishes in 0.15 s against a 5 s deadline (33×). A margin that large does not erode; something different happens.
- Therefore raising the deadline is not a fix and is explicitly out of scope. If the conclusion turns out to be that the deadline is wrong, that requires its own argument and its own approval.
7. First step — an instrument that keys on the failing instance
No fix is proposed yet, because the mechanism is unknown. The first commit is diagnostic only:
- D1. Sample only launchers whose lifetime exceeds ~6 s, so the
failing instance is the only thing recorded, and capture for it:
/proc/<pid>/status(SigBlk,SigIgn,SigCgt,State), the per-threadwchanunder/proc/<pid>/task/*/wchan, and the same for every child.SigIgn/SigBlkanswers directly whetherSIGINTwas ignored or blocked — including whether it was inherited, sinceSIG_IGNsurvives bothforkandexecwhile handlers do not. - D2. Record whether the launcher had already reaped its child at the moment the deadline expired, which separates "the child will not die" from "the launcher will not notice".
- D3. Run the full sweep under D1/D2 until the failure is captured with its diagnostics, and only then propose a fix.
8. Acceptance criteria for the eventual fix
Deliberately written now, so the fix cannot quietly become "make the test pass".
- A1. The named mechanism is stated and demonstrated, not inferred: a witness that fails before the change and passes after, plus a mutation showing the witness bites its own clause.
- A2.
sweep-crdtgreen onmainfor three consecutive full runs — 1/1 is not evidence for a defect that hid from every reduction. - A3. The reduction paradox is explained or explicitly recorded as unexplained. If the fix makes the sweep green without accounting for why subsets always passed, that gap is stated in the record rather than left for the next reader to rediscover.
- A4. No deadline is raised, and no test is skipped, retried, or serialised to obtain green.
- A5. If the mechanism proves to be in
pmacs-gpu's shutdown path, a Ctrl-C on a realpmacs --gpusession tears down the frontend and leaves the daemon running — the product property the test encodes.
9. Coherence impact (COHERENCE.md §20)
- Journey steps touched: none. This is a teardown-correctness and gate-trustworthiness lane; no journey step changes grade.
- Interaction islands: none added.
- Config registry: not touched.
- Background-work attribution: not touched.
Naming these explicitly matters: a reader auditing §20 by grade movement would otherwise conclude this lane touches nothing, when what it restores is the ability of every other lane to prove itself.
10. Out of scope
- Raising or removing the 5 s deadline (bet 3).
- The ~10 daemons each sweep leaks. Real, separately recorded, and not implicated here — green runs already ran at 46–60 leaked daemons.
- The
gpu_initial_target_acceptancebinary including this suite as a module, which makes a reproducing sweep report the failure twice. A tidiness question, not a correctness one.