pmacs/docs/gpu-probe-sigint-framing.md

161 lines
7.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# GPU launcher / probe SIGINT teardown — framing
Revision 1. Status: **awaiting approval. No implementation.**
## 1. The problem, stated as what is observed
`gpu_invocation_acceptance::crdt::ctrl_c_on_launcher_group_does_not_reach_spawned_daemon`
fails inside the `sweep-crdt` gate stage with
```
child did not exit within 5s (tests/gpu_invocation_acceptance.rs:180, called from :1115)
```
The test spawns `pmacs --gpu --socket <s>` in its own process group
(`process_group(0)`, `:1107`), waits for the probe to report
`phase=ready`, sends `SIGINT` to the **group**, and requires the
launcher to exit within five seconds.
This is **pre-existing on `main`**. It is not caused by any feature
branch: the identical `build-crdt && sweep-crdt` pair run at `72da24a`
in a clean worktree with its own target directory fails the same test —
119 test binaries green, one red.
## 2. Why this blocks more than one lane
`sweep-crdt` is stage 15 of the sixteen-stage `--protocol` gate. While it
reds, **no branch can present a green gate**, `main` included. The
`panel-mapping-generation` (§5b) lane is complete with its own fifteen
stages green and is held behind this lane by explicit instruction.
## 3. Ground truth (cited, not recalled)
- **The launcher does not handle signals.** `run_gpu`
(`src/main.rs:324`) builds a `Command` for the GPU binary and blocks
in `command.status()` (`:363`) — a plain `waitpid`. It installs no
handler, so `SIGINT`'s default action should terminate it outright.
- **`pmacs-gpu` does not handle signals either.** Grepping
`SIGINT|signal_hook|sigaction|ctrlc|set_handler|pthread_sigmask|sigprocmask`
across `pmacs-gpu/src` returns **nothing**. The probe should also die
on the default action.
- **The probe never blocks indefinitely.** `run_headless_managed_probe`
(`pmacs-gpu/src/main.rs:1065`) loops on
`event_rx.recv_timeout(Duration::from_millis(50))`, so it wakes twenty
times a second regardless of traffic.
- **The daemon *does* handle signals**, and deliberately:
`src/daemon.rs:629-641` registers `SIGTERM`/`SIGINT` through
`signal_hook::flag`. The daemon is the process the test asserts must
**survive**; it is detached from the launcher's group, which is the
property under test.
- **Only one test in the suite uses `process_group(0)`** — this one.
Taken together the ground truth **deepens** the puzzle rather than
explaining it: two processes with default `SIGINT` disposition, one of
them polling at 50 ms, should both die immediately.
## 4. What has been ruled out, each by measurement
Recorded so this lane does not re-run them.
| hypothesis | how it was refuted |
|---|---|
| machine load | red on a quiet box, load 2.77 at launch |
| tmpfs starving RAM | `/tmp` 21 G → 1.2 G, available 27 G → 45 G; still red |
| leaked test daemons | peak 58, only +8 per sweep; green runs already sat at 46–60 |
| inotify exhaustion | 47 instances in use of 1024 |
| `--workspace` feature unification | same two targets under `--workspace` are green |
| a specific preceding test | **all 37 preceding targets + the suite run green** |
| later targets / other packages | they run at log lines 4848+, after the failure at 3066 |
**The last row is the strange one and is the real shape of this
problem**: the same binaries, in the same order, with the same tests
before it, pass as a subset and fail as part of the whole 119-target
sweep. Reproduction is 5/5 in the full sweep across two trees, and 0/N
in every reduction attempted.
## 5. One retracted claim, kept as a warning
An earlier attempt reported the mechanism as "the launcher blocks in
`do_wait` on a probe child stuck in `futex_do_wait`". **That was
retracted on its own evidence.** The sampler behind it caught 394
launchers and the longest-lived was **5 s total**; the failing instance
must outlive its `SIGINT` by 5 s, so its lifetime would be 8 s or more.
What was described is a *healthy* teardown from another test in the
suite. The pair is worth keeping only as the normal shape.
The lesson binds this lane's first step: **an instrument that samples
every launcher and hopes to catch the failing one is not good enough.**
## 6. Bets
1. The failure is a real teardown defect reachable by a user pressing
Ctrl-C, not a test artifact — because the test asserts an ordinary
product property and the two binaries involved carry no signal
handling at all.
2. It is **not** a timing margin. A green run finishes in 0.15 s against
a 5 s deadline (33×). A margin that large does not erode; something
different happens.
3. Therefore **raising the deadline is not a fix** and is explicitly out
of scope. If the conclusion turns out to be that the deadline is
wrong, that requires its own argument and its own approval.
## 7. First step — an instrument that keys on the failing instance
No fix is proposed yet, because the mechanism is unknown. The first
commit is diagnostic only:
- **D1.** Sample only launchers whose lifetime exceeds ~6 s, so the
failing instance is the *only* thing recorded, and capture for it:
`/proc/<pid>/status` (`SigBlk`, `SigIgn`, `SigCgt`, `State`), the
per-thread `wchan` under `/proc/<pid>/task/*/wchan`, and the same for
every child. `SigIgn`/`SigBlk` answers directly whether `SIGINT` was
ignored or blocked — including whether it was **inherited**, since
`SIG_IGN` survives both `fork` and `exec` while handlers do not.
- **D2.** Record whether the launcher had already reaped its child at
the moment the deadline expired, which separates "the child will not
die" from "the launcher will not notice".
- **D3.** Run the full sweep under D1/D2 until the failure is captured
**with** its diagnostics, and only then propose a fix.
## 8. Acceptance criteria for the eventual fix
Deliberately written now, so the fix cannot quietly become "make the
test pass".
- **A1.** The named mechanism is stated and demonstrated, not inferred:
a witness that fails before the change and passes after, plus a
mutation showing the witness bites its own clause.
- **A2.** `sweep-crdt` green on `main` for **three consecutive full
runs** — 1/1 is not evidence for a defect that hid from every
reduction.
- **A3.** The reduction paradox is explained or explicitly recorded as
unexplained. If the fix makes the sweep green without accounting for
why subsets always passed, that gap is stated in the record rather
than left for the next reader to rediscover.
- **A4.** No deadline is raised, and no test is skipped, retried, or
serialised to obtain green.
- **A5.** If the mechanism proves to be in `pmacs-gpu`'s shutdown path,
a Ctrl-C on a real `pmacs --gpu` session tears down the frontend and
leaves the daemon running — the product property the test encodes.
## 9. Coherence impact (`COHERENCE.md` §20)
- **Journey steps touched: none.** This is a teardown-correctness and
gate-trustworthiness lane; no journey step changes grade.
- **Interaction islands: none added.**
- **Config registry: not touched.**
- **Background-work attribution: not touched.**
Naming these explicitly matters: a reader auditing §20 by grade movement
would otherwise conclude this lane touches nothing, when what it
restores is the ability of every other lane to prove itself.
## 10. Out of scope
- Raising or removing the 5 s deadline (bet 3).
- The ~10 daemons each sweep leaks. Real, separately recorded, and not
implicated here — green runs already ran at 46–60 leaked daemons.
- The `gpu_initial_target_acceptance` binary including this suite as a
module, which makes a reproducing sweep report the failure twice. A
tidiness question, not a correctness one.