351 lines
18 KiB
Markdown
351 lines
18 KiB
Markdown
# GPU launcher / probe SIGINT teardown — framing
|
||
|
||
Revision 4. Status: **awaiting approval. No implementation.**
|
||
|
||
Revisions 1 and 2 were each rejected on five findings. Every correction
|
||
is recorded in place rather than quietly rewritten, because three of
|
||
them were claims this document itself had advanced:
|
||
|
||
- r1 → r2: the ">6 s selector" and the "≥8 s lifetime" arithmetic
|
||
(§5); "two processes with default disposition" (§3); "119 binaries
|
||
green, one red" (§1); an unobtainable A2 (§8); "journey steps
|
||
touched: none" (§9).
|
||
- r2 → r3: **"R9 ran the same binaries" — it did not** (§4); "the probe
|
||
never blocks indefinitely" (§3); the launcher call-site count (§5);
|
||
reduction provenance, now in `docs/probe-sigint-evidence.md`; and
|
||
ledger corrections that had not been made portable (§11).
|
||
- r3 → r4: the interaction table overstated (§4); the red count was 7,
|
||
not 5, and abbreviated argv were not argv (manifest); "workspace
|
||
artifact family" conflated Cargo suffix with byte identity
|
||
(manifest); ledgers still carried the falsified R9 conclusions (§11).
|
||
**And a finding that reframes the lane: the failure has a datable
|
||
onset (§4a) and is not long-standing.**
|
||
|
||
## 1. The problem, stated as what is observed
|
||
|
||
`ctrl_c_on_launcher_group_does_not_reach_spawned_daemon` fails with
|
||
|
||
```
|
||
child did not exit within 5s (tests/gpu_invocation_acceptance.rs:180, called from :1115)
|
||
```
|
||
|
||
The test spawns `pmacs --gpu --socket <s>` in its own process group
|
||
(`process_group(0)`, `:1107`), waits for the probe to report
|
||
`phase=ready`, sends `SIGINT` to the **group** (`:1113`), and requires
|
||
the launcher to exit within five seconds.
|
||
|
||
**It fails in two binaries, not one.** `tests/gpu_initial_target_acceptance.rs`
|
||
includes the suite as a module, so a reproducing sweep reds twice.
|
||
Reference run `20260816T064549Z-2144707/09-sweep-crdt.log`:
|
||
|
||
| | line | result |
|
||
|---|---|---|
|
||
| `gpu_initial_target_acceptance` | 3097 | `FAILED. 14 passed; 1 failed … 5.19s` |
|
||
| `gpu_invocation_acceptance` | 3131 | `FAILED. 14 passed; 1 failed … 5.18s` |
|
||
| green result summaries | — | **119** |
|
||
|
||
So the correct statement is **119 green result summaries and two red
|
||
binaries**. Revision 1 said "119 binaries green, one red", which was
|
||
wrong on both halves.
|
||
|
||
**This is pre-existing on `main`.** The identical `build-crdt &&
|
||
sweep-crdt` pair at `72da24a`, clean worktree, own target directory,
|
||
fails the same test.
|
||
|
||
## 2. Why this blocks more than one lane
|
||
|
||
`sweep-crdt` is stage 15 of the sixteen-stage `--protocol` gate. While
|
||
it reds, **no branch can present a green gate**, `main` included.
|
||
`panel-mapping-generation` (§5b) is complete with its own fifteen
|
||
stages green and is held behind this lane by explicit instruction.
|
||
|
||
## 3. Ground truth (cited), and what it does *not* establish
|
||
|
||
- **Neither binary contains signal-handling code.** `run_gpu`
|
||
(`src/main.rs:324`) blocks in `command.status()` (`:363`) — a plain
|
||
`waitpid` — with no handler installed. Grepping
|
||
`SIGINT|signal_hook|sigaction|ctrlc|set_handler|pthread_sigmask|sigprocmask`
|
||
across `pmacs-gpu/src` returns nothing.
|
||
- **The probe's event loop wakes at least every 50 ms.**
|
||
`run_headless_managed_probe` (`pmacs-gpu/src/main.rs:1065`) loops on
|
||
`event_rx.recv_timeout(Duration::from_millis(50))`. **Revision 2 said
|
||
"never blocks indefinitely", which is false**: the probe's stdin
|
||
reader thread blocks in `read_to_end` (`:1109`) with no timeout, and
|
||
once `ready` the loop has **no deadline of its own** — it leaves only
|
||
when stdin closes (`:1212`). So the process is not bounded; only the
|
||
event wakeup is.
|
||
- **The daemon *does* handle signals, deliberately.**
|
||
`src/daemon.rs:629-641` registers `SIGTERM`/`SIGINT` via
|
||
`signal_hook::flag`. The daemon is the process the test asserts must
|
||
**survive**, detached from the launcher's group.
|
||
|
||
**Correction to revision 1.** Revision 1 concluded from the first two
|
||
bullets that "two processes with default `SIGINT` disposition should
|
||
both die at once". **That does not follow, and it contradicted this
|
||
document's own leading hypothesis.** Absence of handler *code* says
|
||
nothing about runtime *disposition*: `SIG_IGN` is inherited across
|
||
`fork` **and** survives `exec`, so either process can hold a
|
||
non-default disposition it never installed — from the test harness,
|
||
from `cargo`, or from the invoking shell. Inherited ignore is a live
|
||
candidate precisely because the source is silent. What the source
|
||
establishes is narrower: **neither binary sets a disposition itself**,
|
||
so whatever disposition they hold at runtime was inherited, and that is
|
||
measurable rather than arguable.
|
||
|
||
## 4. Reductions attempted
|
||
|
||
**Full provenance lives in `docs/probe-sigint-evidence.md`**, which is
|
||
pushed with this branch: exact command, worktree, HEAD, cleanliness,
|
||
the artifact family actually executed, result, and log digest for every
|
||
physical run. Log bodies stay machine-local under
|
||
`/home/jeans/build/pmacs-gate-targets/probe-sigint-evidence/` — `/tmp`
|
||
is a tmpfs and they were nearly lost to a cleanup mid-lane.
|
||
|
||
Three provenance caveats are recorded there rather than smoothed over:
|
||
**R1 and R2 have no preserved log** (revision 2 cited `gpu3.log` for
|
||
both R2 and R6; that log is R6's three-suite run alone, and counting
|
||
one run as two was wrong); **cleanliness is `UNKNOWN` for every
|
||
pre-manifest run**, because it was not recorded at the time and is not
|
||
inferrable; and **R1–R10 ran in the `panel-mapping-generation`
|
||
worktree**, not at `main`. `D0` re-runs the matrix under a harness that
|
||
captures all of it, at `main`, before any row here is relied on.
|
||
|
||
All rows carry `--features crdt`. Full argv, worktree, HEAD,
|
||
cleanliness and artifact family per run: `docs/probe-sigint-evidence.md`.
|
||
|
||
| # | reduction (after `cargo test`) | runs | result | log |
|
||
|---|---|---|---|---|
|
||
| R1 | `--test gpu_invocation_acceptance ctrl_c_on_launcher_group` | 3 | green, 0.15–0.17 s | **no log preserved** |
|
||
| R2 | `--test gpu_invocation_acceptance` (whole suite) | 1 | green, 15 passed | **no log preserved** |
|
||
| R3 | `--workspace --no-fail-fast -- --skip basedpyright ctrl_c_on_launcher_group` | 1 | green — every binary runs, only this test executes | `filtered.log` |
|
||
| R4 | `--lib --test gpu_invocation_acceptance --no-fail-fast` | 1 | green, 2145 + 15 | `two.log` |
|
||
| R5 | `--test gate_script_acceptance --test gpu_invocation_acceptance` | 1 | green | `suspect.log` |
|
||
| R6 | `--test gpu_font_acceptance --test gpu_initial_target_acceptance --test gpu_invocation_acceptance` | 1 | green | `gpu3.log` |
|
||
| R7 | `--lib --bins` + `--test`×14 (targets 6–19) + the suite | 1 | green | `half1.log` |
|
||
| R8 | `--test`×18 (targets 20–37) + the suite | 1 | green | `half2.log` |
|
||
| R9 | `--lib --bins` + `--test`×32 (targets 6–37) + the suite | 1 | green (**reduction artifacts**) | `prefix.log` |
|
||
| R10 | `--workspace ... --test gpu_initial_target_acceptance --test gpu_invocation_acceptance` | 1 | green (**workspace artifacts**) | `wsonly.log` |
|
||
| F1–F5 | full `--workspace --no-fail-fast -- --skip basedpyright` | 5 | **red, 5/5** | `base-sweep.log` (at `72da24a`), `postclean.log`, `sweep-inst.log`, `sweep-diag.log`, gate `…-2144707` |
|
||
|
||
**Correction to revision 2: R9 did not run the same binaries.** It
|
||
executed `gpu_initial_target_acceptance-91f51d0b5303ff9f` and
|
||
`gpu_invocation_acceptance-6b4b8223dea45247`; the failing sweeps
|
||
executed `-5d9105cb7047aab8` and `-d4dae4f01bcdef62`. Those artifacts
|
||
are **byte-different** (sha256/16 `36912fa2…` vs `1b3cc86c…`, and
|
||
`858d7148…` vs `ede0c07d…`; see `docs/probe-sigint-evidence.md`).
|
||
Cargo's target selection changes the fingerprint, so command shape
|
||
changes the executable. R9 therefore establishes **same target names
|
||
and order**, not same binaries.
|
||
|
||
What the evidence is **consistent with** is an interaction. It does not
|
||
isolate one, because the three rows differ in more than the two columns
|
||
shown — different source heads, different worktrees, unknown
|
||
cleanliness, and, as above, different bytes behind identical Cargo
|
||
suffixes:
|
||
|
||
| prior targets execute | workspace artifact family | result |
|
||
|---|---|---|
|
||
| yes | no | R9 green |
|
||
| no | yes | R10 green |
|
||
| yes | yes | **F1–F5 red** |
|
||
|
||
Neither factor alone reproduced it **in these runs**. That is the
|
||
whole of the claim. `--workspace` artifact selection is **not
|
||
sufficient by itself and not ruled out**; later-selected packages can
|
||
influence Cargo's build graph and fingerprints *before* their test
|
||
executables run, so "their targets execute after the failure at line
|
||
3066" does not exonerate them — that claim is withdrawn. And since
|
||
§4's own preamble says no historical row should be relied on until D0,
|
||
**this table is a description of what was observed, not a finding**.
|
||
Revision 3 asserted it as an interaction while simultaneously
|
||
disclaiming its inputs, which cannot both be true.
|
||
|
||
Also refuted, by measurement: machine load (red on a quiet box, load
|
||
2.77); tmpfs starving RAM (**tested by experiment** — `/tmp` 21 G →
|
||
1.2 G, available 27 G → 45 G, still red); leaked daemons (peak 58, +8
|
||
per sweep, green runs already at 46–60); inotify (47 of 1024).
|
||
|
||
## 4a. The onset is datable — and it reframes the lane
|
||
|
||
`sweep-crdt` appears **17 times** in this target directory's gate logs.
|
||
`ctrl_c` fails in **exactly the last three**, and passed — both copies,
|
||
`... ok` — in the runs before them.
|
||
|
||
| window | runs | `ctrl_c` |
|
||
|---|---|---|
|
||
| 08-14 → 08-15 19:57 | 14 (11 green; 3 red on *other* tests) | **passes** wherever the stage ran |
|
||
| 08-16 06:33 `…-1977433` | red, 2 bins | **fails, both copies** |
|
||
| 08-16 06:45 `…-2144707` | red, 2 bins | **fails, both copies** |
|
||
| 08-17 17:25 `…-2375685` | red, 3 bins | **fails, both copies** |
|
||
|
||
The three earlier red sweeps failed on unrelated rows — protocol and
|
||
version tests, and `composition_overhead_under_ten_percent`. None
|
||
involved `ctrl_c`.
|
||
|
||
**So the failure is not long-standing.** "Pre-existing on `main`"
|
||
remains true — F1 at `72da24a` reproduces it — but "always broken" was
|
||
never established and is now contradicted. Last green containing it:
|
||
`20260815T185708Z`. First red: `20260816T063330Z`. The machine was not
|
||
rebooted across that boundary.
|
||
|
||
**This supersedes the reduction matrix as the lane's first move.** A
|
||
test that passed in this stage fourteen times and then failed three
|
||
times running has a change behind it, and identifying that change is
|
||
worth more than further reduction — which, per §4, has isolated nothing.
|
||
|
||
## 5. Two retracted claims, both mine, kept as warnings
|
||
|
||
**Claim A — "mechanism located".** Reported the launcher blocked in
|
||
`do_wait` on a probe child in `futex_do_wait`.
|
||
|
||
**Claim B — the retraction of A.** Argued A was unsupported because the
|
||
failing launcher "must live ≥ 8 s" while the sampler's longest-lived
|
||
was 5 s.
|
||
|
||
**Claim B's arithmetic is false.** Both reproducing binaries finish in
|
||
**~5.19 s including the five-second timeout** (`:3097`, `:3131`), so
|
||
`phase=ready` is reached in roughly a tenth of a second and the failing
|
||
launcher lives about **5.1 s total** — squarely inside what the sampler
|
||
observed. A ">6 s" selector would therefore have captured **nothing**,
|
||
repeating the very sampling error it was written to correct.
|
||
|
||
So A is **not** refuted by B. A remains **unproven for a different
|
||
reason**: under `--features crdt` the suite spawns root launchers from
|
||
**six** call sites — `:509, :534, :544, :574, :725, :1097`, all inside
|
||
`#[cfg(feature = "crdt")] mod crdt` (`:88`). Eight `--gpu` arguments
|
||
appear in the file, but `:38` and `:65` sit under
|
||
`#[cfg(not(feature = "crdt"))]` (`:26`) and are compiled out of the
|
||
failing configuration. Revision 2 said "five" while citing eight, which
|
||
was wrong twice over. Six is the number; what matters is that it is
|
||
more than one, so a launcher captured by command line alone cannot be
|
||
attributed to *this* test. The `do_wait` /
|
||
`futex_do_wait` pair is consistent with the failing instance and
|
||
consistent with a healthy sibling, and nothing recorded distinguishes
|
||
them.
|
||
|
||
The standing lesson is now the opposite of revision 1's: **do not key
|
||
on process age at all.** Key on identity.
|
||
|
||
## 6. Bets
|
||
|
||
1. **The failure is a real teardown defect** — a user pressing Ctrl-C
|
||
on `pmacs --gpu` sees the same hang. **This is a bet, not a
|
||
finding**, and the current witness does not reach the real GUI
|
||
path: it goes through a wrapper script and `--headless-managed-probe`
|
||
(`:1090-1093`), not a live wgpu frontend. Confirming or dropping this
|
||
bet is D4 below.
|
||
2. It is **not** a timing margin. A green run finishes in 0.15 s against
|
||
a 5 s deadline — 33×. Margins that large do not erode.
|
||
3. Therefore **raising the deadline is not a fix** and is out of scope.
|
||
If the conclusion turns out to be that the deadline is wrong, that
|
||
needs its own argument and its own approval.
|
||
|
||
## 7. First step — diagnostics keyed on identity, not age
|
||
|
||
No fix is proposed; the mechanism is unknown. The first commit is
|
||
diagnostic only, and it must **discriminate** the three live candidates:
|
||
blocked delivery, inherited ignore, and an escaped or wrong process
|
||
group.
|
||
|
||
- **D0a — bisect the onset window** (§4a): `20260815T185708Z` green →
|
||
`20260816T063330Z` red, no reboot between. This is the sharpest lead
|
||
the lane has and comes before any further reduction.
|
||
- **D0b — re-run the §4 matrix with captured provenance**, at `main`,
|
||
recording the artifact hashes actually executed **at run time**.
|
||
Revision 2's strongest claim collapsed because command shape silently
|
||
changed the binary; no further reduction should be trusted until each
|
||
row names the executable it ran.
|
||
- **D1 — key on the PID this test records.** The test already owns
|
||
`launcher.id()`. Capture around its own `kill`, not by scanning for
|
||
age or command line.
|
||
- **D2 — snapshot before *and* after the signal**, for the test parent,
|
||
the launcher, and the probe:
|
||
- `SigIgn`, `SigCgt`, `SigBlk` — **per thread**, from
|
||
`/proc/<pid>/task/*/status`, since `SigBlk` is thread-specific and
|
||
a process-wide reading would hide a blocked delivery on the one
|
||
thread that matters;
|
||
- `SigPnd` and `ShdPnd` — a pending-but-undelivered `SIGINT` is
|
||
exactly what distinguishes blocked delivery from ignore;
|
||
- `PID`, `PPID`, `PGID`, `SID` for each — which settles whether the
|
||
signal was even addressed to the right group, and whether anything
|
||
escaped it.
|
||
A post-failure snapshot alone cannot prove inheritance; the
|
||
before/after pair is what makes the claim provable.
|
||
- **D3 — run the full sweep under D1/D2 until the failure is captured
|
||
*with* its diagnostics.** Only then propose a fix.
|
||
- **D4 — settle bet 1 separately.** Establish whether a real
|
||
`pmacs --gpu` session, not the wrapper/headless probe, reproduces the
|
||
hang. The answer decides whether A5 is an obligation or is dropped.
|
||
|
||
## 8. Acceptance criteria for the eventual fix
|
||
|
||
Written now so the fix cannot quietly become "make the test pass".
|
||
|
||
- **A1.** The mechanism is stated and demonstrated, not inferred: a
|
||
witness failing before the change and passing after, plus a mutation
|
||
showing the witness bites its own clause.
|
||
- **A2.** `sweep-crdt` green for **three consecutive full runs on the
|
||
reviewed fixed head of this branch**. Not "on main" — that is
|
||
unobtainable before this lane is approved, gated and merged, and
|
||
revision 1 stated an impossible precondition. Post-merge
|
||
confirmation on `main` is a follow-up, not a gate on the fix.
|
||
- **A3.** There is no established "R9 paradox" to explain — R9 ran
|
||
different binaries, so the comparison it appeared to make was never
|
||
made. What A3 requires instead: **D0 recreates the subset/full
|
||
comparison under captured provenance**, and whatever it then shows is
|
||
either explained by the fix or explicitly recorded as unexplained. A
|
||
fix that greens the sweep without that comparison having been made
|
||
properly leaves the gap stated, not hidden.
|
||
- **A4.** No deadline raised, no test skipped, retried, or serialised
|
||
to obtain green.
|
||
- **A5.** **Conditional on D4.** If bet 1 holds, this is unconditional:
|
||
Ctrl-C on a real `pmacs --gpu` session tears down the frontend and
|
||
leaves the daemon running. If D4 shows the hang is reachable only
|
||
through the wrapper/headless path, bet 1 is dropped, A5 is struck,
|
||
and the lane is recorded as gate-correctness only.
|
||
|
||
## 9. Coherence impact (`COHERENCE.md` §20)
|
||
|
||
- **Journey step touched: 12(a), "closing is clean."** Ctrl-C teardown
|
||
of a GPU session is exactly that step, whether or not its grade
|
||
moves. **Revision 1 said "journey steps touched: none", which was
|
||
false** — it reasoned from grade movement, which §20 explicitly warns
|
||
against.
|
||
- **Grade movement: none expected.** This restores a property that is
|
||
supposed to hold, rather than opening a new one.
|
||
- **Interaction islands: none added.**
|
||
- **Config registry: not touched. Background-work attribution: not
|
||
touched.**
|
||
- Beyond step 12(a), what this lane restores is every *other* lane's
|
||
ability to prove itself, since no branch can show a green gate while
|
||
stage 15 reds.
|
||
|
||
## 10. Out of scope
|
||
|
||
- Raising or removing the 5 s deadline (bet 3).
|
||
- The ~10 daemons each sweep leaks — real, separately recorded, and not
|
||
implicated: green runs already ran at 46–60 leaked daemons.
|
||
- `gpu_initial_target_acceptance` including the suite as a module. It
|
||
is why the failure reds twice, and it is a tidiness question, not a
|
||
correctness one.
|
||
|
||
## 11. Record corrections owed to other ledgers
|
||
|
||
A correction is not made until it is portable. Two were outstanding
|
||
when revision 2 was reviewed, and both are closed by this revision:
|
||
|
||
- **This branch's ledger** asserted that two default-disposition
|
||
processes "should both die at once" and then withdrew that same claim
|
||
further down. The assertion is removed; only the withdrawal and its
|
||
reasoning remain.
|
||
- **`panel-mapping-generation`** carried "119 binaries green, one
|
||
red", the ≥8 s arithmetic, the "default action" claim and the ">6 s
|
||
selector"; `779a6bd` corrected those. **It still carried more**,
|
||
found on re-review: `--workspace` unification "refuted", R9 running
|
||
the "same binaries", later packages that "cannot be implicated", and
|
||
a cause "cumulative across the preceding 37 binaries". Revision 3's
|
||
claim here that the held lane no longer transports falsified claims
|
||
was **premature**; those are corrected now, and this section should
|
||
be read as a checklist that has been re-verified rather than an
|
||
assurance.
|