From d6bf0c32335bae154d771dad19831245d99f3ea9 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Tue, 4 Aug 2026 22:23:43 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20frame=20macOS=20CI=20signal=20integrity?= =?UTF-8?q?=20=E2=80=94=20a=20signature=20registry,=20then=20hardening?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four red CI incidents across #213 and #214 were each judged "not caused by this PR", and #214's case is airtight: it is docs-only and its tree is byte-identical to a green main. THAT PROVES THE PRs DID NOT CAUSE THEM. It does not prove they are harmless environmental noise, and three of the four have a specific, findable mechanism. This lane separates those two claims, which the current process conflates. FOUR INCIDENTS, THREE TESTS, FOUR SIGNATURES. The registry counts signatures, not test names, because the process test alone produced two with different mechanisms and different causal status — collapsing them under one name is how a possible product defect acquires a flake's immunity. 1. supersede_cancels_in_flight_job_within_50ms — "supersede did not cancel within 50ms". MEASUREMENT DESIGN. Its premise is a 15ms sleep asserted-by-comment to mean "the worker picked the job up"; on a loaded runner it may not have, in which case the test measures the QUEUED path while claiming the running one. And the 50ms clock starts before the second dispatch and is consumed by the test's own tick+sleep pump, so the measured interval is dominated by when THE TEST got scheduled. Widening the number would make it pass and measure nothing more. 2. a_successful_signal_disposition — "leader=exited(signal SIGUSR1)". TEST RACE. Readiness is ProcessEventKind::Started, emitted at SPAWN, not when /bin/sh has installed `trap '' USR1`. USR1's default disposition is terminate, so a signal in that window kills the child. The fixture's own comment states the requirement it does not enforce. 3. a_successful_signal_disposition — "EPERM, measured_group=unobservable(ESRCH), leader=live". NOT a test race: the group-target behaviour #176 and #200 circled and the reap-ledger lane parked disposition changes pending. Recorded UNRESOLVED, POSSIBLE PRODUCT DEFECT, with a diagnosis — never a green rerun — as its retirement condition. 4. terminal_escape_gates — "left: [], right: [49]". TEST RACE. `wait_for_file` returns as soon as `fs::read` succeeds, which succeeds on a ZERO-BYTE file; the probe's `open()` creates the file before `write()` fills it. The predicate is "readable", the assertion is "contains 1" — the same shape as signature 2, fixable at the helper so every caller inherits it. THE EXISTING PROSE IS DUPLICATED AND KEYED BY NAME. Flake claims live in at least six places, disagree in detail, carry no signatures or evidence links, and the handoff's list names three tests — two of which are not among these four incidents, while three of the four are absent from it. A list that is both stale and incomplete is worse than none: it confers "known flaky" on whatever happens to be named and withholds it from everything else. Acceptance 3 therefore AUDITS the existing three: each is carried in with a signature and evidence, or removed with a note. No entry survives on reputation. The rerun rule is REPLACED rather than softened: one rerun reproducing the SAME signature is evidence of intermittence only; a DIFFERENT signature, or the same one twice consecutively, requires investigation or a merge-base control before the red is attributed to the environment. Quarantine, if hardening fails, is a separate STILL-BLOCKING CI step — never #[ignore], continue-on-error, or silent retry-to-green. A quarantine that stops failing the build is a deletion with extra steps. Framing only. Sequencing is registry first, hardening second, per review. Co-Authored-By: Claude Opus 5 (1M context) --- docs/macos-ci-signal-integrity-framing.md | 296 ++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 docs/macos-ci-signal-integrity-framing.md diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md new file mode 100644 index 0000000..2c60b83 --- /dev/null +++ b/docs/macos-ci-signal-integrity-framing.md @@ -0,0 +1,296 @@ +# Framing — macOS CI signal integrity: a signature registry, then hardening + +**Revision 1.** Status: framing only. No branch work beyond this +document. Scouted against `githubsucks/main` @ `bfb97c6`. + +Four red CI incidents across #213 and #214 were each correctly judged +"not caused by this PR" — and #214's case is airtight, because it is +docs-only and its tree is byte-identical to a green `main`. **That +proves the PRs did not cause them. It does not prove they are harmless +environmental noise, and three of the four have a specific, findable +mechanism.** + +This lane separates those two claims, which the current process +conflates. + +--- + +## 0. Coherence impact (COHERENCE §20) + +- **Journey steps touched:** none. +- **Interaction islands:** none. +- **Config registry adoption:** none. +- **Background-work attribution:** none. +- **Why it belongs on the board:** it protects §19's acceptance-test + ratchet and every arc that reads a red run. A flake list that is + wrong, incomplete, or keyed by test *name* rather than *signature* + makes "rerun before concluding" a habit rather than a judgement — and + a real regression arriving in that stream is indistinguishable from + the noise it hides in. + +--- + +## 1. Ground truth (measured at `bfb97c6`) + +### 1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures + +The registry must count **signatures**, not test names. The process test +alone produced two, with different mechanisms and different causal +status — collapsing them under one name would have hidden a possible +product defect behind a known-flaky label. + +| # | test | job / flavor | exact signature | causal status | +|---|---|---|---|---| +| 1 | `async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 | +| 2 | `process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 | +| 3 | *(same test)* | macOS / lua54 | `EPERM, measured_group=unobservable(ESRCH), leader=live` | **UNRESOLVED — possible product defect** — see §1.4 | +| 4 | `vterm_stage2::terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | `left: [], right: [49]` | **test race** — see §1.5 | + +Evidence: +[#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642), +[#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1), +[#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1). + +### 1.2 Signature 1 — the 50ms budget measures more than it claims + +```rust +let first = rt.dispatch_sleep(2_000, Some("search")); +// Let the worker pick the job up so cancel hits a running job. +thread::sleep(Duration::from_millis(15)); +let started = Instant::now(); +let _second = rt.dispatch_sleep(2_000, Some("search")); +while !rt.is_complete(first) { + assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms"); + let _ = rt.tick(); + thread::sleep(Duration::from_millis(1)); +} +``` + +**Two design problems, and neither is fixed by a bigger number.** + +- **The premise is a sleep.** `thread::sleep(15ms)` is asserted-by-comment + to mean "the worker picked the job up". On a loaded runner it may not + have, in which case the test measures cancellation of a *queued* job — + a different code path, already covered by the sibling test — while + claiming to measure a running one. +- **The clock includes the observer.** `started` begins before the + second dispatch, and the budget is consumed by the test's own + `tick()` + `sleep(1ms)` pump loop. Under scheduling pressure the + measured interval is dominated by when *the test* got scheduled, not + by when the worker observed the cancel flag. + +**Widening 50ms to 200ms would make it pass and measure nothing more.** +The open question is what this test should assert: a latency bound needs +a clock the observer does not participate in, or the assertion should be +reformulated as ordering ("the first settles Cancelled before the second +completes") rather than duration. + +### 1.3 Signature 2 — `Started` does not prove the trap is installed + +```rust +spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()]; +let id = sup.spawn(spec).expect("spawn"); +let pid = spawn_started_pid(&mut sup, id); // waits for ProcessEventKind::Started +sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers"); +``` + +`spawn_started_pid` waits for `ProcessEventKind::Started { pid }`, which +is emitted when the process is **spawned** — not when `/bin/sh` has +parsed and installed `trap '' USR1`. **SIGUSR1's default disposition is +terminate**, so a signal delivered inside that window kills the child, +and the record is `exited(signal SIGUSR1)` instead of `Running`. + +The fixture's own comment states the requirement it does not enforce: +*"Ignore USR1 so the successful non-fatal signal cannot end the child +and confuse the state assertion with a real exit."* That is exactly the +confusion observed. + +This is the handoff's **"wait predicate weaker than the assertion"** +race: the predicate is "process exists", the assertion needs "trap +installed". + +### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved + +`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not** a +test race. It is the group-target behaviour the process-signal lanes +have circled three times: + +- #176 established that a group-directed `kill` returned EPERM while the + leader was observed alive by a real `try_wait`, retiring "EPERM cannot + happen for our own children" as a reason to discard the errno. +- #200 added `measured_group` from `getpgid`, the only field able to + disagree — and it is reported here as `unobservable(ESRCH)`, meaning + the group could not be measured at all. +- The reap-ledger lane parked every disposition change pending exactly + this evidence. + +**This lane does not resolve it and must not appear to.** Its registry +entry carries causal status **UNRESOLVED — possible product defect**, +and its retirement condition is a diagnosis, not a green rerun. Folding +it under the same "macOS signal timing" label as signature 2 is how a +real defect acquires a flake's immunity. + +### 1.5 Signature 4 — a readiness predicate satisfied by an empty file + +```rust +fn wait_for_file(path: &Path, timeout: Duration) -> Vec { + loop { + if let Ok(bytes) = fs::read(path) { return bytes; } // succeeds on 0 bytes + ... + } +} +``` + +The probe writes readiness with `open(path,'wb').write(b'1')`. **`open()` +creates the file before `write()` fills it**, so `fs::read` can succeed +on a zero-byte file and `wait_for_file` returns `[]`. The caller then +asserts `== b"1"` and fails `left: [], right: [49]`. + +The predicate is "readable"; the assertion is "contains `1`". This is +the same shape as §1.3 and is fixable at the helper — every caller +inherits the fix. + +### 1.6 The prose is duplicated and keyed by name + +Flake claims currently live in at least six places: the handoff's +hazards list, `docs/active-work.md` (twice), and the reap-ledger, +process-signal, vterm and terminal-config framings. They disagree in +detail, none carries an exact signature or an evidence link, and the +handoff's list names three tests — **two of which are not among the +four incidents seen here**, while three of these four are absent from +it. + +A list that is both stale and incomplete is worse than none: it confers +"known flaky" on whatever happens to be named, and withholds it from +everything else. + +### 1.7 What is NOT established + +- **No incident has been reproduced locally.** All four are macOS-only + and this machine is Linux. The mechanisms in §§1.2–1.5 are read from + source and from CI signatures, not from a local repro. +- **Frequency is unmeasured.** Four incidents across two PRs is not a + rate. The registry records occurrences so a rate can accumulate; + it does not claim one now. +- **Signature 3's cause remains unknown**, by design (§1.4). + +--- + +## 2. Questions + +- **Q#MS1 — where does the registry live?** It must be one file, and + every other mention becomes a pointer. `docs/agent-handoff.md` §5 is + the natural home (it already holds the hazards list), but a dedicated + `docs/ci-flake-registry.md` is easier to keep in one voice and to diff. + **Leaning: a dedicated file, with the handoff pointing at it**, since + the handoff is a briefing and this is a table that will grow. +- **Q#MS2 — what retires an entry?** Proposal: hardening that removes + the mechanism, plus N consecutive green runs of that job on `main`. + N needs a number, and the number is a judgement about how much + evidence "gone" requires. +- **Q#MS3 — does signature 1 get reformulated or re-measured?** §1.2 + argues its budget measures the observer. Reformulating as an ordering + assertion changes what the test proves; keeping a duration means + finding a clock the pump loop does not participate in. **This is the + one question this lane should not answer alone** — it is the + async-runtime lane's design call. +- **Q#MS4 — does hardening ship before or with the registry?** The user + has already answered: **registry now, hardening next.** Recorded here + so the sequencing is visible in the document rather than only in the + conversation. + +--- + +## 3. Bets + +- **Bet 1 — signatures 2 and 4 disappear under hardening**, because both + have a named mechanism and a fix at the readiness predicate. + *Falsified if either recurs after the predicate is strengthened.* +- **Bet 2 — signature 1 does not**, because widening a budget that + measures the observer changes nothing about what it measures. +- **Bet 3 — signature 3 recurs and stays unexplained** until the + process-signal lane resolves the group-target question. The registry's + job is to keep it visible, not to fix it. + +--- + +## 4. Acceptance + +**Stage 1 — the registry (this lane's first PR):** + +1. **One authoritative table**, with a row per **signature**: exact test + path, job and Lua flavor, **exact signature text**, evidence link, + causal status, and retirement condition. +2. **Every duplicate mention becomes a pointer.** The handoff hazards + list, both `active-work.md` mentions, and the four framing docs cite + the registry rather than restating a claim. +3. **The three tests named in the current handoff list are audited**: + each is either carried into the registry with a signature and + evidence, or removed with a note saying it was never substantiated. + No entry survives on reputation. +4. **The rerun rule is replaced**, not softened: + - one rerun that reproduces **the same signature** is evidence of + **intermittence only**; + - a **different signature**, or **the same signature twice + consecutively**, requires investigation or a merge-base control + before the red is attributed to the environment. +5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and + its retirement condition is a diagnosis, never a green rerun. + +**Stage 2 — hardening (a separate PR):** + +6. `wait_for_file` requires a **non-empty** result — or better, the + expected content — so the predicate matches the assertion. Every + caller inherits it. +7. The USR1 fixture proves the **trap is installed**, not merely that + the process started. The child publishes readiness after installing + the trap, and the test waits on that. +8. Signature 1 is **not** fixed by widening the budget (Q#MS3). +9. Both hardened tests run **repeatedly** (a repetition set, as the + reap-ledger lane did) rather than once, because a single green run of + a formerly intermittent test proves nothing. + +**Quarantine — only if hardening fails:** + +10. A quarantined test moves to a **separate, still-blocking CI step**. + Never `#[ignore]`, never `continue-on-error`, never a silent + retry-to-green. A quarantine that stops failing the build is a + deletion with extra steps. + +--- + +## 5. Parked + +- **Resolving signature 3.** It belongs to the process-signal / + reap-ledger lanes, which have already parked three tolerance rules + pending this class of evidence. +- **The `--no-fail-fast` gap and the crdt job's PTY deadlines**, both + recorded in the CI CRDT lane. Related in spirit, separate in scope. +- **A general flake-rate dashboard.** The registry accumulates + occurrences; turning that into a rate with alerting is its own thing. +- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...` and + `m6_8_supervisor_reaps_...` are in the current handoff list and get + audited under acceptance 3, but this lane's incidents are macOS. + +--- + +## 6. Gates + +The standing `CLAUDE.md` suite. Stage 1 is documentation-only and adds +no test; **its verification is acceptance 3** — the audit that no entry +survives on reputation. Stage 2 adds the repetition sets of acceptance 9. + +--- + +## 7. Branch plan + +Two PRs, in this order: + +1. **`macos-ci-signal-integrity`** — the registry, the pointer + rewrites, the audit, and the rerun rule. No code. +2. **A hardening PR** — the `wait_for_file` predicate and the USR1 trap + readiness, each with a repetition set. Signature 1 is referred to the + async-runtime design question rather than patched. + +Quarantine, if it happens, is a third and is scoped by what hardening +fails to fix.