From d6bf0c32335bae154d771dad19831245d99f3ea9 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Tue, 4 Aug 2026 22:23:43 +0200 Subject: [PATCH 1/5] =?UTF-8?q?docs:=20frame=20macOS=20CI=20signal=20integ?= =?UTF-8?q?rity=20=E2=80=94=20a=20signature=20registry,=20then=20hardening?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four red CI incidents across #213 and #214 were each judged "not caused by this PR", and #214's case is airtight: it is docs-only and its tree is byte-identical to a green main. THAT PROVES THE PRs DID NOT CAUSE THEM. It does not prove they are harmless environmental noise, and three of the four have a specific, findable mechanism. This lane separates those two claims, which the current process conflates. FOUR INCIDENTS, THREE TESTS, FOUR SIGNATURES. The registry counts signatures, not test names, because the process test alone produced two with different mechanisms and different causal status — collapsing them under one name is how a possible product defect acquires a flake's immunity. 1. supersede_cancels_in_flight_job_within_50ms — "supersede did not cancel within 50ms". MEASUREMENT DESIGN. Its premise is a 15ms sleep asserted-by-comment to mean "the worker picked the job up"; on a loaded runner it may not have, in which case the test measures the QUEUED path while claiming the running one. And the 50ms clock starts before the second dispatch and is consumed by the test's own tick+sleep pump, so the measured interval is dominated by when THE TEST got scheduled. Widening the number would make it pass and measure nothing more. 2. a_successful_signal_disposition — "leader=exited(signal SIGUSR1)". TEST RACE. Readiness is ProcessEventKind::Started, emitted at SPAWN, not when /bin/sh has installed `trap '' USR1`. USR1's default disposition is terminate, so a signal in that window kills the child. The fixture's own comment states the requirement it does not enforce. 3. a_successful_signal_disposition — "EPERM, measured_group=unobservable(ESRCH), leader=live". NOT a test race: the group-target behaviour #176 and #200 circled and the reap-ledger lane parked disposition changes pending. Recorded UNRESOLVED, POSSIBLE PRODUCT DEFECT, with a diagnosis — never a green rerun — as its retirement condition. 4. terminal_escape_gates — "left: [], right: [49]". TEST RACE. `wait_for_file` returns as soon as `fs::read` succeeds, which succeeds on a ZERO-BYTE file; the probe's `open()` creates the file before `write()` fills it. The predicate is "readable", the assertion is "contains 1" — the same shape as signature 2, fixable at the helper so every caller inherits it. THE EXISTING PROSE IS DUPLICATED AND KEYED BY NAME. Flake claims live in at least six places, disagree in detail, carry no signatures or evidence links, and the handoff's list names three tests — two of which are not among these four incidents, while three of the four are absent from it. A list that is both stale and incomplete is worse than none: it confers "known flaky" on whatever happens to be named and withholds it from everything else. Acceptance 3 therefore AUDITS the existing three: each is carried in with a signature and evidence, or removed with a note. No entry survives on reputation. The rerun rule is REPLACED rather than softened: one rerun reproducing the SAME signature is evidence of intermittence only; a DIFFERENT signature, or the same one twice consecutively, requires investigation or a merge-base control before the red is attributed to the environment. Quarantine, if hardening fails, is a separate STILL-BLOCKING CI step — never #[ignore], continue-on-error, or silent retry-to-green. A quarantine that stops failing the build is a deletion with extra steps. Framing only. Sequencing is registry first, hardening second, per review. Co-Authored-By: Claude Opus 5 (1M context) --- docs/macos-ci-signal-integrity-framing.md | 296 ++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 docs/macos-ci-signal-integrity-framing.md diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md new file mode 100644 index 0000000..2c60b83 --- /dev/null +++ b/docs/macos-ci-signal-integrity-framing.md @@ -0,0 +1,296 @@ +# Framing — macOS CI signal integrity: a signature registry, then hardening + +**Revision 1.** Status: framing only. No branch work beyond this +document. Scouted against `githubsucks/main` @ `bfb97c6`. + +Four red CI incidents across #213 and #214 were each correctly judged +"not caused by this PR" — and #214's case is airtight, because it is +docs-only and its tree is byte-identical to a green `main`. **That +proves the PRs did not cause them. It does not prove they are harmless +environmental noise, and three of the four have a specific, findable +mechanism.** + +This lane separates those two claims, which the current process +conflates. + +--- + +## 0. Coherence impact (COHERENCE §20) + +- **Journey steps touched:** none. +- **Interaction islands:** none. +- **Config registry adoption:** none. +- **Background-work attribution:** none. +- **Why it belongs on the board:** it protects §19's acceptance-test + ratchet and every arc that reads a red run. A flake list that is + wrong, incomplete, or keyed by test *name* rather than *signature* + makes "rerun before concluding" a habit rather than a judgement — and + a real regression arriving in that stream is indistinguishable from + the noise it hides in. + +--- + +## 1. Ground truth (measured at `bfb97c6`) + +### 1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures + +The registry must count **signatures**, not test names. The process test +alone produced two, with different mechanisms and different causal +status — collapsing them under one name would have hidden a possible +product defect behind a known-flaky label. + +| # | test | job / flavor | exact signature | causal status | +|---|---|---|---|---| +| 1 | `async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 | +| 2 | `process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 | +| 3 | *(same test)* | macOS / lua54 | `EPERM, measured_group=unobservable(ESRCH), leader=live` | **UNRESOLVED — possible product defect** — see §1.4 | +| 4 | `vterm_stage2::terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | `left: [], right: [49]` | **test race** — see §1.5 | + +Evidence: +[#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642), +[#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1), +[#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1). + +### 1.2 Signature 1 — the 50ms budget measures more than it claims + +```rust +let first = rt.dispatch_sleep(2_000, Some("search")); +// Let the worker pick the job up so cancel hits a running job. +thread::sleep(Duration::from_millis(15)); +let started = Instant::now(); +let _second = rt.dispatch_sleep(2_000, Some("search")); +while !rt.is_complete(first) { + assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms"); + let _ = rt.tick(); + thread::sleep(Duration::from_millis(1)); +} +``` + +**Two design problems, and neither is fixed by a bigger number.** + +- **The premise is a sleep.** `thread::sleep(15ms)` is asserted-by-comment + to mean "the worker picked the job up". On a loaded runner it may not + have, in which case the test measures cancellation of a *queued* job — + a different code path, already covered by the sibling test — while + claiming to measure a running one. +- **The clock includes the observer.** `started` begins before the + second dispatch, and the budget is consumed by the test's own + `tick()` + `sleep(1ms)` pump loop. Under scheduling pressure the + measured interval is dominated by when *the test* got scheduled, not + by when the worker observed the cancel flag. + +**Widening 50ms to 200ms would make it pass and measure nothing more.** +The open question is what this test should assert: a latency bound needs +a clock the observer does not participate in, or the assertion should be +reformulated as ordering ("the first settles Cancelled before the second +completes") rather than duration. + +### 1.3 Signature 2 — `Started` does not prove the trap is installed + +```rust +spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()]; +let id = sup.spawn(spec).expect("spawn"); +let pid = spawn_started_pid(&mut sup, id); // waits for ProcessEventKind::Started +sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers"); +``` + +`spawn_started_pid` waits for `ProcessEventKind::Started { pid }`, which +is emitted when the process is **spawned** — not when `/bin/sh` has +parsed and installed `trap '' USR1`. **SIGUSR1's default disposition is +terminate**, so a signal delivered inside that window kills the child, +and the record is `exited(signal SIGUSR1)` instead of `Running`. + +The fixture's own comment states the requirement it does not enforce: +*"Ignore USR1 so the successful non-fatal signal cannot end the child +and confuse the state assertion with a real exit."* That is exactly the +confusion observed. + +This is the handoff's **"wait predicate weaker than the assertion"** +race: the predicate is "process exists", the assertion needs "trap +installed". + +### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved + +`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not** a +test race. It is the group-target behaviour the process-signal lanes +have circled three times: + +- #176 established that a group-directed `kill` returned EPERM while the + leader was observed alive by a real `try_wait`, retiring "EPERM cannot + happen for our own children" as a reason to discard the errno. +- #200 added `measured_group` from `getpgid`, the only field able to + disagree — and it is reported here as `unobservable(ESRCH)`, meaning + the group could not be measured at all. +- The reap-ledger lane parked every disposition change pending exactly + this evidence. + +**This lane does not resolve it and must not appear to.** Its registry +entry carries causal status **UNRESOLVED — possible product defect**, +and its retirement condition is a diagnosis, not a green rerun. Folding +it under the same "macOS signal timing" label as signature 2 is how a +real defect acquires a flake's immunity. + +### 1.5 Signature 4 — a readiness predicate satisfied by an empty file + +```rust +fn wait_for_file(path: &Path, timeout: Duration) -> Vec { + loop { + if let Ok(bytes) = fs::read(path) { return bytes; } // succeeds on 0 bytes + ... + } +} +``` + +The probe writes readiness with `open(path,'wb').write(b'1')`. **`open()` +creates the file before `write()` fills it**, so `fs::read` can succeed +on a zero-byte file and `wait_for_file` returns `[]`. The caller then +asserts `== b"1"` and fails `left: [], right: [49]`. + +The predicate is "readable"; the assertion is "contains `1`". This is +the same shape as §1.3 and is fixable at the helper — every caller +inherits the fix. + +### 1.6 The prose is duplicated and keyed by name + +Flake claims currently live in at least six places: the handoff's +hazards list, `docs/active-work.md` (twice), and the reap-ledger, +process-signal, vterm and terminal-config framings. They disagree in +detail, none carries an exact signature or an evidence link, and the +handoff's list names three tests — **two of which are not among the +four incidents seen here**, while three of these four are absent from +it. + +A list that is both stale and incomplete is worse than none: it confers +"known flaky" on whatever happens to be named, and withholds it from +everything else. + +### 1.7 What is NOT established + +- **No incident has been reproduced locally.** All four are macOS-only + and this machine is Linux. The mechanisms in §§1.2–1.5 are read from + source and from CI signatures, not from a local repro. +- **Frequency is unmeasured.** Four incidents across two PRs is not a + rate. The registry records occurrences so a rate can accumulate; + it does not claim one now. +- **Signature 3's cause remains unknown**, by design (§1.4). + +--- + +## 2. Questions + +- **Q#MS1 — where does the registry live?** It must be one file, and + every other mention becomes a pointer. `docs/agent-handoff.md` §5 is + the natural home (it already holds the hazards list), but a dedicated + `docs/ci-flake-registry.md` is easier to keep in one voice and to diff. + **Leaning: a dedicated file, with the handoff pointing at it**, since + the handoff is a briefing and this is a table that will grow. +- **Q#MS2 — what retires an entry?** Proposal: hardening that removes + the mechanism, plus N consecutive green runs of that job on `main`. + N needs a number, and the number is a judgement about how much + evidence "gone" requires. +- **Q#MS3 — does signature 1 get reformulated or re-measured?** §1.2 + argues its budget measures the observer. Reformulating as an ordering + assertion changes what the test proves; keeping a duration means + finding a clock the pump loop does not participate in. **This is the + one question this lane should not answer alone** — it is the + async-runtime lane's design call. +- **Q#MS4 — does hardening ship before or with the registry?** The user + has already answered: **registry now, hardening next.** Recorded here + so the sequencing is visible in the document rather than only in the + conversation. + +--- + +## 3. Bets + +- **Bet 1 — signatures 2 and 4 disappear under hardening**, because both + have a named mechanism and a fix at the readiness predicate. + *Falsified if either recurs after the predicate is strengthened.* +- **Bet 2 — signature 1 does not**, because widening a budget that + measures the observer changes nothing about what it measures. +- **Bet 3 — signature 3 recurs and stays unexplained** until the + process-signal lane resolves the group-target question. The registry's + job is to keep it visible, not to fix it. + +--- + +## 4. Acceptance + +**Stage 1 — the registry (this lane's first PR):** + +1. **One authoritative table**, with a row per **signature**: exact test + path, job and Lua flavor, **exact signature text**, evidence link, + causal status, and retirement condition. +2. **Every duplicate mention becomes a pointer.** The handoff hazards + list, both `active-work.md` mentions, and the four framing docs cite + the registry rather than restating a claim. +3. **The three tests named in the current handoff list are audited**: + each is either carried into the registry with a signature and + evidence, or removed with a note saying it was never substantiated. + No entry survives on reputation. +4. **The rerun rule is replaced**, not softened: + - one rerun that reproduces **the same signature** is evidence of + **intermittence only**; + - a **different signature**, or **the same signature twice + consecutively**, requires investigation or a merge-base control + before the red is attributed to the environment. +5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and + its retirement condition is a diagnosis, never a green rerun. + +**Stage 2 — hardening (a separate PR):** + +6. `wait_for_file` requires a **non-empty** result — or better, the + expected content — so the predicate matches the assertion. Every + caller inherits it. +7. The USR1 fixture proves the **trap is installed**, not merely that + the process started. The child publishes readiness after installing + the trap, and the test waits on that. +8. Signature 1 is **not** fixed by widening the budget (Q#MS3). +9. Both hardened tests run **repeatedly** (a repetition set, as the + reap-ledger lane did) rather than once, because a single green run of + a formerly intermittent test proves nothing. + +**Quarantine — only if hardening fails:** + +10. A quarantined test moves to a **separate, still-blocking CI step**. + Never `#[ignore]`, never `continue-on-error`, never a silent + retry-to-green. A quarantine that stops failing the build is a + deletion with extra steps. + +--- + +## 5. Parked + +- **Resolving signature 3.** It belongs to the process-signal / + reap-ledger lanes, which have already parked three tolerance rules + pending this class of evidence. +- **The `--no-fail-fast` gap and the crdt job's PTY deadlines**, both + recorded in the CI CRDT lane. Related in spirit, separate in scope. +- **A general flake-rate dashboard.** The registry accumulates + occurrences; turning that into a rate with alerting is its own thing. +- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...` and + `m6_8_supervisor_reaps_...` are in the current handoff list and get + audited under acceptance 3, but this lane's incidents are macOS. + +--- + +## 6. Gates + +The standing `CLAUDE.md` suite. Stage 1 is documentation-only and adds +no test; **its verification is acceptance 3** — the audit that no entry +survives on reputation. Stage 2 adds the repetition sets of acceptance 9. + +--- + +## 7. Branch plan + +Two PRs, in this order: + +1. **`macos-ci-signal-integrity`** — the registry, the pointer + rewrites, the audit, and the rerun rule. No code. +2. **A hardening PR** — the `wait_for_file` predicate and the USR1 trap + readiness, each with a repetition set. Signature 1 is referred to the + async-runtime design question rather than patched. + +Quarantine, if it happens, is a third and is scoped by what hardening +fails to fix. From a1c6c93adf7abdaaaf5de9aa5be51db51adf57d0 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Tue, 4 Aug 2026 22:48:37 +0200 Subject: [PATCH 2/5] docs: tighten macOS CI signal-integrity framing Define signatures as normalized selector/job/output matches, preserve historical evidence while centralizing live triage policy, and replace green-run retirement with causal disposition. Also fix the rerun-rule contradiction and move the questions out of the inline-math namespace. --- docs/macos-ci-signal-integrity-framing.md | 144 ++++++++++++++-------- 1 file changed, 90 insertions(+), 54 deletions(-) diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md index 2c60b83..c3ba89d 100644 --- a/docs/macos-ci-signal-integrity-framing.md +++ b/docs/macos-ci-signal-integrity-framing.md @@ -1,7 +1,11 @@ # Framing — macOS CI signal integrity: a signature registry, then hardening -**Revision 1.** Status: framing only. No branch work beyond this -document. Scouted against `githubsucks/main` @ `bfb97c6`. +**Revision 2.** Status: framing only. No branch work beyond this +document. Scouted against `githubsucks/main` @ `bfb97c6`. Revision 2 +separates a machine-matchable signature from verbatim, variable CI +output; preserves historical incident evidence while centralizing live +triage policy; gives retirement a causal rule rather than an arbitrary +green-run count; and corrects the rerun rule. Four red CI incidents across #213 and #214 were each correctly judged "not caused by this PR" — and #214's case is airtight, because it is @@ -39,12 +43,21 @@ alone produced two, with different mechanisms and different causal status — collapsing them under one name would have hidden a possible product defect behind a known-flaky label. -| # | test | job / flavor | exact signature | causal status | +The registry key is an exact **test selector + job/flavor + normalized +match rule**, not a pasted panic block. PIDs, elapsed times, and rendered +OS-error suffixes vary between runs; each row therefore names the +required invariant fragments and the evidence link preserves the +verbatim occurrence. A row matches only when every listed requirement +is satisfied; where a requirement lists alternative platform renderings, +one of those alternatives suffices. A test-name match by itself never +matches a registry entry. + +| # | exact test selector | job / flavor | required signature fragments | causal status | |---|---|---|---|---| -| 1 | `async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 | -| 2 | `process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 | -| 3 | *(same test)* | macOS / lua54 | `EPERM, measured_group=unobservable(ESRCH), leader=live` | **UNRESOLVED — possible product defect** — see §1.4 | -| 4 | `vterm_stage2::terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | `left: [], right: [49]` | **test race** — see §1.5 | +| 1 | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 | +| 2 | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 | +| 3 | *(same test)* | macOS / lua54 | all of `EPERM`, `measured_group=unobservable(` + `ESRCH` / `No such process`, and `leader=live` | **UNRESOLVED — possible product defect** — see §1.4 | +| 4 | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | both `left: []` and `right: [49]` | **test race** — see §1.5 | Evidence: [#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642), @@ -111,9 +124,9 @@ installed". ### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved -`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not** a -test race. It is the group-target behaviour the process-signal lanes -have circled three times: +`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not +established as a test race**. It is the group-target behaviour the +process-signal lanes have circled three times: - #176 established that a group-directed `kill` returned EPERM while the leader was observed alive by a real `try_wait`, retiring "EPERM cannot @@ -147,18 +160,25 @@ on a zero-byte file and `wait_for_file` returns `[]`. The caller then asserts `== b"1"` and fails `left: [], right: [49]`. The predicate is "readable"; the assertion is "contains `1`". This is -the same shape as §1.3 and is fixable at the helper — every caller -inherits the fix. +the same shape as §1.3 and is fixable at the helper. All four callers of +this helper require concrete non-empty content; the similar bottom-panel +helper already rejects empty reads. -### 1.6 The prose is duplicated and keyed by name +### 1.6 Live triage policy and historical evidence are mixed together -Flake claims currently live in at least six places: the handoff's -hazards list, `docs/active-work.md` (twice), and the reap-ledger, -process-signal, vterm and terminal-config framings. They disagree in -detail, none carries an exact signature or an evidence link, and the -handoff's list names three tests — **two of which are not among the -four incidents seen here**, while three of these four are absent from -it. +Flake language currently appears in the handoff's two operational +rules, `docs/active-work.md`, and several landed framing documents. It +is not all duplication. The process-signal and reap-ledger framings, for +example, preserve exact historical occurrences and the reasoning those +lanes built from them; replacing that evidence with a pointer would +make a durable framing depend on a mutable registry. + +The real duplication is **live classification and triage policy**. The +handoff's hazards list names three tests — two of which are not among +the four incidents seen here — while three of these four are absent. +Elsewhere, historical occurrence notes, forward-looking risk warnings, +and current "known flaky" claims are written in the same voice even +though they require different treatment. A list that is both stale and incomplete is worse than none: it confers "known flaky" on whatever happens to be named, and withholds it from @@ -178,26 +198,29 @@ everything else. ## 2. Questions -- **Q#MS1 — where does the registry live?** It must be one file, and - every other mention becomes a pointer. `docs/agent-handoff.md` §5 is - the natural home (it already holds the hazards list), but a dedicated - `docs/ci-flake-registry.md` is easier to keep in one voice and to diff. - **Leaning: a dedicated file, with the handoff pointing at it**, since - the handoff is a briefing and this is a table that will grow. -- **Q#MS2 — what retires an entry?** Proposal: hardening that removes - the mechanism, plus N consecutive green runs of that job on `main`. - N needs a number, and the number is a judgement about how much - evidence "gone" requires. -- **Q#MS3 — does signature 1 get reformulated or re-measured?** §1.2 +- **Q#MCI1 — where does the registry live? DECIDED:** one dedicated + `docs/ci-red-signatures.md`, with the handoff's operational rule + pointing at it. The handoff is a briefing; the registry is an + occurrence ledger whose rows remain available after retirement. +- **Q#MCI2 — what retires an entry? DECIDED:** a mechanism-specific + causal result, not N green runs. A known-race entry requires hardening + that removes the named mechanism plus a discriminating acceptance + witness for the stronger predicate. A measurement-design entry + requires its owning lane to replace or justify the measurement and + pin the resulting claim. An unresolved entry requires diagnosis and + an explicit disposition. Main-job greens remain occurrence evidence, + but cannot retire any row by themselves; retired rows remain in the + history with their disposition. +- **Q#MCI3 — does signature 1 get reformulated or re-measured?** §1.2 argues its budget measures the observer. Reformulating as an ordering assertion changes what the test proves; keeping a duration means finding a clock the pump loop does not participate in. **This is the one question this lane should not answer alone** — it is the async-runtime lane's design call. -- **Q#MS4 — does hardening ship before or with the registry?** The user - has already answered: **registry now, hardening next.** Recorded here - so the sequencing is visible in the document rather than only in the - conversation. +- **Q#MCI4 — does hardening ship before or with the registry? DECIDED:** + the user has already answered: **registry now, hardening next.** + Recorded here so the sequencing is visible in the document rather + than only in the conversation. --- @@ -206,11 +229,15 @@ everything else. - **Bet 1 — signatures 2 and 4 disappear under hardening**, because both have a named mechanism and a fix at the readiness predicate. *Falsified if either recurs after the predicate is strengthened.* -- **Bet 2 — signature 1 does not**, because widening a budget that - measures the observer changes nothing about what it measures. -- **Bet 3 — signature 3 recurs and stays unexplained** until the - process-signal lane resolves the group-target question. The registry's - job is to keep it visible, not to fix it. +- **Bet 2 — a signature-keyed registry refuses name-based immunity.** A + future failure in the process test which does not contain all of row + 2's or row 3's required fragments is a new incident, not a known + flake. *Falsified if the operational rule permits a test-name-only + match.* +- **Bet 3 — green reruns do not erase unresolved evidence.** Signature + 3 remains unresolved until the process-signal lane diagnoses and + disposes it, whether or not later runs pass. *Falsified if a green run + changes that row's status or retirement condition.* --- @@ -219,33 +246,42 @@ everything else. **Stage 1 — the registry (this lane's first PR):** 1. **One authoritative table**, with a row per **signature**: exact test - path, job and Lua flavor, **exact signature text**, evidence link, - causal status, and retirement condition. -2. **Every duplicate mention becomes a pointer.** The handoff hazards - list, both `active-work.md` mentions, and the four framing docs cite - the registry rather than restating a claim. + selector, job and Lua flavor, normalized machine-match rule, evidence + link to the verbatim occurrence, causal status, and mechanism-specific + retirement condition. Matching requires the selector, job/flavor, + and every required fragment; variable values are explicitly + normalized rather than silently abbreviated. +2. **One live triage policy.** Operational duplicate classifications and + rerun rules become pointers to the registry. Historical occurrence + evidence stays where it supports a landed framing; a relevant note + may gain a registry status link, but its evidence and reasoning are + not replaced. Forward-looking risk statements are audited as risks, + not silently promoted to known flakes. 3. **The three tests named in the current handoff list are audited**: each is either carried into the registry with a signature and evidence, or removed with a note saying it was never substantiated. No entry survives on reputation. 4. **The rerun rule is replaced**, not softened: - - one rerun that reproduces **the same signature** is evidence of - **intermittence only**; - - a **different signature**, or **the same signature twice - consecutively**, requires investigation or a merge-base control - before the red is attributed to the environment. + - a **green rerun after a red** establishes intermittence only; it + does not establish environmental cause, harmlessness, or retirement; + - the **same signature on the rerun** is a second occurrence and + remains blocking pending investigation or a merge-base control; + - a **different signature** is a new incident and is judged + independently. A known test name confers no immunity. 5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and its retirement condition is a diagnosis, never a green rerun. **Stage 2 — hardening (a separate PR):** -6. `wait_for_file` requires a **non-empty** result — or better, the - expected content — so the predicate matches the assertion. Every - caller inherits it. +6. The vterm Stage 2 `wait_for_file` requires the **expected content** + (preferred) or at minimum a non-empty result, so the predicate + matches the assertion. Its four callers all require concrete + non-empty content; the bottom-panel helper already rejects empty + reads and is not evidence for widening this change further. 7. The USR1 fixture proves the **trap is installed**, not merely that the process started. The child publishes readiness after installing the trap, and the test waits on that. -8. Signature 1 is **not** fixed by widening the budget (Q#MS3). +8. Signature 1 is **not** fixed by widening the budget (Q#MCI3). 9. Both hardened tests run **repeatedly** (a repetition set, as the reap-ledger lane did) rather than once, because a single green run of a formerly intermittent test proves nothing. From d33bf4df78ca5dc0eeb1de4b7365afac92bed2b5 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Tue, 4 Aug 2026 22:55:25 +0200 Subject: [PATCH 3/5] docs(ci): the red-signature registry, and an audit that found no immunity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 1 of the macOS CI signal-integrity lane. `docs/ci-red-signatures.md` is now the single authority for judging a red CI run. NOT NAMED "FLAKES", DELIBERATELY. One of its rows is a possible product defect, and a filename calling it a flake would confer immunity the evidence does not support. A ROW MATCHES ON SIGNATURE, NEVER ON TEST NAME. All three of selector, job/flavor and every required fragment must hold. Fragments are normalized rather than pasted: PIDs, elapsed times and rendered OS-error suffixes vary between runs, so a verbatim key would match nothing, and where a fragment lists alternatives (ESRCH / "No such process") those are one condition rendered differently by platform. A failure in a listed test WITHOUT that row's fragments is a new incident. The process test is why that rule exists: it produced TWO signatures with different mechanisms and different causal status, and only one is a test bug. Four incidents, three tests, four signatures — the registry counts signatures. THE RERUN RULE IS REPLACED, NOT SOFTENED. A green rerun establishes INTERMITTENCE ONLY — never environmental cause, harmlessness, or retirement. The same signature again is a second occurrence and stays blocking pending investigation or a merge-base control. A different signature is a new incident. RETIREMENT IS CAUSAL. A test race retires by hardening that removes the mechanism plus a discriminating witness; a measurement-design row by its owning lane replacing or justifying the measurement; an unresolved row by diagnosis and disposition. Main-branch greens accumulate as occurrence evidence and retire nothing. THE AUDIT FOUND A THIRD CATEGORY the framing's acceptance 3 did not anticipate. It said each incumbent is either carried in with a signature and evidence, or removed as never substantiated. Both incumbents are in between: the tests are real and the mechanisms plausible — a33's "blue pixels" is a genuine assertion string at pmacs-gpu/src/main.rs:17973, and m6_8 exists and is timing-based — but NEITHER HAS A LINKED OCCURRENCE. Deleting them would discard a real observation; carrying them as peers of the evidenced rows would grant exactly the reputation this lane exists to deny. They are recorded as "mechanism named, no occurrence recorded", with it stated that nothing there confers known-flaky status and that a red matching one is a FIRST recorded occurrence to be investigated. R6 goes further: with no signature ever captured it CANNOT BE MATCHED AT ALL, so a red in that test is a new incident by default. That is the correct outcome for an entry that never carried evidence. LIVE POLICY IS CENTRALIZED; HISTORICAL EVIDENCE IS NOT MOVED. The handoff's hazards rule becomes a pointer. Its landed-lesson block at §5 keeps its own evidence and gains a note that the registry's rerun rule supersedes "rerun isolated". active-work.md's two mentions are verification records for the reap-ledger and bottom-panel lanes and are left alone; only the CI-weakness block's triage half points at the registry, because the job-cost question there is genuinely separate. Verified: fmt, diff-check, clippy with and without crdt, --lib 1896, --lib --features crdt 2081, pmacs-protocol 19, m4 149, required GPU 221. All three tests the registry names pass locally on Linux, which is consistent with every row being macOS-only and is not evidence about any of them. Docs only. Stage 2 (hardening) is a separate PR. Co-Authored-By: Claude Opus 5 (1M context) --- docs/active-work.md | 5 + docs/agent-handoff.md | 26 +++-- docs/ci-red-signatures.md | 207 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+), 7 deletions(-) create mode 100644 docs/ci-red-signatures.md diff --git a/docs/active-work.md b/docs/active-work.md index 733ca6d..24e157b 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -194,6 +194,11 @@ the crdt job. same commit. One of those was a real regression the flip caused; the other was load. **That ambiguity is the problem** — a job where noise and signal look alike trains people to rerun rather than read. + **The triage half of this is now owned by + `docs/ci-red-signatures.md`** — signature-keyed rows and a rerun rule + that refuses to treat a green rerun as an all-clear. What remains here + is the job-cost question: whether the crdt job should carry real-PTY + deadlines at all. Candidate: longer deadlines for real-PTY assertions in this job specifically, or serialize the PTY suites. diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index ca33f92..7c0d432 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -292,11 +292,19 @@ someone forgot. - **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error then` guards make the silence look deliberate. Report through `pmacs.editor.set_status` until the channel is built. -- **Flakes that are not your change.** Judge a red run against these - before bisecting: `process::tests::a_successful_signal_disposition_...` - (macOS-only, signal timing); `a33_headless_terminal_frame_paints_...` - (GPU under parallel load, "0 blue pixels"); `m6_8_supervisor_reaps_...` - (load-sensitive). **Rerun before concluding.** +- **Judging a red CI run: `docs/ci-red-signatures.md` is the authority.** + It keys on **signature**, not test name — a failure in a listed test + that lacks that row's fragments is a NEW incident, not a known one. + The old rule here ("rerun before concluding") is retired: **a green + rerun establishes intermittence only**, never environmental cause or + harmlessness; the same signature again is a second occurrence and + stays blocking. One row is an **unresolved possible product defect** + that no rerun can clear. + + This list previously named three tests. The audit found one of them + had produced **two distinct signatures** with different causes, and + that two of the four incidents actually seen were absent from it — + which is why name-keyed lists are not trustworthy. - **`basedpyright` hangs forever** — always `cargo test --test m4_acceptance -- --skip basedpyright`. - **The crdt sweep needs `cargo build --workspace` first**, or twelve @@ -1996,8 +2004,12 @@ before trusting them: - **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan, `PMACS_REQUIRE_GPU=1` works without lavapipe. - **Flaky-under-load tests — rerun isolated before treating a sweep - failure as a regression.** The m8 daemon tests and the m6 process/PTY - tests (`m6_1_pty_mode_lifecycle_started_then_exited`, + failure as a regression.** *(Historical, from this lane. For LIVE + triage of a red run use `docs/ci-red-signatures.md`, whose rerun rule + supersedes "rerun isolated" — a green rerun proves intermittence only. + `m6_8_supervisor_reaps_all_children_across_cycles` is R6 there, with + no signature ever captured.)* The m8 daemon tests and the m6 + process/PTY tests (`m6_1_pty_mode_lifecycle_started_then_exited`, `m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based; `editor::composition_overhead_under_ten_percent` is a render-ratio microbenchmark that fails ~1/3 even isolated single-threaded (already diff --git a/docs/ci-red-signatures.md b/docs/ci-red-signatures.md new file mode 100644 index 0000000..45adff8 --- /dev/null +++ b/docs/ci-red-signatures.md @@ -0,0 +1,207 @@ +# CI red signatures — the triage registry + +**This file is the single authority for judging a red CI run.** It is an +occurrence ledger, not a flake list: a row records what was seen, what is +known about why, and what would retire it. **A row is not a claim that +the failure is harmless.** + +Deliberately not named "flakes". One of its rows is a possible product +defect, and a filename that called it a flake would confer immunity the +evidence does not support. + +Read this before attributing any red run to the environment. Landed +framing documents keep their own historical evidence and reasoning — +that is not duplication, and it is not superseded by this file. What +lives here is **live triage policy**. + +--- + +## How a row matches + +**A test-name match is never sufficient.** A red run matches a row only +when *all three* hold: + +1. the **exact test selector** matches; +2. the **job / flavor** matches; +3. **every required fragment** is present in the failure output. + +Where a fragment lists alternatives (`ESRCH` / `No such process`), any +one satisfies that requirement — those are the same condition rendered +differently by platform or libc. + +Fragments are **normalized**, never pasted verbatim. PIDs, elapsed +times, thread ids and rendered OS-error suffixes vary between runs; a +verbatim key would match nothing. The evidence link preserves the exact +occurrence. + +**A failure in a listed test that does not carry that row's fragments is +a NEW incident**, judged on its own. The process test below is why this +rule exists: it produced two signatures with different mechanisms and +different causal status, and only one of them is a test bug. + +--- + +## The rerun rule + +This replaces "rerun before concluding", which conflated three different +outcomes. + +- **A green rerun after a red establishes INTERMITTENCE ONLY.** It does + not establish environmental cause, harmlessness, or retirement. +- **The same signature on the rerun is a SECOND OCCURRENCE.** It remains + blocking pending investigation or a merge-base control. +- **A different signature is a NEW INCIDENT**, judged independently. + +A merge-base control — running the same command on the merge base — is +what distinguishes "this branch caused it" from "this tree has it". It +is cheaper than argument and is the correct response to a second +occurrence. + +--- + +## What retires a row + +**Retirement is causal, never a count of green runs.** A row is retired +by removing or explaining its mechanism: + +| causal status | retirement condition | +|---|---| +| **test race** | hardening that removes the named mechanism, plus a discriminating witness for the stronger predicate | +| **measurement design** | the owning lane replaces or justifies the measurement and pins the resulting claim | +| **unresolved** | diagnosis and an explicit disposition | +| **mechanism named, no occurrence** | a linked occurrence promotes it to one of the above; sustained absence is not retirement, because nothing was ever measured | + +Main-branch greens are **occurrence evidence** and accumulate toward a +rate. They retire nothing by themselves. Retired rows stay in this file +with their disposition, so a recurrence is recognisable. + +--- + +## Live rows + +### R1 — supersede cancellation budget + +| field | value | +|---|---| +| **selector** | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | +| **job / flavor** | macOS / luajit | +| **required fragments** | `supersede did not cancel within 50ms` | +| **causal status** | **measurement design** | +| **evidence** | [#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | +| **retirement** | the async-runtime lane replaces or justifies the measurement (Q#MCI3) | + +The test's premise is `thread::sleep(15ms)`, asserted by comment to mean +"the worker picked the job up"; under load it may not have, in which case +the test measures the *queued* path while claiming the running one. And +its 50ms clock starts before the second dispatch and is consumed by the +test's own `tick()` + `sleep(1ms)` pump, so the interval is dominated by +when *the test* was scheduled. **Widening the budget would make it pass +and measure nothing more.** + +### R2 — USR1 delivered before the trap is installed + +| field | value | +|---|---| +| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | +| **job / flavor** | macOS / lua54 | +| **required fragments** | `leader=exited(` + `SIGUSR1` | +| **causal status** | **test race** | +| **evidence** | [#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | +| **retirement** | the fixture proves the trap is installed, with a witness that fails without it | + +Readiness is `ProcessEventKind::Started`, emitted at **spawn** — not when +`/bin/sh` has parsed `trap '' USR1`. SIGUSR1's default disposition is +terminate, so a signal inside that window kills the child. The fixture's +own comment states the requirement it does not enforce. + +### R3 — live-leader EPERM with an unobservable group + +| field | value | +|---|---| +| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | +| **job / flavor** | macOS / lua54 | +| **required fragments** | `EPERM` **and** `measured_group=unobservable(` **and** (`ESRCH` / `No such process`) **and** `leader=live` | +| **causal status** | **UNRESOLVED — possible product defect** | +| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | +| **retirement** | **diagnosis and disposition by the process-signal / reap-ledger lanes. Never a green rerun.** | + +**Same test as R2, different mechanism, different status.** This is the +group-target behaviour #176 and #200 circled and the reap-ledger lane +parked every disposition change pending: a group-directed `kill` returned +EPERM while the leader was observed live, and `measured_group` — the one +field able to disagree — could not be read at all. + +**Do not treat a red matching this row as environmental.** A green rerun +changes nothing about it. + +### R4 — readiness predicate satisfied by an empty file + +| field | value | +|---|---| +| **selector** | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | +| **job / flavor** | macOS / luajit | +| **required fragments** | `left: []` **and** `right: [49]` | +| **causal status** | **test race** | +| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | +| **retirement** | `wait_for_file` requires the expected content, with a witness that fails against a zero-byte file | + +`wait_for_file` returns as soon as `fs::read` succeeds — which succeeds +on a **zero-byte file**. The probe writes readiness with +`open(path,'wb').write(b'1')`, and `open()` creates the file before +`write()` fills it. The predicate is "readable"; the assertion is +"contains `1`" (`49` is ASCII `'1'`). + +--- + +## Audited incumbents — mechanism named, no occurrence recorded + +These were carried in the handoff's hazards list **without evidence**. +The audit found the tests real and the mechanisms plausible, so they are +neither confirmed nor deleted: they are recorded honestly as unmeasured. + +**Nothing here confers "known flaky".** A red matching one of these is a +first recorded occurrence and should be investigated, not reruns-to-green. + +### R5 — GPU terminal cell background did not paint + +| field | value | +|---|---| +| **selector** | `-p pmacs-gpu a33_headless_terminal_frame_paints_cells_without_document_layers` | +| **job / flavor** | GPU Render (headless), under parallel load | +| **required fragments** | `the terminal cell background did not paint` + `blue pixels` | +| **causal status** | **mechanism named, no occurrence recorded** | +| **evidence** | **none linked.** The assertion string is real (`pmacs-gpu/src/main.rs:17973`); no run was ever cited | +| **retirement** | a linked occurrence promotes this row; absence retires nothing, because nothing was measured | + +### R6 — supervisor reap across cycles + +| field | value | +|---|---| +| **selector** | `--test m6_8_multi_repl_acceptance m6_8_supervisor_reaps_all_children_across_cycles` | +| **job / flavor** | not recorded | +| **required fragments** | **not recorded** — no signature was ever captured | +| **causal status** | **mechanism named, no occurrence recorded** ("timing-based", 10 cycles) | +| **evidence** | **none linked** | +| **retirement** | a linked occurrence with a signature promotes this row | + +**R6 cannot currently be matched.** Without required fragments there is +no rule to apply, so a red in this test is a new incident by default. +That is the correct outcome for an entry that never carried evidence. + +--- + +## Occurrence log + +| date | run | row | outcome | +|---|---|---|---| +| 2026-08-04 | [30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | R1 | rerun green — intermittence only | +| 2026-08-04 | [30927084982 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | R2 | rerun green — intermittence only | +| 2026-08-04 | [30932558752 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | R3, R4 | rerun green — intermittence only; **R3 remains unresolved** | + +Four incidents, three tests, **four signatures**. Count signatures: the +process test contributed two, and only one of them is a test bug. + +The #214 occurrence is the strongest available evidence that these are +not caused by the PRs they appeared on — that PR is **docs-only and its +tree is byte-identical to a green `main`**. It is not evidence that any +of them is harmless. From f76897c28460e6a3b85ef974a317eb98859a238d Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 5 Aug 2026 10:52:07 +0200 Subject: [PATCH 4/5] review round 1: audit notes are not rows, and R2's signature was weakened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five corrections, one blocking. BLOCKING — the contract, not the implementation, was what needed changing. Revision 2's acceptance 3 offered a binary: carry an incumbent in with a signature and evidence, or remove it as never substantiated. Stage 1 shipped a THIRD state because both incumbents are neither. The framing is now revision 3 and names all three states, because a governing criterion that says "two" while the branch does five is the framing describing something that does not exist. "MECHANISM NAMED" OVERSTATED THE EVIDENCE, and the phrase is retired. The a33 audit proves an assertion string exists; the m6_8 audit proves a test is timing-based. NEITHER ESTABLISHES A FAILURE MECHANISM — no occurrence was ever observed, so nothing is known about how, or whether, either fails. They are now AUDIT NOTES A1/A2 rather than registry rows, with `R`-numbers reserved for signatures carrying linked evidence. The distinction is not row-versus-weaker-row: a row says "this was seen, here is the evidence", a note says "someone recorded a belief and no occurrence backs it". Both remain unmatchable, so a red in either test is a new incident. R2's SIGNATURE WAS WEAKENED AND IS RESTORED. Splitting `leader=exited(signal SIGUSR1)` into `leader=exited(` plus `SIGUSR1` would match a child that exited by some OTHER disposition while SIGUSR1 appeared elsewhere in the output — precisely the name-style over-matching this registry exists to refuse, reintroduced one level down as fragment-style over-matching. It is one exact fragment again, and the row says why. THE HANDOFF STILL ISSUED LIVE IMPERATIVES. Its historical block opened with "rerun isolated before treating a sweep failure as a regression" and closed with "rerun the test alone before investigating", so the supersession note I added sat between two instructions it contradicted. Both are rewritten as record: the block now reports what that lane OBSERVED, and the retired instruction is marked retired with its reason — an isolated green reproduces nothing about a load-sensitive failure. #215 HAD NO LANE, in the file that requires one for every open PR and records why: #171 drifted 153 commits while invisible there. That is the same defect, caught in review rather than 153 commits later. The lane now carries branch, base, PR, reviewed head, what Stage 1 ships, the verification, what Stage 2 owes, and the recovery worktree command. Both snapshot headers are bumped. Also narrowed a claim the PR body overstated: the registry is NOT macOS-only. All four EVIDENCED rows are macOS, which is a property of these occurrences; A1's job is `GPU Render (headless)`, which runs on Ubuntu, and A2's job was never recorded. A future row from any job belongs in the same table. Verified: fmt, diff-check, clippy, --lib 1896, --lib --features crdt 2081. Co-Authored-By: Claude Opus 5 (1M context) --- docs/active-work.md | 60 ++++++++++++++++++++++ docs/agent-handoff.md | 28 ++++++---- docs/ci-red-signatures.md | 62 ++++++++++++++++------- docs/macos-ci-signal-integrity-framing.md | 53 +++++++++++++++---- 4 files changed, 165 insertions(+), 38 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index 24e157b..969532e 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -13,6 +13,10 @@ CI CRDT coverage**, which had been sitting under "NEEDS A LANE" with no branch and no owner since #166. It is implemented on `ci-crdt-coverage` and its block replaces the old one below. +**Updated 2026-08-05.** One lane opened: **macOS CI signal integrity** +(#215, in review), which this file required a lane for and did not have +until review caught it — the #171 defect recurring. Its block is below. + **Updated 2026-08-04.** Four PRs landed since: the CI CRDT coverage lane #209, Distribution Stage 1 #211 (released as **v1.1.0**), the post-release accuracy pass #212, and **bottom-panel Stage 3 #213 — @@ -157,6 +161,62 @@ form. All four steps ran clean. **The two-argument form still does not work** for a remote-only branch (`fatal: invalid reference`), which is why every lane below spells out the `-b` form. +## macOS CI signal integrity — STAGE 1 IN REVIEW, PR #215 + +**This file requires a lane for every open PR** (see the #171/#174 note +above: an open PR is exactly the volatile work this file records, and +#171 drifted 153 commits while invisible here). #215 had none until this +entry — the same defect, caught in review. + +- **Branch `macos-ci-signal-integrity`**, base `githubsucks/main` @ + `bfb97c6`. Framing `docs/macos-ci-signal-integrity-framing.md` + **revision 3**. +- **PR: ** — Stage 1, + docs only. Reviewed at head `d33bf4d` with all 14 checks green + ([run 30950108477](https://github.com/levineuwirth/pmacs/actions/runs/30950108477)); + one review round since. +- **What Stage 1 ships:** `docs/ci-red-signatures.md`, the single + authority for judging a red CI run. Rows key on **signature** — + selector + job/flavor + every required fragment, normalized — so **a + test-name match confers nothing**. The rerun rule is replaced: a green + rerun establishes **intermittence only**; the same signature again is + a second occurrence and stays blocking; a different signature is a new + incident. Retirement is **causal**, never a count of green runs. +- **Four incidents, three tests, four signatures.** The process test + produced two — one a test race (R2), one an **unresolved possible + product defect** (R3) that no rerun can clear. Counting by test name + would have hidden the second behind the first. +- **The audit found a third state the framing did not allow**, which is + why the framing is at revision 3: two incumbents have a historical + claim but **no linked occurrence ever**. They are **audit notes A1/A2**, + not rows — unmatchable by construction, so a red in either is a new + incident. The registry is therefore *stricter* than the list it + replaces: nothing is pre-excused. +- **Verification:** fmt, diff-check, clippy with and without `crdt`, + `--lib` 1896, `--lib --features crdt` 2081, pmacs-protocol 19, m4 149, + required GPU 221. All three named tests pass locally on Linux — + consistent with R1–R4 being macOS occurrences, and **not evidence + about any of them**. + +### Stage 2 — hardening, NOT started + +Waits on #215's merge. Owns R2's trap-readiness fix and R4's +`wait_for_file` predicate, each with a **repetition set** rather than a +single green run, plus a discriminating witness that fails without the +fix. **R1 is referred to the async-runtime lane** (Q#MCI3) rather than +patched: widening its budget would make it pass and measure nothing +more. **R3 stays unresolved** and belongs to the process-signal / +reap-ledger lanes. + +Recovery from a clean checkout: + +```sh +git fetch githubsucks +git worktree add ../pmacs-ci-signals \ + -b macos-ci-signal-integrity \ + githubsucks/macos-ci-signal-integrity +``` + ## CI CRDT coverage — MERGED (#209); kept for its three follow-ons **Rewritten, not removed.** Rule 4 removes a lane when its ARC is done; diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index 7c0d432..b165193 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1,6 +1,10 @@ # Agent handoff — cross-machine continuity -**Last updated: 2026-08-04, as bottom-panel Stage 3 #213 — the adopter +**Last updated: 2026-08-05.** The live CI-triage rule in §5 now points +at `docs/ci-red-signatures.md` (PR #215, in review), which keys on +signature rather than test name; the hazards list this file used to +carry is retired, and its two unevidenced entries are audit notes there. +Previously **2026-08-04, as bottom-panel Stage 3 #213 — the adopter default flip, which COMPLETES ARC 7: omitting `display` now means the panel, and the workbench's panel half is done on both frontends. Beneath it the post-release accuracy pass #212 and Distribution Stage 1 #211 — @@ -2003,12 +2007,14 @@ before trusting them: separate proposals, each owed its own evidence. - **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan, `PMACS_REQUIRE_GPU=1` works without lavapipe. -- **Flaky-under-load tests — rerun isolated before treating a sweep - failure as a regression.** *(Historical, from this lane. For LIVE - triage of a red run use `docs/ci-red-signatures.md`, whose rerun rule - supersedes "rerun isolated" — a green rerun proves intermittence only. - `m6_8_supervisor_reaps_all_children_across_cycles` is R6 there, with - no signature ever captured.)* The m8 daemon tests and the m6 +- **Flaky-under-load tests — what this lane OBSERVED.** *(Historical + record. This bullet no longer states a triage rule: for judging a red + run, `docs/ci-red-signatures.md` is the authority, and its rule + supersedes "rerun isolated" — a green rerun establishes intermittence + only, never environmental cause. + `m6_8_supervisor_reaps_all_children_across_cycles` is **A2** there — + an audit note, not a matchable row, because no signature was ever + captured.)* The m8 daemon tests and the m6 process/PTY tests (`m6_1_pty_mode_lifecycle_started_then_exited`, `m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based; `editor::composition_overhead_under_ten_percent` is a render-ratio @@ -2017,9 +2023,11 @@ before trusting them: `real_tui_terminal_smoke_restores_host_after_output_input_resize_scroll_copy_and_bell`; the complete failed-job rerun passed. The required-GPU gate also failed once in `headless_diag_face_recolors_band_counter_despite_unchanged_text`, then - passed both an isolated single-thread rerun and the full 139-test rerun. A - lone timing failure → rerun the test alone (`-- --test-threads=1`) before - investigating. Run the workspace + passed both an isolated single-thread rerun and the full 139-test rerun. + *(That "rerun the test alone before investigating" instruction is + retired — an isolated green reproduces nothing about a load-sensitive + failure and establishes intermittence at most. See the registry's + rerun rule.)* Run the workspace sweep as ONE `cargo test` invocation piped to a full log — a double invocation + `grep -c "test result: ok"` can mask real failures with a misleading `0`. diff --git a/docs/ci-red-signatures.md b/docs/ci-red-signatures.md index 45adff8..3c702f9 100644 --- a/docs/ci-red-signatures.md +++ b/docs/ci-red-signatures.md @@ -69,7 +69,11 @@ by removing or explaining its mechanism: | **test race** | hardening that removes the named mechanism, plus a discriminating witness for the stronger predicate | | **measurement design** | the owning lane replaces or justifies the measurement and pins the resulting claim | | **unresolved** | diagnosis and an explicit disposition | -| **mechanism named, no occurrence** | a linked occurrence promotes it to one of the above; sustained absence is not retirement, because nothing was ever measured | + +**Audit notes (`A`-numbers) have no retirement condition**, because they +have nothing to retire — see that section. A linked occurrence promotes +one into an `R` row; absence retires nothing, because nothing was ever +measured. Main-branch greens are **occurrence evidence** and accumulate toward a rate. They retire nothing by themselves. Retired rows stay in this file @@ -104,7 +108,7 @@ and measure nothing more.** |---|---| | **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | | **job / flavor** | macOS / lua54 | -| **required fragments** | `leader=exited(` + `SIGUSR1` | +| **required fragments** | `leader=exited(signal SIGUSR1)` — **one exact fragment, not two loose ones**. Split into `leader=exited(` and `SIGUSR1` it would match a child that exited by some *other* disposition while `SIGUSR1` appeared elsewhere in the output | | **causal status** | **test race** | | **evidence** | [#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | | **retirement** | the fixture proves the trap is installed, with a witness that fails without it | @@ -153,40 +157,54 @@ on a **zero-byte file**. The probe writes readiness with --- -## Audited incumbents — mechanism named, no occurrence recorded +## Audit notes — historical claims with no linked occurrence -These were carried in the handoff's hazards list **without evidence**. -The audit found the tests real and the mechanisms plausible, so they are -neither confirmed nor deleted: they are recorded honestly as unmeasured. +**These are NOT registry rows.** They carry `A`-numbers, not `R`-numbers, +because nothing here can be matched against a red run and nothing here +confers any status. -**Nothing here confers "known flaky".** A red matching one of these is a -first recorded occurrence and should be investigated, not reruns-to-green. +They were named in the handoff's hazards list without evidence. The audit +found the tests real and the claims recorded in good faith — but **an +assertion string existing is not a mechanism, and "timing-based" is not +an observation.** No occurrence of either was ever linked, so nothing is +known about how either fails, or whether either has failed. -### R5 — GPU terminal cell background did not paint +Deleting them would discard a real recorded belief. Listing them beside +the evidenced rows would grant the reputation this file exists to deny. +So they are stated as what they are: **claims awaiting a first +occurrence.** A red in either test is a first recorded occurrence, to be +investigated and then promoted to an `R` row — not matched against +anything here. + +### A1 — GPU terminal cell background did not paint | field | value | |---|---| | **selector** | `-p pmacs-gpu a33_headless_terminal_frame_paints_cells_without_document_layers` | | **job / flavor** | GPU Render (headless), under parallel load | | **required fragments** | `the terminal cell background did not paint` + `blue pixels` | -| **causal status** | **mechanism named, no occurrence recorded** | -| **evidence** | **none linked.** The assertion string is real (`pmacs-gpu/src/main.rs:17973`); no run was ever cited | -| **retirement** | a linked occurrence promotes this row; absence retires nothing, because nothing was measured | +| **status** | **historical claim, no linked occurrence** | +| **what IS established** | the test exists and the assertion string is real (`pmacs-gpu/src/main.rs:17973`). That is all | +| **what is NOT** | any mechanism, and any occurrence. No run was ever cited | +| **promotion** | a linked occurrence makes this an `R` row with a signature. Absence retires nothing, because nothing was measured | -### R6 — supervisor reap across cycles +### A2 — supervisor reap across cycles | field | value | |---|---| | **selector** | `--test m6_8_multi_repl_acceptance m6_8_supervisor_reaps_all_children_across_cycles` | | **job / flavor** | not recorded | | **required fragments** | **not recorded** — no signature was ever captured | -| **causal status** | **mechanism named, no occurrence recorded** ("timing-based", 10 cycles) | -| **evidence** | **none linked** | -| **retirement** | a linked occurrence with a signature promotes this row | +| **status** | **historical claim, no linked occurrence** | +| **what IS established** | the test exists and runs 10 cycles; the handoff called it "timing-based" | +| **what is NOT** | any mechanism, any signature, any occurrence | +| **promotion** | a linked occurrence *with a captured signature* makes this an `R` row | -**R6 cannot currently be matched.** Without required fragments there is -no rule to apply, so a red in this test is a new incident by default. -That is the correct outcome for an entry that never carried evidence. +**A2 cannot be matched, and neither can A1** — that is what makes them +notes rather than rows. A red in either test is a new incident by +default. That is the correct outcome for an entry that never carried +evidence, and it means this file is **stricter** than the list it +replaces: nothing is pre-excused. --- @@ -201,6 +219,12 @@ That is the correct outcome for an entry that never carried evidence. Four incidents, three tests, **four signatures**. Count signatures: the process test contributed two, and only one of them is a test bug. +**All four *evidenced* rows (R1–R4) are macOS.** That is a property of +these occurrences, not of the file: **A1's job is `GPU Render +(headless)`, which runs on Ubuntu**, and **A2's job was never +recorded**. Nothing here is macOS-only by construction, and a future +row from any job belongs in the same table. + The #214 occurrence is the strongest available evidence that these are not caused by the PRs they appeared on — that PR is **docs-only and its tree is byte-identical to a green `main`**. It is not evidence that any diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md index c3ba89d..b7d1a40 100644 --- a/docs/macos-ci-signal-integrity-framing.md +++ b/docs/macos-ci-signal-integrity-framing.md @@ -1,7 +1,27 @@ # Framing — macOS CI signal integrity: a signature registry, then hardening -**Revision 2.** Status: framing only. No branch work beyond this -document. Scouted against `githubsucks/main` @ `bfb97c6`. Revision 2 +**Revision 3.** Status: **Stage 1 implemented** on +`macos-ci-signal-integrity`, PR #215. Scouted against +`githubsucks/main` @ `bfb97c6`. + +**Revision 2 → 3** exists because the implementation discovered a state +the contract did not allow, and the contract — not the implementation — +was what needed changing. Acceptance 3 offered a binary: carry an +incumbent in with a signature and evidence, or remove it as never +substantiated. **Both incumbents are neither**, and shipping a third +state while the governing criterion still said "two" would have made the +framing describe something the branch does not do. + +Revision 3 also **retires the phrase "mechanism named"**, which +overstated what the audit found. The a33 audit proves an assertion +string and a historical claim exist; the m6_8 audit proves a test is +timing-based. +**Neither establishes a failure mechanism** — no occurrence was ever +observed, so nothing is known about how either fails. They become +**audit notes A1/A2**, not registry rows, and `R`-numbers are reserved +for signatures with linked evidence. + +Revision 2 separates a machine-matchable signature from verbatim, variable CI output; preserves historical incident evidence while centralizing live triage policy; gives retirement a causal rule rather than an arbitrary @@ -257,10 +277,21 @@ everything else. may gain a registry status link, but its evidence and reasoning are not replaced. Forward-looking risk statements are audited as risks, not silently promoted to known flakes. -3. **The three tests named in the current handoff list are audited**: - each is either carried into the registry with a signature and - evidence, or removed with a note saying it was never substantiated. - No entry survives on reputation. +3. **The three tests named in the current handoff list are audited**, + into one of **three** states — the third was found by doing the audit + and is why this is revision 3: + - **carried as a registry row** (`R`-numbered) when a signature and a + linked occurrence both exist; + - **removed** when nothing substantiates it at all; + - **recorded as an audit note** (`A`-numbered) when a *historical + claim* exists but **no occurrence was ever linked**. An audit note + is not a registry row, cannot be matched against a red run, and + confers nothing. + + **No entry survives on reputation, and an audit note is not a weaker + row — it is a different kind of statement.** A row says "this was + seen, here is the evidence"; a note says "someone recorded a belief + and no occurrence backs it. 4. **The rerun rule is replaced**, not softened: - a **green rerun after a red** establishes intermittence only; it does not establish environmental cause, harmlessness, or retirement; @@ -304,9 +335,13 @@ everything else. recorded in the CI CRDT lane. Related in spirit, separate in scope. - **A general flake-rate dashboard.** The registry accumulates occurrences; turning that into a rate with alerting is its own thing. -- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...` and - `m6_8_supervisor_reaps_...` are in the current handoff list and get - audited under acceptance 3, but this lane's incidents are macOS. +- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...` + and `m6_8_supervisor_reaps_...` were audited under acceptance 3 and + became **audit notes A1/A2** — claims with no linked occurrence. + **A1's job runs on Ubuntu**, so the registry is not macOS-only even + now; A2's job was never recorded. This lane's four *evidenced* + incidents are macOS, which is a fact about them and not about the + file. --- From 2e0617fdddc943eb81d5a361d4292ff0c67e3ba3 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 5 Aug 2026 11:11:57 +0200 Subject: [PATCH 5/5] review round 2: precision fixes, and a checkpoint table that cannot go stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections. "NO OCCURRENCE WAS EVER OBSERVED" OVERSTATED WHAT THE AUDIT CAN SAY. Someone may well have seen one of these fail and simply not recorded it; what is established is the absence of a RECORD. The framing now says "linked or captured" and states the distinction explicitly, because an audit that claims to know what nobody saw is making the same kind of unfounded assertion this lane exists to remove — one level up. A missing closing quotation mark in acceptance 3. THE LANE CHECKPOINTED THE REVIEWED HEAD, WHICH GOES STALE ON THE NEXT PUSH — the exact mechanism by which #171 became invisible while it drifted 153 commits. Recording one head is a snapshot; what a resume ledger needs is the sequence. It is now a table of head -> CI run -> result, newest last, with the rule stated: the branch tip is authoritative over any row, verified by `git rev-parse`, because the table is written by hand and the tip is not. Verified: fmt, diff-check. Co-Authored-By: Claude Opus 5 (1M context) --- docs/active-work.md | 16 +++++++++++++--- docs/macos-ci-signal-integrity-framing.md | 7 +++++-- 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index 969532e..1ba0422 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -172,9 +172,19 @@ entry — the same defect, caught in review. `bfb97c6`. Framing `docs/macos-ci-signal-integrity-framing.md` **revision 3**. - **PR: ** — Stage 1, - docs only. Reviewed at head `d33bf4d` with all 14 checks green - ([run 30950108477](https://github.com/levineuwirth/pmacs/actions/runs/30950108477)); - one review round since. + docs only. **Checkpoints, newest last** — a lane that records only the + *reviewed* head goes stale the moment a review round is pushed, which + is how #171 became invisible: + + | head | CI run | result | + |---|---|---| + | `d33bf4d` | [30950108477](https://github.com/levineuwirth/pmacs/actions/runs/30950108477) | 14/14 green — the reviewed head | + | `f76897c` | [30990816890](https://github.com/levineuwirth/pmacs/actions/runs/30990816890) | review round 1 | + | *(tip)* | — | review round 2: framing wording, the missing quote, this table | + + **The branch tip is authoritative over any row here.** Verify with + `git rev-parse githubsucks/macos-ci-signal-integrity` rather than + trusting the newest line. - **What Stage 1 ships:** `docs/ci-red-signatures.md`, the single authority for judging a red CI run. Rows key on **signature** — selector + job/flavor + every required fragment, normalized — so **a diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md index b7d1a40..56da71a 100644 --- a/docs/macos-ci-signal-integrity-framing.md +++ b/docs/macos-ci-signal-integrity-framing.md @@ -17,7 +17,10 @@ overstated what the audit found. The a33 audit proves an assertion string and a historical claim exist; the m6_8 audit proves a test is timing-based. **Neither establishes a failure mechanism** — no occurrence was ever -observed, so nothing is known about how either fails. They become +**linked or captured**, so nothing is known about how either fails. +(Not "never observed": someone may well have seen one and not recorded +it. What is established is the absence of a *record*, which is the only +thing this audit can speak to.) They become **audit notes A1/A2**, not registry rows, and `R`-numbers are reserved for signatures with linked evidence. @@ -291,7 +294,7 @@ everything else. **No entry survives on reputation, and an audit note is not a weaker row — it is a different kind of statement.** A row says "this was seen, here is the evidence"; a note says "someone recorded a belief - and no occurrence backs it. + and no occurrence backs it." 4. **The rerun rule is replaced**, not softened: - a **green rerun after a red** establishes intermittence only; it does not establish environmental cause, harmlessness, or retirement;