diff --git a/docs/active-work.md b/docs/active-work.md index 733ca6d..1ba0422 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -13,6 +13,10 @@ CI CRDT coverage**, which had been sitting under "NEEDS A LANE" with no branch and no owner since #166. It is implemented on `ci-crdt-coverage` and its block replaces the old one below. +**Updated 2026-08-05.** One lane opened: **macOS CI signal integrity** +(#215, in review), which this file required a lane for and did not have +until review caught it — the #171 defect recurring. Its block is below. + **Updated 2026-08-04.** Four PRs landed since: the CI CRDT coverage lane #209, Distribution Stage 1 #211 (released as **v1.1.0**), the post-release accuracy pass #212, and **bottom-panel Stage 3 #213 — @@ -157,6 +161,72 @@ form. All four steps ran clean. **The two-argument form still does not work** for a remote-only branch (`fatal: invalid reference`), which is why every lane below spells out the `-b` form. +## macOS CI signal integrity — STAGE 1 IN REVIEW, PR #215 + +**This file requires a lane for every open PR** (see the #171/#174 note +above: an open PR is exactly the volatile work this file records, and +#171 drifted 153 commits while invisible here). #215 had none until this +entry — the same defect, caught in review. + +- **Branch `macos-ci-signal-integrity`**, base `githubsucks/main` @ + `bfb97c6`. Framing `docs/macos-ci-signal-integrity-framing.md` + **revision 3**. +- **PR: ** — Stage 1, + docs only. **Checkpoints, newest last** — a lane that records only the + *reviewed* head goes stale the moment a review round is pushed, which + is how #171 became invisible: + + | head | CI run | result | + |---|---|---| + | `d33bf4d` | [30950108477](https://github.com/levineuwirth/pmacs/actions/runs/30950108477) | 14/14 green — the reviewed head | + | `f76897c` | [30990816890](https://github.com/levineuwirth/pmacs/actions/runs/30990816890) | review round 1 | + | *(tip)* | — | review round 2: framing wording, the missing quote, this table | + + **The branch tip is authoritative over any row here.** Verify with + `git rev-parse githubsucks/macos-ci-signal-integrity` rather than + trusting the newest line. +- **What Stage 1 ships:** `docs/ci-red-signatures.md`, the single + authority for judging a red CI run. Rows key on **signature** — + selector + job/flavor + every required fragment, normalized — so **a + test-name match confers nothing**. The rerun rule is replaced: a green + rerun establishes **intermittence only**; the same signature again is + a second occurrence and stays blocking; a different signature is a new + incident. Retirement is **causal**, never a count of green runs. +- **Four incidents, three tests, four signatures.** The process test + produced two — one a test race (R2), one an **unresolved possible + product defect** (R3) that no rerun can clear. Counting by test name + would have hidden the second behind the first. +- **The audit found a third state the framing did not allow**, which is + why the framing is at revision 3: two incumbents have a historical + claim but **no linked occurrence ever**. They are **audit notes A1/A2**, + not rows — unmatchable by construction, so a red in either is a new + incident. The registry is therefore *stricter* than the list it + replaces: nothing is pre-excused. +- **Verification:** fmt, diff-check, clippy with and without `crdt`, + `--lib` 1896, `--lib --features crdt` 2081, pmacs-protocol 19, m4 149, + required GPU 221. All three named tests pass locally on Linux — + consistent with R1–R4 being macOS occurrences, and **not evidence + about any of them**. + +### Stage 2 — hardening, NOT started + +Waits on #215's merge. Owns R2's trap-readiness fix and R4's +`wait_for_file` predicate, each with a **repetition set** rather than a +single green run, plus a discriminating witness that fails without the +fix. **R1 is referred to the async-runtime lane** (Q#MCI3) rather than +patched: widening its budget would make it pass and measure nothing +more. **R3 stays unresolved** and belongs to the process-signal / +reap-ledger lanes. + +Recovery from a clean checkout: + +```sh +git fetch githubsucks +git worktree add ../pmacs-ci-signals \ + -b macos-ci-signal-integrity \ + githubsucks/macos-ci-signal-integrity +``` + ## CI CRDT coverage — MERGED (#209); kept for its three follow-ons **Rewritten, not removed.** Rule 4 removes a lane when its ARC is done; @@ -194,6 +264,11 @@ the crdt job. same commit. One of those was a real regression the flip caused; the other was load. **That ambiguity is the problem** — a job where noise and signal look alike trains people to rerun rather than read. + **The triage half of this is now owned by + `docs/ci-red-signatures.md`** — signature-keyed rows and a rerun rule + that refuses to treat a green rerun as an all-clear. What remains here + is the job-cost question: whether the crdt job should carry real-PTY + deadlines at all. Candidate: longer deadlines for real-PTY assertions in this job specifically, or serialize the PTY suites. diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index ca33f92..b165193 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1,6 +1,10 @@ # Agent handoff — cross-machine continuity -**Last updated: 2026-08-04, as bottom-panel Stage 3 #213 — the adopter +**Last updated: 2026-08-05.** The live CI-triage rule in §5 now points +at `docs/ci-red-signatures.md` (PR #215, in review), which keys on +signature rather than test name; the hazards list this file used to +carry is retired, and its two unevidenced entries are audit notes there. +Previously **2026-08-04, as bottom-panel Stage 3 #213 — the adopter default flip, which COMPLETES ARC 7: omitting `display` now means the panel, and the workbench's panel half is done on both frontends. Beneath it the post-release accuracy pass #212 and Distribution Stage 1 #211 — @@ -292,11 +296,19 @@ someone forgot. - **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error then` guards make the silence look deliberate. Report through `pmacs.editor.set_status` until the channel is built. -- **Flakes that are not your change.** Judge a red run against these - before bisecting: `process::tests::a_successful_signal_disposition_...` - (macOS-only, signal timing); `a33_headless_terminal_frame_paints_...` - (GPU under parallel load, "0 blue pixels"); `m6_8_supervisor_reaps_...` - (load-sensitive). **Rerun before concluding.** +- **Judging a red CI run: `docs/ci-red-signatures.md` is the authority.** + It keys on **signature**, not test name — a failure in a listed test + that lacks that row's fragments is a NEW incident, not a known one. + The old rule here ("rerun before concluding") is retired: **a green + rerun establishes intermittence only**, never environmental cause or + harmlessness; the same signature again is a second occurrence and + stays blocking. One row is an **unresolved possible product defect** + that no rerun can clear. + + This list previously named three tests. The audit found one of them + had produced **two distinct signatures** with different causes, and + that two of the four incidents actually seen were absent from it — + which is why name-keyed lists are not trustworthy. - **`basedpyright` hangs forever** — always `cargo test --test m4_acceptance -- --skip basedpyright`. - **The crdt sweep needs `cargo build --workspace` first**, or twelve @@ -1995,9 +2007,15 @@ before trusting them: separate proposals, each owed its own evidence. - **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan, `PMACS_REQUIRE_GPU=1` works without lavapipe. -- **Flaky-under-load tests — rerun isolated before treating a sweep - failure as a regression.** The m8 daemon tests and the m6 process/PTY - tests (`m6_1_pty_mode_lifecycle_started_then_exited`, +- **Flaky-under-load tests — what this lane OBSERVED.** *(Historical + record. This bullet no longer states a triage rule: for judging a red + run, `docs/ci-red-signatures.md` is the authority, and its rule + supersedes "rerun isolated" — a green rerun establishes intermittence + only, never environmental cause. + `m6_8_supervisor_reaps_all_children_across_cycles` is **A2** there — + an audit note, not a matchable row, because no signature was ever + captured.)* The m8 daemon tests and the m6 + process/PTY tests (`m6_1_pty_mode_lifecycle_started_then_exited`, `m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based; `editor::composition_overhead_under_ten_percent` is a render-ratio microbenchmark that fails ~1/3 even isolated single-threaded (already @@ -2005,9 +2023,11 @@ before trusting them: `real_tui_terminal_smoke_restores_host_after_output_input_resize_scroll_copy_and_bell`; the complete failed-job rerun passed. The required-GPU gate also failed once in `headless_diag_face_recolors_band_counter_despite_unchanged_text`, then - passed both an isolated single-thread rerun and the full 139-test rerun. A - lone timing failure → rerun the test alone (`-- --test-threads=1`) before - investigating. Run the workspace + passed both an isolated single-thread rerun and the full 139-test rerun. + *(That "rerun the test alone before investigating" instruction is + retired — an isolated green reproduces nothing about a load-sensitive + failure and establishes intermittence at most. See the registry's + rerun rule.)* Run the workspace sweep as ONE `cargo test` invocation piped to a full log — a double invocation + `grep -c "test result: ok"` can mask real failures with a misleading `0`. diff --git a/docs/ci-red-signatures.md b/docs/ci-red-signatures.md new file mode 100644 index 0000000..3c702f9 --- /dev/null +++ b/docs/ci-red-signatures.md @@ -0,0 +1,231 @@ +# CI red signatures — the triage registry + +**This file is the single authority for judging a red CI run.** It is an +occurrence ledger, not a flake list: a row records what was seen, what is +known about why, and what would retire it. **A row is not a claim that +the failure is harmless.** + +Deliberately not named "flakes". One of its rows is a possible product +defect, and a filename that called it a flake would confer immunity the +evidence does not support. + +Read this before attributing any red run to the environment. Landed +framing documents keep their own historical evidence and reasoning — +that is not duplication, and it is not superseded by this file. What +lives here is **live triage policy**. + +--- + +## How a row matches + +**A test-name match is never sufficient.** A red run matches a row only +when *all three* hold: + +1. the **exact test selector** matches; +2. the **job / flavor** matches; +3. **every required fragment** is present in the failure output. + +Where a fragment lists alternatives (`ESRCH` / `No such process`), any +one satisfies that requirement — those are the same condition rendered +differently by platform or libc. + +Fragments are **normalized**, never pasted verbatim. PIDs, elapsed +times, thread ids and rendered OS-error suffixes vary between runs; a +verbatim key would match nothing. The evidence link preserves the exact +occurrence. + +**A failure in a listed test that does not carry that row's fragments is +a NEW incident**, judged on its own. The process test below is why this +rule exists: it produced two signatures with different mechanisms and +different causal status, and only one of them is a test bug. + +--- + +## The rerun rule + +This replaces "rerun before concluding", which conflated three different +outcomes. + +- **A green rerun after a red establishes INTERMITTENCE ONLY.** It does + not establish environmental cause, harmlessness, or retirement. +- **The same signature on the rerun is a SECOND OCCURRENCE.** It remains + blocking pending investigation or a merge-base control. +- **A different signature is a NEW INCIDENT**, judged independently. + +A merge-base control — running the same command on the merge base — is +what distinguishes "this branch caused it" from "this tree has it". It +is cheaper than argument and is the correct response to a second +occurrence. + +--- + +## What retires a row + +**Retirement is causal, never a count of green runs.** A row is retired +by removing or explaining its mechanism: + +| causal status | retirement condition | +|---|---| +| **test race** | hardening that removes the named mechanism, plus a discriminating witness for the stronger predicate | +| **measurement design** | the owning lane replaces or justifies the measurement and pins the resulting claim | +| **unresolved** | diagnosis and an explicit disposition | + +**Audit notes (`A`-numbers) have no retirement condition**, because they +have nothing to retire — see that section. A linked occurrence promotes +one into an `R` row; absence retires nothing, because nothing was ever +measured. + +Main-branch greens are **occurrence evidence** and accumulate toward a +rate. They retire nothing by themselves. Retired rows stay in this file +with their disposition, so a recurrence is recognisable. + +--- + +## Live rows + +### R1 — supersede cancellation budget + +| field | value | +|---|---| +| **selector** | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | +| **job / flavor** | macOS / luajit | +| **required fragments** | `supersede did not cancel within 50ms` | +| **causal status** | **measurement design** | +| **evidence** | [#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | +| **retirement** | the async-runtime lane replaces or justifies the measurement (Q#MCI3) | + +The test's premise is `thread::sleep(15ms)`, asserted by comment to mean +"the worker picked the job up"; under load it may not have, in which case +the test measures the *queued* path while claiming the running one. And +its 50ms clock starts before the second dispatch and is consumed by the +test's own `tick()` + `sleep(1ms)` pump, so the interval is dominated by +when *the test* was scheduled. **Widening the budget would make it pass +and measure nothing more.** + +### R2 — USR1 delivered before the trap is installed + +| field | value | +|---|---| +| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | +| **job / flavor** | macOS / lua54 | +| **required fragments** | `leader=exited(signal SIGUSR1)` — **one exact fragment, not two loose ones**. Split into `leader=exited(` and `SIGUSR1` it would match a child that exited by some *other* disposition while `SIGUSR1` appeared elsewhere in the output | +| **causal status** | **test race** | +| **evidence** | [#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | +| **retirement** | the fixture proves the trap is installed, with a witness that fails without it | + +Readiness is `ProcessEventKind::Started`, emitted at **spawn** — not when +`/bin/sh` has parsed `trap '' USR1`. SIGUSR1's default disposition is +terminate, so a signal inside that window kills the child. The fixture's +own comment states the requirement it does not enforce. + +### R3 — live-leader EPERM with an unobservable group + +| field | value | +|---|---| +| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | +| **job / flavor** | macOS / lua54 | +| **required fragments** | `EPERM` **and** `measured_group=unobservable(` **and** (`ESRCH` / `No such process`) **and** `leader=live` | +| **causal status** | **UNRESOLVED — possible product defect** | +| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | +| **retirement** | **diagnosis and disposition by the process-signal / reap-ledger lanes. Never a green rerun.** | + +**Same test as R2, different mechanism, different status.** This is the +group-target behaviour #176 and #200 circled and the reap-ledger lane +parked every disposition change pending: a group-directed `kill` returned +EPERM while the leader was observed live, and `measured_group` — the one +field able to disagree — could not be read at all. + +**Do not treat a red matching this row as environmental.** A green rerun +changes nothing about it. + +### R4 — readiness predicate satisfied by an empty file + +| field | value | +|---|---| +| **selector** | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | +| **job / flavor** | macOS / luajit | +| **required fragments** | `left: []` **and** `right: [49]` | +| **causal status** | **test race** | +| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | +| **retirement** | `wait_for_file` requires the expected content, with a witness that fails against a zero-byte file | + +`wait_for_file` returns as soon as `fs::read` succeeds — which succeeds +on a **zero-byte file**. The probe writes readiness with +`open(path,'wb').write(b'1')`, and `open()` creates the file before +`write()` fills it. The predicate is "readable"; the assertion is +"contains `1`" (`49` is ASCII `'1'`). + +--- + +## Audit notes — historical claims with no linked occurrence + +**These are NOT registry rows.** They carry `A`-numbers, not `R`-numbers, +because nothing here can be matched against a red run and nothing here +confers any status. + +They were named in the handoff's hazards list without evidence. The audit +found the tests real and the claims recorded in good faith — but **an +assertion string existing is not a mechanism, and "timing-based" is not +an observation.** No occurrence of either was ever linked, so nothing is +known about how either fails, or whether either has failed. + +Deleting them would discard a real recorded belief. Listing them beside +the evidenced rows would grant the reputation this file exists to deny. +So they are stated as what they are: **claims awaiting a first +occurrence.** A red in either test is a first recorded occurrence, to be +investigated and then promoted to an `R` row — not matched against +anything here. + +### A1 — GPU terminal cell background did not paint + +| field | value | +|---|---| +| **selector** | `-p pmacs-gpu a33_headless_terminal_frame_paints_cells_without_document_layers` | +| **job / flavor** | GPU Render (headless), under parallel load | +| **required fragments** | `the terminal cell background did not paint` + `blue pixels` | +| **status** | **historical claim, no linked occurrence** | +| **what IS established** | the test exists and the assertion string is real (`pmacs-gpu/src/main.rs:17973`). That is all | +| **what is NOT** | any mechanism, and any occurrence. No run was ever cited | +| **promotion** | a linked occurrence makes this an `R` row with a signature. Absence retires nothing, because nothing was measured | + +### A2 — supervisor reap across cycles + +| field | value | +|---|---| +| **selector** | `--test m6_8_multi_repl_acceptance m6_8_supervisor_reaps_all_children_across_cycles` | +| **job / flavor** | not recorded | +| **required fragments** | **not recorded** — no signature was ever captured | +| **status** | **historical claim, no linked occurrence** | +| **what IS established** | the test exists and runs 10 cycles; the handoff called it "timing-based" | +| **what is NOT** | any mechanism, any signature, any occurrence | +| **promotion** | a linked occurrence *with a captured signature* makes this an `R` row | + +**A2 cannot be matched, and neither can A1** — that is what makes them +notes rather than rows. A red in either test is a new incident by +default. That is the correct outcome for an entry that never carried +evidence, and it means this file is **stricter** than the list it +replaces: nothing is pre-excused. + +--- + +## Occurrence log + +| date | run | row | outcome | +|---|---|---|---| +| 2026-08-04 | [30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | R1 | rerun green — intermittence only | +| 2026-08-04 | [30927084982 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | R2 | rerun green — intermittence only | +| 2026-08-04 | [30932558752 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | R3, R4 | rerun green — intermittence only; **R3 remains unresolved** | + +Four incidents, three tests, **four signatures**. Count signatures: the +process test contributed two, and only one of them is a test bug. + +**All four *evidenced* rows (R1–R4) are macOS.** That is a property of +these occurrences, not of the file: **A1's job is `GPU Render +(headless)`, which runs on Ubuntu**, and **A2's job was never +recorded**. Nothing here is macOS-only by construction, and a future +row from any job belongs in the same table. + +The #214 occurrence is the strongest available evidence that these are +not caused by the PRs they appeared on — that PR is **docs-only and its +tree is byte-identical to a green `main`**. It is not evidence that any +of them is harmless. diff --git a/docs/macos-ci-signal-integrity-framing.md b/docs/macos-ci-signal-integrity-framing.md new file mode 100644 index 0000000..56da71a --- /dev/null +++ b/docs/macos-ci-signal-integrity-framing.md @@ -0,0 +1,370 @@ +# Framing — macOS CI signal integrity: a signature registry, then hardening + +**Revision 3.** Status: **Stage 1 implemented** on +`macos-ci-signal-integrity`, PR #215. Scouted against +`githubsucks/main` @ `bfb97c6`. + +**Revision 2 → 3** exists because the implementation discovered a state +the contract did not allow, and the contract — not the implementation — +was what needed changing. Acceptance 3 offered a binary: carry an +incumbent in with a signature and evidence, or remove it as never +substantiated. **Both incumbents are neither**, and shipping a third +state while the governing criterion still said "two" would have made the +framing describe something the branch does not do. + +Revision 3 also **retires the phrase "mechanism named"**, which +overstated what the audit found. The a33 audit proves an assertion +string and a historical claim exist; the m6_8 audit proves a test is +timing-based. +**Neither establishes a failure mechanism** — no occurrence was ever +**linked or captured**, so nothing is known about how either fails. +(Not "never observed": someone may well have seen one and not recorded +it. What is established is the absence of a *record*, which is the only +thing this audit can speak to.) They become +**audit notes A1/A2**, not registry rows, and `R`-numbers are reserved +for signatures with linked evidence. + +Revision 2 +separates a machine-matchable signature from verbatim, variable CI +output; preserves historical incident evidence while centralizing live +triage policy; gives retirement a causal rule rather than an arbitrary +green-run count; and corrects the rerun rule. + +Four red CI incidents across #213 and #214 were each correctly judged +"not caused by this PR" — and #214's case is airtight, because it is +docs-only and its tree is byte-identical to a green `main`. **That +proves the PRs did not cause them. It does not prove they are harmless +environmental noise, and three of the four have a specific, findable +mechanism.** + +This lane separates those two claims, which the current process +conflates. + +--- + +## 0. Coherence impact (COHERENCE §20) + +- **Journey steps touched:** none. +- **Interaction islands:** none. +- **Config registry adoption:** none. +- **Background-work attribution:** none. +- **Why it belongs on the board:** it protects §19's acceptance-test + ratchet and every arc that reads a red run. A flake list that is + wrong, incomplete, or keyed by test *name* rather than *signature* + makes "rerun before concluding" a habit rather than a judgement — and + a real regression arriving in that stream is indistinguishable from + the noise it hides in. + +--- + +## 1. Ground truth (measured at `bfb97c6`) + +### 1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures + +The registry must count **signatures**, not test names. The process test +alone produced two, with different mechanisms and different causal +status — collapsing them under one name would have hidden a possible +product defect behind a known-flaky label. + +The registry key is an exact **test selector + job/flavor + normalized +match rule**, not a pasted panic block. PIDs, elapsed times, and rendered +OS-error suffixes vary between runs; each row therefore names the +required invariant fragments and the evidence link preserves the +verbatim occurrence. A row matches only when every listed requirement +is satisfied; where a requirement lists alternative platform renderings, +one of those alternatives suffices. A test-name match by itself never +matches a registry entry. + +| # | exact test selector | job / flavor | required signature fragments | causal status | +|---|---|---|---|---| +| 1 | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 | +| 2 | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 | +| 3 | *(same test)* | macOS / lua54 | all of `EPERM`, `measured_group=unobservable(` + `ESRCH` / `No such process`, and `leader=live` | **UNRESOLVED — possible product defect** — see §1.4 | +| 4 | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | both `left: []` and `right: [49]` | **test race** — see §1.5 | + +Evidence: +[#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642), +[#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1), +[#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1). + +### 1.2 Signature 1 — the 50ms budget measures more than it claims + +```rust +let first = rt.dispatch_sleep(2_000, Some("search")); +// Let the worker pick the job up so cancel hits a running job. +thread::sleep(Duration::from_millis(15)); +let started = Instant::now(); +let _second = rt.dispatch_sleep(2_000, Some("search")); +while !rt.is_complete(first) { + assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms"); + let _ = rt.tick(); + thread::sleep(Duration::from_millis(1)); +} +``` + +**Two design problems, and neither is fixed by a bigger number.** + +- **The premise is a sleep.** `thread::sleep(15ms)` is asserted-by-comment + to mean "the worker picked the job up". On a loaded runner it may not + have, in which case the test measures cancellation of a *queued* job — + a different code path, already covered by the sibling test — while + claiming to measure a running one. +- **The clock includes the observer.** `started` begins before the + second dispatch, and the budget is consumed by the test's own + `tick()` + `sleep(1ms)` pump loop. Under scheduling pressure the + measured interval is dominated by when *the test* got scheduled, not + by when the worker observed the cancel flag. + +**Widening 50ms to 200ms would make it pass and measure nothing more.** +The open question is what this test should assert: a latency bound needs +a clock the observer does not participate in, or the assertion should be +reformulated as ordering ("the first settles Cancelled before the second +completes") rather than duration. + +### 1.3 Signature 2 — `Started` does not prove the trap is installed + +```rust +spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()]; +let id = sup.spawn(spec).expect("spawn"); +let pid = spawn_started_pid(&mut sup, id); // waits for ProcessEventKind::Started +sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers"); +``` + +`spawn_started_pid` waits for `ProcessEventKind::Started { pid }`, which +is emitted when the process is **spawned** — not when `/bin/sh` has +parsed and installed `trap '' USR1`. **SIGUSR1's default disposition is +terminate**, so a signal delivered inside that window kills the child, +and the record is `exited(signal SIGUSR1)` instead of `Running`. + +The fixture's own comment states the requirement it does not enforce: +*"Ignore USR1 so the successful non-fatal signal cannot end the child +and confuse the state assertion with a real exit."* That is exactly the +confusion observed. + +This is the handoff's **"wait predicate weaker than the assertion"** +race: the predicate is "process exists", the assertion needs "trap +installed". + +### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved + +`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not +established as a test race**. It is the group-target behaviour the +process-signal lanes have circled three times: + +- #176 established that a group-directed `kill` returned EPERM while the + leader was observed alive by a real `try_wait`, retiring "EPERM cannot + happen for our own children" as a reason to discard the errno. +- #200 added `measured_group` from `getpgid`, the only field able to + disagree — and it is reported here as `unobservable(ESRCH)`, meaning + the group could not be measured at all. +- The reap-ledger lane parked every disposition change pending exactly + this evidence. + +**This lane does not resolve it and must not appear to.** Its registry +entry carries causal status **UNRESOLVED — possible product defect**, +and its retirement condition is a diagnosis, not a green rerun. Folding +it under the same "macOS signal timing" label as signature 2 is how a +real defect acquires a flake's immunity. + +### 1.5 Signature 4 — a readiness predicate satisfied by an empty file + +```rust +fn wait_for_file(path: &Path, timeout: Duration) -> Vec { + loop { + if let Ok(bytes) = fs::read(path) { return bytes; } // succeeds on 0 bytes + ... + } +} +``` + +The probe writes readiness with `open(path,'wb').write(b'1')`. **`open()` +creates the file before `write()` fills it**, so `fs::read` can succeed +on a zero-byte file and `wait_for_file` returns `[]`. The caller then +asserts `== b"1"` and fails `left: [], right: [49]`. + +The predicate is "readable"; the assertion is "contains `1`". This is +the same shape as §1.3 and is fixable at the helper. All four callers of +this helper require concrete non-empty content; the similar bottom-panel +helper already rejects empty reads. + +### 1.6 Live triage policy and historical evidence are mixed together + +Flake language currently appears in the handoff's two operational +rules, `docs/active-work.md`, and several landed framing documents. It +is not all duplication. The process-signal and reap-ledger framings, for +example, preserve exact historical occurrences and the reasoning those +lanes built from them; replacing that evidence with a pointer would +make a durable framing depend on a mutable registry. + +The real duplication is **live classification and triage policy**. The +handoff's hazards list names three tests — two of which are not among +the four incidents seen here — while three of these four are absent. +Elsewhere, historical occurrence notes, forward-looking risk warnings, +and current "known flaky" claims are written in the same voice even +though they require different treatment. + +A list that is both stale and incomplete is worse than none: it confers +"known flaky" on whatever happens to be named, and withholds it from +everything else. + +### 1.7 What is NOT established + +- **No incident has been reproduced locally.** All four are macOS-only + and this machine is Linux. The mechanisms in §§1.2–1.5 are read from + source and from CI signatures, not from a local repro. +- **Frequency is unmeasured.** Four incidents across two PRs is not a + rate. The registry records occurrences so a rate can accumulate; + it does not claim one now. +- **Signature 3's cause remains unknown**, by design (§1.4). + +--- + +## 2. Questions + +- **Q#MCI1 — where does the registry live? DECIDED:** one dedicated + `docs/ci-red-signatures.md`, with the handoff's operational rule + pointing at it. The handoff is a briefing; the registry is an + occurrence ledger whose rows remain available after retirement. +- **Q#MCI2 — what retires an entry? DECIDED:** a mechanism-specific + causal result, not N green runs. A known-race entry requires hardening + that removes the named mechanism plus a discriminating acceptance + witness for the stronger predicate. A measurement-design entry + requires its owning lane to replace or justify the measurement and + pin the resulting claim. An unresolved entry requires diagnosis and + an explicit disposition. Main-job greens remain occurrence evidence, + but cannot retire any row by themselves; retired rows remain in the + history with their disposition. +- **Q#MCI3 — does signature 1 get reformulated or re-measured?** §1.2 + argues its budget measures the observer. Reformulating as an ordering + assertion changes what the test proves; keeping a duration means + finding a clock the pump loop does not participate in. **This is the + one question this lane should not answer alone** — it is the + async-runtime lane's design call. +- **Q#MCI4 — does hardening ship before or with the registry? DECIDED:** + the user has already answered: **registry now, hardening next.** + Recorded here so the sequencing is visible in the document rather + than only in the conversation. + +--- + +## 3. Bets + +- **Bet 1 — signatures 2 and 4 disappear under hardening**, because both + have a named mechanism and a fix at the readiness predicate. + *Falsified if either recurs after the predicate is strengthened.* +- **Bet 2 — a signature-keyed registry refuses name-based immunity.** A + future failure in the process test which does not contain all of row + 2's or row 3's required fragments is a new incident, not a known + flake. *Falsified if the operational rule permits a test-name-only + match.* +- **Bet 3 — green reruns do not erase unresolved evidence.** Signature + 3 remains unresolved until the process-signal lane diagnoses and + disposes it, whether or not later runs pass. *Falsified if a green run + changes that row's status or retirement condition.* + +--- + +## 4. Acceptance + +**Stage 1 — the registry (this lane's first PR):** + +1. **One authoritative table**, with a row per **signature**: exact test + selector, job and Lua flavor, normalized machine-match rule, evidence + link to the verbatim occurrence, causal status, and mechanism-specific + retirement condition. Matching requires the selector, job/flavor, + and every required fragment; variable values are explicitly + normalized rather than silently abbreviated. +2. **One live triage policy.** Operational duplicate classifications and + rerun rules become pointers to the registry. Historical occurrence + evidence stays where it supports a landed framing; a relevant note + may gain a registry status link, but its evidence and reasoning are + not replaced. Forward-looking risk statements are audited as risks, + not silently promoted to known flakes. +3. **The three tests named in the current handoff list are audited**, + into one of **three** states — the third was found by doing the audit + and is why this is revision 3: + - **carried as a registry row** (`R`-numbered) when a signature and a + linked occurrence both exist; + - **removed** when nothing substantiates it at all; + - **recorded as an audit note** (`A`-numbered) when a *historical + claim* exists but **no occurrence was ever linked**. An audit note + is not a registry row, cannot be matched against a red run, and + confers nothing. + + **No entry survives on reputation, and an audit note is not a weaker + row — it is a different kind of statement.** A row says "this was + seen, here is the evidence"; a note says "someone recorded a belief + and no occurrence backs it." +4. **The rerun rule is replaced**, not softened: + - a **green rerun after a red** establishes intermittence only; it + does not establish environmental cause, harmlessness, or retirement; + - the **same signature on the rerun** is a second occurrence and + remains blocking pending investigation or a merge-base control; + - a **different signature** is a new incident and is judged + independently. A known test name confers no immunity. +5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and + its retirement condition is a diagnosis, never a green rerun. + +**Stage 2 — hardening (a separate PR):** + +6. The vterm Stage 2 `wait_for_file` requires the **expected content** + (preferred) or at minimum a non-empty result, so the predicate + matches the assertion. Its four callers all require concrete + non-empty content; the bottom-panel helper already rejects empty + reads and is not evidence for widening this change further. +7. The USR1 fixture proves the **trap is installed**, not merely that + the process started. The child publishes readiness after installing + the trap, and the test waits on that. +8. Signature 1 is **not** fixed by widening the budget (Q#MCI3). +9. Both hardened tests run **repeatedly** (a repetition set, as the + reap-ledger lane did) rather than once, because a single green run of + a formerly intermittent test proves nothing. + +**Quarantine — only if hardening fails:** + +10. A quarantined test moves to a **separate, still-blocking CI step**. + Never `#[ignore]`, never `continue-on-error`, never a silent + retry-to-green. A quarantine that stops failing the build is a + deletion with extra steps. + +--- + +## 5. Parked + +- **Resolving signature 3.** It belongs to the process-signal / + reap-ledger lanes, which have already parked three tolerance rules + pending this class of evidence. +- **The `--no-fail-fast` gap and the crdt job's PTY deadlines**, both + recorded in the CI CRDT lane. Related in spirit, separate in scope. +- **A general flake-rate dashboard.** The registry accumulates + occurrences; turning that into a rate with alerting is its own thing. +- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...` + and `m6_8_supervisor_reaps_...` were audited under acceptance 3 and + became **audit notes A1/A2** — claims with no linked occurrence. + **A1's job runs on Ubuntu**, so the registry is not macOS-only even + now; A2's job was never recorded. This lane's four *evidenced* + incidents are macOS, which is a fact about them and not about the + file. + +--- + +## 6. Gates + +The standing `CLAUDE.md` suite. Stage 1 is documentation-only and adds +no test; **its verification is acceptance 3** — the audit that no entry +survives on reputation. Stage 2 adds the repetition sets of acceptance 9. + +--- + +## 7. Branch plan + +Two PRs, in this order: + +1. **`macos-ci-signal-integrity`** — the registry, the pointer + rewrites, the audit, and the rerun rule. No code. +2. **A hardening PR** — the `wait_for_file` predicate and the USR1 trap + readiness, each with a repetition set. Signature 1 is referred to the + async-runtime design question rather than patched. + +Quarantine, if it happens, is a third and is scoped by what hardening +fails to fix.