neuwirth
/
pmacs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #215 from levineuwirth/macos-ci-signal-integrity 

CI red-signature registry — Stage 1 of macOS signal integrity
This commit is contained in:
Levi Neuwirth 2026-08-05 11:07:53 +00:00 committed by GitHub
parent bfb97c67c6 2e0617fddd
commit 12f2970ad8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 708 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

75
docs/active-work.md
View File

@ -13,6 +13,10 @@ CI CRDT coverage**, which had been sitting under "NEEDS A LANE" with no
branch and no owner since #166. It is implemented on branch and no owner since #166. It is implemented on
`ci-crdt-coverage` and its block replaces the old one below. `ci-crdt-coverage` and its block replaces the old one below.
**Updated 2026-08-05.** One lane opened: **macOS CI signal integrity**
(#215, in review), which this file required a lane for and did not have
until review caught it — the #171 defect recurring. Its block is below.
**Updated 2026-08-04.** Four PRs landed since: the CI CRDT coverage **Updated 2026-08-04.** Four PRs landed since: the CI CRDT coverage
lane #209, Distribution Stage 1 #211 (released as **v1.1.0**), the lane #209, Distribution Stage 1 #211 (released as **v1.1.0**), the
post-release accuracy pass #212, and **bottom-panel Stage 3 #213 — post-release accuracy pass #212, and **bottom-panel Stage 3 #213 —
@ -157,6 +161,72 @@ form. All four steps ran clean. **The two-argument form still does not
work** for a remote-only branch (`fatal: invalid reference`), which is work** for a remote-only branch (`fatal: invalid reference`), which is
why every lane below spells out the `-b` form. why every lane below spells out the `-b` form.
## macOS CI signal integrity — STAGE 1 IN REVIEW, PR #215
**This file requires a lane for every open PR** (see the #171/#174 note
above: an open PR is exactly the volatile work this file records, and
#171 drifted 153 commits while invisible here). #215 had none until this
entry — the same defect, caught in review.
- **Branch `macos-ci-signal-integrity`**, base `githubsucks/main` @
`bfb97c6`. Framing `docs/macos-ci-signal-integrity-framing.md`
**revision 3**.
- **PR: <https://github.com/levineuwirth/pmacs/pull/215>** — Stage 1,
docs only. **Checkpoints, newest last** — a lane that records only the
*reviewed* head goes stale the moment a review round is pushed, which
is how #171 became invisible:
| head | CI run | result |
|---|---|---|
| `d33bf4d` | [30950108477](https://github.com/levineuwirth/pmacs/actions/runs/30950108477) | 14/14 green — the reviewed head |
| `f76897c` | [30990816890](https://github.com/levineuwirth/pmacs/actions/runs/30990816890) | review round 1 |
| *(tip)* | — | review round 2: framing wording, the missing quote, this table |
**The branch tip is authoritative over any row here.** Verify with
`git rev-parse githubsucks/macos-ci-signal-integrity` rather than
trusting the newest line.
- **What Stage 1 ships:** `docs/ci-red-signatures.md`, the single
authority for judging a red CI run. Rows key on **signature** —
selector + job/flavor + every required fragment, normalized — so **a
test-name match confers nothing**. The rerun rule is replaced: a green
rerun establishes **intermittence only**; the same signature again is
a second occurrence and stays blocking; a different signature is a new
incident. Retirement is **causal**, never a count of green runs.
- **Four incidents, three tests, four signatures.** The process test
produced two — one a test race (R2), one an **unresolved possible
product defect** (R3) that no rerun can clear. Counting by test name
would have hidden the second behind the first.
- **The audit found a third state the framing did not allow**, which is
why the framing is at revision 3: two incumbents have a historical
claim but **no linked occurrence ever**. They are **audit notes A1/A2**,
not rows — unmatchable by construction, so a red in either is a new
incident. The registry is therefore *stricter* than the list it
replaces: nothing is pre-excused.
- **Verification:** fmt, diff-check, clippy with and without `crdt`,
`--lib` 1896, `--lib --features crdt` 2081, pmacs-protocol 19, m4 149,
required GPU 221. All three named tests pass locally on Linux —
consistent with R1–R4 being macOS occurrences, and **not evidence
about any of them**.
### Stage 2 — hardening, NOT started
Waits on #215's merge. Owns R2's trap-readiness fix and R4's
`wait_for_file` predicate, each with a **repetition set** rather than a
single green run, plus a discriminating witness that fails without the
fix. **R1 is referred to the async-runtime lane** (Q#MCI3) rather than
patched: widening its budget would make it pass and measure nothing
more. **R3 stays unresolved** and belongs to the process-signal /
reap-ledger lanes.
Recovery from a clean checkout:
```sh
git fetch githubsucks
git worktree add ../pmacs-ci-signals \
-b macos-ci-signal-integrity \
githubsucks/macos-ci-signal-integrity
```
## CI CRDT coverage — MERGED (#209); kept for its three follow-ons ## CI CRDT coverage — MERGED (#209); kept for its three follow-ons
**Rewritten, not removed.** Rule 4 removes a lane when its ARC is done; **Rewritten, not removed.** Rule 4 removes a lane when its ARC is done;
@ -194,6 +264,11 @@ the crdt job.
same commit. One of those was a real regression the flip caused; the same commit. One of those was a real regression the flip caused; the
other was load. **That ambiguity is the problem** — a job where noise other was load. **That ambiguity is the problem** — a job where noise
and signal look alike trains people to rerun rather than read. and signal look alike trains people to rerun rather than read.
**The triage half of this is now owned by
`docs/ci-red-signatures.md`** — signature-keyed rows and a rerun rule
that refuses to treat a green rerun as an all-clear. What remains here
is the job-cost question: whether the crdt job should carry real-PTY
deadlines at all.
Candidate: longer deadlines for real-PTY assertions in this job Candidate: longer deadlines for real-PTY assertions in this job
specifically, or serialize the PTY suites. specifically, or serialize the PTY suites.

44
docs/agent-handoff.md
View File

@ -1,6 +1,10 @@
# Agent handoff — cross-machine continuity # Agent handoff — cross-machine continuity
**Last updated: 2026-08-04, as bottom-panel Stage 3 #213 — the adopter **Last updated: 2026-08-05.** The live CI-triage rule in §5 now points
at `docs/ci-red-signatures.md` (PR #215, in review), which keys on
signature rather than test name; the hazards list this file used to
carry is retired, and its two unevidenced entries are audit notes there.
Previously **2026-08-04, as bottom-panel Stage 3 #213 — the adopter
default flip, which COMPLETES ARC 7: omitting `display` now means the default flip, which COMPLETES ARC 7: omitting `display` now means the
panel, and the workbench's panel half is done on both frontends. Beneath panel, and the workbench's panel half is done on both frontends. Beneath
it the post-release accuracy pass #212 and Distribution Stage 1 #211 — it the post-release accuracy pass #212 and Distribution Stage 1 #211 —
@ -292,11 +296,19 @@ someone forgot.
- **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error - **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error
then` guards make the silence look deliberate. Report through then` guards make the silence look deliberate. Report through
`pmacs.editor.set_status` until the channel is built. `pmacs.editor.set_status` until the channel is built.
- **Flakes that are not your change.** Judge a red run against these - **Judging a red CI run: `docs/ci-red-signatures.md` is the authority.**
before bisecting: `process::tests::a_successful_signal_disposition_...` It keys on **signature**, not test name — a failure in a listed test
(macOS-only, signal timing); `a33_headless_terminal_frame_paints_...` that lacks that row's fragments is a NEW incident, not a known one.
(GPU under parallel load, "0 blue pixels"); `m6_8_supervisor_reaps_...` The old rule here ("rerun before concluding") is retired: **a green
(load-sensitive). **Rerun before concluding.** rerun establishes intermittence only**, never environmental cause or
harmlessness; the same signature again is a second occurrence and
stays blocking. One row is an **unresolved possible product defect**
that no rerun can clear.
This list previously named three tests. The audit found one of them
had produced **two distinct signatures** with different causes, and
that two of the four incidents actually seen were absent from it —
which is why name-keyed lists are not trustworthy.
- **`basedpyright` hangs forever** — always - **`basedpyright` hangs forever** — always
`cargo test --test m4_acceptance -- --skip basedpyright`. `cargo test --test m4_acceptance -- --skip basedpyright`.
- **The crdt sweep needs `cargo build --workspace` first**, or twelve - **The crdt sweep needs `cargo build --workspace` first**, or twelve
@ -1995,9 +2007,15 @@ before trusting them:
separate proposals, each owed its own evidence. separate proposals, each owed its own evidence.
- **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan, - **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan,
`PMACS_REQUIRE_GPU=1` works without lavapipe. `PMACS_REQUIRE_GPU=1` works without lavapipe.
- **Flaky-under-load tests — rerun isolated before treating a sweep - **Flaky-under-load tests — what this lane OBSERVED.** *(Historical
failure as a regression.** The m8 daemon tests and the m6 process/PTY record. This bullet no longer states a triage rule: for judging a red
tests (`m6_1_pty_mode_lifecycle_started_then_exited`, run, `docs/ci-red-signatures.md` is the authority, and its rule
supersedes "rerun isolated" — a green rerun establishes intermittence
only, never environmental cause.
`m6_8_supervisor_reaps_all_children_across_cycles` is **A2** there —
an audit note, not a matchable row, because no signature was ever
captured.)* The m8 daemon tests and the m6
process/PTY tests (`m6_1_pty_mode_lifecycle_started_then_exited`,
`m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based; `m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based;
`editor::composition_overhead_under_ten_percent` is a render-ratio `editor::composition_overhead_under_ten_percent` is a render-ratio
microbenchmark that fails ~1/3 even isolated single-threaded (already microbenchmark that fails ~1/3 even isolated single-threaded (already
@ -2005,9 +2023,11 @@ before trusting them:
`real_tui_terminal_smoke_restores_host_after_output_input_resize_scroll_copy_and_bell`; `real_tui_terminal_smoke_restores_host_after_output_input_resize_scroll_copy_and_bell`;
the complete failed-job rerun passed. The required-GPU gate also failed once the complete failed-job rerun passed. The required-GPU gate also failed once
in `headless_diag_face_recolors_band_counter_despite_unchanged_text`, then in `headless_diag_face_recolors_band_counter_despite_unchanged_text`, then
passed both an isolated single-thread rerun and the full 139-test rerun. A passed both an isolated single-thread rerun and the full 139-test rerun.
lone timing failure → rerun the test alone (`-- --test-threads=1`) before *(That "rerun the test alone before investigating" instruction is
investigating. Run the workspace retired — an isolated green reproduces nothing about a load-sensitive
failure and establishes intermittence at most. See the registry's
rerun rule.)* Run the workspace
sweep as ONE `cargo test` invocation piped to a full log — a double sweep as ONE `cargo test` invocation piped to a full log — a double
invocation + `grep -c "test result: ok"` can mask real failures with a invocation + `grep -c "test result: ok"` can mask real failures with a
misleading `0`. misleading `0`.

231
docs/ci-red-signatures.md Normal file
View File

@ -0,0 +1,231 @@
# CI red signatures — the triage registry
**This file is the single authority for judging a red CI run.** It is an
occurrence ledger, not a flake list: a row records what was seen, what is
known about why, and what would retire it. **A row is not a claim that
the failure is harmless.**
Deliberately not named "flakes". One of its rows is a possible product
defect, and a filename that called it a flake would confer immunity the
evidence does not support.
Read this before attributing any red run to the environment. Landed
framing documents keep their own historical evidence and reasoning —
that is not duplication, and it is not superseded by this file. What
lives here is **live triage policy**.
---
## How a row matches
**A test-name match is never sufficient.** A red run matches a row only
when *all three* hold:
1. the **exact test selector** matches;
2. the **job / flavor** matches;
3. **every required fragment** is present in the failure output.
Where a fragment lists alternatives (`ESRCH` / `No such process`), any
one satisfies that requirement — those are the same condition rendered
differently by platform or libc.
Fragments are **normalized**, never pasted verbatim. PIDs, elapsed
times, thread ids and rendered OS-error suffixes vary between runs; a
verbatim key would match nothing. The evidence link preserves the exact
occurrence.
**A failure in a listed test that does not carry that row's fragments is
a NEW incident**, judged on its own. The process test below is why this
rule exists: it produced two signatures with different mechanisms and
different causal status, and only one of them is a test bug.
---
## The rerun rule
This replaces "rerun before concluding", which conflated three different
outcomes.
- **A green rerun after a red establishes INTERMITTENCE ONLY.** It does
not establish environmental cause, harmlessness, or retirement.
- **The same signature on the rerun is a SECOND OCCURRENCE.** It remains
blocking pending investigation or a merge-base control.
- **A different signature is a NEW INCIDENT**, judged independently.
A merge-base control — running the same command on the merge base — is
what distinguishes "this branch caused it" from "this tree has it". It
is cheaper than argument and is the correct response to a second
occurrence.
---
## What retires a row
**Retirement is causal, never a count of green runs.** A row is retired
by removing or explaining its mechanism:
| causal status | retirement condition |
|---|---|
| **test race** | hardening that removes the named mechanism, plus a discriminating witness for the stronger predicate |
| **measurement design** | the owning lane replaces or justifies the measurement and pins the resulting claim |
| **unresolved** | diagnosis and an explicit disposition |
**Audit notes (`A`-numbers) have no retirement condition**, because they
have nothing to retire — see that section. A linked occurrence promotes
one into an `R` row; absence retires nothing, because nothing was ever
measured.
Main-branch greens are **occurrence evidence** and accumulate toward a
rate. They retire nothing by themselves. Retired rows stay in this file
with their disposition, so a recurrence is recognisable.
---
## Live rows
### R1 — supersede cancellation budget
| field | value |
|---|---|
| **selector** | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` |
| **job / flavor** | macOS / luajit |
| **required fragments** | `supersede did not cancel within 50ms` |
| **causal status** | **measurement design** |
| **evidence** | [#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) |
| **retirement** | the async-runtime lane replaces or justifies the measurement (Q#MCI3) |
The test's premise is `thread::sleep(15ms)`, asserted by comment to mean
"the worker picked the job up"; under load it may not have, in which case
the test measures the *queued* path while claiming the running one. And
its 50ms clock starts before the second dispatch and is consumed by the
test's own `tick()` + `sleep(1ms)` pump, so the interval is dominated by
when *the test* was scheduled. **Widening the budget would make it pass
and measure nothing more.**
### R2 — USR1 delivered before the trap is installed
| field | value |
|---|---|
| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` |
| **job / flavor** | macOS / lua54 |
| **required fragments** | `leader=exited(signal SIGUSR1)` — **one exact fragment, not two loose ones**. Split into `leader=exited(` and `SIGUSR1` it would match a child that exited by some *other* disposition while `SIGUSR1` appeared elsewhere in the output |
| **causal status** | **test race** |
| **evidence** | [#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) |
| **retirement** | the fixture proves the trap is installed, with a witness that fails without it |
Readiness is `ProcessEventKind::Started`, emitted at **spawn** — not when
`/bin/sh` has parsed `trap '' USR1`. SIGUSR1's default disposition is
terminate, so a signal inside that window kills the child. The fixture's
own comment states the requirement it does not enforce.
### R3 — live-leader EPERM with an unobservable group
| field | value |
|---|---|
| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` |
| **job / flavor** | macOS / lua54 |
| **required fragments** | `EPERM` **and** `measured_group=unobservable(` **and** (`ESRCH` / `No such process`) **and** `leader=live` |
| **causal status** | **UNRESOLVED — possible product defect** |
| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) |
| **retirement** | **diagnosis and disposition by the process-signal / reap-ledger lanes. Never a green rerun.** |
**Same test as R2, different mechanism, different status.** This is the
group-target behaviour #176 and #200 circled and the reap-ledger lane
parked every disposition change pending: a group-directed `kill` returned
EPERM while the leader was observed live, and `measured_group` — the one
field able to disagree — could not be read at all.
**Do not treat a red matching this row as environmental.** A green rerun
changes nothing about it.
### R4 — readiness predicate satisfied by an empty file
| field | value |
|---|---|
| **selector** | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` |
| **job / flavor** | macOS / luajit |
| **required fragments** | `left: []` **and** `right: [49]` |
| **causal status** | **test race** |
| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) |
| **retirement** | `wait_for_file` requires the expected content, with a witness that fails against a zero-byte file |
`wait_for_file` returns as soon as `fs::read` succeeds — which succeeds
on a **zero-byte file**. The probe writes readiness with
`open(path,'wb').write(b'1')`, and `open()` creates the file before
`write()` fills it. The predicate is "readable"; the assertion is
"contains `1`" (`49` is ASCII `'1'`).
---
## Audit notes — historical claims with no linked occurrence
**These are NOT registry rows.** They carry `A`-numbers, not `R`-numbers,
because nothing here can be matched against a red run and nothing here
confers any status.
They were named in the handoff's hazards list without evidence. The audit
found the tests real and the claims recorded in good faith — but **an
assertion string existing is not a mechanism, and "timing-based" is not
an observation.** No occurrence of either was ever linked, so nothing is
known about how either fails, or whether either has failed.
Deleting them would discard a real recorded belief. Listing them beside
the evidenced rows would grant the reputation this file exists to deny.
So they are stated as what they are: **claims awaiting a first
occurrence.** A red in either test is a first recorded occurrence, to be
investigated and then promoted to an `R` row — not matched against
anything here.
### A1 — GPU terminal cell background did not paint
| field | value |
|---|---|
| **selector** | `-p pmacs-gpu a33_headless_terminal_frame_paints_cells_without_document_layers` |
| **job / flavor** | GPU Render (headless), under parallel load |
| **required fragments** | `the terminal cell background did not paint` + `blue pixels` |
| **status** | **historical claim, no linked occurrence** |
| **what IS established** | the test exists and the assertion string is real (`pmacs-gpu/src/main.rs:17973`). That is all |
| **what is NOT** | any mechanism, and any occurrence. No run was ever cited |
| **promotion** | a linked occurrence makes this an `R` row with a signature. Absence retires nothing, because nothing was measured |
### A2 — supervisor reap across cycles
| field | value |
|---|---|
| **selector** | `--test m6_8_multi_repl_acceptance m6_8_supervisor_reaps_all_children_across_cycles` |
| **job / flavor** | not recorded |
| **required fragments** | **not recorded** — no signature was ever captured |
| **status** | **historical claim, no linked occurrence** |
| **what IS established** | the test exists and runs 10 cycles; the handoff called it "timing-based" |
| **what is NOT** | any mechanism, any signature, any occurrence |
| **promotion** | a linked occurrence *with a captured signature* makes this an `R` row |
**A2 cannot be matched, and neither can A1** — that is what makes them
notes rather than rows. A red in either test is a new incident by
default. That is the correct outcome for an entry that never carried
evidence, and it means this file is **stricter** than the list it
replaces: nothing is pre-excused.
---
## Occurrence log
| date | run | row | outcome |
|---|---|---|---|
| 2026-08-04 | [30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | R1 | rerun green — intermittence only |
| 2026-08-04 | [30927084982 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | R2 | rerun green — intermittence only |
| 2026-08-04 | [30932558752 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | R3, R4 | rerun green — intermittence only; **R3 remains unresolved** |
Four incidents, three tests, **four signatures**. Count signatures: the
process test contributed two, and only one of them is a test bug.
**All four *evidenced* rows (R1–R4) are macOS.** That is a property of
these occurrences, not of the file: **A1's job is `GPU Render
(headless)`, which runs on Ubuntu**, and **A2's job was never
recorded**. Nothing here is macOS-only by construction, and a future
row from any job belongs in the same table.
The #214 occurrence is the strongest available evidence that these are
not caused by the PRs they appeared on — that PR is **docs-only and its
tree is byte-identical to a green `main`**. It is not evidence that any
of them is harmless.

370
docs/macos-ci-signal-integrity-framing.md Normal file
View File

@ -0,0 +1,370 @@
# Framing — macOS CI signal integrity: a signature registry, then hardening
**Revision 3.** Status: **Stage 1 implemented** on
`macos-ci-signal-integrity`, PR #215. Scouted against
`githubsucks/main` @ `bfb97c6`.
**Revision 2 → 3** exists because the implementation discovered a state
the contract did not allow, and the contract — not the implementation —
was what needed changing. Acceptance 3 offered a binary: carry an
incumbent in with a signature and evidence, or remove it as never
substantiated. **Both incumbents are neither**, and shipping a third
state while the governing criterion still said "two" would have made the
framing describe something the branch does not do.
Revision 3 also **retires the phrase "mechanism named"**, which
overstated what the audit found. The a33 audit proves an assertion
string and a historical claim exist; the m6_8 audit proves a test is
timing-based.
**Neither establishes a failure mechanism** — no occurrence was ever
**linked or captured**, so nothing is known about how either fails.
(Not "never observed": someone may well have seen one and not recorded
it. What is established is the absence of a *record*, which is the only
thing this audit can speak to.) They become
**audit notes A1/A2**, not registry rows, and `R`-numbers are reserved
for signatures with linked evidence.
Revision 2
separates a machine-matchable signature from verbatim, variable CI
output; preserves historical incident evidence while centralizing live
triage policy; gives retirement a causal rule rather than an arbitrary
green-run count; and corrects the rerun rule.
Four red CI incidents across #213 and #214 were each correctly judged
"not caused by this PR" — and #214's case is airtight, because it is
docs-only and its tree is byte-identical to a green `main`. **That
proves the PRs did not cause them. It does not prove they are harmless
environmental noise, and three of the four have a specific, findable
mechanism.**
This lane separates those two claims, which the current process
conflates.
---
## 0. Coherence impact (COHERENCE §20)
- **Journey steps touched:** none.
- **Interaction islands:** none.
- **Config registry adoption:** none.
- **Background-work attribution:** none.
- **Why it belongs on the board:** it protects §19's acceptance-test
ratchet and every arc that reads a red run. A flake list that is
wrong, incomplete, or keyed by test *name* rather than *signature*
makes "rerun before concluding" a habit rather than a judgement — and
a real regression arriving in that stream is indistinguishable from
the noise it hides in.
---
## 1. Ground truth (measured at `bfb97c6`)
### 1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures
The registry must count **signatures**, not test names. The process test
alone produced two, with different mechanisms and different causal
status — collapsing them under one name would have hidden a possible
product defect behind a known-flaky label.
The registry key is an exact **test selector + job/flavor + normalized
match rule**, not a pasted panic block. PIDs, elapsed times, and rendered
OS-error suffixes vary between runs; each row therefore names the
required invariant fragments and the evidence link preserves the
verbatim occurrence. A row matches only when every listed requirement
is satisfied; where a requirement lists alternative platform renderings,
one of those alternatives suffices. A test-name match by itself never
matches a registry entry.
| # | exact test selector | job / flavor | required signature fragments | causal status |
|---|---|---|---|---|
| 1 | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 |
| 2 | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 |
| 3 | *(same test)* | macOS / lua54 | all of `EPERM`, `measured_group=unobservable(` + `ESRCH` / `No such process`, and `leader=live` | **UNRESOLVED — possible product defect** — see §1.4 |
| 4 | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | both `left: []` and `right: [49]` | **test race** — see §1.5 |
Evidence:
[#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642),
[#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1),
[#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1).
### 1.2 Signature 1 — the 50ms budget measures more than it claims
```rust
let first = rt.dispatch_sleep(2_000, Some("search"));
// Let the worker pick the job up so cancel hits a running job.
thread::sleep(Duration::from_millis(15));
let started = Instant::now();
let _second = rt.dispatch_sleep(2_000, Some("search"));
while !rt.is_complete(first) {
assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms");
let _ = rt.tick();
thread::sleep(Duration::from_millis(1));
}
```
**Two design problems, and neither is fixed by a bigger number.**
- **The premise is a sleep.** `thread::sleep(15ms)` is asserted-by-comment
to mean "the worker picked the job up". On a loaded runner it may not
have, in which case the test measures cancellation of a *queued* job —
a different code path, already covered by the sibling test — while
claiming to measure a running one.
- **The clock includes the observer.** `started` begins before the
second dispatch, and the budget is consumed by the test's own
`tick()` + `sleep(1ms)` pump loop. Under scheduling pressure the
measured interval is dominated by when *the test* got scheduled, not
by when the worker observed the cancel flag.
**Widening 50ms to 200ms would make it pass and measure nothing more.**
The open question is what this test should assert: a latency bound needs
a clock the observer does not participate in, or the assertion should be
reformulated as ordering ("the first settles Cancelled before the second
completes") rather than duration.
### 1.3 Signature 2 — `Started` does not prove the trap is installed
```rust
spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()];
let id = sup.spawn(spec).expect("spawn");
let pid = spawn_started_pid(&mut sup, id); // waits for ProcessEventKind::Started
sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers");
```
`spawn_started_pid` waits for `ProcessEventKind::Started { pid }`, which
is emitted when the process is **spawned** — not when `/bin/sh` has
parsed and installed `trap '' USR1`. **SIGUSR1's default disposition is
terminate**, so a signal delivered inside that window kills the child,
and the record is `exited(signal SIGUSR1)` instead of `Running`.
The fixture's own comment states the requirement it does not enforce:
*"Ignore USR1 so the successful non-fatal signal cannot end the child
and confuse the state assertion with a real exit."* That is exactly the
confusion observed.
This is the handoff's **"wait predicate weaker than the assertion"**
race: the predicate is "process exists", the assertion needs "trap
installed".
### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved
`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not
established as a test race**. It is the group-target behaviour the
process-signal lanes have circled three times:
- #176 established that a group-directed `kill` returned EPERM while the
leader was observed alive by a real `try_wait`, retiring "EPERM cannot
happen for our own children" as a reason to discard the errno.
- #200 added `measured_group` from `getpgid`, the only field able to
disagree — and it is reported here as `unobservable(ESRCH)`, meaning
the group could not be measured at all.
- The reap-ledger lane parked every disposition change pending exactly
this evidence.
**This lane does not resolve it and must not appear to.** Its registry
entry carries causal status **UNRESOLVED — possible product defect**,
and its retirement condition is a diagnosis, not a green rerun. Folding
it under the same "macOS signal timing" label as signature 2 is how a
real defect acquires a flake's immunity.
### 1.5 Signature 4 — a readiness predicate satisfied by an empty file
```rust
fn wait_for_file(path: &Path, timeout: Duration) -> Vec<u8> {
loop {
if let Ok(bytes) = fs::read(path) { return bytes; } // succeeds on 0 bytes
...
}
}
```
The probe writes readiness with `open(path,'wb').write(b'1')`. **`open()`
creates the file before `write()` fills it**, so `fs::read` can succeed
on a zero-byte file and `wait_for_file` returns `[]`. The caller then
asserts `== b"1"` and fails `left: [], right: [49]`.
The predicate is "readable"; the assertion is "contains `1`". This is
the same shape as §1.3 and is fixable at the helper. All four callers of
this helper require concrete non-empty content; the similar bottom-panel
helper already rejects empty reads.
### 1.6 Live triage policy and historical evidence are mixed together
Flake language currently appears in the handoff's two operational
rules, `docs/active-work.md`, and several landed framing documents. It
is not all duplication. The process-signal and reap-ledger framings, for
example, preserve exact historical occurrences and the reasoning those
lanes built from them; replacing that evidence with a pointer would
make a durable framing depend on a mutable registry.
The real duplication is **live classification and triage policy**. The
handoff's hazards list names three tests — two of which are not among
the four incidents seen here — while three of these four are absent.
Elsewhere, historical occurrence notes, forward-looking risk warnings,
and current "known flaky" claims are written in the same voice even
though they require different treatment.
A list that is both stale and incomplete is worse than none: it confers
"known flaky" on whatever happens to be named, and withholds it from
everything else.
### 1.7 What is NOT established
- **No incident has been reproduced locally.** All four are macOS-only
and this machine is Linux. The mechanisms in §§1.2–1.5 are read from
source and from CI signatures, not from a local repro.
- **Frequency is unmeasured.** Four incidents across two PRs is not a
rate. The registry records occurrences so a rate can accumulate;
it does not claim one now.
- **Signature 3's cause remains unknown**, by design (§1.4).
---
## 2. Questions
- **Q#MCI1 — where does the registry live? DECIDED:** one dedicated
`docs/ci-red-signatures.md`, with the handoff's operational rule
pointing at it. The handoff is a briefing; the registry is an
occurrence ledger whose rows remain available after retirement.
- **Q#MCI2 — what retires an entry? DECIDED:** a mechanism-specific
causal result, not N green runs. A known-race entry requires hardening
that removes the named mechanism plus a discriminating acceptance
witness for the stronger predicate. A measurement-design entry
requires its owning lane to replace or justify the measurement and
pin the resulting claim. An unresolved entry requires diagnosis and
an explicit disposition. Main-job greens remain occurrence evidence,
but cannot retire any row by themselves; retired rows remain in the
history with their disposition.
- **Q#MCI3 — does signature 1 get reformulated or re-measured?** §1.2
argues its budget measures the observer. Reformulating as an ordering
assertion changes what the test proves; keeping a duration means
finding a clock the pump loop does not participate in. **This is the
one question this lane should not answer alone** — it is the
async-runtime lane's design call.
- **Q#MCI4 — does hardening ship before or with the registry? DECIDED:**
the user has already answered: **registry now, hardening next.**
Recorded here so the sequencing is visible in the document rather
than only in the conversation.
---
## 3. Bets
- **Bet 1 — signatures 2 and 4 disappear under hardening**, because both
have a named mechanism and a fix at the readiness predicate.
*Falsified if either recurs after the predicate is strengthened.*
- **Bet 2 — a signature-keyed registry refuses name-based immunity.** A
future failure in the process test which does not contain all of row
2's or row 3's required fragments is a new incident, not a known
flake. *Falsified if the operational rule permits a test-name-only
match.*
- **Bet 3 — green reruns do not erase unresolved evidence.** Signature
3 remains unresolved until the process-signal lane diagnoses and
disposes it, whether or not later runs pass. *Falsified if a green run
changes that row's status or retirement condition.*
---
## 4. Acceptance
**Stage 1 — the registry (this lane's first PR):**
1. **One authoritative table**, with a row per **signature**: exact test
selector, job and Lua flavor, normalized machine-match rule, evidence
link to the verbatim occurrence, causal status, and mechanism-specific
retirement condition. Matching requires the selector, job/flavor,
and every required fragment; variable values are explicitly
normalized rather than silently abbreviated.
2. **One live triage policy.** Operational duplicate classifications and
rerun rules become pointers to the registry. Historical occurrence
evidence stays where it supports a landed framing; a relevant note
may gain a registry status link, but its evidence and reasoning are
not replaced. Forward-looking risk statements are audited as risks,
not silently promoted to known flakes.
3. **The three tests named in the current handoff list are audited**,
into one of **three** states — the third was found by doing the audit
and is why this is revision 3:
- **carried as a registry row** (`R`-numbered) when a signature and a
linked occurrence both exist;
- **removed** when nothing substantiates it at all;
- **recorded as an audit note** (`A`-numbered) when a *historical
claim* exists but **no occurrence was ever linked**. An audit note
is not a registry row, cannot be matched against a red run, and
confers nothing.
**No entry survives on reputation, and an audit note is not a weaker
row — it is a different kind of statement.** A row says "this was
seen, here is the evidence"; a note says "someone recorded a belief
and no occurrence backs it."
4. **The rerun rule is replaced**, not softened:
- a **green rerun after a red** establishes intermittence only; it
does not establish environmental cause, harmlessness, or retirement;
- the **same signature on the rerun** is a second occurrence and
remains blocking pending investigation or a merge-base control;
- a **different signature** is a new incident and is judged
independently. A known test name confers no immunity.
5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and
its retirement condition is a diagnosis, never a green rerun.
**Stage 2 — hardening (a separate PR):**
6. The vterm Stage 2 `wait_for_file` requires the **expected content**
(preferred) or at minimum a non-empty result, so the predicate
matches the assertion. Its four callers all require concrete
non-empty content; the bottom-panel helper already rejects empty
reads and is not evidence for widening this change further.
7. The USR1 fixture proves the **trap is installed**, not merely that
the process started. The child publishes readiness after installing
the trap, and the test waits on that.
8. Signature 1 is **not** fixed by widening the budget (Q#MCI3).
9. Both hardened tests run **repeatedly** (a repetition set, as the
reap-ledger lane did) rather than once, because a single green run of
a formerly intermittent test proves nothing.
**Quarantine — only if hardening fails:**
10. A quarantined test moves to a **separate, still-blocking CI step**.
Never `#[ignore]`, never `continue-on-error`, never a silent
retry-to-green. A quarantine that stops failing the build is a
deletion with extra steps.
---
## 5. Parked
- **Resolving signature 3.** It belongs to the process-signal /
reap-ledger lanes, which have already parked three tolerance rules
pending this class of evidence.
- **The `--no-fail-fast` gap and the crdt job's PTY deadlines**, both
recorded in the CI CRDT lane. Related in spirit, separate in scope.
- **A general flake-rate dashboard.** The registry accumulates
occurrences; turning that into a rate with alerting is its own thing.
- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...`
and `m6_8_supervisor_reaps_...` were audited under acceptance 3 and
became **audit notes A1/A2** — claims with no linked occurrence.
**A1's job runs on Ubuntu**, so the registry is not macOS-only even
now; A2's job was never recorded. This lane's four *evidenced*
incidents are macOS, which is a fact about them and not about the
file.
---
## 6. Gates
The standing `CLAUDE.md` suite. Stage 1 is documentation-only and adds
no test; **its verification is acceptance 3** — the audit that no entry
survives on reputation. Stage 2 adds the repetition sets of acceptance 9.
---
## 7. Branch plan
Two PRs, in this order:
1. **`macos-ci-signal-integrity`** — the registry, the pointer
rewrites, the audit, and the rerun rule. No code.
2. **A hardening PR** — the `wait_for_file` predicate and the USR1 trap
readiness, each with a repetition set. Signature 1 is referred to the
async-runtime design question rather than patched.
Quarantine, if it happens, is a third and is scoped by what hardening
fails to fix.