neuwirth
/
pmacs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(ci): the red-signature registry, and an audit that found no immunity 

Stage 1 of the macOS CI signal-integrity lane. `docs/ci-red-signatures.md`
is now the single authority for judging a red CI run.

NOT NAMED "FLAKES", DELIBERATELY. One of its rows is a possible product
defect, and a filename calling it a flake would confer immunity the
evidence does not support.

A ROW MATCHES ON SIGNATURE, NEVER ON TEST NAME. All three of selector,
job/flavor and every required fragment must hold. Fragments are
normalized rather than pasted: PIDs, elapsed times and rendered
OS-error suffixes vary between runs, so a verbatim key would match
nothing, and where a fragment lists alternatives (ESRCH / "No such
process") those are one condition rendered differently by platform. A
failure in a listed test WITHOUT that row's fragments is a new incident.

The process test is why that rule exists: it produced TWO signatures
with different mechanisms and different causal status, and only one is a
test bug. Four incidents, three tests, four signatures — the registry
counts signatures.

THE RERUN RULE IS REPLACED, NOT SOFTENED. A green rerun establishes
INTERMITTENCE ONLY — never environmental cause, harmlessness, or
retirement. The same signature again is a second occurrence and stays
blocking pending investigation or a merge-base control. A different
signature is a new incident.

RETIREMENT IS CAUSAL. A test race retires by hardening that removes the
mechanism plus a discriminating witness; a measurement-design row by its
owning lane replacing or justifying the measurement; an unresolved row
by diagnosis and disposition. Main-branch greens accumulate as
occurrence evidence and retire nothing.

THE AUDIT FOUND A THIRD CATEGORY the framing's acceptance 3 did not
anticipate. It said each incumbent is either carried in with a signature
and evidence, or removed as never substantiated. Both incumbents are in
between: the tests are real and the mechanisms plausible — a33's "blue
pixels" is a genuine assertion string at pmacs-gpu/src/main.rs:17973,
and m6_8 exists and is timing-based — but NEITHER HAS A LINKED
OCCURRENCE. Deleting them would discard a real observation; carrying
them as peers of the evidenced rows would grant exactly the reputation
this lane exists to deny.

They are recorded as "mechanism named, no occurrence recorded", with it
stated that nothing there confers known-flaky status and that a red
matching one is a FIRST recorded occurrence to be investigated. R6 goes
further: with no signature ever captured it CANNOT BE MATCHED AT ALL, so
a red in that test is a new incident by default. That is the correct
outcome for an entry that never carried evidence.

LIVE POLICY IS CENTRALIZED; HISTORICAL EVIDENCE IS NOT MOVED. The
handoff's hazards rule becomes a pointer. Its landed-lesson block at
§5 keeps its own evidence and gains a note that the registry's rerun
rule supersedes "rerun isolated". active-work.md's two mentions are
verification records for the reap-ledger and bottom-panel lanes and are
left alone; only the CI-weakness block's triage half points at the
registry, because the job-cost question there is genuinely separate.

Verified: fmt, diff-check, clippy with and without crdt, --lib 1896,
--lib --features crdt 2081, pmacs-protocol 19, m4 149, required GPU 221.
All three tests the registry names pass locally on Linux, which is
consistent with every row being macOS-only and is not evidence about
any of them.

Docs only. Stage 2 (hardening) is a separate PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Levi Neuwirth 2026-08-04 22:55:25 +02:00
parent a1c6c93adf
commit d33bf4df78
No known key found for this signature in database
3 changed files with 231 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/active-work.md
View File

@ -194,6 +194,11 @@ the crdt job.
same commit. One of those was a real regression the flip caused; the same commit. One of those was a real regression the flip caused; the
other was load. **That ambiguity is the problem** — a job where noise other was load. **That ambiguity is the problem** — a job where noise
and signal look alike trains people to rerun rather than read. and signal look alike trains people to rerun rather than read.
**The triage half of this is now owned by
`docs/ci-red-signatures.md`** — signature-keyed rows and a rerun rule
that refuses to treat a green rerun as an all-clear. What remains here
is the job-cost question: whether the crdt job should carry real-PTY
deadlines at all.
Candidate: longer deadlines for real-PTY assertions in this job Candidate: longer deadlines for real-PTY assertions in this job
specifically, or serialize the PTY suites. specifically, or serialize the PTY suites.

26
docs/agent-handoff.md
View File

@ -292,11 +292,19 @@ someone forgot.
- **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error - **`pmacs.error` is undefined in production.** Fifteen `if pmacs.error
then` guards make the silence look deliberate. Report through then` guards make the silence look deliberate. Report through
`pmacs.editor.set_status` until the channel is built. `pmacs.editor.set_status` until the channel is built.
- **Flakes that are not your change.** Judge a red run against these - **Judging a red CI run: `docs/ci-red-signatures.md` is the authority.**
before bisecting: `process::tests::a_successful_signal_disposition_...` It keys on **signature**, not test name — a failure in a listed test
(macOS-only, signal timing); `a33_headless_terminal_frame_paints_...` that lacks that row's fragments is a NEW incident, not a known one.
(GPU under parallel load, "0 blue pixels"); `m6_8_supervisor_reaps_...` The old rule here ("rerun before concluding") is retired: **a green
(load-sensitive). **Rerun before concluding.** rerun establishes intermittence only**, never environmental cause or
harmlessness; the same signature again is a second occurrence and
stays blocking. One row is an **unresolved possible product defect**
that no rerun can clear.
This list previously named three tests. The audit found one of them
had produced **two distinct signatures** with different causes, and
that two of the four incidents actually seen were absent from it —
which is why name-keyed lists are not trustworthy.
- **`basedpyright` hangs forever** — always - **`basedpyright` hangs forever** — always
`cargo test --test m4_acceptance -- --skip basedpyright`. `cargo test --test m4_acceptance -- --skip basedpyright`.
- **The crdt sweep needs `cargo build --workspace` first**, or twelve - **The crdt sweep needs `cargo build --workspace` first**, or twelve
@ -1996,8 +2004,12 @@ before trusting them:
- **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan, - **GPU on the laptop**: AMD Radeon 780M (RADV) — native Vulkan,
`PMACS_REQUIRE_GPU=1` works without lavapipe. `PMACS_REQUIRE_GPU=1` works without lavapipe.
- **Flaky-under-load tests — rerun isolated before treating a sweep - **Flaky-under-load tests — rerun isolated before treating a sweep
failure as a regression.** The m8 daemon tests and the m6 process/PTY failure as a regression.** *(Historical, from this lane. For LIVE
tests (`m6_1_pty_mode_lifecycle_started_then_exited`, triage of a red run use `docs/ci-red-signatures.md`, whose rerun rule
supersedes "rerun isolated" — a green rerun proves intermittence only.
`m6_8_supervisor_reaps_all_children_across_cycles` is R6 there, with
no signature ever captured.)* The m8 daemon tests and the m6
process/PTY tests (`m6_1_pty_mode_lifecycle_started_then_exited`,
`m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based; `m6_8_supervisor_reaps_all_children_across_cycles`) are timing-based;
`editor::composition_overhead_under_ten_percent` is a render-ratio `editor::composition_overhead_under_ten_percent` is a render-ratio
microbenchmark that fails ~1/3 even isolated single-threaded (already microbenchmark that fails ~1/3 even isolated single-threaded (already

207
docs/ci-red-signatures.md Normal file
View File

@ -0,0 +1,207 @@
# CI red signatures — the triage registry
**This file is the single authority for judging a red CI run.** It is an
occurrence ledger, not a flake list: a row records what was seen, what is
known about why, and what would retire it. **A row is not a claim that
the failure is harmless.**
Deliberately not named "flakes". One of its rows is a possible product
defect, and a filename that called it a flake would confer immunity the
evidence does not support.
Read this before attributing any red run to the environment. Landed
framing documents keep their own historical evidence and reasoning —
that is not duplication, and it is not superseded by this file. What
lives here is **live triage policy**.
---
## How a row matches
**A test-name match is never sufficient.** A red run matches a row only
when *all three* hold:
1. the **exact test selector** matches;
2. the **job / flavor** matches;
3. **every required fragment** is present in the failure output.
Where a fragment lists alternatives (`ESRCH` / `No such process`), any
one satisfies that requirement — those are the same condition rendered
differently by platform or libc.
Fragments are **normalized**, never pasted verbatim. PIDs, elapsed
times, thread ids and rendered OS-error suffixes vary between runs; a
verbatim key would match nothing. The evidence link preserves the exact
occurrence.
**A failure in a listed test that does not carry that row's fragments is
a NEW incident**, judged on its own. The process test below is why this
rule exists: it produced two signatures with different mechanisms and
different causal status, and only one of them is a test bug.
---
## The rerun rule
This replaces "rerun before concluding", which conflated three different
outcomes.
- **A green rerun after a red establishes INTERMITTENCE ONLY.** It does
not establish environmental cause, harmlessness, or retirement.
- **The same signature on the rerun is a SECOND OCCURRENCE.** It remains
blocking pending investigation or a merge-base control.
- **A different signature is a NEW INCIDENT**, judged independently.
A merge-base control — running the same command on the merge base — is
what distinguishes "this branch caused it" from "this tree has it". It
is cheaper than argument and is the correct response to a second
occurrence.
---
## What retires a row
**Retirement is causal, never a count of green runs.** A row is retired
by removing or explaining its mechanism:
| causal status | retirement condition |
|---|---|
| **test race** | hardening that removes the named mechanism, plus a discriminating witness for the stronger predicate |
| **measurement design** | the owning lane replaces or justifies the measurement and pins the resulting claim |
| **unresolved** | diagnosis and an explicit disposition |
| **mechanism named, no occurrence** | a linked occurrence promotes it to one of the above; sustained absence is not retirement, because nothing was ever measured |
Main-branch greens are **occurrence evidence** and accumulate toward a
rate. They retire nothing by themselves. Retired rows stay in this file
with their disposition, so a recurrence is recognisable.
---
## Live rows
### R1 — supersede cancellation budget
| field | value |
|---|---|
| **selector** | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` |
| **job / flavor** | macOS / luajit |
| **required fragments** | `supersede did not cancel within 50ms` |
| **causal status** | **measurement design** |
| **evidence** | [#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) |
| **retirement** | the async-runtime lane replaces or justifies the measurement (Q#MCI3) |
The test's premise is `thread::sleep(15ms)`, asserted by comment to mean
"the worker picked the job up"; under load it may not have, in which case
the test measures the *queued* path while claiming the running one. And
its 50ms clock starts before the second dispatch and is consumed by the
test's own `tick()` + `sleep(1ms)` pump, so the interval is dominated by
when *the test* was scheduled. **Widening the budget would make it pass
and measure nothing more.**
### R2 — USR1 delivered before the trap is installed
| field | value |
|---|---|
| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` |
| **job / flavor** | macOS / lua54 |
| **required fragments** | `leader=exited(` + `SIGUSR1` |
| **causal status** | **test race** |
| **evidence** | [#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) |
| **retirement** | the fixture proves the trap is installed, with a witness that fails without it |
Readiness is `ProcessEventKind::Started`, emitted at **spawn** — not when
`/bin/sh` has parsed `trap '' USR1`. SIGUSR1's default disposition is
terminate, so a signal inside that window kills the child. The fixture's
own comment states the requirement it does not enforce.
### R3 — live-leader EPERM with an unobservable group
| field | value |
|---|---|
| **selector** | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` |
| **job / flavor** | macOS / lua54 |
| **required fragments** | `EPERM` **and** `measured_group=unobservable(` **and** (`ESRCH` / `No such process`) **and** `leader=live` |
| **causal status** | **UNRESOLVED — possible product defect** |
| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) |
| **retirement** | **diagnosis and disposition by the process-signal / reap-ledger lanes. Never a green rerun.** |
**Same test as R2, different mechanism, different status.** This is the
group-target behaviour #176 and #200 circled and the reap-ledger lane
parked every disposition change pending: a group-directed `kill` returned
EPERM while the leader was observed live, and `measured_group` — the one
field able to disagree — could not be read at all.
**Do not treat a red matching this row as environmental.** A green rerun
changes nothing about it.
### R4 — readiness predicate satisfied by an empty file
| field | value |
|---|---|
| **selector** | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` |
| **job / flavor** | macOS / luajit |
| **required fragments** | `left: []` **and** `right: [49]` |
| **causal status** | **test race** |
| **evidence** | [#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) |
| **retirement** | `wait_for_file` requires the expected content, with a witness that fails against a zero-byte file |
`wait_for_file` returns as soon as `fs::read` succeeds — which succeeds
on a **zero-byte file**. The probe writes readiness with
`open(path,'wb').write(b'1')`, and `open()` creates the file before
`write()` fills it. The predicate is "readable"; the assertion is
"contains `1`" (`49` is ASCII `'1'`).
---
## Audited incumbents — mechanism named, no occurrence recorded
These were carried in the handoff's hazards list **without evidence**.
The audit found the tests real and the mechanisms plausible, so they are
neither confirmed nor deleted: they are recorded honestly as unmeasured.
**Nothing here confers "known flaky".** A red matching one of these is a
first recorded occurrence and should be investigated, not reruns-to-green.
### R5 — GPU terminal cell background did not paint
| field | value |
|---|---|
| **selector** | `-p pmacs-gpu a33_headless_terminal_frame_paints_cells_without_document_layers` |
| **job / flavor** | GPU Render (headless), under parallel load |
| **required fragments** | `the terminal cell background did not paint` + `blue pixels` |
| **causal status** | **mechanism named, no occurrence recorded** |
| **evidence** | **none linked.** The assertion string is real (`pmacs-gpu/src/main.rs:17973`); no run was ever cited |
| **retirement** | a linked occurrence promotes this row; absence retires nothing, because nothing was measured |
### R6 — supervisor reap across cycles
| field | value |
|---|---|
| **selector** | `--test m6_8_multi_repl_acceptance m6_8_supervisor_reaps_all_children_across_cycles` |
| **job / flavor** | not recorded |
| **required fragments** | **not recorded** — no signature was ever captured |
| **causal status** | **mechanism named, no occurrence recorded** ("timing-based", 10 cycles) |
| **evidence** | **none linked** |
| **retirement** | a linked occurrence with a signature promotes this row |
**R6 cannot currently be matched.** Without required fragments there is
no rule to apply, so a red in this test is a new incident by default.
That is the correct outcome for an entry that never carried evidence.
---
## Occurrence log
| date | run | row | outcome |
|---|---|---|---|
| 2026-08-04 | [30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642) | R1 | rerun green — intermittence only |
| 2026-08-04 | [30927084982 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1) | R2 | rerun green — intermittence only |
| 2026-08-04 | [30932558752 att.1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1) | R3, R4 | rerun green — intermittence only; **R3 remains unresolved** |
Four incidents, three tests, **four signatures**. Count signatures: the
process test contributed two, and only one of them is a test bug.
The #214 occurrence is the strongest available evidence that these are
not caused by the PRs they appeared on — that PR is **docs-only and its
tree is byte-identical to a green `main`**. It is not evidence that any
of them is harmless.