pmacs/docs/macos-ci-signal-integrity-framing.md

# Framing — macOS CI signal integrity: a signature registry, then hardening

**Revision 4.** Status: **Stage 1 MERGED** as #215 (`main` @ `12f2970`);
**Stage 2 implemented** on `ci-signal-hardening`. Scouted against
`githubsucks/main` @ `bfb97c6`.

**Revision 4 records implementation findings, not a new design round.**
The acceptance in §4 is unchanged and was approved at revision 3; §8
below is what building Stage 2 established, including two places where
this document's own account of a mechanism was imprecise.

**Revision 2 → 3** exists because the implementation discovered a state
the contract did not allow, and the contract — not the implementation —
was what needed changing. Acceptance 3 offered a binary: carry an
incumbent in with a signature and evidence, or remove it as never
substantiated. **Both incumbents are neither**, and shipping a third
state while the governing criterion still said "two" would have made the
framing describe something the branch does not do.

Revision 3 also **retires the phrase "mechanism named"**, which
overstated what the audit found. The a33 audit proves an assertion
string and a historical claim exist; the m6_8 audit proves a test is
timing-based.
**Neither establishes a failure mechanism** — no occurrence was ever
**linked or captured**, so nothing is known about how either fails.
(Not "never observed": someone may well have seen one and not recorded
it. What is established is the absence of a *record*, which is the only
thing this audit can speak to.) They become
**audit notes A1/A2**, not registry rows, and `R`-numbers are reserved
for signatures with linked evidence.

Revision 2
separates a machine-matchable signature from verbatim, variable CI
output; preserves historical incident evidence while centralizing live
triage policy; gives retirement a causal rule rather than an arbitrary
green-run count; and corrects the rerun rule.

Four red CI incidents across #213 and #214 were each correctly judged
"not caused by this PR" — and #214's case is airtight, because it is
docs-only and its tree is byte-identical to a green `main`. **That
proves the PRs did not cause them. It does not prove they are harmless
environmental noise, and three of the four have a specific, findable
mechanism.**

This lane separates those two claims, which the current process
conflates.

---

## 0. Coherence impact (COHERENCE §20)

- **Journey steps touched:** none.
- **Interaction islands:** none.
- **Config registry adoption:** none.
- **Background-work attribution:** none.
- **Why it belongs on the board:** it protects §19's acceptance-test
  ratchet and every arc that reads a red run. A flake list that is
  wrong, incomplete, or keyed by test *name* rather than *signature*
  makes "rerun before concluding" a habit rather than a judgement — and
  a real regression arriving in that stream is indistinguishable from
  the noise it hides in.

---

## 1. Ground truth (measured at `bfb97c6`)

### 1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures

The registry must count **signatures**, not test names. The process test
alone produced two, with different mechanisms and different causal
status — collapsing them under one name would have hidden a possible
product defect behind a known-flaky label.

The registry key is an exact **test selector + job/flavor + normalized
match rule**, not a pasted panic block. PIDs, elapsed times, and rendered
OS-error suffixes vary between runs; each row therefore names the
required invariant fragments and the evidence link preserves the
verbatim occurrence. A row matches only when every listed requirement
is satisfied; where a requirement lists alternative platform renderings,
one of those alternatives suffices. A test-name match by itself never
matches a registry entry.

| # | exact test selector | job / flavor | required signature fragments | causal status |
|---|---|---|---|---|
| 1 | `--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms` | macOS / luajit | `supersede did not cancel within 50ms` | **measurement design** — see §1.2 |
| 2 | `--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal` | macOS / lua54 | `leader=exited(signal SIGUSR1)` | **test race** — see §1.3 |
| 3 | *(same test)* | macOS / lua54 | all of `EPERM`, `measured_group=unobservable(` + `ESRCH` / `No such process`, and `leader=live` | **UNRESOLVED — possible product defect** — see §1.4 |
| 4 | `--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt` | macOS / luajit | both `left: []` and `right: [49]` | **test race** — see §1.5 |

Evidence:
[#213 run 30826884642](https://github.com/levineuwirth/pmacs/actions/runs/30826884642),
[#213 run 30927084982 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30927084982/attempts/1),
[#214 run 30932558752 attempt 1](https://github.com/levineuwirth/pmacs/actions/runs/30932558752/attempts/1).

### 1.2 Signature 1 — the 50ms budget measures more than it claims

```rust
let first = rt.dispatch_sleep(2_000, Some("search"));
// Let the worker pick the job up so cancel hits a running job.
thread::sleep(Duration::from_millis(15));
let started = Instant::now();
let _second = rt.dispatch_sleep(2_000, Some("search"));
while !rt.is_complete(first) {
    assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms");
    let _ = rt.tick();
    thread::sleep(Duration::from_millis(1));
}
```

**Two design problems, and neither is fixed by a bigger number.**

- **The premise is a sleep.** `thread::sleep(15ms)` is asserted-by-comment
  to mean "the worker picked the job up". On a loaded runner it may not
  have, in which case the test measures cancellation of a *queued* job —
  a different code path, already covered by the sibling test — while
  claiming to measure a running one.
- **The clock includes the observer.** `started` begins before the
  second dispatch, and the budget is consumed by the test's own
  `tick()` + `sleep(1ms)` pump loop. Under scheduling pressure the
  measured interval is dominated by when *the test* got scheduled, not
  by when the worker observed the cancel flag.

**Widening 50ms to 200ms would make it pass and measure nothing more.**
The open question is what this test should assert: a latency bound needs
a clock the observer does not participate in, or the assertion should be
reformulated as ordering ("the first settles Cancelled before the second
completes") rather than duration.

### 1.3 Signature 2 — `Started` does not prove the trap is installed

```rust
spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()];
let id = sup.spawn(spec).expect("spawn");
let pid = spawn_started_pid(&mut sup, id);   // waits for ProcessEventKind::Started
sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers");
```

`spawn_started_pid` waits for `ProcessEventKind::Started { pid }`, which
is emitted when the process is **spawned** — not when `/bin/sh` has
parsed and installed `trap '' USR1`. **SIGUSR1's default disposition is
terminate**, so a signal delivered inside that window kills the child,
and the record is `exited(signal SIGUSR1)` instead of `Running`.

The fixture's own comment states the requirement it does not enforce:
*"Ignore USR1 so the successful non-fatal signal cannot end the child
and confuse the state assertion with a real exit."* That is exactly the
confusion observed.

This is the handoff's **"wait predicate weaker than the assertion"**
race: the predicate is "process exists", the assertion needs "trap
installed".

### 1.4 Signature 3 — the live-leader EPERM, deliberately unresolved

`EPERM, measured_group=unobservable(ESRCH), leader=live` is **not
established as a test race**. It is the group-target behaviour the
process-signal lanes have circled three times:

- #176 established that a group-directed `kill` returned EPERM while the
  leader was observed alive by a real `try_wait`, retiring "EPERM cannot
  happen for our own children" as a reason to discard the errno.
- #200 added `measured_group` from `getpgid`, the only field able to
  disagree — and it is reported here as `unobservable(ESRCH)`, meaning
  the group could not be measured at all.
- The reap-ledger lane parked every disposition change pending exactly
  this evidence.

**This lane does not resolve it and must not appear to.** Its registry
entry carries causal status **UNRESOLVED — possible product defect**,
and its retirement condition is a diagnosis, not a green rerun. Folding
it under the same "macOS signal timing" label as signature 2 is how a
real defect acquires a flake's immunity.

### 1.5 Signature 4 — a readiness predicate satisfied by an empty file

```rust
fn wait_for_file(path: &Path, timeout: Duration) -> Vec<u8> {
    loop {
        if let Ok(bytes) = fs::read(path) { return bytes; }   // succeeds on 0 bytes
        ...
    }
}
```

The probe writes readiness with `open(path,'wb').write(b'1')`. **`open()`
creates the file before `write()` fills it**, so `fs::read` can succeed
on a zero-byte file and `wait_for_file` returns `[]`. The caller then
asserts `== b"1"` and fails `left: [], right: [49]`.

The predicate is "readable"; the assertion is "contains `1`". This is
the same shape as §1.3 and is fixable at the helper. All four callers of
this helper require concrete non-empty content; the similar bottom-panel
helper already rejects empty reads.

### 1.6 Live triage policy and historical evidence are mixed together

Flake language currently appears in the handoff's two operational
rules, `docs/active-work.md`, and several landed framing documents. It
is not all duplication. The process-signal and reap-ledger framings, for
example, preserve exact historical occurrences and the reasoning those
lanes built from them; replacing that evidence with a pointer would
make a durable framing depend on a mutable registry.

The real duplication is **live classification and triage policy**. The
handoff's hazards list names three tests — two of which are not among
the four incidents seen here — while three of these four are absent.
Elsewhere, historical occurrence notes, forward-looking risk warnings,
and current "known flaky" claims are written in the same voice even
though they require different treatment.

A list that is both stale and incomplete is worse than none: it confers
"known flaky" on whatever happens to be named, and withholds it from
everything else.

### 1.7 What is NOT established

- **No incident has been reproduced locally.** All four are macOS-only
  and this machine is Linux. The mechanisms in §§1.2–1.5 are read from
  source and from CI signatures, not from a local repro.
- **Frequency is unmeasured.** Four incidents across two PRs is not a
  rate. The registry records occurrences so a rate can accumulate;
  it does not claim one now.
- **Signature 3's cause remains unknown**, by design (§1.4).

---

## 2. Questions

- **Q#MCI1 — where does the registry live? DECIDED:** one dedicated
  `docs/ci-red-signatures.md`, with the handoff's operational rule
  pointing at it. The handoff is a briefing; the registry is an
  occurrence ledger whose rows remain available after retirement.
- **Q#MCI2 — what retires an entry? DECIDED:** a mechanism-specific
  causal result, not N green runs. A known-race entry requires hardening
  that removes the named mechanism plus a discriminating acceptance
  witness for the stronger predicate. A measurement-design entry
  requires its owning lane to replace or justify the measurement and
  pin the resulting claim. An unresolved entry requires diagnosis and
  an explicit disposition. Main-job greens remain occurrence evidence,
  but cannot retire any row by themselves; retired rows remain in the
  history with their disposition.
- **Q#MCI3 — does signature 1 get reformulated or re-measured?** §1.2
  argues its budget measures the observer. Reformulating as an ordering
  assertion changes what the test proves; keeping a duration means
  finding a clock the pump loop does not participate in. **This is the
  one question this lane should not answer alone** — it is the
  async-runtime lane's design call.
- **Q#MCI4 — does hardening ship before or with the registry? DECIDED:**
  the user has already answered: **registry now, hardening next.**
  Recorded here so the sequencing is visible in the document rather
  than only in the conversation.

---

## 3. Bets

- **Bet 1 — signatures 2 and 4 disappear under hardening**, because both
  have a named mechanism and a fix at the readiness predicate.
  *Falsified if either recurs after the predicate is strengthened.*
- **Bet 2 — a signature-keyed registry refuses name-based immunity.** A
  future failure in the process test which does not contain all of row
  2's or row 3's required fragments is a new incident, not a known
  flake. *Falsified if the operational rule permits a test-name-only
  match.*
- **Bet 3 — green reruns do not erase unresolved evidence.** Signature
  3 remains unresolved until the process-signal lane diagnoses and
  disposes it, whether or not later runs pass. *Falsified if a green run
  changes that row's status or retirement condition.*

---

## 4. Acceptance

**Stage 1 — the registry (this lane's first PR):**

1. **One authoritative table**, with a row per **signature**: exact test
   selector, job and Lua flavor, normalized machine-match rule, evidence
   link to the verbatim occurrence, causal status, and mechanism-specific
   retirement condition. Matching requires the selector, job/flavor,
   and every required fragment; variable values are explicitly
   normalized rather than silently abbreviated.
2. **One live triage policy.** Operational duplicate classifications and
   rerun rules become pointers to the registry. Historical occurrence
   evidence stays where it supports a landed framing; a relevant note
   may gain a registry status link, but its evidence and reasoning are
   not replaced. Forward-looking risk statements are audited as risks,
   not silently promoted to known flakes.
3. **The three tests named in the current handoff list are audited**,
   into one of **three** states — the third was found by doing the audit
   and is why this is revision 3:
   - **carried as a registry row** (`R`-numbered) when a signature and a
     linked occurrence both exist;
   - **removed** when nothing substantiates it at all;
   - **recorded as an audit note** (`A`-numbered) when a *historical
     claim* exists but **no occurrence was ever linked**. An audit note
     is not a registry row, cannot be matched against a red run, and
     confers nothing.

   **No entry survives on reputation, and an audit note is not a weaker
   row — it is a different kind of statement.** A row says "this was
   seen, here is the evidence"; a note says "someone recorded a belief
   and no occurrence backs it."
4. **The rerun rule is replaced**, not softened:
   - a **green rerun after a red** establishes intermittence only; it
     does not establish environmental cause, harmlessness, or retirement;
   - the **same signature on the rerun** is a second occurrence and
     remains blocking pending investigation or a merge-base control;
   - a **different signature** is a new incident and is judged
     independently. A known test name confers no immunity.
5. Signature 3 is recorded **UNRESOLVED — possible product defect**, and
   its retirement condition is a diagnosis, never a green rerun.

**Stage 2 — hardening (a separate PR):**

6. The vterm Stage 2 `wait_for_file` requires the **expected content**
   (preferred) or at minimum a non-empty result, so the predicate
   matches the assertion. Its four callers all require concrete
   non-empty content; the bottom-panel helper already rejects empty
   reads and is not evidence for widening this change further.
7. The USR1 fixture proves the **trap is installed**, not merely that
   the process started. The child publishes readiness after installing
   the trap, and the test waits on that.
8. Signature 1 is **not** fixed by widening the budget (Q#MCI3).
9. Both hardened tests run **repeatedly** (a repetition set, as the
   reap-ledger lane did) rather than once, because a single green run of
   a formerly intermittent test proves nothing.

**Quarantine — only if hardening fails:**

10. A quarantined test moves to a **separate, still-blocking CI step**.
    Never `#[ignore]`, never `continue-on-error`, never a silent
    retry-to-green. A quarantine that stops failing the build is a
    deletion with extra steps.

---

## 5. Parked

- **Resolving signature 3.** It belongs to the process-signal /
  reap-ledger lanes, which have already parked three tolerance rules
  pending this class of evidence.
- **The `--no-fail-fast` gap and the crdt job's PTY deadlines**, both
  recorded in the CI CRDT lane. Related in spirit, separate in scope.
- **A general flake-rate dashboard.** The registry accumulates
  occurrences; turning that into a rate with alerting is its own thing.
- **Linux and GPU flakes.** `a33_headless_terminal_frame_paints_...`
  and `m6_8_supervisor_reaps_...` were audited under acceptance 3 and
  became **audit notes A1/A2** — claims with no linked occurrence.
  **A1's job runs on Ubuntu**, so the registry is not macOS-only even
  now; A2's job was never recorded. This lane's four *evidenced*
  incidents are macOS, which is a fact about them and not about the
  file.

---

## 6. Gates

The standing `CLAUDE.md` suite. Stage 1 is documentation-only and adds
no test; **its verification is acceptance 3** — the audit that no entry
survives on reputation. Stage 2 adds the repetition sets of acceptance 9.

---

## 7. Stage 2 implementation notes (revision 4)

Stage 2 landed acceptance 6–9 on `ci-signal-hardening`. Four findings,
two of which correct this document.

### 7.1 §1.3 named the right window and the wrong assertion

`leader=exited(signal SIGUSR1)` is rendered **only on a failed `kill`**,
by `signal_failure_report`. The USR1 cannot be the call that failed — it
is the call that did the killing — so the failing call is the **SIGTERM
that follows**, and the assertion that blew up is
`.expect("TERM delivers")`, not the `Running` state check. §1.3's "the
record is `exited(signal SIGUSR1)` instead of `Running`" reads as though
`ProcessState` carries that value; it does not, and nothing ticks
between the two calls, so the state assertion could not have seen an
exit. **The row's fragment and its mechanism were both right.** Why a
group-directed TERM then found no group is *not* established, and the
fix does not depend on it.

### 7.2 The fixture had a second dependency nobody had named

These signals are **group-directed**, so the fixture's `sleep 30` — if
the shell *forks* it — is an **untrapped member of the same group**, and
the USR1 kills it even when the trap is installed correctly. That the
old fixture survived at all depended on bash and dash suppressing the
fork for the last command of a `-c` script when no non-ignored trap is
set. Verified locally, and the suppression is visible in `ps`: with
`trap '' USR1` present, the only member of the group is the **shell's
own pid** running `sleep` — the shell replaced itself, so there was
never an untrapped child to kill.

**An optimization is not a guarantee.** The fixture now says `exec`, so
the group holds exactly one process and the ignored disposition survives
by POSIX. This is a second readiness-shaped defect in the same fixture:
the test depended on a state it never established.

### 7.3 §1.5 scoped the fix one function too narrowly

`wait_for_published_file` — in the same suite, one function above
`wait_for_file` — gates the real-TUI smoke's `assert_eq!(…, b"1")` on
the identical "`fs::read` succeeded" predicate. §1.5's note that the
*bottom-panel* helper already rejects empty reads is about a different
file and correctly refuses scope creep there; it does not cover this
one. Fixing only `wait_for_file` would have left R4's mechanism live
under a different selector, where the registry would have had to judge
the recurrence a **new incident**.

### 7.4 What a witness had to be, in each case

- **R4's witness reproduces the row.** The file is created empty and
  filled later, so the window is certain rather than load-dependent, and
  with the old predicate restored it fails `left: []`, `right: [49]` —
  the row's two required fragments, verbatim.
- **R2's witness could not reproduce the row on Linux**, because the old
  fixture passes here; the pre-trap window is real everywhere but only
  macOS ever reported the failure. So the witness **widens the window
  deliberately** (the fixture sleeps before `trap`) and proves survival
  by the child's **exit disposition** — `Signaled { signal: "SIGUSR1" }`
  versus `"SIGTERM"` — rather than by an absence observed within a
  window, which would have been another timing assumption in a lane
  about timing assumptions.
- Bet 1 is therefore **supported but not yet resolved**: both mechanisms
  are gone and both are witnessed, but the bet is falsified only by a
  recurrence in CI, which only time can supply.

## 8. Branch plan

Two PRs, in this order — **both now exist**:

1. **`macos-ci-signal-integrity`** — the registry, the pointer
   rewrites, the audit, and the rerun rule. No code. **Merged as #215.**
2. **`ci-signal-hardening`** — the `wait_for_file` predicate and the
   USR1 trap readiness, each with a repetition set. Signature 1 is
   referred to the async-runtime design question rather than patched.

Quarantine, if it happens, is a third and is scoped by what hardening
fails to fix.