21 KiB

Raw Permalink Blame History

Framing — macOS CI signal integrity: a signature registry, then hardening

Revision 4. Status: Stage 1 MERGED as #215 (main @ 12f2970); Stage 2 implemented on ci-signal-hardening. Scouted against githubsucks/main @ bfb97c6.

Revision 4 records implementation findings, not a new design round. The acceptance in §4 is unchanged and was approved at revision 3; §8 below is what building Stage 2 established, including two places where this document's own account of a mechanism was imprecise.

Revision 2 → 3 exists because the implementation discovered a state the contract did not allow, and the contract — not the implementation — was what needed changing. Acceptance 3 offered a binary: carry an incumbent in with a signature and evidence, or remove it as never substantiated. Both incumbents are neither, and shipping a third state while the governing criterion still said "two" would have made the framing describe something the branch does not do.

Revision 3 also retires the phrase "mechanism named", which overstated what the audit found. The a33 audit proves an assertion string and a historical claim exist; the m6_8 audit proves a test is timing-based. Neither establishes a failure mechanism — no occurrence was ever linked or captured, so nothing is known about how either fails. (Not "never observed": someone may well have seen one and not recorded it. What is established is the absence of a record, which is the only thing this audit can speak to.) They become audit notes A1/A2, not registry rows, and R-numbers are reserved for signatures with linked evidence.

Revision 2 separates a machine-matchable signature from verbatim, variable CI output; preserves historical incident evidence while centralizing live triage policy; gives retirement a causal rule rather than an arbitrary green-run count; and corrects the rerun rule.

Four red CI incidents across #213 and #214 were each correctly judged "not caused by this PR" — and #214's case is airtight, because it is docs-only and its tree is byte-identical to a green main. That proves the PRs did not cause them. It does not prove they are harmless environmental noise, and three of the four have a specific, findable mechanism.

This lane separates those two claims, which the current process conflates.

0. Coherence impact (COHERENCE §20)

Journey steps touched: none.
Interaction islands: none.
Config registry adoption: none.
Background-work attribution: none.
Why it belongs on the board: it protects §19's acceptance-test ratchet and every arc that reads a red run. A flake list that is wrong, incomplete, or keyed by test name rather than signature makes "rerun before concluding" a habit rather than a judgement — and a real regression arriving in that stream is indistinguishable from the noise it hides in.

1. Ground truth (measured at `bfb97c6`)

1.1 The accounting: FOUR incidents, THREE tests, FOUR signatures

The registry must count signatures, not test names. The process test alone produced two, with different mechanisms and different causal status — collapsing them under one name would have hidden a possible product defect behind a known-flaky label.

The registry key is an exact test selector + job/flavor + normalized match rule, not a pasted panic block. PIDs, elapsed times, and rendered OS-error suffixes vary between runs; each row therefore names the required invariant fragments and the evidence link preserves the verbatim occurrence. A row matches only when every listed requirement is satisfied; where a requirement lists alternative platform renderings, one of those alternatives suffices. A test-name match by itself never matches a registry entry.

#	exact test selector	job / flavor	required signature fragments	causal status
1	`--lib async_runtime::tests::supersede_cancels_in_flight_job_within_50ms`	macOS / luajit	`supersede did not cancel within 50ms`	measurement design — see §1.2
2	`--lib process::tests::a_successful_signal_disposition_depends_on_whether_it_is_fatal`	macOS / lua54	`leader=exited(signal SIGUSR1)`	test race — see §1.3
3	(same test)	macOS / lua54	all of `EPERM`, `measured_group=unobservable(` + `ESRCH` / `No such process`, and `leader=live`	UNRESOLVED — possible product defect — see §1.4
4	`--test vterm_stage2_acceptance terminal_escape_gates_local_bindings_and_double_escape_sends_interrupt`	macOS / luajit	both `left: []` and `right: [49]`	test race — see §1.5

Evidence: #213 run 30826884642, #213 run 30927084982 attempt 1, #214 run 30932558752 attempt 1.

1.2 Signature 1 — the 50ms budget measures more than it claims

let first = rt.dispatch_sleep(2_000, Some("search"));
// Let the worker pick the job up so cancel hits a running job.
thread::sleep(Duration::from_millis(15));
let started = Instant::now();
let _second = rt.dispatch_sleep(2_000, Some("search"));
while !rt.is_complete(first) {
    assert!(started.elapsed() < Duration::from_millis(50), "supersede did not cancel within 50ms");
    let _ = rt.tick();
    thread::sleep(Duration::from_millis(1));
}

Two design problems, and neither is fixed by a bigger number.

The premise is a sleep. thread::sleep(15ms) is asserted-by-comment to mean "the worker picked the job up". On a loaded runner it may not have, in which case the test measures cancellation of a queued job — a different code path, already covered by the sibling test — while claiming to measure a running one.
The clock includes the observer. started begins before the second dispatch, and the budget is consumed by the test's own tick() + sleep(1ms) pump loop. Under scheduling pressure the measured interval is dominated by when the test got scheduled, not by when the worker observed the cancel flag.

Widening 50ms to 200ms would make it pass and measure nothing more. The open question is what this test should assert: a latency bound needs a clock the observer does not participate in, or the assertion should be reformulated as ordering ("the first settles Cancelled before the second completes") rather than duration.

1.3 Signature 2 — `Started` does not prove the trap is installed

spec.args = vec!["-c".into(), "trap '' USR1; sleep 30".into()];
let id = sup.spawn(spec).expect("spawn");
let pid = spawn_started_pid(&mut sup, id);   // waits for ProcessEventKind::Started
sup.signal(id, Signal::SIGUSR1).expect("USR1 delivers");

spawn_started_pid waits for ProcessEventKind::Started { pid }, which is emitted when the process is spawned — not when /bin/sh has parsed and installed trap '' USR1. SIGUSR1's default disposition is terminate, so a signal delivered inside that window kills the child, and the record is exited(signal SIGUSR1) instead of Running.

The fixture's own comment states the requirement it does not enforce: "Ignore USR1 so the successful non-fatal signal cannot end the child and confuse the state assertion with a real exit." That is exactly the confusion observed.

This is the handoff's "wait predicate weaker than the assertion" race: the predicate is "process exists", the assertion needs "trap installed".

1.4 Signature 3 — the live-leader EPERM, deliberately unresolved

EPERM, measured_group=unobservable(ESRCH), leader=live is not established as a test race. It is the group-target behaviour the process-signal lanes have circled three times:

#176 established that a group-directed kill returned EPERM while the leader was observed alive by a real try_wait, retiring "EPERM cannot happen for our own children" as a reason to discard the errno.
#200 added measured_group from getpgid, the only field able to disagree — and it is reported here as unobservable(ESRCH), meaning the group could not be measured at all.
The reap-ledger lane parked every disposition change pending exactly this evidence.

This lane does not resolve it and must not appear to. Its registry entry carries causal status UNRESOLVED — possible product defect, and its retirement condition is a diagnosis, not a green rerun. Folding it under the same "macOS signal timing" label as signature 2 is how a real defect acquires a flake's immunity.

1.5 Signature 4 — a readiness predicate satisfied by an empty file

fn wait_for_file(path: &Path, timeout: Duration) -> Vec<u8> {
    loop {
        if let Ok(bytes) = fs::read(path) { return bytes; }   // succeeds on 0 bytes
        ...
    }
}

The probe writes readiness with open(path,'wb').write(b'1'). open() creates the file before write() fills it, so fs::read can succeed on a zero-byte file and wait_for_file returns []. The caller then asserts == b"1" and fails left: [], right: [49].

The predicate is "readable"; the assertion is "contains 1". This is the same shape as §1.3 and is fixable at the helper. All four callers of this helper require concrete non-empty content; the similar bottom-panel helper already rejects empty reads.

1.6 Live triage policy and historical evidence are mixed together

Flake language currently appears in the handoff's two operational rules, docs/active-work.md, and several landed framing documents. It is not all duplication. The process-signal and reap-ledger framings, for example, preserve exact historical occurrences and the reasoning those lanes built from them; replacing that evidence with a pointer would make a durable framing depend on a mutable registry.

The real duplication is live classification and triage policy. The handoff's hazards list names three tests — two of which are not among the four incidents seen here — while three of these four are absent. Elsewhere, historical occurrence notes, forward-looking risk warnings, and current "known flaky" claims are written in the same voice even though they require different treatment.

A list that is both stale and incomplete is worse than none: it confers "known flaky" on whatever happens to be named, and withholds it from everything else.

1.7 What is NOT established

No incident has been reproduced locally. All four are macOS-only and this machine is Linux. The mechanisms in §§1.2–1.5 are read from source and from CI signatures, not from a local repro.
Frequency is unmeasured. Four incidents across two PRs is not a rate. The registry records occurrences so a rate can accumulate; it does not claim one now.
Signature 3's cause remains unknown, by design (§1.4).

2. Questions

Q#MCI1 — where does the registry live? DECIDED: one dedicated docs/ci-red-signatures.md, with the handoff's operational rule pointing at it. The handoff is a briefing; the registry is an occurrence ledger whose rows remain available after retirement.
Q#MCI2 — what retires an entry? DECIDED: a mechanism-specific causal result, not N green runs. A known-race entry requires hardening that removes the named mechanism plus a discriminating acceptance witness for the stronger predicate. A measurement-design entry requires its owning lane to replace or justify the measurement and pin the resulting claim. An unresolved entry requires diagnosis and an explicit disposition. Main-job greens remain occurrence evidence, but cannot retire any row by themselves; retired rows remain in the history with their disposition.
Q#MCI3 — does signature 1 get reformulated or re-measured? §1.2 argues its budget measures the observer. Reformulating as an ordering assertion changes what the test proves; keeping a duration means finding a clock the pump loop does not participate in. This is the one question this lane should not answer alone — it is the async-runtime lane's design call.
Q#MCI4 — does hardening ship before or with the registry? DECIDED: the user has already answered: registry now, hardening next. Recorded here so the sequencing is visible in the document rather than only in the conversation.

3. Bets

Bet 1 — signatures 2 and 4 disappear under hardening, because both have a named mechanism and a fix at the readiness predicate. Falsified if either recurs after the predicate is strengthened.
Bet 2 — a signature-keyed registry refuses name-based immunity. A future failure in the process test which does not contain all of row 2's or row 3's required fragments is a new incident, not a known flake. Falsified if the operational rule permits a test-name-only match.
Bet 3 — green reruns do not erase unresolved evidence. Signature 3 remains unresolved until the process-signal lane diagnoses and disposes it, whether or not later runs pass. Falsified if a green run changes that row's status or retirement condition.

4. Acceptance

Stage 1 — the registry (this lane's first PR):

One authoritative table, with a row per signature: exact test selector, job and Lua flavor, normalized machine-match rule, evidence link to the verbatim occurrence, causal status, and mechanism-specific retirement condition. Matching requires the selector, job/flavor, and every required fragment; variable values are explicitly normalized rather than silently abbreviated.
One live triage policy. Operational duplicate classifications and rerun rules become pointers to the registry. Historical occurrence evidence stays where it supports a landed framing; a relevant note may gain a registry status link, but its evidence and reasoning are not replaced. Forward-looking risk statements are audited as risks, not silently promoted to known flakes.
The three tests named in the current handoff list are audited, into one of three states — the third was found by doing the audit and is why this is revision 3:
- carried as a registry row (R-numbered) when a signature and a linked occurrence both exist;
- removed when nothing substantiates it at all;
- recorded as an audit note (A-numbered) when a historical claim exists but no occurrence was ever linked. An audit note is not a registry row, cannot be matched against a red run, and confers nothing.
No entry survives on reputation, and an audit note is not a weaker row — it is a different kind of statement. A row says "this was seen, here is the evidence"; a note says "someone recorded a belief and no occurrence backs it."
The rerun rule is replaced, not softened:
- a green rerun after a red establishes intermittence only; it does not establish environmental cause, harmlessness, or retirement;
- the same signature on the rerun is a second occurrence and remains blocking pending investigation or a merge-base control;
- a different signature is a new incident and is judged independently. A known test name confers no immunity.
Signature 3 is recorded UNRESOLVED — possible product defect, and its retirement condition is a diagnosis, never a green rerun.

Stage 2 — hardening (a separate PR):

The vterm Stage 2 wait_for_file requires the expected content (preferred) or at minimum a non-empty result, so the predicate matches the assertion. Its four callers all require concrete non-empty content; the bottom-panel helper already rejects empty reads and is not evidence for widening this change further.
The USR1 fixture proves the trap is installed, not merely that the process started. The child publishes readiness after installing the trap, and the test waits on that.
Signature 1 is not fixed by widening the budget (Q#MCI3).
Both hardened tests run repeatedly (a repetition set, as the reap-ledger lane did) rather than once, because a single green run of a formerly intermittent test proves nothing.

Quarantine — only if hardening fails:

A quarantined test moves to a separate, still-blocking CI step. Never #[ignore], never continue-on-error, never a silent retry-to-green. A quarantine that stops failing the build is a deletion with extra steps.

5. Parked

Resolving signature 3. It belongs to the process-signal / reap-ledger lanes, which have already parked three tolerance rules pending this class of evidence.
The --no-fail-fast gap and the crdt job's PTY deadlines, both recorded in the CI CRDT lane. Related in spirit, separate in scope.
A general flake-rate dashboard. The registry accumulates occurrences; turning that into a rate with alerting is its own thing.
Linux and GPU flakes. a33_headless_terminal_frame_paints_... and m6_8_supervisor_reaps_... were audited under acceptance 3 and became audit notes A1/A2 — claims with no linked occurrence. A1's job runs on Ubuntu, so the registry is not macOS-only even now; A2's job was never recorded. This lane's four evidenced incidents are macOS, which is a fact about them and not about the file.

6. Gates

The standing CLAUDE.md suite. Stage 1 is documentation-only and adds no test; its verification is acceptance 3 — the audit that no entry survives on reputation. Stage 2 adds the repetition sets of acceptance 9.

7. Stage 2 implementation notes (revision 4)

Stage 2 landed acceptance 6–9 on ci-signal-hardening. Four findings, two of which correct this document.

7.1 §1.3 named the right window and the wrong assertion

leader=exited(signal SIGUSR1) is rendered only on a failed kill, by signal_failure_report. The USR1 cannot be the call that failed — it is the call that did the killing — so the failing call is the SIGTERM that follows, and the assertion that blew up is .expect("TERM delivers"), not the Running state check. §1.3's "the record is exited(signal SIGUSR1) instead of Running" reads as though ProcessState carries that value; it does not, and nothing ticks between the two calls, so the state assertion could not have seen an exit. The row's fragment and its mechanism were both right. Why a group-directed TERM then found no group is not established, and the fix does not depend on it.

7.2 The fixture had a second dependency nobody had named

These signals are group-directed, so the fixture's sleep 30 — if the shell forks it — is an untrapped member of the same group, and the USR1 kills it even when the trap is installed correctly. That the old fixture survived at all depended on bash and dash suppressing the fork for the last command of a -c script when no non-ignored trap is set. Verified locally, and the suppression is visible in ps: with trap '' USR1 present, the only member of the group is the shell's own pid running sleep — the shell replaced itself, so there was never an untrapped child to kill.

An optimization is not a guarantee. The fixture now says exec, so the group holds exactly one process and the ignored disposition survives by POSIX. This is a second readiness-shaped defect in the same fixture: the test depended on a state it never established.

7.3 §1.5 scoped the fix one function too narrowly

wait_for_published_file — in the same suite, one function above wait_for_file — gates the real-TUI smoke's assert_eq!(…, b"1") on the identical "fs::read succeeded" predicate. §1.5's note that the bottom-panel helper already rejects empty reads is about a different file and correctly refuses scope creep there; it does not cover this one. Fixing only wait_for_file would have left R4's mechanism live under a different selector, where the registry would have had to judge the recurrence a new incident.

7.4 What a witness had to be, in each case

R4's witness reproduces the row. The file is created empty and filled later, so the window is certain rather than load-dependent, and with the old predicate restored it fails left: [], right: [49] — the row's two required fragments, verbatim.
R2's witness could not reproduce the row on Linux, because the old fixture passes here; the pre-trap window is real everywhere but only macOS ever reported the failure. So the witness widens the window deliberately (the fixture sleeps before trap) and proves survival by the child's exit disposition — Signaled { signal: "SIGUSR1" } versus "SIGTERM" — rather than by an absence observed within a window, which would have been another timing assumption in a lane about timing assumptions.
Bet 1 is therefore supported but not yet resolved: both mechanisms are gone and both are witnessed, but the bet is falsified only by a recurrence in CI, which only time can supply.

8. Branch plan

Two PRs, in this order — both now exist:

macos-ci-signal-integrity — the registry, the pointer rewrites, the audit, and the rerun rule. No code. Merged as #215.
ci-signal-hardening — the wait_for_file predicate and the USR1 trap readiness, each with a repetition set. Signature 1 is referred to the async-runtime design question rather than patched.

Quarantine, if it happens, is a third and is scoped by what hardening fails to fix.

21 KiB Raw Permalink Blame History Unescape Escape