From 7ce03018f82f0eb8b119bfebd0824f0289e8347e Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Mon, 31 Aug 2026 19:40:57 +0200 Subject: [PATCH] docs(ci-reds): R7 has TWELVE occurrences --- absorb the two owed ones `docs/active-work.md` recorded two full-fragment R7 occurrences from 2026-08-15 (logs 20260815T095532Z and T100719Z, `attach.rs:1728`) under a heading saying they were "owed to the registry by whichever branch merges second". Both branches merged. Nothing carried them across, and they sat there for sixteen days, so R7's count read two low even after yesterday's renumbering. With those absorbed and the duplicate "fourth" fixed, the sequence is: August 29 = ninth and tenth, August 30 = eleventh, August 31 = twelfth. The parse-budget lane block, which still said sixth and seventh, is updated too. The deferral itself was reasonable --- this file has been bitten by two branches inventing the same row id --- but not discharging it was not. The lesson recorded is narrower than "absorb faster": an entry parked under "owed to the registry" needs an owner named in the same sentence, or it belongs to nobody. The summary cell's line-specific claim was also incomplete: occurrences one through FOUR report `attach.rs:1680`, not the first three. And U18 over-corrected. `GONOSUMDB` is a real Go variable --- `go help environment` documents `GOPRIVATE, GONOPROXY, GONOSUMDB` as module prefixes "that should not be compared against the checksum database", which is exactly the step that failed. It is technically applicable; whether the authentication tradeoff is acceptable is a different question. Discarding a real knob while correcting an invented one is its own error and is recorded as one. --- docs/active-work.md | 28 ++++++++------ docs/ci-red-signatures.md | 80 +++++++++++++++++++++++++++++---------- 2 files changed, 76 insertions(+), 32 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index 6149293..fc07ad3 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -369,14 +369,16 @@ touching crdt-gated code does not rediscover it at CI. **EIGHT registry rows moved on this lane** — R7, R6, U6, U14, U15, U16, U17 and U18. *(This heading said "four" while listing more; corrected.)* -- **R7's ninth and TENTH occurrences** — the ninth is the green/red - pair whose heads differ by one markdown file, which excludes the - SOURCE TREE and nothing more; an earlier write-up of mine narrowed the - cause to three gate-state candidates and that overstatement is - withdrawn in the row. **The row also carried TWO blocks numbered - "fourth"** — D3 on 2026-08-11 and TMPDIR isolation on 2026-08-13 — so - every later ordinal was one low. Renumbered by date, and the row's - summary now states the total rather than "three"; +- **R7's eleventh and TWELFTH occurrences** — the eleventh is the + green/red pair whose heads differ by one markdown file, which excludes + the SOURCE TREE and nothing more; an earlier write-up of mine narrowed + the cause to three gate-state candidates and that overstatement is + withdrawn in the row. **The row's numbering was wrong twice over**: it + carried TWO blocks labelled "fourth" (D3 2026-08-11, TMPDIR + 2026-08-13), **and** two full-fragment occurrences of 2026-08-15 sat + in this file marked "owed to the registry" and were never absorbed. + Renumbered by date with both fixed, and the row's summary now states + the total and which `attach.rs` line each group reports; - **U6 went from one occurrence to five** — four on 2026-08-30, two out of gate and two in. Its first reproduction ever. **A direction claim I made here ("runs the OPPOSITE way to R7", resting on four green @@ -556,7 +558,7 @@ from #171 and #215. U13 is not this lane's and is left as recorded, but the pattern is now a pattern rather than an oversight, and each occurrence costs a review round to establish nothing. -- **R7 gained its sixth and seventh occurrences here**, on a branch +- **R7 gained its ninth and tenth occurrences here** (recorded at the time as its sixth and seventh; the row was renumbered on 2026-08-31 after two duplicate ordinals and two unabsorbed 2026-08-15 occurrences were found), on a branch touching no `pmacs-gpu` file, and the registry gained a **bounded observation window** so the in-gate/out-of-gate ratio cannot drift with review activity — plus a correction: seventeen green @@ -1359,8 +1361,12 @@ from #171 and #215. branches' entries "merged **without a conflict**, producing duplicate ids across four sites". **The rows below are owed to the registry by whichever branch merges second**, numbered after the other's. - - **R7, two occurrences on this branch** (2026-08-15, `gpu` step, - logs `20260815T095532Z` and `20260815T100719Z`). Fragments verified + - **R7, two occurrences on this branch — ABSORBED 2026-08-31 as the + row's seventh and eighth** (2026-08-15, `gpu` step, + logs `20260815T095532Z` and `20260815T100719Z`). **They sat here + unabsorbed for sixteen days** while both branches merged, which is + why R7's count read two low; the deferral was reasonable, not + discharging it was not. Fragments verified both times: `transient sequence must attach: Attach(Handshake(Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })))` at `pmacs-gpu/src/attach.rs:1728`. One machine, one day, one branch, diff --git a/docs/ci-red-signatures.md b/docs/ci-red-signatures.md index 55c7ac1..0ddf7d2 100644 --- a/docs/ci-red-signatures.md +++ b/docs/ci-red-signatures.md @@ -595,26 +595,26 @@ Stage 4; the lane touches no `pmacs-gpu` code at all. | **selector** | `-p pmacs-gpu attach::tests::managed_retry_survives_transients_and_uses_the_successful_stream` | | **job / flavor** | local (Linux), `cargo test --workspace --features crdt --no-fail-fast`, i.e. under full-sweep load | | **required fragments** | `transient sequence must attach` + `Handshake(Io(` + `BrokenPipe` (or `code: 32`) | -| **status** | **TENTH OCCURRENCE 2026-08-31 — causal status still UNRESOLVED.** The ninth carries the strongest tree exclusion this row has had, and it supersedes the sixth's: two consecutive gate runs on ONE worktree whose heads differ by a single markdown file, the first all-green and the second red. **NOTE: this row carried TWO blocks numbered "fourth" — D3 on 2026-08-11 and TMPDIR isolation on 2026-08-13 — so every later ordinal was one low until 2026-08-31. Renumbered by date; the count below is the total** | -| **what IS established** | **TEN occurrences**, of which the first three sit at `pmacs-gpu/src/attach.rs:1680` (the line moves as `attach.rs` changes; this row treats a `:LINE` suffix as occurrence-specific), the second and third with all three fragments **verified** rather than inferred. The test drives a scripted transient-then-success sequence over a real socket pair. **The added GPU test is not the mechanism** — see the third-occurrence control below | +| **status** | **TWELFTH OCCURRENCE 2026-08-31 — causal status still UNRESOLVED.** The eleventh carries the strongest tree exclusion this row has had, and it supersedes the sixth's: two consecutive gate runs on ONE worktree whose heads differ by a single markdown file, the first all-green and the second red. **NOTE: the numbering was wrong twice over.** The row carried TWO blocks labelled "fourth" (D3 on 2026-08-11, TMPDIR isolation on 2026-08-13), and **two further full-fragment occurrences of 2026-08-15 sat in `docs/active-work.md` marked "owed to the registry" and were never absorbed** (logs `20260815T095532Z`, `20260815T100719Z`). Renumbered by date with both defects fixed; the count below is the total | +| **what IS established** | **TWELVE occurrences.** The line moves as `attach.rs` changes and this row treats a `:LINE` suffix as occurrence-specific: occurrences **one through four** report `pmacs-gpu/src/attach.rs:1680`, the sixth through eighth `:1728`, and the ninth through twelfth `:1889`. The second and third carry all three fragments **verified** rather than inferred; so do the seventh and eighth. The test drives a scripted transient-then-success sequence over a real socket pair. **The added GPU test is not the mechanism** — see the third-occurrence control below | | **what is NOT** | whether the broken pipe is the *fixture's* writer closing early or a real retry-path defect. **This row is not a claim that it is harmless** | | **rerun evidence** | occurrence 1: 6 isolated runs green, plus a full `--workspace --features crdt` sweep green (113 targets). Occurrence 2: **30 green on the observing branch** (15 isolated selector, 15 full `-p pmacs-gpu`) **plus a 15-run merge-base control, also green**. Occurrence 3: 5 isolated selector runs green, 10 full `-p pmacs-gpu` runs green **with** the added test, and **1 failure in 10 with the added test `#[ignore]`d** — the first rerun in this row's history that reproduced anything. Per the rerun rule the green runs establish intermittence only; the red control run is what carries the exclusion | | **retirement** | hardening that removes the named mechanism plus a discriminating witness — or a diagnosis showing the fixture, not the code, closes the pipe | -**Tenth occurrence — the same lane, 2026-08-31, local (Linux), `gpu` +**Twelfth occurrence — the same lane, 2026-08-31, local (Linux), `gpu` step**, log `20260831T141818Z-2974002`. All three required fragments, same selector, same `pmacs-gpu/src/attach.rs:1889`, same `283 passed; 1 failed`. The other seven stages were green. **Recorded, and it adds nothing but a count — deliberately.** The -ninth occurrence's method note says the remaining candidates have to be +eleventh occurrence's method note says the remaining candidates have to be varied **inside** the gate, one per run; that has not been done here and this lane is not the place to start. `/proc/loadavg` at the failure read `3.35 8.88 5.93` — a condition, recorded because U15 made load a measured quantity in this file, **not** a cause, and R7 is not a budget row. The observing commit is documentation only. -**Ninth occurrence — the CRDT identity-undo lane, 2026-08-30, local +**Eleventh occurrence — the CRDT identity-undo lane, 2026-08-30, local (Linux), `gpu` step.** All three required fragments present in the durable log (`pmacs-fdccc423/gate-logs/20260830T155621Z-3005460/06-gpu.log`): @@ -649,7 +649,7 @@ and nothing else: `283 passed; 1 failed` in each. * **Rerun: isolated selector green five times** (`1 passed`, 0.00s each). Per this file's rerun rule that establishes **intermittence - only** — and per the eighth occurrence's correction, running the + only** — and per the tenth occurrence's correction, running the selector outside the gate excludes nothing at all, because nothing outside the gate has ever reproduced this failure. * **No ratio is claimed from this occurrence.** It is one in-gate @@ -664,10 +664,10 @@ and nothing else: `283 passed; 1 failed` in each. (`20260830T160242Z-3095339`), one commit later. Recorded because omitting it would be selective, not because it resolves anything: per the rerun rule a green run establishes intermittence only, and the - eighth occurrence already falsified "in-gate always fails". + tenth occurrence already falsified "in-gate always fails". **What this changes about the method, stated at the strength it -carries.** The eighth occurrence's narrowing said the remaining +carries.** The tenth occurrence's narrowing said the remaining candidates must be varied INSIDE the gate, one per run. This pair sharpens **one** exclusion and nothing else: **the Rust source tree is not the variable.** @@ -684,7 +684,7 @@ else the machine was doing at 15:48 versus 15:56. A socket handshake racing a `BrokenPipe` is exactly the kind of failure those can drive, and holding the tree fixed says nothing about any of them. -**Seventh occurrence — the parse-budget diagnosability lane, 2026-08-29, +**Ninth occurrence — the parse-budget diagnosability lane, 2026-08-29, local (Linux), `gpu` step.** All three required fragments present in the durable log (`pmacs-parse-budget-9c27ecfe/gate-logs/20260829T144541Z-350549/06-gpu.log`): @@ -696,7 +696,7 @@ kind: BrokenPipe, message: "Broken pipe" }))) at `pmacs-gpu/src/attach.rs:1889`, `283 passed; 1 failed`. -* **The tree exclusion is as strong as the sixth's.** The observing +* **The tree exclusion is as strong as the sixth's.** (Occurrences seven and eight, below, are on that same lane's branch.) The observing lane's entire diff is `src/async_runtime.rs`, `tests/m4_acceptance.rs` and three docs — **no `pmacs-gpu` file, and no file `pmacs-gpu` links against beyond the workspace it always did.** @@ -708,13 +708,13 @@ at `pmacs-gpu/src/attach.rs:1889`, `283 passed; 1 failed`. `07-sweep.log` ends in `Terminated`. That stage's absence says nothing, and the run as a whole is **not** a gate result. Only the `gpu` stage's failure is, because it completed and reported. -**AN EIGHTH OCCURRENCE FOLLOWED IMMEDIATELY**, on the next gate run of +**A TENTH OCCURRENCE FOLLOWED IMMEDIATELY**, on the next gate run of the same worktree at head `45d438c` (`20260829T150011Z-429115/06-gpu.log`), same selector, same three fragments, `283 passed; 1 failed`. **That run's other seven stages were green**, `sweep` included and complete — 121 result lines, none with a failure — so this pair is not confounded by a truncation the way the -seventh was. +ninth was. **Two consecutive in-gate failures is new for this row**, whose prior five were spread across lanes and months. It prompted a narrowing, and @@ -769,6 +769,30 @@ lanes and trees, and these locate the asymmetry in the *runner* while showing that the obvious way to probe it — reproducing gate conditions outside the gate — cannot work. +**Seventh and eighth occurrences — the same §5b branch, later the same +day, 2026-08-15, local (Linux), `gpu` step.** Logs +`20260815T095532Z` and `20260815T100719Z`. **All three fragments +verified both times** — +`transient sequence must attach: Attach(Handshake(Io(Os { code: 32, +kind: BrokenPipe, message: "Broken pipe" })))` at +`pmacs-gpu/src/attach.rs:1728`, the same line as the sixth. One machine, +one day, one branch, **with a green full-gate run between them**. +Isolated reruns green. + +**These sat unabsorbed for sixteen days, and that is the finding worth +keeping.** `docs/active-work.md` recorded them under a heading saying +they were "**owed to the registry by whichever branch merges second**", +deliberately held back to avoid inventing a row id against an unseen +neighbour — a real hazard this file has been bitten by, when two +branches' entries merged without a conflict and produced duplicate ids +across four sites. **The deferral was reasonable; not discharging it was +not.** Both branches merged, and nothing carried them across, so R7's +count read two low until 2026-08-31. + +*The lesson is narrower than "absorb faster": an entry parked under +"owed to the registry" needs an owner named in the same sentence, or it +belongs to nobody.* + **Sixth occurrence — panel cell-mapping generation (§5b) framing, 2026-08-15, local (Linux).** The `scripts/gate` **`gpu` step** again, the same flavor as occurrence 2, inside a `--protocol` run @@ -790,9 +814,11 @@ the same flavor as occurrence 2, inside a `--protocol` run not exonerate the tree — though here there is no tree change to exonerate. -**What six occurrences now support, stated carefully:** the failure is -**not lane-correlated**. It has appeared under three flavors across -six unrelated lanes, once on a diff that touches no code whatsoever. +**What the first six occurrences supported, stated carefully** — and it +is unchanged by the six that followed: the failure is **not +lane-correlated**. It has appeared under three flavors across six +unrelated lanes by this point, once on a diff that touches no code +whatsoever. That is evidence about *where the cause is not*, and still says nothing about what it is. **The retirement condition is unchanged.** @@ -1662,12 +1688,24 @@ three were wrong.** They are corrected here rather than deleted: vendor directories**, so pinning that way needs a different installation path entirely. -**No replacement knob is named, because none was verified.** The one -option above that certainly applies is **retrying the step**; anything -else requires someone to check the current toolchain's actual switches -first. Choosing among them is not this lane's work — but inventing an -environment variable to fill a sentence is the failure this file exists -to catch, so it is recorded as one. +**`GONOSUMDB` was thrown out with them, and should not have been.** It +is real, and `go help environment` on the toolchain in this checkout +documents `GOPRIVATE, GONOPROXY, GONOSUMDB` as glob patterns of module +prefixes "that should always be fetched directly **or that should not be +compared against the checksum database**" — precisely the step that +failed here. So it is **technically applicable**. Whether exempting a +dependency from checksum verification is an acceptable authentication +tradeoff to buy CI stability is a real question, and a different one +from whether the knob exists. The same page notes `GOINSECURE` does +**not** disable that validation, which is the kind of near-miss that +made the original sentence wrong. + +**The options that stand, then:** `GONOSUMDB`/`GOPRIVATE` with the +tradeoff stated, or **retrying the step**. Choosing between them is not +this lane's work. Both directions of this paragraph's history are +recorded because both were errors: **inventing an environment variable +to fill a sentence**, and then **discarding a real one while correcting +it**. **No rerun was performed**, and deliberately: U3's lesson is to read the log first, and the log is now read and quoted above. Whether a rerun