docs: measure the CI coverage gap and record the vterm as-framed audit

The lane opened in the previous commit was scoped to the Vterm Stage 3
acceptance. Measuring it properly shows the problem is much larger and
not vterm-specific.

Comparing cargo test --list under CI's exact flags against the same
flags plus crdt: 3,024 versus 3,288. 264 tests are dark in CI, and the
single worst line is the library itself at 177 -- cargo test --lib
--features crdt is a required local gate that CI has never run. Ten
suites run zero or one test, including gpu_initial_target (#148's entire
acceptance, 1 of 14), gpu_invocation (#141's, 1 of 14), and a37, the
Stage 3 real-daemon/real-PTY/real-wgpu path that #135 built precisely
because a decoded-message fixture would prove none of the three fit
together.

The lane now carries the per-target table, the verified flag combination
for the fix, a two-part fix shape (a crdt leg on the test job, plus the
GPU-requiring suites onto the existing gpu-render job that already has
lavapipe), and an explicit instruction to sort deliberate exclusions
from accidental ones first -- some of the 264 are perf suites that are
ignored by default and belong to their own jobs, while m10_10_perf has
no ignore attribute and no job naming it.

docs/vterm-framing.md gains an as-framed audit section. The arc is
structurally complete and every test named in the Stage 2 verification
map exists, but criterion 22's "without thrash" clause was never pinned
anywhere -- the word appears nowhere in src or tests -- and that clause
describes exactly the defect #166 fixed. Of the nine Stage 3 tests, only
three drive a real daemon, so the six that construct EditorState
directly could never see a dispatcher-loop defect; a31 passes on the
broken tree for that reason. Four of the nine, including a37 and Stage 3
review round 1's own presence regression guard, do not run in CI at all.
The section also records what was not audited: section 11's blanket
claim about deferral safety covers roughly twenty items and none were
spot-checked.

docs/gpu-terminal-input-framing.md scores bet B2 true now that the
reporter has confirmed typing works, and retracts Q#GT5. The bash fixture
behind it does not reproduce in real use and was almost certainly
measuring its own timing rather than a product behaviour; it is marked
retracted rather than deleted so nobody re-derives it from an earlier
revision.

docs/agent-handoff.md section 5 gains the lesson the confirmation cost:
a daemon-side fix is not deployed until the daemon is restarted from a
tree containing it, and rebuilding a binary does nothing to a running
process.

No code changes.
This commit is contained in:
Levi Neuwirth 2026-07-25 17:17:23 -04:00
parent 8c924ebf41
commit 40709fafb9
4 changed files with 129 additions and 30 deletions

View File

@ -227,32 +227,68 @@ If it does not, stop and repair the remote/fetch configuration.
buffer a directory should resolve *to*, and `pmacs .` should route into buffer a directory should resolve *to*, and `pmacs .` should route into
it rather than growing a second directory surface. it rather than growing a second directory surface.
## Stage 3 acceptance is dark in CI — NEEDS A LANE ## The CRDT half of the test corpus is dark in CI — NEEDS A LANE
- **No branch, no framing yet.** Found while gating #166; deliberately kept - **No branch, no framing yet.** Found while gating #166, then measured
out of it so a CI change would not arrive after review approval. properly during the vterm as-framed audit. Deliberately kept out of #166 so
- `.github/workflows/ci.yml` **never enables the `crdt` feature** (grep the a CI change would not arrive after review approval.
workflow directory: zero hits). Every `#[cfg(feature = "crdt")]` acceptance - **Root cause:** `.github/workflows/ci.yml` never enables the `crdt` feature
test is therefore not merely skipped in CI — it is **not compiled**. anywhere — zero hits across the workflow directory. The `test` job runs
- That covers the whole Vterm Stage 3 real-path acceptance, including `a37` `cargo test --all-targets --no-default-features --features luajit|lua54`.
(real daemon + real PTY + real wgpu), which has been dark since #135, and Every `#[cfg(feature = "crdt")]` test is therefore **not compiled** in CI,
the two tests #166 added beside it. not merely skipped.
- The `gpu-render` job is the only one with lavapipe and - **Measured, `--list` under CI's exact flags versus the same flags plus
`PMACS_REQUIRE_GPU=1`, and it runs `cargo test -p pmacs-gpu`, which never `crdt`: 3,024 vs 3,288 — 264 tests dark.** Per target:
reaches the `pmacs` crate's acceptance suites.
- The shape of the fix is one step on the `gpu-render` job: | dark | CI | full | target |
`cargo test --features crdt --test vterm_stage3_acceptance -- --test-threads=1`. |---:|---:|---:|---|
It needs its own lane rather than a drive-by because it would run `a37` | 177 | 1,832 | 2,009 | **the library itself** (`src/lib.rs`) |
under lavapipe **for the first time**, and neither its timing budgets nor | 21 | 15 | 36 | `m5_5_acceptance` |
its wgpu path have ever been exercised on that adapter or on macOS CI. | 13 | 1 | 14 | `gpu_invocation_acceptance` |
- Worth auditing at the same time: which *other* `crdt`-gated acceptance | 13 | 1 | 14 | `gpu_initial_target_acceptance` |
suites are dark for the same reason. This is a coverage question about the | 8 | 0 | 8 | `m10_11_acceptance` |
gate list itself, not about any one suite. | 6 | 0 | 6 | `auto_pair_crdt_acceptance` |
| 6 | 0 | 6 | `m10_2_perf` |
| 4 | 5 | 9 | `vterm_stage3_acceptance` |
| 4 | 0 | 4 | `m10_10_perf` |
| 3 | 0 | 3 | `compile_mode_crdt_acceptance` |
| 2 | 22 | 24 | `theme_faces_acceptance` |
| 2 | 0 | 2 | `m11_5_semantic_acceptance` |
| 1 | 9 | 10 | `vterm_stage1_acceptance` |
| 1 | 7 | 8 | `statusline_segments_acceptance` |
| 1 | 10 | 11 | `gpu_font_acceptance` |
| 1 | 0 | 1 | `auto_indent_crdt_acceptance` |
| 1 | 0 | 1 | `m10_11_perf` |
- **The single worst line is the library.** `cargo test --lib --features crdt`
is a REQUIRED local gate in `CLAUDE.md`, and CI has never run it. 177
library tests — the whole CRDT half — are developer-machine-only.
- **Ten suites run zero or one test in CI**, including `gpu_initial_target`
(#148's entire acceptance, 1/14), `gpu_invocation` (#141's, 1/14), and
`a37`, the Vterm Stage 3 real-daemon/real-PTY/real-wgpu path that #135
built specifically because "a decoded-message fixture would prove none of
the three fit together".
- **Sort deliberate from accidental before proposing a fix.** Some of the 264
are perf suites that are `#[ignore]`d by default and belong to their own
jobs (`m10_2_perf` 6, `m10_11_perf` 1). `m10_10_perf` has **no** `#[ignore]`
and no CI job naming it, so it looks accidental. This classification is not
finished and is the lane's first task.
- **Fix shape, two parts** (the flag combination is verified to work:
`--no-default-features --features luajit,crdt` lists 10 vterm Stage 1 tests
versus 9 without):
1. a `crdt` leg on the `test` job for the non-GPU suites and the library;
2. the GPU-requiring `crdt` suites onto the existing `gpu-render` job, which
already has lavapipe and `PMACS_REQUIRE_GPU=1`
`vterm_stage3_acceptance`, `gpu_invocation_acceptance`,
`gpu_initial_target_acceptance`, `gpu_font_acceptance`.
- **Expect first-run failures, and budget for them.** These would execute in
CI for the first time ever: real PTY timing on CI runners, wgpu under
lavapipe, and daemon-socket tests at unfamiliar concurrency. Start
ubuntu-only and decide about macOS from evidence. A red first run is the
lane working, not the lane failing.
- Mitigating fact, verified rather than assumed: #166's three unit pins are - Mitigating fact, verified rather than assumed: #166's three unit pins are
**not** `crdt`-gated and do run under CI's exact flags **not** `crdt`-gated and do run under CI's exact flags, including the
(`--no-default-features --features luajit|lua54`), including the
controller-release pin whose only job is catching the plausible wrong fix. controller-release pin whose only job is catching the plausible wrong fix.
The regression protection is live; the real-daemon evidence is local-only.
## Bottom-panel lane (window placement + side windows) — Stage 1 IN REVIEW ## Bottom-panel lane (window placement + side windows) — Stage 1 IN REVIEW

View File

@ -745,6 +745,15 @@ final variant — its own round-trip cannot detect a discriminant shift.
## 5. Hard-won ops lessons ## 5. Hard-won ops lessons
- **A daemon-side fix is not deployed until the daemon is restarted from a
tree that contains it.** #166's reporter rebuilt and saw no change: the
running daemon had been started from a shared checkout still on a pre-fix
branch, and `pmacs --gpu` attaches to whatever process already owns the
socket. Rebuilding a binary does nothing to a running process. When
validating a daemon-side fix by hand, check the running process's binary
path and start time against the tree you think you fixed —
`ps -eo pid,lstart,args | grep '[p]macs --daemon'` — before concluding the
fix failed.
- **Two operations that must be alternatives are not made alternatives by - **Two operations that must be alternatives are not made alternatives by
being adjacent.** The dispatcher applied its grid and semantic being adjacent.** The dispatcher applied its grid and semantic
terminal-layout syncs to every attached frontend; a semantic session terminal-layout syncs to every attached frontend; a semantic session

View File

@ -81,7 +81,11 @@ character:
| | frames for a static screen | typed `Z` ever visible at the prompt | | | frames for a static screen | typed `Z` ever visible at the prompt |
|---|---|---| |---|---|---|
| `main` today | **730** in a 20 s window | **no** | | `main` today | **730** in a 20 s window | **no** |
| with the guard | **2** | (see Q#GT5 — a separate question) | | with the fix | **2** | yes |
Bet B2 is **scored TRUE**: with the fix deployed, the reporter confirmed
typing into a GPU terminal works. The earlier caveat here pointed at Q#GT5,
which is now retracted — see "Deferred (named)".
The TUI is unaffected: a grid session has no semantic terminal declaration, so The TUI is unaffected: a grid session has no semantic terminal declaration, so
only one arm ever runs for it. This is a **frontend-kind** defect, which is only one arm ever runs for it. This is a **frontend-kind** defect, which is
@ -281,11 +285,12 @@ change. Stays v20.
snapshot that signals the switch-away). Hence the split in Q#GT1. Recorded snapshot that signals the switch-away). Hence the split in Q#GT1. Recorded
rather than deleted: the failure mode is one a reviewer or a future rather than deleted: the failure mode is one a reviewer or a future
simplification will re-propose. simplification will re-propose.
- **B2.** The user's reported symptom is this defect. *Partially scored: the - **B2 — SCORED TRUE 2026-07-25.** "The user's reported symptom is this
storm is proven and GUI-only, and its shape (line editor unusable, output defect." Confirmed in real use after the fix was deployed: typing into a GPU
still flowing) matches the report. Not fully scored until the user, or an terminal works. The confirmation needed a daemon **restart** built from a
acceptance running the **user's own shell**, confirms typing works after the tree containing the fix — the first attempt reported no change because a
fix. Q#GT5 is the reason this bet is stated rather than assumed.* pre-fix daemon still owned the socket, which is worth remembering whenever a
daemon-side fix is being validated by hand.
- **B3.** No other pair of per-frontend-kind daemon operations is applied as - **B3.** No other pair of per-frontend-kind daemon operations is applied as
siblings rather than alternatives. *Scored by an explicit audit of the siblings rather than alternatives. *Scored by an explicit audit of the
dispatcher's per-frontend loop during implementation — this defect's shape dispatcher's per-frontend loop during implementation — this defect's shape
@ -294,7 +299,14 @@ change. Stays v20.
## Deferred (named) ## Deferred (named)
- Interactive-shell echo on a raw-mode PTY (Q#GT5) — its own scout. - ~~Interactive-shell echo on a raw-mode PTY (Q#GT5)~~ — **RETRACTED
2026-07-25.** The observation behind it (a `bash --norc -i` fixture not
echoing typed characters) does not reproduce in real use: with the fix
deployed, typing into a GPU terminal echoes normally. The fixture was almost
certainly measuring its own timing — polling a published screen snapshot
before readline had finished initialising — not a product behaviour. Recorded
as retracted rather than deleted so nobody re-derives it from the framing's
earlier revision and spends a scout on it.
- **A geometry change appears to clear the visible screen.** Observed while - **A geometry change appears to clear the visible screen.** Observed while
building acceptance 4: after the probe's deliberate 25×92 → 20×71 resize, building acceptance 4: after the probe's deliberate 25×92 → 20×71 resize,
the next frame's visible grid is entirely blank even though the content the next frame's visible grid is entirely blank even though the content

View File

@ -1674,6 +1674,48 @@ GPU assertions remain in `pmacs-protocol` and `pmacs-gpu` respectively.
- **37:** one real-daemon/real-PTY/headless-wgpu acceptance path; it is not - **37:** one real-daemon/real-PTY/headless-wgpu acceptance path; it is not
replaced by a decoded-message fixture. replaced by a decoded-message fixture.
### 0.12 As-framed audit, 2026-07-25 (after #166)
Prompted by a GPU terminal input defect that shipped in Stage 3 and was fixed
in #166. The arc is structurally complete — all 37 criteria have
implementations, and every test named in the Stage 2 verification map exists —
but the audit found two gaps worth recording against the criteria themselves.
**Criterion 22's "without thrash" was never pinned.** The criterion reads
"unchanged, zero, passive, and failed resize cases preserve prior geometry
*without thrash*". The word appears nowhere in `src/` or `tests/`. The suite
pinned the four enumerated single-arm cases and never the cross-arm
interaction — which is exactly where the thrash lived: the daemon applied
both the grid and the semantic terminal-layout sync to every attached
frontend, so a semantic session's PTY was resized twice per tick forever.
Criterion 31's "only the exact durable controller changes PTY geometry" was
violated in the same event, in spirit rather than letter: the controller was
the right frontend, but the geometry came from the grid projection. #166 adds
the settle pins; the gap was open from #135 (2026-07-22) until then.
**Why the Stage 3 suite could not see it.** Of its nine tests, only three
drive a real daemon; the other six construct `EditorState` directly and never
execute the dispatcher loop where the defect lived. `a31`, which is about two
semantic frontends sharing one session, therefore passes on the broken tree.
The same structural blindness explains why `bottom_panel_stage1_acceptance`
was unaffected. A criterion about *dispatcher* behavior needs a test that
runs the dispatcher.
**Four of the nine Stage 3 tests do not run in CI at all**, because they are
`#[cfg(feature = "crdt")]` and the workflow never enables that feature:
`a37`, the two added by #166, and
`terminal_mode_keeps_reporting_presence_so_peers_drop_the_stale_caret` — which
is Stage 3 review round 1's own regression guard. Stage 1's
`read_only_empty_crdt_bootstrap_is_immutable_against_remote_content`, the CRDT
half of criterion 14, is dark for the same reason. Stage 2 is fully covered
(6/6). This is not a vterm problem: 264 tests workspace-wide are dark,
including 177 in the library. It has its own lane in `docs/active-work.md`.
**Not audited:** §11's blanket claim that "deferral means graceful ignore or
documented absence, never escape leakage, panic, unbounded allocation, or
child leak". That covers roughly twenty deferred items and none were
spot-checked. It remains an unproven claim rather than a known gap.
## 10. Gates and bite verification ## 10. Gates and bite verification
Every PR runs the standing full gates from `AGENTS.md`, sequentially, plus its Every PR runs the standing full gates from `AGENTS.md`, sequentially, plus its