docs: measure the CI coverage gap and record the vterm as-framed audit
The lane opened in the previous commit was scoped to the Vterm Stage 3 acceptance. Measuring it properly shows the problem is much larger and not vterm-specific. Comparing cargo test --list under CI's exact flags against the same flags plus crdt: 3,024 versus 3,288. 264 tests are dark in CI, and the single worst line is the library itself at 177 -- cargo test --lib --features crdt is a required local gate that CI has never run. Ten suites run zero or one test, including gpu_initial_target (#148's entire acceptance, 1 of 14), gpu_invocation (#141's, 1 of 14), and a37, the Stage 3 real-daemon/real-PTY/real-wgpu path that #135 built precisely because a decoded-message fixture would prove none of the three fit together. The lane now carries the per-target table, the verified flag combination for the fix, a two-part fix shape (a crdt leg on the test job, plus the GPU-requiring suites onto the existing gpu-render job that already has lavapipe), and an explicit instruction to sort deliberate exclusions from accidental ones first -- some of the 264 are perf suites that are ignored by default and belong to their own jobs, while m10_10_perf has no ignore attribute and no job naming it. docs/vterm-framing.md gains an as-framed audit section. The arc is structurally complete and every test named in the Stage 2 verification map exists, but criterion 22's "without thrash" clause was never pinned anywhere -- the word appears nowhere in src or tests -- and that clause describes exactly the defect #166 fixed. Of the nine Stage 3 tests, only three drive a real daemon, so the six that construct EditorState directly could never see a dispatcher-loop defect; a31 passes on the broken tree for that reason. Four of the nine, including a37 and Stage 3 review round 1's own presence regression guard, do not run in CI at all. The section also records what was not audited: section 11's blanket claim about deferral safety covers roughly twenty items and none were spot-checked. docs/gpu-terminal-input-framing.md scores bet B2 true now that the reporter has confirmed typing works, and retracts Q#GT5. The bash fixture behind it does not reproduce in real use and was almost certainly measuring its own timing rather than a product behaviour; it is marked retracted rather than deleted so nobody re-derives it from an earlier revision. docs/agent-handoff.md section 5 gains the lesson the confirmation cost: a daemon-side fix is not deployed until the daemon is restarted from a tree containing it, and rebuilding a binary does nothing to a running process. No code changes.
This commit is contained in:
parent
8c924ebf41
commit
40709fafb9
|
|
@ -227,32 +227,68 @@ If it does not, stop and repair the remote/fetch configuration.
|
||||||
buffer a directory should resolve *to*, and `pmacs .` should route into
|
buffer a directory should resolve *to*, and `pmacs .` should route into
|
||||||
it rather than growing a second directory surface.
|
it rather than growing a second directory surface.
|
||||||
|
|
||||||
## Stage 3 acceptance is dark in CI — NEEDS A LANE
|
## The CRDT half of the test corpus is dark in CI — NEEDS A LANE
|
||||||
|
|
||||||
- **No branch, no framing yet.** Found while gating #166; deliberately kept
|
- **No branch, no framing yet.** Found while gating #166, then measured
|
||||||
out of it so a CI change would not arrive after review approval.
|
properly during the vterm as-framed audit. Deliberately kept out of #166 so
|
||||||
- `.github/workflows/ci.yml` **never enables the `crdt` feature** (grep the
|
a CI change would not arrive after review approval.
|
||||||
workflow directory: zero hits). Every `#[cfg(feature = "crdt")]` acceptance
|
- **Root cause:** `.github/workflows/ci.yml` never enables the `crdt` feature
|
||||||
test is therefore not merely skipped in CI — it is **not compiled**.
|
anywhere — zero hits across the workflow directory. The `test` job runs
|
||||||
- That covers the whole Vterm Stage 3 real-path acceptance, including `a37`
|
`cargo test --all-targets --no-default-features --features luajit|lua54`.
|
||||||
(real daemon + real PTY + real wgpu), which has been dark since #135, and
|
Every `#[cfg(feature = "crdt")]` test is therefore **not compiled** in CI,
|
||||||
the two tests #166 added beside it.
|
not merely skipped.
|
||||||
- The `gpu-render` job is the only one with lavapipe and
|
- **Measured, `--list` under CI's exact flags versus the same flags plus
|
||||||
`PMACS_REQUIRE_GPU=1`, and it runs `cargo test -p pmacs-gpu`, which never
|
`crdt`: 3,024 vs 3,288 — 264 tests dark.** Per target:
|
||||||
reaches the `pmacs` crate's acceptance suites.
|
|
||||||
- The shape of the fix is one step on the `gpu-render` job:
|
| dark | CI | full | target |
|
||||||
`cargo test --features crdt --test vterm_stage3_acceptance -- --test-threads=1`.
|
|---:|---:|---:|---|
|
||||||
It needs its own lane rather than a drive-by because it would run `a37`
|
| 177 | 1,832 | 2,009 | **the library itself** (`src/lib.rs`) |
|
||||||
under lavapipe **for the first time**, and neither its timing budgets nor
|
| 21 | 15 | 36 | `m5_5_acceptance` |
|
||||||
its wgpu path have ever been exercised on that adapter or on macOS CI.
|
| 13 | 1 | 14 | `gpu_invocation_acceptance` |
|
||||||
- Worth auditing at the same time: which *other* `crdt`-gated acceptance
|
| 13 | 1 | 14 | `gpu_initial_target_acceptance` |
|
||||||
suites are dark for the same reason. This is a coverage question about the
|
| 8 | 0 | 8 | `m10_11_acceptance` |
|
||||||
gate list itself, not about any one suite.
|
| 6 | 0 | 6 | `auto_pair_crdt_acceptance` |
|
||||||
|
| 6 | 0 | 6 | `m10_2_perf` |
|
||||||
|
| 4 | 5 | 9 | `vterm_stage3_acceptance` |
|
||||||
|
| 4 | 0 | 4 | `m10_10_perf` |
|
||||||
|
| 3 | 0 | 3 | `compile_mode_crdt_acceptance` |
|
||||||
|
| 2 | 22 | 24 | `theme_faces_acceptance` |
|
||||||
|
| 2 | 0 | 2 | `m11_5_semantic_acceptance` |
|
||||||
|
| 1 | 9 | 10 | `vterm_stage1_acceptance` |
|
||||||
|
| 1 | 7 | 8 | `statusline_segments_acceptance` |
|
||||||
|
| 1 | 10 | 11 | `gpu_font_acceptance` |
|
||||||
|
| 1 | 0 | 1 | `auto_indent_crdt_acceptance` |
|
||||||
|
| 1 | 0 | 1 | `m10_11_perf` |
|
||||||
|
|
||||||
|
- **The single worst line is the library.** `cargo test --lib --features crdt`
|
||||||
|
is a REQUIRED local gate in `CLAUDE.md`, and CI has never run it. 177
|
||||||
|
library tests — the whole CRDT half — are developer-machine-only.
|
||||||
|
- **Ten suites run zero or one test in CI**, including `gpu_initial_target`
|
||||||
|
(#148's entire acceptance, 1/14), `gpu_invocation` (#141's, 1/14), and
|
||||||
|
`a37`, the Vterm Stage 3 real-daemon/real-PTY/real-wgpu path that #135
|
||||||
|
built specifically because "a decoded-message fixture would prove none of
|
||||||
|
the three fit together".
|
||||||
|
- **Sort deliberate from accidental before proposing a fix.** Some of the 264
|
||||||
|
are perf suites that are `#[ignore]`d by default and belong to their own
|
||||||
|
jobs (`m10_2_perf` 6, `m10_11_perf` 1). `m10_10_perf` has **no** `#[ignore]`
|
||||||
|
and no CI job naming it, so it looks accidental. This classification is not
|
||||||
|
finished and is the lane's first task.
|
||||||
|
- **Fix shape, two parts** (the flag combination is verified to work:
|
||||||
|
`--no-default-features --features luajit,crdt` lists 10 vterm Stage 1 tests
|
||||||
|
versus 9 without):
|
||||||
|
1. a `crdt` leg on the `test` job for the non-GPU suites and the library;
|
||||||
|
2. the GPU-requiring `crdt` suites onto the existing `gpu-render` job, which
|
||||||
|
already has lavapipe and `PMACS_REQUIRE_GPU=1` —
|
||||||
|
`vterm_stage3_acceptance`, `gpu_invocation_acceptance`,
|
||||||
|
`gpu_initial_target_acceptance`, `gpu_font_acceptance`.
|
||||||
|
- **Expect first-run failures, and budget for them.** These would execute in
|
||||||
|
CI for the first time ever: real PTY timing on CI runners, wgpu under
|
||||||
|
lavapipe, and daemon-socket tests at unfamiliar concurrency. Start
|
||||||
|
ubuntu-only and decide about macOS from evidence. A red first run is the
|
||||||
|
lane working, not the lane failing.
|
||||||
- Mitigating fact, verified rather than assumed: #166's three unit pins are
|
- Mitigating fact, verified rather than assumed: #166's three unit pins are
|
||||||
**not** `crdt`-gated and do run under CI's exact flags
|
**not** `crdt`-gated and do run under CI's exact flags, including the
|
||||||
(`--no-default-features --features luajit|lua54`), including the
|
|
||||||
controller-release pin whose only job is catching the plausible wrong fix.
|
controller-release pin whose only job is catching the plausible wrong fix.
|
||||||
The regression protection is live; the real-daemon evidence is local-only.
|
|
||||||
|
|
||||||
## Bottom-panel lane (window placement + side windows) — Stage 1 IN REVIEW
|
## Bottom-panel lane (window placement + side windows) — Stage 1 IN REVIEW
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -745,6 +745,15 @@ final variant — its own round-trip cannot detect a discriminant shift.
|
||||||
|
|
||||||
## 5. Hard-won ops lessons
|
## 5. Hard-won ops lessons
|
||||||
|
|
||||||
|
- **A daemon-side fix is not deployed until the daemon is restarted from a
|
||||||
|
tree that contains it.** #166's reporter rebuilt and saw no change: the
|
||||||
|
running daemon had been started from a shared checkout still on a pre-fix
|
||||||
|
branch, and `pmacs --gpu` attaches to whatever process already owns the
|
||||||
|
socket. Rebuilding a binary does nothing to a running process. When
|
||||||
|
validating a daemon-side fix by hand, check the running process's binary
|
||||||
|
path and start time against the tree you think you fixed —
|
||||||
|
`ps -eo pid,lstart,args | grep '[p]macs --daemon'` — before concluding the
|
||||||
|
fix failed.
|
||||||
- **Two operations that must be alternatives are not made alternatives by
|
- **Two operations that must be alternatives are not made alternatives by
|
||||||
being adjacent.** The dispatcher applied its grid and semantic
|
being adjacent.** The dispatcher applied its grid and semantic
|
||||||
terminal-layout syncs to every attached frontend; a semantic session
|
terminal-layout syncs to every attached frontend; a semantic session
|
||||||
|
|
|
||||||
|
|
@ -81,7 +81,11 @@ character:
|
||||||
| | frames for a static screen | typed `Z` ever visible at the prompt |
|
| | frames for a static screen | typed `Z` ever visible at the prompt |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `main` today | **730** in a 20 s window | **no** |
|
| `main` today | **730** in a 20 s window | **no** |
|
||||||
| with the guard | **2** | (see Q#GT5 — a separate question) |
|
| with the fix | **2** | yes |
|
||||||
|
|
||||||
|
Bet B2 is **scored TRUE**: with the fix deployed, the reporter confirmed
|
||||||
|
typing into a GPU terminal works. The earlier caveat here pointed at Q#GT5,
|
||||||
|
which is now retracted — see "Deferred (named)".
|
||||||
|
|
||||||
The TUI is unaffected: a grid session has no semantic terminal declaration, so
|
The TUI is unaffected: a grid session has no semantic terminal declaration, so
|
||||||
only one arm ever runs for it. This is a **frontend-kind** defect, which is
|
only one arm ever runs for it. This is a **frontend-kind** defect, which is
|
||||||
|
|
@ -281,11 +285,12 @@ change. Stays v20.
|
||||||
snapshot that signals the switch-away). Hence the split in Q#GT1. Recorded
|
snapshot that signals the switch-away). Hence the split in Q#GT1. Recorded
|
||||||
rather than deleted: the failure mode is one a reviewer or a future
|
rather than deleted: the failure mode is one a reviewer or a future
|
||||||
simplification will re-propose.
|
simplification will re-propose.
|
||||||
- **B2.** The user's reported symptom is this defect. *Partially scored: the
|
- **B2 — SCORED TRUE 2026-07-25.** "The user's reported symptom is this
|
||||||
storm is proven and GUI-only, and its shape (line editor unusable, output
|
defect." Confirmed in real use after the fix was deployed: typing into a GPU
|
||||||
still flowing) matches the report. Not fully scored until the user, or an
|
terminal works. The confirmation needed a daemon **restart** built from a
|
||||||
acceptance running the **user's own shell**, confirms typing works after the
|
tree containing the fix — the first attempt reported no change because a
|
||||||
fix. Q#GT5 is the reason this bet is stated rather than assumed.*
|
pre-fix daemon still owned the socket, which is worth remembering whenever a
|
||||||
|
daemon-side fix is being validated by hand.
|
||||||
- **B3.** No other pair of per-frontend-kind daemon operations is applied as
|
- **B3.** No other pair of per-frontend-kind daemon operations is applied as
|
||||||
siblings rather than alternatives. *Scored by an explicit audit of the
|
siblings rather than alternatives. *Scored by an explicit audit of the
|
||||||
dispatcher's per-frontend loop during implementation — this defect's shape
|
dispatcher's per-frontend loop during implementation — this defect's shape
|
||||||
|
|
@ -294,7 +299,14 @@ change. Stays v20.
|
||||||
|
|
||||||
## Deferred (named)
|
## Deferred (named)
|
||||||
|
|
||||||
- Interactive-shell echo on a raw-mode PTY (Q#GT5) — its own scout.
|
- ~~Interactive-shell echo on a raw-mode PTY (Q#GT5)~~ — **RETRACTED
|
||||||
|
2026-07-25.** The observation behind it (a `bash --norc -i` fixture not
|
||||||
|
echoing typed characters) does not reproduce in real use: with the fix
|
||||||
|
deployed, typing into a GPU terminal echoes normally. The fixture was almost
|
||||||
|
certainly measuring its own timing — polling a published screen snapshot
|
||||||
|
before readline had finished initialising — not a product behaviour. Recorded
|
||||||
|
as retracted rather than deleted so nobody re-derives it from the framing's
|
||||||
|
earlier revision and spends a scout on it.
|
||||||
- **A geometry change appears to clear the visible screen.** Observed while
|
- **A geometry change appears to clear the visible screen.** Observed while
|
||||||
building acceptance 4: after the probe's deliberate 25×92 → 20×71 resize,
|
building acceptance 4: after the probe's deliberate 25×92 → 20×71 resize,
|
||||||
the next frame's visible grid is entirely blank even though the content
|
the next frame's visible grid is entirely blank even though the content
|
||||||
|
|
|
||||||
|
|
@ -1674,6 +1674,48 @@ GPU assertions remain in `pmacs-protocol` and `pmacs-gpu` respectively.
|
||||||
- **37:** one real-daemon/real-PTY/headless-wgpu acceptance path; it is not
|
- **37:** one real-daemon/real-PTY/headless-wgpu acceptance path; it is not
|
||||||
replaced by a decoded-message fixture.
|
replaced by a decoded-message fixture.
|
||||||
|
|
||||||
|
### 0.12 As-framed audit, 2026-07-25 (after #166)
|
||||||
|
|
||||||
|
Prompted by a GPU terminal input defect that shipped in Stage 3 and was fixed
|
||||||
|
in #166. The arc is structurally complete — all 37 criteria have
|
||||||
|
implementations, and every test named in the Stage 2 verification map exists —
|
||||||
|
but the audit found two gaps worth recording against the criteria themselves.
|
||||||
|
|
||||||
|
**Criterion 22's "without thrash" was never pinned.** The criterion reads
|
||||||
|
"unchanged, zero, passive, and failed resize cases preserve prior geometry
|
||||||
|
*without thrash*". The word appears nowhere in `src/` or `tests/`. The suite
|
||||||
|
pinned the four enumerated single-arm cases and never the cross-arm
|
||||||
|
interaction — which is exactly where the thrash lived: the daemon applied
|
||||||
|
both the grid and the semantic terminal-layout sync to every attached
|
||||||
|
frontend, so a semantic session's PTY was resized twice per tick forever.
|
||||||
|
Criterion 31's "only the exact durable controller changes PTY geometry" was
|
||||||
|
violated in the same event, in spirit rather than letter: the controller was
|
||||||
|
the right frontend, but the geometry came from the grid projection. #166 adds
|
||||||
|
the settle pins; the gap was open from #135 (2026-07-22) until then.
|
||||||
|
|
||||||
|
**Why the Stage 3 suite could not see it.** Of its nine tests, only three
|
||||||
|
drive a real daemon; the other six construct `EditorState` directly and never
|
||||||
|
execute the dispatcher loop where the defect lived. `a31`, which is about two
|
||||||
|
semantic frontends sharing one session, therefore passes on the broken tree.
|
||||||
|
The same structural blindness explains why `bottom_panel_stage1_acceptance`
|
||||||
|
was unaffected. A criterion about *dispatcher* behavior needs a test that
|
||||||
|
runs the dispatcher.
|
||||||
|
|
||||||
|
**Four of the nine Stage 3 tests do not run in CI at all**, because they are
|
||||||
|
`#[cfg(feature = "crdt")]` and the workflow never enables that feature:
|
||||||
|
`a37`, the two added by #166, and
|
||||||
|
`terminal_mode_keeps_reporting_presence_so_peers_drop_the_stale_caret` — which
|
||||||
|
is Stage 3 review round 1's own regression guard. Stage 1's
|
||||||
|
`read_only_empty_crdt_bootstrap_is_immutable_against_remote_content`, the CRDT
|
||||||
|
half of criterion 14, is dark for the same reason. Stage 2 is fully covered
|
||||||
|
(6/6). This is not a vterm problem: 264 tests workspace-wide are dark,
|
||||||
|
including 177 in the library. It has its own lane in `docs/active-work.md`.
|
||||||
|
|
||||||
|
**Not audited:** §11's blanket claim that "deferral means graceful ignore or
|
||||||
|
documented absence, never escape leakage, panic, unbounded allocation, or
|
||||||
|
child leak". That covers roughly twenty deferred items and none were
|
||||||
|
spot-checked. It remains an unproven claim rather than a known gap.
|
||||||
|
|
||||||
## 10. Gates and bite verification
|
## 10. Gates and bite verification
|
||||||
|
|
||||||
Every PR runs the standing full gates from `AGENTS.md`, sequentially, plus its
|
Every PR runs the standing full gates from `AGENTS.md`, sequentially, plus its
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue