Review finding (P2): docs/active-work.md and docs/agent-handoff.md both
still said "PR not yet open". This file's own rule is that an open PR
gets a lane recording its state — "When a PR is opened, give it a lane",
written after #171 drifted 153 commits while invisible here. Both now
carry PR #209, its six commits, and the first CI run.
That run is worth recording rather than merely citing, because it
discharges the one risk the framing could not close locally. ALL 14
CHECKS GREEN, including both new jobs (Test (crdt) 12m20s, M10 Perf
Gates (crdt) 5m40s) and the macOS/luajit leg that is the documented
flake surface. This was the first time in the project's history that
any of these tests executed in CI.
ACCEPTANCE 8 HOLDS AGAINST THE REAL RUN. Test (crdt) reported 3,717
passed / 0 failed / 30 ignored: the 3,746 all-targets census (with
basedpyright NOT skipped, as it is locally) plus one doc test, less the
30 ignored. The job demonstrably compiled and ran the crdt corpus rather
than reporting green over nothing — which was the whole point of writing
that criterion while the local sweep was already green.
A trap for the next reader is recorded with it: DO NOT SUBTRACT THE TWO
JOBS' TOTALS. Test (ubuntu/luajit) reports 3,485 and Test (crdt) 3,747,
a difference of 262 rather than 279, because the jobs run different
SETS — the non-crdt job adds pmacs-protocol's 17 tests. The dark count
is the all-targets comparison, 3,746 - 3,467. The wrong number looks
entirely plausible.
Chasing that discrepancy found a real gap, and one this lane's own tool
could never have surfaced: PMACS-PROTOCOL HAS ITS OWN `crdt` FEATURE.
It gates no tests, so a per-test census reports 17 either way and scores
the crate as unaffected — but it changes `cfg!(feature = "crdt")`
EXPRESSIONS inside InstanceCapabilities::default and
FrontendCapabilities::default, so those same 17 tests exercise different
runtime values under it. CI had only ever run the non-crdt ones. Closed
with an explicit `-p pmacs-protocol --all-targets --features crdt` step,
verified 17/17.
The blind spot is structural, not an omission, so it is documented at
the tool rather than fixed in it: scripts/feature-census censuses the
workspace DEFAULT MEMBER, because that is what a bare
`cargo test --all-targets` builds. Sibling crates are invisible to it no
matter what configs are passed. The durable lesson: A FEATURE CAN MATTER
TO A CRATE A PER-TEST CENSUS SCORES AS UNAFFECTED. Check siblings by
hand.
Verified: ci.yml parses at 10 jobs, script syntax clean, fmt,
diff-check, and the new protocol step green locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/active-work.md says the dark-test figure "moves with every merge
and must be re-measured, not quoted." That instruction has never had a
tool, so every re-measurement was a hand-rolled `--list` pipeline
written from scratch.
Hand-rolling it is not safe. Writing this lane's census by hand, the
first attempt filtered libtest's list with `/ : test$/` — but the output
is `name: test` with NO space before the colon, so it matched nothing,
reported zero targets, and looked like a clean run. A census that
silently reports nothing is the same failure class as the dark tests
themselves: no signal, presented as a result.
scripts/feature-census diffs `cargo test --list` between two feature
configurations and reports what the second has that the first cannot
see. Its header records each parsing trap, because every one of them was
hit while writing it:
* `name: test` has no space before the colon.
* `--list` also emits `: benchmark` lines.
* cargo's `Running` lines have two shapes — `unittests src/lib.rs` and
`tests/foo.rs` — so a fixed field index handles one and mangles the
other.
* a target with zero tests prints its `Running` line and nothing else,
so counting only test lines DROPS it from the diff — losing exactly
the finding worth surfacing.
* `--list` includes #[ignore]d tests, which are dark in the same sense
but are NOT recovered by adding a feature to an ordinary test job.
That last one needed a second correction after the script was running.
Counting only B's ignored set attributed pre-existing ignores to the
feature: `rope::tests::perf_smoke_*` are ignored under both configs and
are not "dark and ignored." Both sides now get an ignored pass and the
figure is the difference, which is what turns a flat "279 dark" into
"268 recovered by a plain leg, 11 needing --ignored."
The script also corrected a claim in this lane's own framing doc. The
framing said eight test binaries contain zero tests under CI's flags,
derived from a target-count difference (93 vs 101). The truth is that
ELEVEN targets run with zero tests under those flags; eight of them gain
tests under crdt and three are helper binaries with no tests in either
configuration. Two different true statements, and the framing had
merged them.
Fail-closed on a build failure (exit 3) rather than reporting a census.
A configuration that does not compile yields no test list, which is
indistinguishable by counting from "this configuration has no tests" and
would render as a spectacular and entirely false "every test is dark."
That is not a small error; it is a number that would get quoted.
All five documented exit codes are exercised rather than asserted: 0 on
a clean census and a holding --covers claim, 1 when the claim fails
(both for a test present under both configs and for a misspelled name),
2 on usage, 3 on a configuration that fails to build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Levi Neuwirth