docs/active-work.md says the dark-test figure "moves with every merge
and must be re-measured, not quoted." That instruction has never had a
tool, so every re-measurement was a hand-rolled `--list` pipeline
written from scratch.
Hand-rolling it is not safe. Writing this lane's census by hand, the
first attempt filtered libtest's list with `/ : test$/` — but the output
is `name: test` with NO space before the colon, so it matched nothing,
reported zero targets, and looked like a clean run. A census that
silently reports nothing is the same failure class as the dark tests
themselves: no signal, presented as a result.
scripts/feature-census diffs `cargo test --list` between two feature
configurations and reports what the second has that the first cannot
see. Its header records each parsing trap, because every one of them was
hit while writing it:
* `name: test` has no space before the colon.
* `--list` also emits `: benchmark` lines.
* cargo's `Running` lines have two shapes — `unittests src/lib.rs` and
`tests/foo.rs` — so a fixed field index handles one and mangles the
other.
* a target with zero tests prints its `Running` line and nothing else,
so counting only test lines DROPS it from the diff — losing exactly
the finding worth surfacing.
* `--list` includes #[ignore]d tests, which are dark in the same sense
but are NOT recovered by adding a feature to an ordinary test job.
That last one needed a second correction after the script was running.
Counting only B's ignored set attributed pre-existing ignores to the
feature: `rope::tests::perf_smoke_*` are ignored under both configs and
are not "dark and ignored." Both sides now get an ignored pass and the
figure is the difference, which is what turns a flat "279 dark" into
"268 recovered by a plain leg, 11 needing --ignored."
The script also corrected a claim in this lane's own framing doc. The
framing said eight test binaries contain zero tests under CI's flags,
derived from a target-count difference (93 vs 101). The truth is that
ELEVEN targets run with zero tests under those flags; eight of them gain
tests under crdt and three are helper binaries with no tests in either
configuration. Two different true statements, and the framing had
merged them.
Fail-closed on a build failure (exit 3) rather than reporting a census.
A configuration that does not compile yields no test list, which is
indistinguishable by counting from "this configuration has no tests" and
would render as a spectacular and entirely false "every test is dark."
That is not a small error; it is a number that would get quoted.
All five documented exit codes are exercised rather than asserted: 0 on
a clean census and a holding --covers claim, 1 when the claim fails
(both for a test present under both configs and for a misspelled name),
2 on usage, 3 on a configuration that fails to build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Levi Neuwirth
a776bc337b