review round 2: MIXED must not exit 0, and its assumed trigger is wrong

P2 --- the MIXED arm said "OK" and exited 0 while its likeliest cause
is vacuity. Reaching it means cargo failed, at least one `test result:`
line exists, and none says FAILED --- i.e. the named tests may have run
CLEAN against the old tree, the vacuous outcome, with an unrelated
failure supplying the exit status. The message admitted as much while
the exit code contradicted it, so a scripted caller, or a framing doc
quoting "bite exited 0", would record a certified bite. It now exits 4
and is labelled INCONCLUSIVE rather than OK. The exit ladder in the
header gains the code.

While exercising that arm I found the review's stated trigger --- and
my own comment repeating it --- is WRONG, which is worth more than the
fix. `--test A --test B`, where B's swapped file will not build, does
NOT produce MIXED: cargo builds every named target before running any,
so B's build failure stops A from running too and the swapped run
emits no summary at all. Verified by running exactly that shape
against an ancient `tests/m4_acceptance.rs`; it lands in COMPILE, as
it should. What genuinely reaches MIXED is a run that prints summaries
and *then* fails --- doc-tests failing to compile after the lib tests
pass, or a harness dying after its summary line.

So the arm stays as defence but is documented as defensive, and the
ledger lists it as the one path NOT reproduced rather than implying
the validation was complete. Manufacturing a doc-test failure to reach
it would test the arm at the cost of a contrived fixture; saying which
paths were executed and which were not is the more useful record.

Two wording fixes from the same round. The COMPILE message said the
file "does not build against the current tree", but the no-summary
condition also covers a harness that built and then died before
printing one; it now says "produced no test summary (build failure, or
a harness that died before printing one)" so the label never overstates
the observation. And the colour comment no longer overclaims: exporting
CARGO_TERM_COLOR=never defeats an ambient variable, but an explicit
`--color always` in the passed-through args still beats the
environment. Self-inflicted and fail-closed, and now said so.
This commit is contained in:
Levi Neuwirth 2026-07-29 11:05:55 -04:00
parent 1485f7610b
commit 8d93fc6978
2 changed files with 54 additions and 16 deletions

View File

@ -543,12 +543,26 @@ has **no branch and no framing yet**.
tree **and** that at least one ran (a filter matching nothing exits
0, so passing alone is insufficient), exiting 3 as `NO CONTROL`
otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`.
- **Validated on all three paths, not just the happy one:** a
zero-match filter reports `NO CONTROL`; a genuine bite reports
`control OK` then `OK (assertion)`; and a deliberately broken test
that fails on both trees reports `NO CONTROL` at exit 3 — **the old
script printed `bite: OK` for that last case**, which is the whole
point.
- **Validated by execution, not by reading**, and the results are
listed with the one arm that was NOT reproduced:
- zero-match filter -> `NO CONTROL`, exit 3;
- genuine bite -> `control OK` then `OK (assertion)`, exit 0;
- test broken so it fails on BOTH trees -> `NO CONTROL`, exit 3.
**The old script printed `bite: OK` here**, which is the whole
point of the lane;
- assertion failure whose own output prints `error[E0308]` and
`error: could not compile` at column 0 -> still `OK (assertion)`,
which is why classification reads libtest's summary rather than
compiler text;
- swapped file that will not build -> `OK (COMPILE)`;
- **`INCONCLUSIVE (MIXED)` was not reproduced.** Its assumed trigger
`--test A --test B` where B's swapped file fails to build — was
tested and **does not reach it**: cargo builds every named target
before running any, so B's failure stops A from running and the
run yields no summary at all (the COMPILE arm). The arm stays as
defence for the genuinely reachable causes (doc-tests failing to
compile after lib tests pass; a harness dying after its summary),
and the script says it was not manufactured.
- **Handoff correction, verified rather than inherited.**
`docs/agent-handoff.md` §5 claimed the script "restores by `git
checkout --`, which reverts the file to HEAD", destroying

View File

@ -12,7 +12,9 @@
# Exit status: 0 when the named tests pass now and fail against
# <ref>'s version of <path> (the fix bites), 1 when they still pass
# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE
# CONTROL fails. The working-tree file is restored on every exit path,
# CONTROL fails, 4 when the swapped run is INCONCLUSIVE (see the MIXED
# arm below --- non-zero because its likeliest cause is vacuity, not
# success). The working-tree file is restored on every exit path,
# including interrupts.
#
# THE POSITIVE CONTROL, and why it is not optional. Before this
@ -76,9 +78,12 @@ fi
# nothing until it finishes --- that is the trade, not a bug. Do not
# "fix" it back to streaming without giving the parser another source.
#
# Colour is pinned off on both runs: `--color always` in the
# passed-through args would otherwise wrap the summary lines in escape
# codes, and every anchored match here would silently stop matching.
# Colour is pinned off on both runs, so an ambient CARGO_TERM_COLOR
# cannot wrap the summary lines in escape codes and silently stop every
# anchored match here from matching. Note the limit: an explicit
# `--color always` in the passed-through args still beats the
# environment. That is self-inflicted, and the failure is fail-closed
# (a zeroed count reads as NO CONTROL), but do not pass it.
export CARGO_TERM_COLOR=never
# Sum the `N passed` figures across every `test result:` line, so a
@ -142,12 +147,31 @@ fi
if printf '%s\n' "$swapped_out" | grep -q '^test result: FAILED'; then
echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)"
elif printf '%s\n' "$swapped_out" | grep -q '^test result:'; then
# Some target ran to completion without a failure, yet cargo still
# exited non-zero --- typically a *different* target in the same
# invocation failed to build. Neither clean result applies.
echo "bite: OK (MIXED) --- some tests ran and none failed, but the run still failed." >&2
echo "bite: narrow the invocation to one target before trusting this." >&2
# Some target ran to completion with no failure, yet cargo still
# exited non-zero.
#
# NOT reachable the obvious way, which was checked rather than
# assumed: `--test A --test B` where B's swapped file will not
# build does NOT land here. Cargo builds every named target before
# running any, so B's build failure stops A from running too and
# there is no summary at all --- that is the COMPILE arm below.
# What does reach here is a run that prints summaries and *then*
# fails: doc-tests failing to compile after the lib tests pass, or
# a harness dying after its summary line. This arm is therefore
# defensive, and its trigger was not manufactured for validation.
#
# It exits NON-ZERO on purpose. The likeliest reading is not
# success but VACUITY: the named tests ran clean against the old
# tree (exactly the vacuous outcome) while an unrelated failure
# supplied the exit status. Reporting "OK" and exiting 0 here would
# let a scripted caller --- or a framing doc quoting "bite exited
# 0" --- record a certified bite for a run that certified nothing.
echo "bite: INCONCLUSIVE (MIXED) --- some tests ran, none failed, yet the run failed." >&2
echo "bite: the named tests may have PASSED against $ref:$path with an unrelated" >&2
echo "bite: build error supplying the exit status. Narrow to one target and re-run." >&2
exit 4
else
echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2
echo "bite: OK (COMPILE) --- $ref:$path produced no test summary" >&2
echo "bite: (build failure, or a harness that died before printing one)." >&2
echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2
fi