review round 2: MIXED must not exit 0, and its assumed trigger is wrong
P2 --- the MIXED arm said "OK" and exited 0 while its likeliest cause is vacuity. Reaching it means cargo failed, at least one `test result:` line exists, and none says FAILED --- i.e. the named tests may have run CLEAN against the old tree, the vacuous outcome, with an unrelated failure supplying the exit status. The message admitted as much while the exit code contradicted it, so a scripted caller, or a framing doc quoting "bite exited 0", would record a certified bite. It now exits 4 and is labelled INCONCLUSIVE rather than OK. The exit ladder in the header gains the code. While exercising that arm I found the review's stated trigger --- and my own comment repeating it --- is WRONG, which is worth more than the fix. `--test A --test B`, where B's swapped file will not build, does NOT produce MIXED: cargo builds every named target before running any, so B's build failure stops A from running too and the swapped run emits no summary at all. Verified by running exactly that shape against an ancient `tests/m4_acceptance.rs`; it lands in COMPILE, as it should. What genuinely reaches MIXED is a run that prints summaries and *then* fails --- doc-tests failing to compile after the lib tests pass, or a harness dying after its summary line. So the arm stays as defence but is documented as defensive, and the ledger lists it as the one path NOT reproduced rather than implying the validation was complete. Manufacturing a doc-test failure to reach it would test the arm at the cost of a contrived fixture; saying which paths were executed and which were not is the more useful record. Two wording fixes from the same round. The COMPILE message said the file "does not build against the current tree", but the no-summary condition also covers a harness that built and then died before printing one; it now says "produced no test summary (build failure, or a harness that died before printing one)" so the label never overstates the observation. And the colour comment no longer overclaims: exporting CARGO_TERM_COLOR=never defeats an ambient variable, but an explicit `--color always` in the passed-through args still beats the environment. Self-inflicted and fail-closed, and now said so.
This commit is contained in:
parent
1485f7610b
commit
8d93fc6978
|
|
@ -543,12 +543,26 @@ has **no branch and no framing yet**.
|
||||||
tree **and** that at least one ran (a filter matching nothing exits
|
tree **and** that at least one ran (a filter matching nothing exits
|
||||||
0, so passing alone is insufficient), exiting 3 as `NO CONTROL`
|
0, so passing alone is insufficient), exiting 3 as `NO CONTROL`
|
||||||
otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`.
|
otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`.
|
||||||
- **Validated on all three paths, not just the happy one:** a
|
- **Validated by execution, not by reading**, and the results are
|
||||||
zero-match filter reports `NO CONTROL`; a genuine bite reports
|
listed with the one arm that was NOT reproduced:
|
||||||
`control OK` then `OK (assertion)`; and a deliberately broken test
|
- zero-match filter -> `NO CONTROL`, exit 3;
|
||||||
that fails on both trees reports `NO CONTROL` at exit 3 — **the old
|
- genuine bite -> `control OK` then `OK (assertion)`, exit 0;
|
||||||
script printed `bite: OK` for that last case**, which is the whole
|
- test broken so it fails on BOTH trees -> `NO CONTROL`, exit 3.
|
||||||
point.
|
**The old script printed `bite: OK` here**, which is the whole
|
||||||
|
point of the lane;
|
||||||
|
- assertion failure whose own output prints `error[E0308]` and
|
||||||
|
`error: could not compile` at column 0 -> still `OK (assertion)`,
|
||||||
|
which is why classification reads libtest's summary rather than
|
||||||
|
compiler text;
|
||||||
|
- swapped file that will not build -> `OK (COMPILE)`;
|
||||||
|
- **`INCONCLUSIVE (MIXED)` was not reproduced.** Its assumed trigger
|
||||||
|
— `--test A --test B` where B's swapped file fails to build — was
|
||||||
|
tested and **does not reach it**: cargo builds every named target
|
||||||
|
before running any, so B's failure stops A from running and the
|
||||||
|
run yields no summary at all (the COMPILE arm). The arm stays as
|
||||||
|
defence for the genuinely reachable causes (doc-tests failing to
|
||||||
|
compile after lib tests pass; a harness dying after its summary),
|
||||||
|
and the script says it was not manufactured.
|
||||||
- **Handoff correction, verified rather than inherited.**
|
- **Handoff correction, verified rather than inherited.**
|
||||||
`docs/agent-handoff.md` §5 claimed the script "restores by `git
|
`docs/agent-handoff.md` §5 claimed the script "restores by `git
|
||||||
checkout --`, which reverts the file to HEAD", destroying
|
checkout --`, which reverts the file to HEAD", destroying
|
||||||
|
|
|
||||||
44
scripts/bite
44
scripts/bite
|
|
@ -12,7 +12,9 @@
|
||||||
# Exit status: 0 when the named tests pass now and fail against
|
# Exit status: 0 when the named tests pass now and fail against
|
||||||
# <ref>'s version of <path> (the fix bites), 1 when they still pass
|
# <ref>'s version of <path> (the fix bites), 1 when they still pass
|
||||||
# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE
|
# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE
|
||||||
# CONTROL fails. The working-tree file is restored on every exit path,
|
# CONTROL fails, 4 when the swapped run is INCONCLUSIVE (see the MIXED
|
||||||
|
# arm below --- non-zero because its likeliest cause is vacuity, not
|
||||||
|
# success). The working-tree file is restored on every exit path,
|
||||||
# including interrupts.
|
# including interrupts.
|
||||||
#
|
#
|
||||||
# THE POSITIVE CONTROL, and why it is not optional. Before this
|
# THE POSITIVE CONTROL, and why it is not optional. Before this
|
||||||
|
|
@ -76,9 +78,12 @@ fi
|
||||||
# nothing until it finishes --- that is the trade, not a bug. Do not
|
# nothing until it finishes --- that is the trade, not a bug. Do not
|
||||||
# "fix" it back to streaming without giving the parser another source.
|
# "fix" it back to streaming without giving the parser another source.
|
||||||
#
|
#
|
||||||
# Colour is pinned off on both runs: `--color always` in the
|
# Colour is pinned off on both runs, so an ambient CARGO_TERM_COLOR
|
||||||
# passed-through args would otherwise wrap the summary lines in escape
|
# cannot wrap the summary lines in escape codes and silently stop every
|
||||||
# codes, and every anchored match here would silently stop matching.
|
# anchored match here from matching. Note the limit: an explicit
|
||||||
|
# `--color always` in the passed-through args still beats the
|
||||||
|
# environment. That is self-inflicted, and the failure is fail-closed
|
||||||
|
# (a zeroed count reads as NO CONTROL), but do not pass it.
|
||||||
export CARGO_TERM_COLOR=never
|
export CARGO_TERM_COLOR=never
|
||||||
|
|
||||||
# Sum the `N passed` figures across every `test result:` line, so a
|
# Sum the `N passed` figures across every `test result:` line, so a
|
||||||
|
|
@ -142,12 +147,31 @@ fi
|
||||||
if printf '%s\n' "$swapped_out" | grep -q '^test result: FAILED'; then
|
if printf '%s\n' "$swapped_out" | grep -q '^test result: FAILED'; then
|
||||||
echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)"
|
echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)"
|
||||||
elif printf '%s\n' "$swapped_out" | grep -q '^test result:'; then
|
elif printf '%s\n' "$swapped_out" | grep -q '^test result:'; then
|
||||||
# Some target ran to completion without a failure, yet cargo still
|
# Some target ran to completion with no failure, yet cargo still
|
||||||
# exited non-zero --- typically a *different* target in the same
|
# exited non-zero.
|
||||||
# invocation failed to build. Neither clean result applies.
|
#
|
||||||
echo "bite: OK (MIXED) --- some tests ran and none failed, but the run still failed." >&2
|
# NOT reachable the obvious way, which was checked rather than
|
||||||
echo "bite: narrow the invocation to one target before trusting this." >&2
|
# assumed: `--test A --test B` where B's swapped file will not
|
||||||
|
# build does NOT land here. Cargo builds every named target before
|
||||||
|
# running any, so B's build failure stops A from running too and
|
||||||
|
# there is no summary at all --- that is the COMPILE arm below.
|
||||||
|
# What does reach here is a run that prints summaries and *then*
|
||||||
|
# fails: doc-tests failing to compile after the lib tests pass, or
|
||||||
|
# a harness dying after its summary line. This arm is therefore
|
||||||
|
# defensive, and its trigger was not manufactured for validation.
|
||||||
|
#
|
||||||
|
# It exits NON-ZERO on purpose. The likeliest reading is not
|
||||||
|
# success but VACUITY: the named tests ran clean against the old
|
||||||
|
# tree (exactly the vacuous outcome) while an unrelated failure
|
||||||
|
# supplied the exit status. Reporting "OK" and exiting 0 here would
|
||||||
|
# let a scripted caller --- or a framing doc quoting "bite exited
|
||||||
|
# 0" --- record a certified bite for a run that certified nothing.
|
||||||
|
echo "bite: INCONCLUSIVE (MIXED) --- some tests ran, none failed, yet the run failed." >&2
|
||||||
|
echo "bite: the named tests may have PASSED against $ref:$path with an unrelated" >&2
|
||||||
|
echo "bite: build error supplying the exit status. Narrow to one target and re-run." >&2
|
||||||
|
exit 4
|
||||||
else
|
else
|
||||||
echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2
|
echo "bite: OK (COMPILE) --- $ref:$path produced no test summary" >&2
|
||||||
|
echo "bite: (build failure, or a harness that died before printing one)." >&2
|
||||||
echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2
|
echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2
|
||||||
fi
|
fi
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue