From 29d02ae1b0aa442c667ba89e60607a2ef765b927 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 29 Jul 2026 10:12:38 -0400 Subject: [PATCH 1/3] tooling: give scripts/bite a positive control, and correct the handoff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lane 6 of the test-improvement arc, pulled ahead of its own priority list because every later lane's evidence is bite-shaped. A verifier that cannot fail is worth no more than the claims it certifies. The defect. The script ran only the SWAPPED tree, so "the tests failed against the old file" was the only thing it ever checked. It could not distinguish a load-bearing fix from a broken test: a typo, an unrelated compile break, or a filter matching nothing all make the swapped run fail, and each printed `bite: OK`. The control now asserts two things, because passing alone is not enough --- `cargo test` with a filter matching zero tests exits 0, and a compile error in the old tree would then still read as OK: * the named tests PASS against the working tree, and * at least one test actually RAN, exiting 3 as NO CONTROL otherwise. The swapped run additionally distinguishes `OK (assertion)` from `OK (COMPILE)`, since an old file that will not build against the current tree is much weaker evidence: the tests may never have run at all. Validated on all three paths rather than the happy one alone. A zero-match filter reports NO CONTROL. A genuine bite --- PR #190's `rd1`, which is known to fail against `main` --- reports `control OK --- 1 test(s) pass` then `OK (assertion)`. A test deliberately broken so it fails on BOTH trees reports NO CONTROL at exit 3; the previous script printed `bite: OK` for exactly that case. Handoff correction, verified rather than inherited. §5 claimed the script "restores by `git checkout --`, which reverts the file to HEAD", destroying uncommitted work, and cited a wiped review round during #165. The mechanism description is false: the script copies the file to a mktemp path before swapping and restores from that copy under an `EXIT INT TERM` trap, touching no git state beyond a read-only `git show`. The commit-before-gating rule is kept --- it stands on gate-results-describe-the-pushed-tree --- but the false mechanism is removed, because a reader who believes bite eats uncommitted work will reach for `git stash` to protect themselves, which is the repo-global trap the script exists to avoid. No product code. No tests changed. --- docs/active-work.md | 36 ++++++++++++++++++ docs/agent-handoff.md | 38 ++++++++++++++----- scripts/bite | 87 ++++++++++++++++++++++++++++++++++++------- 3 files changed, 139 insertions(+), 22 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index 2dfbaf0..b67334d 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -521,6 +521,42 @@ has **no branch and no framing yet**. `FrontendView.fold_projection` to `true` for semantic frontends, which Stage 2 deliberately left `false` (Q#FD21). +## Test-improvement arc, lane 6 — `scripts/bite` positive control + +- Portable branch: `githubsucks/bite-positive-control`, worktree + `../pmacs-bite`, based on `main` @ `300cbc4`. First lane of the + test-improvement arc scoped by `TEST_IMPROVEMENT.md` §8. Tooling and + documentation only — **no product code, no tests changed.** +- Sequenced **first**, ahead of the arc's own priority list, because + every later lane's evidence is bite-shaped. A verifier that cannot + fail is not worth more than the claims it certifies. +- **The defect:** the script ran only the swapped tree, so a failing + swapped run was the only thing it checked. A test that fails + everywhere — a typo, an unrelated compile break — therefore printed + `bite: OK`. Now it asserts the named tests pass against the working + tree **and** that at least one ran (a filter matching nothing exits + 0, so passing alone is insufficient), exiting 3 as `NO CONTROL` + otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`. +- **Validated on all three paths, not just the happy one:** a + zero-match filter reports `NO CONTROL`; a genuine bite reports + `control OK` then `OK (assertion)`; and a deliberately broken test + that fails on both trees reports `NO CONTROL` at exit 3 — **the old + script printed `bite: OK` for that last case**, which is the whole + point. +- **Handoff correction, verified rather than inherited.** + `docs/agent-handoff.md` §5 claimed the script "restores by `git + checkout --`, which reverts the file to HEAD", destroying + uncommitted work. False: it copies to a `mktemp` path before the + swap and restores from that copy under an `EXIT INT TERM` trap. The + commit-before-gating rule stands on other grounds and is kept; the + false mechanism is removed, because it would push the next reader + toward `git stash` — the repo-global trap the script exists to + avoid. +- Recovery from a clean checkout: + `git fetch githubsucks && git worktree add ../pmacs-bite + -b bite-positive-control githubsucks/bite-positive-control`. + + ## Parked lane: kill-ring browser + persistence - Portable branch: `githubsucks/kill-ring-browser` diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index f514cec..7c80ee9 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1569,16 +1569,36 @@ round-trip cannot detect a discriminant shift. your tree (happened during #111: a failed `stash push` chained into `stash pop`, which grabbed the user's PR-#17-era entry). For run-tests-against-an-old-version swaps, use `scripts/bite` — a - trap-guarded one-file swap over read-only `git show`, with an - inverted verdict (exit 0 iff the tests FAIL against the old + trap-guarded one-file swap over read-only `git show`, with a + two-sided verdict (the tests must PASS now and FAIL against the old version), making bite-verification machine-checkable. -- **A fix must be COMMITTED before it is bitten.** `scripts/bite` - restores by `git checkout --`, which reverts the file to **HEAD**, not - to the state it found — so any uncommitted work in a bitten file is - destroyed. A whole review round's fixes were wiped this way during - #165. Corollary for a NEW file: the swap-over-`git show` mode does not - apply at all, so its claims must be bitten by hand-editing, which makes - the commit-first rule load-bearing rather than hygienic. +- **`scripts/bite` is only as good as its positive control, and it + had none until it was given one.** Before that it ran *only* the + swapped tree, so it could not tell "my fix is load-bearing" from + "my test is broken": a test that fails everywhere made the swapped + run fail, and a failing swapped run was the only thing checked, so + it printed `bite: OK` and certified nothing. It now asserts the + named tests pass **and** that at least one actually ran (a filter + matching nothing exits 0, so passing alone is not enough), exiting + 3 as `NO CONTROL` otherwise. It also distinguishes `OK (assertion)` + from `OK (COMPILE)` — an old file that does not build against the + current tree is a much weaker result, because the tests may never + have run at all. **Read which one it printed.** +- **A fix must be COMMITTED before it is bitten** — but *not* for the + reason previously recorded here. This file used to say `scripts/bite` + "restores by `git checkout --`, which reverts the file to HEAD", and + that a review round's fixes were wiped that way during #165. **That + mechanism description is false and was verified false:** the script + copies the file to a `mktemp` path before the swap and restores from + that copy, under an `EXIT INT TERM` trap, so uncommitted work in the + bitten file survives. It has never touched git state beyond a + read-only `git show`. The rule still stands on its own merits — + gate results must describe the pushed tree, and a `cargo fmt` after + a commit splits worktree from branch — but do not repeat the + destroys-your-work claim, which will push the next reader toward + `git stash` to protect themselves, straight into the repo-global + trap above. Corollary for a NEW file: the swap-over-`git show` mode + does not apply at all, so its claims must be bitten by hand-editing. - **A CONFLICTING PR silently runs no CI at all.** GitHub builds `pull_request` workflow runs against the PR's **merge ref**, which it does not create while the branch conflicts with its base. So pushes diff --git a/scripts/bite b/scripts/bite index cccd3eb..00b367e 100755 --- a/scripts/bite +++ b/scripts/bite @@ -1,6 +1,7 @@ #!/bin/sh -# scripts/bite --- prove a fix's tests BITE: run them against an older -# version of ONE file and succeed only if they FAIL there. +# scripts/bite --- prove a fix's tests BITE: run them against the +# working tree (they must PASS), then against an older version of ONE +# file (they must FAIL). # # scripts/bite [cargo-test-args...] # @@ -8,21 +9,40 @@ # scripts/bite HEAD~1 builtin/runtime/editops.lua \ # --test editops_acceptance -- capitalize trim_on_save_unexpected # -# Exit status: 0 when the named tests fail against 's version of -# (the fix bites), 1 when they still pass (vacuous), 2 on -# usage/setup errors. The working-tree file is restored on every exit -# path, including interrupts. +# Exit status: 0 when the named tests pass now and fail against +# 's version of (the fix bites), 1 when they still pass +# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE +# CONTROL fails. The working-tree file is restored on every exit path, +# including interrupts. # -# Caveat: a COMPILE error of the old tree also counts as "fails" — -# correct, but weaker evidence than a clean assertion failure; eyeball -# the output when the swapped file is Rust rather than Lua. +# THE POSITIVE CONTROL, and why it is not optional. Before this +# existed the script ran only the swapped tree, so it could not +# distinguish "my fix is load-bearing" from "my test is broken": a +# typo, an unrelated compile break, or a filter that matches nothing +# all made the swapped run fail, and a failing swapped run was the +# only thing it checked. It printed `bite: OK` and certified nothing. +# So the control asserts two things, because passing alone is not +# enough — `cargo test` with a filter matching zero tests exits 0, and +# then a compile error in the old tree would still read as OK: +# +# * the named tests PASS against the working tree, and +# * at least one test actually RAN. +# +# COMPILE errors of the old tree still count as "fails", and that is +# correct but weaker than a clean assertion failure --- the swapped +# file may simply not build against the current tree. The script now +# says which kind it saw instead of leaving it to be eyeballed. Prefer +# an assertion failure; treat a compile failure as a prompt to narrow +# the swap to a file that compiles both ways. # # Why this exists: bite-verification is step 4 of the working method, -# and the obvious shortcut — git stash — is a trap here. The stash +# and the obvious shortcut --- git stash --- is a trap here. The stash # namespace is REPO-GLOBAL: shared across every worktree and with # humans, so a scripted push/pop can collide with (or pop!) someone # else's stashed work. This helper never touches git state beyond a -# read-only `git show`. +# read-only `git show`, and it restores the working file from a +# mktemp copy --- NOT from git, so uncommitted work in is +# preserved. set -eu @@ -40,6 +60,34 @@ if [ ! -f "$path" ]; then exit 2 fi +# Sum the `N passed` figures across every `test result:` line, so a +# multi-target invocation is counted correctly rather than only its +# last binary. +count_passed() { + sed -n 's/^test result: ok\. \([0-9][0-9]*\) passed.*/\1/p' \ + | awk '{ total += $1 } END { print total + 0 }' +} + +# --- Positive control: the tests must pass, and must exist, NOW. --- +echo "bite: positive control --- running against the working tree" >&2 +control_status=0 +control_out=$(cargo test "$@" 2>&1) || control_status=$? +printf '%s\n' "$control_out" + +if [ "$control_status" -ne 0 ]; then + echo "bite: NO CONTROL --- tests do not pass against the working tree." >&2 + echo "bite: fix the tests first; a failure here makes the swapped run meaningless." >&2 + exit 3 +fi + +control_ran=$(printf '%s\n' "$control_out" | count_passed) +if [ "$control_ran" -eq 0 ]; then + echo "bite: NO CONTROL --- the filter matched no tests (0 passed)." >&2 + echo "bite: check the test names; an empty filter passes everywhere and proves nothing." >&2 + exit 3 +fi +echo "bite: control OK --- $control_ran test(s) pass against the working tree" >&2 + saved=$(mktemp "${TMPDIR:-/tmp}/bite.XXXXXX") cp -- "$path" "$saved" restore() { @@ -52,9 +100,22 @@ trap restore EXIT INT TERM # script works from any directory inside the repo. git show "$ref:./$path" > "$path" -if cargo test "$@"; then +echo "bite: swapping in $ref:$path" >&2 +swapped_status=0 +swapped_out=$(cargo test "$@" 2>&1) || swapped_status=$? +printf '%s\n' "$swapped_out" + +if [ "$swapped_status" -eq 0 ]; then echo "bite: VACUOUS --- tests still pass against $ref:$path" >&2 exit 1 fi -echo "bite: OK --- tests fail against $ref:$path (the fix bites)" +# Distinguish the two ways the swapped run can fail. An assertion +# failure is the evidence we want; a compile failure only says the old +# file does not build here. +if printf '%s\n' "$swapped_out" | grep -q '^error\[E[0-9]*\]\|^error: could not compile'; then + echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2 + echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2 +else + echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" +fi From 1485f7610b0469669f2397a21f9c3932192aac1a Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 29 Jul 2026 10:55:42 -0400 Subject: [PATCH 2/3] review round 1: classify from libtest, and drop a dangling doc citation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P1 --- the classifier could mislabel a genuine bite, and this repo manufactures the trigger. It grepped the swapped run's output for `^error\[E...\]` / `^error: could not compile`, but a failing test can print those lines itself: `tests/compile_mode_acceptance.rs:572` has a fixture that emits `error[E0308]: mismatched types` at column 0, and any suite that dumps buffer or PTY text on failure does the same. A clean assertion failure --- the strong evidence --- would then be stamped `OK (COMPILE)`, "weaker evidence", while the new handoff text tells the reader to trust the label. The verdict was never wrong; the label was. Classification now reads libtest's own summary instead: `test result: FAILED` present means the harness ran and tests failed; no `test result:` line at all means nothing ran. A third `MIXED` arm covers "some target ran clean but cargo still failed", which the two-way split silently folded into the wrong side. This also removes two latent portability bugs for free: `\|` alternation in a BRE is a GNU extension, so on BSD grep every failure would have labelled as assertion; and cargo's coloured `error:` output would have evaded the grep entirely. libtest's summary lines stay uncoloured when piped, and `CARGO_TERM_COLOR=never` is now exported so `--color always` in the passed-through args cannot break the parse either --- that would otherwise have zeroed `count_passed` and produced a spurious NO CONTROL. Verified rather than reasoned: a test made to print BOTH `error[E0308]: mismatched types` and ``error: could not compile `probe` `` at column 0, then failed by swap, is still classified `OK (assertion)`. P2 --- the ledger cited a file that is not in the tree. The lane scoped itself by `TEST_IMPROVEMENT.md` §8, which is untracked and machine-local: not on `main`, not in this PR's tree. That is a dangling reference for any other machine, which is the failure this ledger exists to prevent. The entry is now self-contained and says so; if that audit is to scope later lanes it needs committing in its own PR first. P3 --- a comment records that both runs are captured rather than streamed, and why, so nobody "fixes" it back to streaming and breaks the parser. Double blank line before the parked lane collapsed. Also, on review's prompting: the #165 wiped-round incident is now UNEXPLAINED, and the handoff says so rather than leaving a gap the next reader will fill by re-deriving the false mechanism. Work was lost; not by the cause this file blamed. SIGKILL bypassing the trap and a stash collision in the same round are both candidates, and the note declines to pick one. --- docs/active-work.md | 17 +++++++++++------ docs/agent-handoff.md | 9 ++++++++- scripts/bite | 44 +++++++++++++++++++++++++++++++++++++------ 3 files changed, 57 insertions(+), 13 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index b67334d..a19af89 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -524,12 +524,18 @@ has **no branch and no framing yet**. ## Test-improvement arc, lane 6 — `scripts/bite` positive control - Portable branch: `githubsucks/bite-positive-control`, worktree - `../pmacs-bite`, based on `main` @ `300cbc4`. First lane of the - test-improvement arc scoped by `TEST_IMPROVEMENT.md` §8. Tooling and + `../pmacs-bite`, based on `main` @ `300cbc4`. Tooling and documentation only — **no product code, no tests changed.** -- Sequenced **first**, ahead of the arc's own priority list, because - every later lane's evidence is bite-shaped. A verifier that cannot - fail is not worth more than the claims it certifies. +- **Self-contained deliberately.** An earlier draft of this entry + scoped the lane by citing a `TEST_IMPROVEMENT.md` at the repo root. + That file is **untracked and machine-local** — not on `main`, not in + this PR's tree — so the citation was a dangling reference for any + other machine, which is exactly what this ledger exists to prevent. + The lane is described here on its own terms instead. If that audit + is to scope later lanes it needs committing first, in its own PR. +- Sequenced ahead of the rest of the testing work because every later + lane's evidence is bite-shaped. A verifier that cannot fail is not + worth more than the claims it certifies. - **The defect:** the script ran only the swapped tree, so a failing swapped run was the only thing it checked. A test that fails everywhere — a typo, an unrelated compile break — therefore printed @@ -556,7 +562,6 @@ has **no branch and no framing yet**. `git fetch githubsucks && git worktree add ../pmacs-bite -b bite-positive-control githubsucks/bite-positive-control`. - ## Parked lane: kill-ring browser + persistence - Portable branch: `githubsucks/kill-ring-browser` diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index 7c80ee9..5307799 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1597,7 +1597,14 @@ round-trip cannot detect a discriminant shift. a commit splits worktree from branch — but do not repeat the destroys-your-work claim, which will push the next reader toward `git stash` to protect themselves, straight into the repo-global - trap above. Corollary for a NEW file: the swap-over-`git show` mode + trap above. **The #165 incident itself is now unexplained**, and + that is recorded rather than papered over: work really was lost, but + not by the mechanism this file blamed. A `SIGKILL` bypasses the trap + and would leave the swapped file in place, which is one candidate; + so is a stash collision, given the same round. Do not invent a + mechanism to close the gap — an unexplained incident is safer than a + confident wrong cause, which is what produced this correction. + Corollary for a NEW file: the swap-over-`git show` mode does not apply at all, so its claims must be bitten by hand-editing. - **A CONFLICTING PR silently runs no CI at all.** GitHub builds `pull_request` workflow runs against the PR's **merge ref**, which it diff --git a/scripts/bite b/scripts/bite index 00b367e..212674f 100755 --- a/scripts/bite +++ b/scripts/bite @@ -35,6 +35,16 @@ # an assertion failure; treat a compile failure as a prompt to narrow # the swap to a file that compiles both ways. # +# That classification reads libtest's own `test result:` summary and +# deliberately does NOT grep for compiler text. A test may PRINT +# compiler output on failure --- `compile_mode_acceptance` has a +# fixture that emits `error[E0308]: mismatched types` at column 0, and +# the compile-mode and editops suites are this script's primary +# consumers --- so a `^error[E...]` grep would stamp "weaker evidence" +# on exactly the strong result it was meant to identify. `test result:` +# is emitted whenever the harness ran, is absent when the swapped file +# did not build, and stays uncolored when piped. +# # Why this exists: bite-verification is step 4 of the working method, # and the obvious shortcut --- git stash --- is a trap here. The stash # namespace is REPO-GLOBAL: shared across every worktree and with @@ -60,6 +70,17 @@ if [ ! -f "$path" ]; then exit 2 fi +# Both cargo runs are CAPTURED rather than streamed, because their +# output is parsed (passed-counts for the control, `test result:` for +# the classification). A long acceptance suite therefore prints +# nothing until it finishes --- that is the trade, not a bug. Do not +# "fix" it back to streaming without giving the parser another source. +# +# Colour is pinned off on both runs: `--color always` in the +# passed-through args would otherwise wrap the summary lines in escape +# codes, and every anchored match here would silently stop matching. +export CARGO_TERM_COLOR=never + # Sum the `N passed` figures across every `test result:` line, so a # multi-target invocation is counted correctly rather than only its # last binary. @@ -110,12 +131,23 @@ if [ "$swapped_status" -eq 0 ]; then exit 1 fi -# Distinguish the two ways the swapped run can fail. An assertion -# failure is the evidence we want; a compile failure only says the old -# file does not build here. -if printf '%s\n' "$swapped_out" | grep -q '^error\[E[0-9]*\]\|^error: could not compile'; then +# Distinguish the ways the swapped run can fail. An assertion failure +# is the evidence we want; a build failure only says the old file does +# not compile here, and the tests may never have run at all. +# +# Classified from libtest's summary line, NOT from compiler text --- a +# failing test can print `error[E0308]: ...` at column 0 itself (see +# the header). `test result: FAILED` means the harness ran and tests +# failed; no `test result:` line at all means nothing ran. +if printf '%s\n' "$swapped_out" | grep -q '^test result: FAILED'; then + echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" +elif printf '%s\n' "$swapped_out" | grep -q '^test result:'; then + # Some target ran to completion without a failure, yet cargo still + # exited non-zero --- typically a *different* target in the same + # invocation failed to build. Neither clean result applies. + echo "bite: OK (MIXED) --- some tests ran and none failed, but the run still failed." >&2 + echo "bite: narrow the invocation to one target before trusting this." >&2 +else echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2 echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2 -else - echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" fi From 8d93fc6978f09414aa536957339f5b14752dfe96 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 29 Jul 2026 11:05:55 -0400 Subject: [PATCH 3/3] review round 2: MIXED must not exit 0, and its assumed trigger is wrong P2 --- the MIXED arm said "OK" and exited 0 while its likeliest cause is vacuity. Reaching it means cargo failed, at least one `test result:` line exists, and none says FAILED --- i.e. the named tests may have run CLEAN against the old tree, the vacuous outcome, with an unrelated failure supplying the exit status. The message admitted as much while the exit code contradicted it, so a scripted caller, or a framing doc quoting "bite exited 0", would record a certified bite. It now exits 4 and is labelled INCONCLUSIVE rather than OK. The exit ladder in the header gains the code. While exercising that arm I found the review's stated trigger --- and my own comment repeating it --- is WRONG, which is worth more than the fix. `--test A --test B`, where B's swapped file will not build, does NOT produce MIXED: cargo builds every named target before running any, so B's build failure stops A from running too and the swapped run emits no summary at all. Verified by running exactly that shape against an ancient `tests/m4_acceptance.rs`; it lands in COMPILE, as it should. What genuinely reaches MIXED is a run that prints summaries and *then* fails --- doc-tests failing to compile after the lib tests pass, or a harness dying after its summary line. So the arm stays as defence but is documented as defensive, and the ledger lists it as the one path NOT reproduced rather than implying the validation was complete. Manufacturing a doc-test failure to reach it would test the arm at the cost of a contrived fixture; saying which paths were executed and which were not is the more useful record. Two wording fixes from the same round. The COMPILE message said the file "does not build against the current tree", but the no-summary condition also covers a harness that built and then died before printing one; it now says "produced no test summary (build failure, or a harness that died before printing one)" so the label never overstates the observation. And the colour comment no longer overclaims: exporting CARGO_TERM_COLOR=never defeats an ambient variable, but an explicit `--color always` in the passed-through args still beats the environment. Self-inflicted and fail-closed, and now said so. --- docs/active-work.md | 26 ++++++++++++++++++++------ scripts/bite | 44 ++++++++++++++++++++++++++++++++++---------- 2 files changed, 54 insertions(+), 16 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index a19af89..b298948 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -543,12 +543,26 @@ has **no branch and no framing yet**. tree **and** that at least one ran (a filter matching nothing exits 0, so passing alone is insufficient), exiting 3 as `NO CONTROL` otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`. -- **Validated on all three paths, not just the happy one:** a - zero-match filter reports `NO CONTROL`; a genuine bite reports - `control OK` then `OK (assertion)`; and a deliberately broken test - that fails on both trees reports `NO CONTROL` at exit 3 — **the old - script printed `bite: OK` for that last case**, which is the whole - point. +- **Validated by execution, not by reading**, and the results are + listed with the one arm that was NOT reproduced: + - zero-match filter -> `NO CONTROL`, exit 3; + - genuine bite -> `control OK` then `OK (assertion)`, exit 0; + - test broken so it fails on BOTH trees -> `NO CONTROL`, exit 3. + **The old script printed `bite: OK` here**, which is the whole + point of the lane; + - assertion failure whose own output prints `error[E0308]` and + `error: could not compile` at column 0 -> still `OK (assertion)`, + which is why classification reads libtest's summary rather than + compiler text; + - swapped file that will not build -> `OK (COMPILE)`; + - **`INCONCLUSIVE (MIXED)` was not reproduced.** Its assumed trigger + — `--test A --test B` where B's swapped file fails to build — was + tested and **does not reach it**: cargo builds every named target + before running any, so B's failure stops A from running and the + run yields no summary at all (the COMPILE arm). The arm stays as + defence for the genuinely reachable causes (doc-tests failing to + compile after lib tests pass; a harness dying after its summary), + and the script says it was not manufactured. - **Handoff correction, verified rather than inherited.** `docs/agent-handoff.md` §5 claimed the script "restores by `git checkout --`, which reverts the file to HEAD", destroying diff --git a/scripts/bite b/scripts/bite index 212674f..8bc3c6b 100755 --- a/scripts/bite +++ b/scripts/bite @@ -12,7 +12,9 @@ # Exit status: 0 when the named tests pass now and fail against # 's version of (the fix bites), 1 when they still pass # there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE -# CONTROL fails. The working-tree file is restored on every exit path, +# CONTROL fails, 4 when the swapped run is INCONCLUSIVE (see the MIXED +# arm below --- non-zero because its likeliest cause is vacuity, not +# success). The working-tree file is restored on every exit path, # including interrupts. # # THE POSITIVE CONTROL, and why it is not optional. Before this @@ -76,9 +78,12 @@ fi # nothing until it finishes --- that is the trade, not a bug. Do not # "fix" it back to streaming without giving the parser another source. # -# Colour is pinned off on both runs: `--color always` in the -# passed-through args would otherwise wrap the summary lines in escape -# codes, and every anchored match here would silently stop matching. +# Colour is pinned off on both runs, so an ambient CARGO_TERM_COLOR +# cannot wrap the summary lines in escape codes and silently stop every +# anchored match here from matching. Note the limit: an explicit +# `--color always` in the passed-through args still beats the +# environment. That is self-inflicted, and the failure is fail-closed +# (a zeroed count reads as NO CONTROL), but do not pass it. export CARGO_TERM_COLOR=never # Sum the `N passed` figures across every `test result:` line, so a @@ -142,12 +147,31 @@ fi if printf '%s\n' "$swapped_out" | grep -q '^test result: FAILED'; then echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" elif printf '%s\n' "$swapped_out" | grep -q '^test result:'; then - # Some target ran to completion without a failure, yet cargo still - # exited non-zero --- typically a *different* target in the same - # invocation failed to build. Neither clean result applies. - echo "bite: OK (MIXED) --- some tests ran and none failed, but the run still failed." >&2 - echo "bite: narrow the invocation to one target before trusting this." >&2 + # Some target ran to completion with no failure, yet cargo still + # exited non-zero. + # + # NOT reachable the obvious way, which was checked rather than + # assumed: `--test A --test B` where B's swapped file will not + # build does NOT land here. Cargo builds every named target before + # running any, so B's build failure stops A from running too and + # there is no summary at all --- that is the COMPILE arm below. + # What does reach here is a run that prints summaries and *then* + # fails: doc-tests failing to compile after the lib tests pass, or + # a harness dying after its summary line. This arm is therefore + # defensive, and its trigger was not manufactured for validation. + # + # It exits NON-ZERO on purpose. The likeliest reading is not + # success but VACUITY: the named tests ran clean against the old + # tree (exactly the vacuous outcome) while an unrelated failure + # supplied the exit status. Reporting "OK" and exiting 0 here would + # let a scripted caller --- or a framing doc quoting "bite exited + # 0" --- record a certified bite for a run that certified nothing. + echo "bite: INCONCLUSIVE (MIXED) --- some tests ran, none failed, yet the run failed." >&2 + echo "bite: the named tests may have PASSED against $ref:$path with an unrelated" >&2 + echo "bite: build error supplying the exit status. Narrow to one target and re-run." >&2 + exit 4 else - echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2 + echo "bite: OK (COMPILE) --- $ref:$path produced no test summary" >&2 + echo "bite: (build failure, or a harness that died before printing one)." >&2 echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2 fi