From 29d02ae1b0aa442c667ba89e60607a2ef765b927 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 29 Jul 2026 10:12:38 -0400 Subject: [PATCH] tooling: give scripts/bite a positive control, and correct the handoff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lane 6 of the test-improvement arc, pulled ahead of its own priority list because every later lane's evidence is bite-shaped. A verifier that cannot fail is worth no more than the claims it certifies. The defect. The script ran only the SWAPPED tree, so "the tests failed against the old file" was the only thing it ever checked. It could not distinguish a load-bearing fix from a broken test: a typo, an unrelated compile break, or a filter matching nothing all make the swapped run fail, and each printed `bite: OK`. The control now asserts two things, because passing alone is not enough --- `cargo test` with a filter matching zero tests exits 0, and a compile error in the old tree would then still read as OK: * the named tests PASS against the working tree, and * at least one test actually RAN, exiting 3 as NO CONTROL otherwise. The swapped run additionally distinguishes `OK (assertion)` from `OK (COMPILE)`, since an old file that will not build against the current tree is much weaker evidence: the tests may never have run at all. Validated on all three paths rather than the happy one alone. A zero-match filter reports NO CONTROL. A genuine bite --- PR #190's `rd1`, which is known to fail against `main` --- reports `control OK --- 1 test(s) pass` then `OK (assertion)`. A test deliberately broken so it fails on BOTH trees reports NO CONTROL at exit 3; the previous script printed `bite: OK` for exactly that case. Handoff correction, verified rather than inherited. §5 claimed the script "restores by `git checkout --`, which reverts the file to HEAD", destroying uncommitted work, and cited a wiped review round during #165. The mechanism description is false: the script copies the file to a mktemp path before swapping and restores from that copy under an `EXIT INT TERM` trap, touching no git state beyond a read-only `git show`. The commit-before-gating rule is kept --- it stands on gate-results-describe-the-pushed-tree --- but the false mechanism is removed, because a reader who believes bite eats uncommitted work will reach for `git stash` to protect themselves, which is the repo-global trap the script exists to avoid. No product code. No tests changed. --- docs/active-work.md | 36 ++++++++++++++++++ docs/agent-handoff.md | 38 ++++++++++++++----- scripts/bite | 87 ++++++++++++++++++++++++++++++++++++------- 3 files changed, 139 insertions(+), 22 deletions(-) diff --git a/docs/active-work.md b/docs/active-work.md index 2dfbaf0..b67334d 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -521,6 +521,42 @@ has **no branch and no framing yet**. `FrontendView.fold_projection` to `true` for semantic frontends, which Stage 2 deliberately left `false` (Q#FD21). +## Test-improvement arc, lane 6 — `scripts/bite` positive control + +- Portable branch: `githubsucks/bite-positive-control`, worktree + `../pmacs-bite`, based on `main` @ `300cbc4`. First lane of the + test-improvement arc scoped by `TEST_IMPROVEMENT.md` §8. Tooling and + documentation only — **no product code, no tests changed.** +- Sequenced **first**, ahead of the arc's own priority list, because + every later lane's evidence is bite-shaped. A verifier that cannot + fail is not worth more than the claims it certifies. +- **The defect:** the script ran only the swapped tree, so a failing + swapped run was the only thing it checked. A test that fails + everywhere — a typo, an unrelated compile break — therefore printed + `bite: OK`. Now it asserts the named tests pass against the working + tree **and** that at least one ran (a filter matching nothing exits + 0, so passing alone is insufficient), exiting 3 as `NO CONTROL` + otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`. +- **Validated on all three paths, not just the happy one:** a + zero-match filter reports `NO CONTROL`; a genuine bite reports + `control OK` then `OK (assertion)`; and a deliberately broken test + that fails on both trees reports `NO CONTROL` at exit 3 — **the old + script printed `bite: OK` for that last case**, which is the whole + point. +- **Handoff correction, verified rather than inherited.** + `docs/agent-handoff.md` §5 claimed the script "restores by `git + checkout --`, which reverts the file to HEAD", destroying + uncommitted work. False: it copies to a `mktemp` path before the + swap and restores from that copy under an `EXIT INT TERM` trap. The + commit-before-gating rule stands on other grounds and is kept; the + false mechanism is removed, because it would push the next reader + toward `git stash` — the repo-global trap the script exists to + avoid. +- Recovery from a clean checkout: + `git fetch githubsucks && git worktree add ../pmacs-bite + -b bite-positive-control githubsucks/bite-positive-control`. + + ## Parked lane: kill-ring browser + persistence - Portable branch: `githubsucks/kill-ring-browser` diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index f514cec..7c80ee9 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1569,16 +1569,36 @@ round-trip cannot detect a discriminant shift. your tree (happened during #111: a failed `stash push` chained into `stash pop`, which grabbed the user's PR-#17-era entry). For run-tests-against-an-old-version swaps, use `scripts/bite` — a - trap-guarded one-file swap over read-only `git show`, with an - inverted verdict (exit 0 iff the tests FAIL against the old + trap-guarded one-file swap over read-only `git show`, with a + two-sided verdict (the tests must PASS now and FAIL against the old version), making bite-verification machine-checkable. -- **A fix must be COMMITTED before it is bitten.** `scripts/bite` - restores by `git checkout --`, which reverts the file to **HEAD**, not - to the state it found — so any uncommitted work in a bitten file is - destroyed. A whole review round's fixes were wiped this way during - #165. Corollary for a NEW file: the swap-over-`git show` mode does not - apply at all, so its claims must be bitten by hand-editing, which makes - the commit-first rule load-bearing rather than hygienic. +- **`scripts/bite` is only as good as its positive control, and it + had none until it was given one.** Before that it ran *only* the + swapped tree, so it could not tell "my fix is load-bearing" from + "my test is broken": a test that fails everywhere made the swapped + run fail, and a failing swapped run was the only thing checked, so + it printed `bite: OK` and certified nothing. It now asserts the + named tests pass **and** that at least one actually ran (a filter + matching nothing exits 0, so passing alone is not enough), exiting + 3 as `NO CONTROL` otherwise. It also distinguishes `OK (assertion)` + from `OK (COMPILE)` — an old file that does not build against the + current tree is a much weaker result, because the tests may never + have run at all. **Read which one it printed.** +- **A fix must be COMMITTED before it is bitten** — but *not* for the + reason previously recorded here. This file used to say `scripts/bite` + "restores by `git checkout --`, which reverts the file to HEAD", and + that a review round's fixes were wiped that way during #165. **That + mechanism description is false and was verified false:** the script + copies the file to a `mktemp` path before the swap and restores from + that copy, under an `EXIT INT TERM` trap, so uncommitted work in the + bitten file survives. It has never touched git state beyond a + read-only `git show`. The rule still stands on its own merits — + gate results must describe the pushed tree, and a `cargo fmt` after + a commit splits worktree from branch — but do not repeat the + destroys-your-work claim, which will push the next reader toward + `git stash` to protect themselves, straight into the repo-global + trap above. Corollary for a NEW file: the swap-over-`git show` mode + does not apply at all, so its claims must be bitten by hand-editing. - **A CONFLICTING PR silently runs no CI at all.** GitHub builds `pull_request` workflow runs against the PR's **merge ref**, which it does not create while the branch conflicts with its base. So pushes diff --git a/scripts/bite b/scripts/bite index cccd3eb..00b367e 100755 --- a/scripts/bite +++ b/scripts/bite @@ -1,6 +1,7 @@ #!/bin/sh -# scripts/bite --- prove a fix's tests BITE: run them against an older -# version of ONE file and succeed only if they FAIL there. +# scripts/bite --- prove a fix's tests BITE: run them against the +# working tree (they must PASS), then against an older version of ONE +# file (they must FAIL). # # scripts/bite [cargo-test-args...] # @@ -8,21 +9,40 @@ # scripts/bite HEAD~1 builtin/runtime/editops.lua \ # --test editops_acceptance -- capitalize trim_on_save_unexpected # -# Exit status: 0 when the named tests fail against 's version of -# (the fix bites), 1 when they still pass (vacuous), 2 on -# usage/setup errors. The working-tree file is restored on every exit -# path, including interrupts. +# Exit status: 0 when the named tests pass now and fail against +# 's version of (the fix bites), 1 when they still pass +# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE +# CONTROL fails. The working-tree file is restored on every exit path, +# including interrupts. # -# Caveat: a COMPILE error of the old tree also counts as "fails" — -# correct, but weaker evidence than a clean assertion failure; eyeball -# the output when the swapped file is Rust rather than Lua. +# THE POSITIVE CONTROL, and why it is not optional. Before this +# existed the script ran only the swapped tree, so it could not +# distinguish "my fix is load-bearing" from "my test is broken": a +# typo, an unrelated compile break, or a filter that matches nothing +# all made the swapped run fail, and a failing swapped run was the +# only thing it checked. It printed `bite: OK` and certified nothing. +# So the control asserts two things, because passing alone is not +# enough — `cargo test` with a filter matching zero tests exits 0, and +# then a compile error in the old tree would still read as OK: +# +# * the named tests PASS against the working tree, and +# * at least one test actually RAN. +# +# COMPILE errors of the old tree still count as "fails", and that is +# correct but weaker than a clean assertion failure --- the swapped +# file may simply not build against the current tree. The script now +# says which kind it saw instead of leaving it to be eyeballed. Prefer +# an assertion failure; treat a compile failure as a prompt to narrow +# the swap to a file that compiles both ways. # # Why this exists: bite-verification is step 4 of the working method, -# and the obvious shortcut — git stash — is a trap here. The stash +# and the obvious shortcut --- git stash --- is a trap here. The stash # namespace is REPO-GLOBAL: shared across every worktree and with # humans, so a scripted push/pop can collide with (or pop!) someone # else's stashed work. This helper never touches git state beyond a -# read-only `git show`. +# read-only `git show`, and it restores the working file from a +# mktemp copy --- NOT from git, so uncommitted work in is +# preserved. set -eu @@ -40,6 +60,34 @@ if [ ! -f "$path" ]; then exit 2 fi +# Sum the `N passed` figures across every `test result:` line, so a +# multi-target invocation is counted correctly rather than only its +# last binary. +count_passed() { + sed -n 's/^test result: ok\. \([0-9][0-9]*\) passed.*/\1/p' \ + | awk '{ total += $1 } END { print total + 0 }' +} + +# --- Positive control: the tests must pass, and must exist, NOW. --- +echo "bite: positive control --- running against the working tree" >&2 +control_status=0 +control_out=$(cargo test "$@" 2>&1) || control_status=$? +printf '%s\n' "$control_out" + +if [ "$control_status" -ne 0 ]; then + echo "bite: NO CONTROL --- tests do not pass against the working tree." >&2 + echo "bite: fix the tests first; a failure here makes the swapped run meaningless." >&2 + exit 3 +fi + +control_ran=$(printf '%s\n' "$control_out" | count_passed) +if [ "$control_ran" -eq 0 ]; then + echo "bite: NO CONTROL --- the filter matched no tests (0 passed)." >&2 + echo "bite: check the test names; an empty filter passes everywhere and proves nothing." >&2 + exit 3 +fi +echo "bite: control OK --- $control_ran test(s) pass against the working tree" >&2 + saved=$(mktemp "${TMPDIR:-/tmp}/bite.XXXXXX") cp -- "$path" "$saved" restore() { @@ -52,9 +100,22 @@ trap restore EXIT INT TERM # script works from any directory inside the repo. git show "$ref:./$path" > "$path" -if cargo test "$@"; then +echo "bite: swapping in $ref:$path" >&2 +swapped_status=0 +swapped_out=$(cargo test "$@" 2>&1) || swapped_status=$? +printf '%s\n' "$swapped_out" + +if [ "$swapped_status" -eq 0 ]; then echo "bite: VACUOUS --- tests still pass against $ref:$path" >&2 exit 1 fi -echo "bite: OK --- tests fail against $ref:$path (the fix bites)" +# Distinguish the two ways the swapped run can fail. An assertion +# failure is the evidence we want; a compile failure only says the old +# file does not build here. +if printf '%s\n' "$swapped_out" | grep -q '^error\[E[0-9]*\]\|^error: could not compile'; then + echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2 + echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2 +else + echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" +fi