tooling: give scripts/bite a positive control, and correct the handoff

Lane 6 of the test-improvement arc, pulled ahead of its own priority
list because every later lane's evidence is bite-shaped. A verifier
that cannot fail is worth no more than the claims it certifies.

The defect. The script ran only the SWAPPED tree, so "the tests failed
against the old file" was the only thing it ever checked. It could not
distinguish a load-bearing fix from a broken test: a typo, an unrelated
compile break, or a filter matching nothing all make the swapped run
fail, and each printed `bite: OK`.

The control now asserts two things, because passing alone is not
enough --- `cargo test` with a filter matching zero tests exits 0, and
a compile error in the old tree would then still read as OK:

  * the named tests PASS against the working tree, and
  * at least one test actually RAN,

exiting 3 as NO CONTROL otherwise. The swapped run additionally
distinguishes `OK (assertion)` from `OK (COMPILE)`, since an old file
that will not build against the current tree is much weaker evidence:
the tests may never have run at all.

Validated on all three paths rather than the happy one alone. A
zero-match filter reports NO CONTROL. A genuine bite --- PR #190's
`rd1`, which is known to fail against `main` --- reports `control OK
--- 1 test(s) pass` then `OK (assertion)`. A test deliberately broken
so it fails on BOTH trees reports NO CONTROL at exit 3; the previous
script printed `bite: OK` for exactly that case.

Handoff correction, verified rather than inherited. §5 claimed the
script "restores by `git checkout --`, which reverts the file to HEAD",
destroying uncommitted work, and cited a wiped review round during
#165. The mechanism description is false: the script copies the file to
a mktemp path before swapping and restores from that copy under an
`EXIT INT TERM` trap, touching no git state beyond a read-only
`git show`. The commit-before-gating rule is kept --- it stands on
gate-results-describe-the-pushed-tree --- but the false mechanism is
removed, because a reader who believes bite eats uncommitted work will
reach for `git stash` to protect themselves, which is the repo-global
trap the script exists to avoid.

No product code. No tests changed.
This commit is contained in:
Levi Neuwirth 2026-07-29 10:12:38 -04:00
parent 300cbc4ef0
commit 29d02ae1b0
3 changed files with 139 additions and 22 deletions

View File

@ -521,6 +521,42 @@ has **no branch and no framing yet**.
`FrontendView.fold_projection` to `true` for semantic frontends, which `FrontendView.fold_projection` to `true` for semantic frontends, which
Stage 2 deliberately left `false` (Q#FD21). Stage 2 deliberately left `false` (Q#FD21).
## Test-improvement arc, lane 6 — `scripts/bite` positive control
- Portable branch: `githubsucks/bite-positive-control`, worktree
`../pmacs-bite`, based on `main` @ `300cbc4`. First lane of the
test-improvement arc scoped by `TEST_IMPROVEMENT.md` §8. Tooling and
documentation only — **no product code, no tests changed.**
- Sequenced **first**, ahead of the arc's own priority list, because
every later lane's evidence is bite-shaped. A verifier that cannot
fail is not worth more than the claims it certifies.
- **The defect:** the script ran only the swapped tree, so a failing
swapped run was the only thing it checked. A test that fails
everywhere — a typo, an unrelated compile break — therefore printed
`bite: OK`. Now it asserts the named tests pass against the working
tree **and** that at least one ran (a filter matching nothing exits
0, so passing alone is insufficient), exiting 3 as `NO CONTROL`
otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`.
- **Validated on all three paths, not just the happy one:** a
zero-match filter reports `NO CONTROL`; a genuine bite reports
`control OK` then `OK (assertion)`; and a deliberately broken test
that fails on both trees reports `NO CONTROL` at exit 3 — **the old
script printed `bite: OK` for that last case**, which is the whole
point.
- **Handoff correction, verified rather than inherited.**
`docs/agent-handoff.md` §5 claimed the script "restores by `git
checkout --`, which reverts the file to HEAD", destroying
uncommitted work. False: it copies to a `mktemp` path before the
swap and restores from that copy under an `EXIT INT TERM` trap. The
commit-before-gating rule stands on other grounds and is kept; the
false mechanism is removed, because it would push the next reader
toward `git stash` — the repo-global trap the script exists to
avoid.
- Recovery from a clean checkout:
`git fetch githubsucks && git worktree add ../pmacs-bite
-b bite-positive-control githubsucks/bite-positive-control`.
## Parked lane: kill-ring browser + persistence ## Parked lane: kill-ring browser + persistence
- Portable branch: `githubsucks/kill-ring-browser` - Portable branch: `githubsucks/kill-ring-browser`

View File

@ -1569,16 +1569,36 @@ round-trip cannot detect a discriminant shift.
your tree (happened during #111: a failed `stash push` chained your tree (happened during #111: a failed `stash push` chained
into `stash pop`, which grabbed the user's PR-#17-era entry). For into `stash pop`, which grabbed the user's PR-#17-era entry). For
run-tests-against-an-old-version swaps, use `scripts/bite` — a run-tests-against-an-old-version swaps, use `scripts/bite` — a
trap-guarded one-file swap over read-only `git show`, with an trap-guarded one-file swap over read-only `git show`, with a
inverted verdict (exit 0 iff the tests FAIL against the old two-sided verdict (the tests must PASS now and FAIL against the old
version), making bite-verification machine-checkable. version), making bite-verification machine-checkable.
- **A fix must be COMMITTED before it is bitten.** `scripts/bite` - **`scripts/bite` is only as good as its positive control, and it
restores by `git checkout --`, which reverts the file to **HEAD**, not had none until it was given one.** Before that it ran *only* the
to the state it found — so any uncommitted work in a bitten file is swapped tree, so it could not tell "my fix is load-bearing" from
destroyed. A whole review round's fixes were wiped this way during "my test is broken": a test that fails everywhere made the swapped
#165. Corollary for a NEW file: the swap-over-`git show` mode does not run fail, and a failing swapped run was the only thing checked, so
apply at all, so its claims must be bitten by hand-editing, which makes it printed `bite: OK` and certified nothing. It now asserts the
the commit-first rule load-bearing rather than hygienic. named tests pass **and** that at least one actually ran (a filter
matching nothing exits 0, so passing alone is not enough), exiting
3 as `NO CONTROL` otherwise. It also distinguishes `OK (assertion)`
from `OK (COMPILE)` — an old file that does not build against the
current tree is a much weaker result, because the tests may never
have run at all. **Read which one it printed.**
- **A fix must be COMMITTED before it is bitten** — but *not* for the
reason previously recorded here. This file used to say `scripts/bite`
"restores by `git checkout --`, which reverts the file to HEAD", and
that a review round's fixes were wiped that way during #165. **That
mechanism description is false and was verified false:** the script
copies the file to a `mktemp` path before the swap and restores from
that copy, under an `EXIT INT TERM` trap, so uncommitted work in the
bitten file survives. It has never touched git state beyond a
read-only `git show`. The rule still stands on its own merits —
gate results must describe the pushed tree, and a `cargo fmt` after
a commit splits worktree from branch — but do not repeat the
destroys-your-work claim, which will push the next reader toward
`git stash` to protect themselves, straight into the repo-global
trap above. Corollary for a NEW file: the swap-over-`git show` mode
does not apply at all, so its claims must be bitten by hand-editing.
- **A CONFLICTING PR silently runs no CI at all.** GitHub builds - **A CONFLICTING PR silently runs no CI at all.** GitHub builds
`pull_request` workflow runs against the PR's **merge ref**, which it `pull_request` workflow runs against the PR's **merge ref**, which it
does not create while the branch conflicts with its base. So pushes does not create while the branch conflicts with its base. So pushes

View File

@ -1,6 +1,7 @@
#!/bin/sh #!/bin/sh
# scripts/bite --- prove a fix's tests BITE: run them against an older # scripts/bite --- prove a fix's tests BITE: run them against the
# version of ONE file and succeed only if they FAIL there. # working tree (they must PASS), then against an older version of ONE
# file (they must FAIL).
# #
# scripts/bite <ref> <path> [cargo-test-args...] # scripts/bite <ref> <path> [cargo-test-args...]
# #
@ -8,21 +9,40 @@
# scripts/bite HEAD~1 builtin/runtime/editops.lua \ # scripts/bite HEAD~1 builtin/runtime/editops.lua \
# --test editops_acceptance -- capitalize trim_on_save_unexpected # --test editops_acceptance -- capitalize trim_on_save_unexpected
# #
# Exit status: 0 when the named tests fail against <ref>'s version of # Exit status: 0 when the named tests pass now and fail against
# <path> (the fix bites), 1 when they still pass (vacuous), 2 on # <ref>'s version of <path> (the fix bites), 1 when they still pass
# usage/setup errors. The working-tree file is restored on every exit # there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE
# path, including interrupts. # CONTROL fails. The working-tree file is restored on every exit path,
# including interrupts.
# #
# Caveat: a COMPILE error of the old tree also counts as "fails" — # THE POSITIVE CONTROL, and why it is not optional. Before this
# correct, but weaker evidence than a clean assertion failure; eyeball # existed the script ran only the swapped tree, so it could not
# the output when the swapped file is Rust rather than Lua. # distinguish "my fix is load-bearing" from "my test is broken": a
# typo, an unrelated compile break, or a filter that matches nothing
# all made the swapped run fail, and a failing swapped run was the
# only thing it checked. It printed `bite: OK` and certified nothing.
# So the control asserts two things, because passing alone is not
# enough — `cargo test` with a filter matching zero tests exits 0, and
# then a compile error in the old tree would still read as OK:
#
# * the named tests PASS against the working tree, and
# * at least one test actually RAN.
#
# COMPILE errors of the old tree still count as "fails", and that is
# correct but weaker than a clean assertion failure --- the swapped
# file may simply not build against the current tree. The script now
# says which kind it saw instead of leaving it to be eyeballed. Prefer
# an assertion failure; treat a compile failure as a prompt to narrow
# the swap to a file that compiles both ways.
# #
# Why this exists: bite-verification is step 4 of the working method, # Why this exists: bite-verification is step 4 of the working method,
# and the obvious shortcut — git stash — is a trap here. The stash # and the obvious shortcut --- git stash --- is a trap here. The stash
# namespace is REPO-GLOBAL: shared across every worktree and with # namespace is REPO-GLOBAL: shared across every worktree and with
# humans, so a scripted push/pop can collide with (or pop!) someone # humans, so a scripted push/pop can collide with (or pop!) someone
# else's stashed work. This helper never touches git state beyond a # else's stashed work. This helper never touches git state beyond a
# read-only `git show`. # read-only `git show`, and it restores the working file from a
# mktemp copy --- NOT from git, so uncommitted work in <path> is
# preserved.
set -eu set -eu
@ -40,6 +60,34 @@ if [ ! -f "$path" ]; then
exit 2 exit 2
fi fi
# Sum the `N passed` figures across every `test result:` line, so a
# multi-target invocation is counted correctly rather than only its
# last binary.
count_passed() {
sed -n 's/^test result: ok\. \([0-9][0-9]*\) passed.*/\1/p' \
| awk '{ total += $1 } END { print total + 0 }'
}
# --- Positive control: the tests must pass, and must exist, NOW. ---
echo "bite: positive control --- running against the working tree" >&2
control_status=0
control_out=$(cargo test "$@" 2>&1) || control_status=$?
printf '%s\n' "$control_out"
if [ "$control_status" -ne 0 ]; then
echo "bite: NO CONTROL --- tests do not pass against the working tree." >&2
echo "bite: fix the tests first; a failure here makes the swapped run meaningless." >&2
exit 3
fi
control_ran=$(printf '%s\n' "$control_out" | count_passed)
if [ "$control_ran" -eq 0 ]; then
echo "bite: NO CONTROL --- the filter matched no tests (0 passed)." >&2
echo "bite: check the test names; an empty filter passes everywhere and proves nothing." >&2
exit 3
fi
echo "bite: control OK --- $control_ran test(s) pass against the working tree" >&2
saved=$(mktemp "${TMPDIR:-/tmp}/bite.XXXXXX") saved=$(mktemp "${TMPDIR:-/tmp}/bite.XXXXXX")
cp -- "$path" "$saved" cp -- "$path" "$saved"
restore() { restore() {
@ -52,9 +100,22 @@ trap restore EXIT INT TERM
# script works from any directory inside the repo. # script works from any directory inside the repo.
git show "$ref:./$path" > "$path" git show "$ref:./$path" > "$path"
if cargo test "$@"; then echo "bite: swapping in $ref:$path" >&2
swapped_status=0
swapped_out=$(cargo test "$@" 2>&1) || swapped_status=$?
printf '%s\n' "$swapped_out"
if [ "$swapped_status" -eq 0 ]; then
echo "bite: VACUOUS --- tests still pass against $ref:$path" >&2 echo "bite: VACUOUS --- tests still pass against $ref:$path" >&2
exit 1 exit 1
fi fi
echo "bite: OK --- tests fail against $ref:$path (the fix bites)" # Distinguish the two ways the swapped run can fail. An assertion
# failure is the evidence we want; a compile failure only says the old
# file does not build here.
if printf '%s\n' "$swapped_out" | grep -q '^error\[E[0-9]*\]\|^error: could not compile'; then
echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2
echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2
else
echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)"
fi