diff --git a/docs/active-work.md b/docs/active-work.md index 2dfbaf0..b67334d 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -521,6 +521,42 @@ has **no branch and no framing yet**. `FrontendView.fold_projection` to `true` for semantic frontends, which Stage 2 deliberately left `false` (Q#FD21). +## Test-improvement arc, lane 6 — `scripts/bite` positive control + +- Portable branch: `githubsucks/bite-positive-control`, worktree + `../pmacs-bite`, based on `main` @ `300cbc4`. First lane of the + test-improvement arc scoped by `TEST_IMPROVEMENT.md` §8. Tooling and + documentation only — **no product code, no tests changed.** +- Sequenced **first**, ahead of the arc's own priority list, because + every later lane's evidence is bite-shaped. A verifier that cannot + fail is not worth more than the claims it certifies. +- **The defect:** the script ran only the swapped tree, so a failing + swapped run was the only thing it checked. A test that fails + everywhere — a typo, an unrelated compile break — therefore printed + `bite: OK`. Now it asserts the named tests pass against the working + tree **and** that at least one ran (a filter matching nothing exits + 0, so passing alone is insufficient), exiting 3 as `NO CONTROL` + otherwise, and it labels `OK (assertion)` versus `OK (COMPILE)`. +- **Validated on all three paths, not just the happy one:** a + zero-match filter reports `NO CONTROL`; a genuine bite reports + `control OK` then `OK (assertion)`; and a deliberately broken test + that fails on both trees reports `NO CONTROL` at exit 3 — **the old + script printed `bite: OK` for that last case**, which is the whole + point. +- **Handoff correction, verified rather than inherited.** + `docs/agent-handoff.md` §5 claimed the script "restores by `git + checkout --`, which reverts the file to HEAD", destroying + uncommitted work. False: it copies to a `mktemp` path before the + swap and restores from that copy under an `EXIT INT TERM` trap. The + commit-before-gating rule stands on other grounds and is kept; the + false mechanism is removed, because it would push the next reader + toward `git stash` — the repo-global trap the script exists to + avoid. +- Recovery from a clean checkout: + `git fetch githubsucks && git worktree add ../pmacs-bite + -b bite-positive-control githubsucks/bite-positive-control`. + + ## Parked lane: kill-ring browser + persistence - Portable branch: `githubsucks/kill-ring-browser` diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index f514cec..7c80ee9 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1569,16 +1569,36 @@ round-trip cannot detect a discriminant shift. your tree (happened during #111: a failed `stash push` chained into `stash pop`, which grabbed the user's PR-#17-era entry). For run-tests-against-an-old-version swaps, use `scripts/bite` — a - trap-guarded one-file swap over read-only `git show`, with an - inverted verdict (exit 0 iff the tests FAIL against the old + trap-guarded one-file swap over read-only `git show`, with a + two-sided verdict (the tests must PASS now and FAIL against the old version), making bite-verification machine-checkable. -- **A fix must be COMMITTED before it is bitten.** `scripts/bite` - restores by `git checkout --`, which reverts the file to **HEAD**, not - to the state it found — so any uncommitted work in a bitten file is - destroyed. A whole review round's fixes were wiped this way during - #165. Corollary for a NEW file: the swap-over-`git show` mode does not - apply at all, so its claims must be bitten by hand-editing, which makes - the commit-first rule load-bearing rather than hygienic. +- **`scripts/bite` is only as good as its positive control, and it + had none until it was given one.** Before that it ran *only* the + swapped tree, so it could not tell "my fix is load-bearing" from + "my test is broken": a test that fails everywhere made the swapped + run fail, and a failing swapped run was the only thing checked, so + it printed `bite: OK` and certified nothing. It now asserts the + named tests pass **and** that at least one actually ran (a filter + matching nothing exits 0, so passing alone is not enough), exiting + 3 as `NO CONTROL` otherwise. It also distinguishes `OK (assertion)` + from `OK (COMPILE)` — an old file that does not build against the + current tree is a much weaker result, because the tests may never + have run at all. **Read which one it printed.** +- **A fix must be COMMITTED before it is bitten** — but *not* for the + reason previously recorded here. This file used to say `scripts/bite` + "restores by `git checkout --`, which reverts the file to HEAD", and + that a review round's fixes were wiped that way during #165. **That + mechanism description is false and was verified false:** the script + copies the file to a `mktemp` path before the swap and restores from + that copy, under an `EXIT INT TERM` trap, so uncommitted work in the + bitten file survives. It has never touched git state beyond a + read-only `git show`. The rule still stands on its own merits — + gate results must describe the pushed tree, and a `cargo fmt` after + a commit splits worktree from branch — but do not repeat the + destroys-your-work claim, which will push the next reader toward + `git stash` to protect themselves, straight into the repo-global + trap above. Corollary for a NEW file: the swap-over-`git show` mode + does not apply at all, so its claims must be bitten by hand-editing. - **A CONFLICTING PR silently runs no CI at all.** GitHub builds `pull_request` workflow runs against the PR's **merge ref**, which it does not create while the branch conflicts with its base. So pushes diff --git a/scripts/bite b/scripts/bite index cccd3eb..00b367e 100755 --- a/scripts/bite +++ b/scripts/bite @@ -1,6 +1,7 @@ #!/bin/sh -# scripts/bite --- prove a fix's tests BITE: run them against an older -# version of ONE file and succeed only if they FAIL there. +# scripts/bite --- prove a fix's tests BITE: run them against the +# working tree (they must PASS), then against an older version of ONE +# file (they must FAIL). # # scripts/bite [cargo-test-args...] # @@ -8,21 +9,40 @@ # scripts/bite HEAD~1 builtin/runtime/editops.lua \ # --test editops_acceptance -- capitalize trim_on_save_unexpected # -# Exit status: 0 when the named tests fail against 's version of -# (the fix bites), 1 when they still pass (vacuous), 2 on -# usage/setup errors. The working-tree file is restored on every exit -# path, including interrupts. +# Exit status: 0 when the named tests pass now and fail against +# 's version of (the fix bites), 1 when they still pass +# there (vacuous), 2 on usage/setup errors, 3 when the POSITIVE +# CONTROL fails. The working-tree file is restored on every exit path, +# including interrupts. # -# Caveat: a COMPILE error of the old tree also counts as "fails" — -# correct, but weaker evidence than a clean assertion failure; eyeball -# the output when the swapped file is Rust rather than Lua. +# THE POSITIVE CONTROL, and why it is not optional. Before this +# existed the script ran only the swapped tree, so it could not +# distinguish "my fix is load-bearing" from "my test is broken": a +# typo, an unrelated compile break, or a filter that matches nothing +# all made the swapped run fail, and a failing swapped run was the +# only thing it checked. It printed `bite: OK` and certified nothing. +# So the control asserts two things, because passing alone is not +# enough — `cargo test` with a filter matching zero tests exits 0, and +# then a compile error in the old tree would still read as OK: +# +# * the named tests PASS against the working tree, and +# * at least one test actually RAN. +# +# COMPILE errors of the old tree still count as "fails", and that is +# correct but weaker than a clean assertion failure --- the swapped +# file may simply not build against the current tree. The script now +# says which kind it saw instead of leaving it to be eyeballed. Prefer +# an assertion failure; treat a compile failure as a prompt to narrow +# the swap to a file that compiles both ways. # # Why this exists: bite-verification is step 4 of the working method, -# and the obvious shortcut — git stash — is a trap here. The stash +# and the obvious shortcut --- git stash --- is a trap here. The stash # namespace is REPO-GLOBAL: shared across every worktree and with # humans, so a scripted push/pop can collide with (or pop!) someone # else's stashed work. This helper never touches git state beyond a -# read-only `git show`. +# read-only `git show`, and it restores the working file from a +# mktemp copy --- NOT from git, so uncommitted work in is +# preserved. set -eu @@ -40,6 +60,34 @@ if [ ! -f "$path" ]; then exit 2 fi +# Sum the `N passed` figures across every `test result:` line, so a +# multi-target invocation is counted correctly rather than only its +# last binary. +count_passed() { + sed -n 's/^test result: ok\. \([0-9][0-9]*\) passed.*/\1/p' \ + | awk '{ total += $1 } END { print total + 0 }' +} + +# --- Positive control: the tests must pass, and must exist, NOW. --- +echo "bite: positive control --- running against the working tree" >&2 +control_status=0 +control_out=$(cargo test "$@" 2>&1) || control_status=$? +printf '%s\n' "$control_out" + +if [ "$control_status" -ne 0 ]; then + echo "bite: NO CONTROL --- tests do not pass against the working tree." >&2 + echo "bite: fix the tests first; a failure here makes the swapped run meaningless." >&2 + exit 3 +fi + +control_ran=$(printf '%s\n' "$control_out" | count_passed) +if [ "$control_ran" -eq 0 ]; then + echo "bite: NO CONTROL --- the filter matched no tests (0 passed)." >&2 + echo "bite: check the test names; an empty filter passes everywhere and proves nothing." >&2 + exit 3 +fi +echo "bite: control OK --- $control_ran test(s) pass against the working tree" >&2 + saved=$(mktemp "${TMPDIR:-/tmp}/bite.XXXXXX") cp -- "$path" "$saved" restore() { @@ -52,9 +100,22 @@ trap restore EXIT INT TERM # script works from any directory inside the repo. git show "$ref:./$path" > "$path" -if cargo test "$@"; then +echo "bite: swapping in $ref:$path" >&2 +swapped_status=0 +swapped_out=$(cargo test "$@" 2>&1) || swapped_status=$? +printf '%s\n' "$swapped_out" + +if [ "$swapped_status" -eq 0 ]; then echo "bite: VACUOUS --- tests still pass against $ref:$path" >&2 exit 1 fi -echo "bite: OK --- tests fail against $ref:$path (the fix bites)" +# Distinguish the two ways the swapped run can fail. An assertion +# failure is the evidence we want; a compile failure only says the old +# file does not build here. +if printf '%s\n' "$swapped_out" | grep -q '^error\[E[0-9]*\]\|^error: could not compile'; then + echo "bite: OK (COMPILE) --- $ref:$path does not build against the current tree." >&2 + echo "bite: weaker evidence than an assertion failure --- the tests may never have run." >&2 +else + echo "bite: OK (assertion) --- tests fail against $ref:$path (the fix bites)" +fi