`verify-store-recovery.sh` looked for `store-crash-driver` and `store-bench`
under `$repo_root/target/debug`. `CARGO_TARGET_DIR`, a `build.target-dir` in a
config.toml, and a shared workspace target all move that, and on a host that
sets one the script builds successfully and then exits 70 saying the driver
"was not built" - which reads as a compilation failure and is not one. It asks
cargo now.
The working directory deliberately does not follow. It holds crash roots whose
filesystem is part of what the campaign measures, which is why the script
already refuses tmpfs; a shared target directory may be on a different one. The
distinction is commented so the remaining `$repo_root/target` is not read as a
missed substitution.
Two scope claims were behind the code and one of them undersold the work.
Section 7 said the harness satisfied two of four run conditions. It satisfies
three: index sealing earned `index_maintenance = runs_sealed`, derived from the
`IndexRun` files read back off the device. `checkpointing` has moved from
`unimplemented` to `enabled_not_reached` - the mechanism exists and short runs
do not accumulate enough work to trip it - so what remains is a run long enough
to take a checkpoint, not a mechanism to build.
Section 6.6 item 5 said the emitter wrote a schema-invalid bundle. That closed
with 062797d. The passage is kept rather than deleted because the mechanism is
the point: the gate was red for a schema the store satisfied, since three
`store-bench` unit tests validate the emitted bundle against the schema rather
than against substrings. That is schema conformance living inside the gate,
which is what review 2026-07-24-B was after.
New carry-forward: `ENGINE_MAX_INDEX_RUNS` outlived its reason. It is 1,000,000
against a store default of 64 and still carries the comment "raised because
index-delta sealing is unimplemented", but sealing landed in fef8520. Two
consequences follow that did not before - the emitted `index_run_ceiling` names
a reason that is no longer true, and at that ceiling the fan-out trigger never
fires, so `runs_sealed` above is earned by entry pressure alone. It is left as a
decision rather than taken: dropping the raise needs the measured run to survive
the real ceiling, and renaming the value is an amendment to the lead-owned
schema, whose own rule is to request it rather than emit it.
Verified on this box at b4e4c7e: `check-phase1.sh` reports GATE_EXIT=0, and
`verify-store-recovery.sh --cycles 2` reports matrix=pass, bundle=schema-valid
on both paths, acknowledged_loss=0, torn_transactions=0, VERIFY_EXIT=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bundle's verification flags were assertions about methodology that
nothing checked. This makes them conditional on machine-readable
declarations of what the run actually did, and fixes five cases where the
emitter stated something it had not established.
bench/result-schema.json gains a required run_conditions block of ten
closed enumerations -- initialization and mutation path, checkpoint and
index state, index-run ceiling, receipt reconciliation, objects_new
source, commit-id uniqueness, build profile, environment fidelity. It is
not disclosure beside the claims; it is what the claims are conditioned
on, so a harness can only assert what its declaration permits. A prose
caveat field was rejected: free text is not a condition a consumer can
check, and a bundle whose caveats live only in a report reads as
unconditional to everyone who receives it.
The branch conditional forbids the three newly earnable claims on the
journal-drive path and pins its four provenance declarations to the only
values that seam can make. Forbidding the claims alone left the hole one
field over -- a drive bundle could otherwise declare exact receipt
reconciliation it has no receipts to perform.
Emitter defects, each found by reading the schema against the code:
- Setup traffic was inside the measured interval. Both counter baselines
were read only at the end, so repository creation -- which goes through
submit, and therefore fences and signs -- was counted as measured work
while the bundle asserted setup_traffic_excluded. A false exclusion
claim is worse than a wrong number: a wrong number invites scrutiny
and this deflects it.
- A zero-work run produced a schema-valid bundle asserting uniqueness
over zero ids and three-objects-per-commit over zero commits. Both are
vacuously true, which is why they must not be earnable that way: the
result is indistinguishable from a measured run by the consumer the
schema exists to serve. Refused by name at two altitudes.
- An ACK-journal write failure ended the run quietly. It set a stop flag
without recording a refusal, so neither the fatal guard nor the
zero-work guard saw it, and the bundle omitted a committed transaction
while still counting its fence and its signature -- one counted
transaction against two fences and two signings. It is now a fatal
incomplete-accounting refusal carrying the original errno, because the
commit happened: folding it into the refused count would report a
transaction the store committed as one it declined.
- Widening that class to "a failure that produces a value nobody read"
found three more. A shard with no counters summed to zero fences,
silently shrinking the total that bounds every durability claim. A
digest of an unreadable file returned the digest of empty input -- a
well-formed 64-hex value indistinguishable from a real one, feeding
five attested fields. An unreadable /proc/meminfo published one byte
of RAM. All three refuse now.
- Index steady state was inferred from any directory entry, so one stray
file declared the index sealed. Entries are parsed back as index runs
against the root's own uuid; an unparseable entry is reported as
unvalidatable rather than lowering a count, and a backlog is refused
because neither named value describes sealing that did not keep up.
deployment.tmpfs, persistent_data_mount, and hardware.filesystem were
constants -- the emitter could assert deployment facts it had never
checked. They are read from /proc/mounts now. Both deployment fields relax
to booleans so a diagnostic run is representable at all: it was previously
not disqualified but unencodable, and a schema that can only express
successful runs is not a record of what was measured. outcome=pass
requires reference fidelity at every gate, and a diagnostic run may never
carry a pass verdict.
Hardware profiles are derived, never accepted. store-bench parses the
whole frozen profile tables and names a profile only by exact comparison,
iterating every pinned fact rather than every supplied one -- so a fact
the emitter does not model eliminates the profile instead of being
invisible. That took the honest unobserved list on this host from 7 facts
to 24, which is the inversion working. A deployed-node harness supplies
privileged facts as evidence to compare, never as a label. The outcome
derivation now also requires the checkpoint and index conditions, because
fidelity was the only thing preventing a pass and would have stopped being
so the moment profile recognition started working, at which point the
emitter would have produced a pass its own validator rejects.
operation_receipts_reconciled is expressible and deliberately not emitted:
the bench accepts any Committed status without comparing the payload, and
the digest it records is of the operation id rather than the receipt.
Contract review 2026-07-28-C records the amendment and the ceiling it does
not close: run_conditions is self-reported, and only the index-run ceiling
is cross-checked against an independent value. Scope 5.1 records why a
full filesystem is indistinguishable from a concurrency flake by symptom,
and that an I/O error must reach a report with its errno intact -- the
same requirement as the incomplete-accounting refusal above.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh reports
bundle=schema-valid on both paths, zero_work_run=refused, and
unaccounted_ack_run=refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the Wave A interfaces, frame format, durability ordering, and
crash/fault fixtures per doc/phase1-storage-spine-scope.md section 5. Wave B
may now start against these.
D0 lands crates/levcs-store: the sealed public API, the file-ownership split,
the single durability syscall funnel with its counters and fault hooks, the
17-entry failpoint registry in compiler-enforced correspondence with
oracle::AppendFailpoint, and the journal-level drive seam. Wave A lands the
frame codec and journal/segment lifecycle (A1), the recovery index and
checkpoints (A2), and the durable-ingest benchmark and crash harness (A3).
The adversarial review found five defects behind a green gate, three of them
blockers, all closed here. Two were the same shape: drive::reopen_through_
recovery had reimplemented a simplified recovery and called none of recovery.rs
-- so it adopted segment footer sequences without validating frame bytes,
journal_id, or root_uuid, and it double-adopted interrupted seals. The seam
between two packages was untested precisely because each package's own tests
passed. reopen_through_recovery now delegates rather than decides, and
DriveRecovery carries the recovery report verbatim so tests assert the
disposition and not merely its effect: a double adoption and a correct replay
produce the same adopted set, which is how the defect stayed invisible.
Also closed: store-bench now schema-validates its own emitted artifact with a
five-mutation negative control instead of matching JSON substrings; the ACK
reconciler distinguishes duplicates and regressions from forward gaps;
checkpoint writes and a journal truncation are routed through the durability
funnel, whose guard now covers writes and truncations rather than only sync,
rename, and unlink.
bench/result-schema.json is amended (contract review 2026-07-24-B, second and
third amendments): per-gate latency ceilings conditional on outcome so a failed
run is representable, and the verification claims split per gate so a storage
run cannot certify an object graph it never touches. Not-applicable claims are
forbidden rather than falsified; applicable-but-not-performed report false.
Every relaxation is re-pinned in the else branch and asserted member by member,
after an edit in this series silently un-pinned all eleven validation flags and
was caught only by revalidating against constructed bundles.
Arming the fault registry now requires a FaultSerial token, so the invariant is
a compile error rather than a comment. The file where this was diagnosed
carried a header saying it was deliberately the only test in it, and a second
test had been added under that comment anyway -- an 8-in-40 failure rate that
read as flakiness.
Evidence at this commit: check-phase1.sh GATE_EXIT=0 across all four feature
configurations, 124 test binaries, zero failures; verify-store-recovery.sh
--cycles 100 with recovery_failures=0, acknowledged_loss=0,
torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio
40/40 at four test threads; golden corpus byte-stable; fmt clean.
Carry-forwards, explicitly not Wave A blockers and recorded in scope section 5:
extend the crash matrix to generate sealed-frame corruption and cross-shard
journal movement, since it structurally cannot express the class the first
blocker belonged to; and wire GroupBuilder through B1's production path, since
deliverable 4-A1.2 is presently asserted only over a type nothing calls.
Charter item 9 -- ask every package what of its work is correct but uncalled --
is accepted for every subsequent wave. It, not the review, found the class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG
Levi Neuwirth