`--fail-ack-append-after N` states that the run met an acknowledgment-journal
failure at append N. A run that ended before reaching N — too short a window,
too few groups, a store that refused every submit — met none of it, and still
emitted an ordinary bundle: the flag was on the command line, the failure
never happened, and nothing in the output said so. Same class of defect as
the ack write failure this flag exists to induce, one level up.
So the arming is treated as a claim and `fired` as the counter, compared at
run completion on both seams, before any total is read. The negative control
is the load-bearing half of the test: a check that refused every armed run
would satisfy the refusal assertion, so the same seam runs with a reachable
target and must still report the *injected* failure rather than the arming
one. The two refusals are asserted to stay distinguishable.
The usage statement said "emits no bundle by construction", which was only
true when the target was reachable. It now says so unconditionally, which is
what the code does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKzM69CHmBuDcA3qN1jFdh
A bundle's verification flags were assertions about methodology that
nothing checked. This makes them conditional on machine-readable
declarations of what the run actually did, and fixes five cases where the
emitter stated something it had not established.
bench/result-schema.json gains a required run_conditions block of ten
closed enumerations -- initialization and mutation path, checkpoint and
index state, index-run ceiling, receipt reconciliation, objects_new
source, commit-id uniqueness, build profile, environment fidelity. It is
not disclosure beside the claims; it is what the claims are conditioned
on, so a harness can only assert what its declaration permits. A prose
caveat field was rejected: free text is not a condition a consumer can
check, and a bundle whose caveats live only in a report reads as
unconditional to everyone who receives it.
The branch conditional forbids the three newly earnable claims on the
journal-drive path and pins its four provenance declarations to the only
values that seam can make. Forbidding the claims alone left the hole one
field over -- a drive bundle could otherwise declare exact receipt
reconciliation it has no receipts to perform.
Emitter defects, each found by reading the schema against the code:
- Setup traffic was inside the measured interval. Both counter baselines
were read only at the end, so repository creation -- which goes through
submit, and therefore fences and signs -- was counted as measured work
while the bundle asserted setup_traffic_excluded. A false exclusion
claim is worse than a wrong number: a wrong number invites scrutiny
and this deflects it.
- A zero-work run produced a schema-valid bundle asserting uniqueness
over zero ids and three-objects-per-commit over zero commits. Both are
vacuously true, which is why they must not be earnable that way: the
result is indistinguishable from a measured run by the consumer the
schema exists to serve. Refused by name at two altitudes.
- An ACK-journal write failure ended the run quietly. It set a stop flag
without recording a refusal, so neither the fatal guard nor the
zero-work guard saw it, and the bundle omitted a committed transaction
while still counting its fence and its signature -- one counted
transaction against two fences and two signings. It is now a fatal
incomplete-accounting refusal carrying the original errno, because the
commit happened: folding it into the refused count would report a
transaction the store committed as one it declined.
- Widening that class to "a failure that produces a value nobody read"
found three more. A shard with no counters summed to zero fences,
silently shrinking the total that bounds every durability claim. A
digest of an unreadable file returned the digest of empty input -- a
well-formed 64-hex value indistinguishable from a real one, feeding
five attested fields. An unreadable /proc/meminfo published one byte
of RAM. All three refuse now.
- Index steady state was inferred from any directory entry, so one stray
file declared the index sealed. Entries are parsed back as index runs
against the root's own uuid; an unparseable entry is reported as
unvalidatable rather than lowering a count, and a backlog is refused
because neither named value describes sealing that did not keep up.
deployment.tmpfs, persistent_data_mount, and hardware.filesystem were
constants -- the emitter could assert deployment facts it had never
checked. They are read from /proc/mounts now. Both deployment fields relax
to booleans so a diagnostic run is representable at all: it was previously
not disqualified but unencodable, and a schema that can only express
successful runs is not a record of what was measured. outcome=pass
requires reference fidelity at every gate, and a diagnostic run may never
carry a pass verdict.
Hardware profiles are derived, never accepted. store-bench parses the
whole frozen profile tables and names a profile only by exact comparison,
iterating every pinned fact rather than every supplied one -- so a fact
the emitter does not model eliminates the profile instead of being
invisible. That took the honest unobserved list on this host from 7 facts
to 24, which is the inversion working. A deployed-node harness supplies
privileged facts as evidence to compare, never as a label. The outcome
derivation now also requires the checkpoint and index conditions, because
fidelity was the only thing preventing a pass and would have stopped being
so the moment profile recognition started working, at which point the
emitter would have produced a pass its own validator rejects.
operation_receipts_reconciled is expressible and deliberately not emitted:
the bench accepts any Committed status without comparing the payload, and
the digest it records is of the operation id rather than the receipt.
Contract review 2026-07-28-C records the amendment and the ceiling it does
not close: run_conditions is self-reported, and only the index-run ceiling
is cross-checked against an independent value. Scope 5.1 records why a
full filesystem is indistinguishable from a concurrency flake by symptom,
and that an I/O error must reach a report with its errno intact -- the
same requirement as the incomplete-accounting refusal above.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh reports
bundle=schema-valid on both paths, zero_work_run=refused, and
unaccounted_ack_run=refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.6 deliverables 1-3, partial. This is evidence, not a freeze: no
Wave B deliverable set is complete and the throughput figure below is not
a publishable bundle.
The revert-and-observe-red acceptance for cd37f8b's two generators was
never actually performed -- the tests asserted it in their doc comments,
which is a claim. It is performed now, in a throwaway copy, one mutation
at a time, and each reddens exactly one test for the right reason. With
segment frame validation reverted to pushing footer offsets straight into
the adopted set, the store still opens and recovery still completes: it
reports recovery_ok=true and adopts the corrupted frame as authority, so
the red is exit 0 where 65 is required rather than a store that failed to
open. With the shard-index check removed from journal binding, the moved
journal is adopted whole and root_uuid alone does not catch it -- the Wave
A blocker reproduced.
The eight Wave B failpoint rows now drive StoreEngine::submit in-process
with the full six-field expectation asserted against the frozen oracle,
field by field rather than by struct comparison. No pending-wave-b row
remains. The four fields Wave A could not reach come from four independent
observations: the store naming itself poisoned, the two-root status read,
a receipt obtainable at all, and a different transaction submitted to the
same shard before any reopen. The last two are not the same question -- a
shard can be unpoisoned and still refuse a later append because its writer
thread died, which is exactly what phase-aware panic ownership fixed and
what this now checks from outside. AfterRootCasBeforeWaiterWake is
observed as a genuinely hung submit whose receipt is still retrievable,
so a waiter that never wakes is proved to be a hung request rather than an
absent transaction. Every row also asserts its group's fence count against
the public durability snapshot.
Two flake campaigns were measured rather than rerun: 4 failures in 40, then
7 in 40, from two distinct causes. One is a finding -- publishing a group
adds an index delta layer and none are sealed, so submit refuses after
exactly max_index_runs publications for the life of an engine. Both fixed
structurally; 200/200 and 40/40 after.
store-bench emit-skeleton now defaults to the submit path, with the journal
seam retained under --path drive for comparison. The signer is real, the
ref CAS is evaluated by the sequencer, and objects_new is summed from the
store's own receipts rather than multiplied out of the transaction count.
Explicit blockers, retained rather than worked around:
- StoreEngine::open still refuses startup state 1, so the benchmark seeds
its root by a non-production path. Seeding a store off the production
path in order to measure the production path is the charter item 8
smell; the disclosure is recorded in the fixture, a const doc, and the
module docs, and a test asserts open still refuses so it cannot go
stale in the safe direction.
- P2 is blocked three ways -- checkpointing disabled, no steady state
under the index-run ceiling, and no warmup/repetition/trim protocol.
The rate emitted is a debug build on tmpfs, marked preliminary.
- The 100 SIGKILL cycles still drive the journal seam, so kill -9 never
lands inside a real publication.
- Four schema claims became earnable and are requested, not emitted;
bench/result-schema.json is lead-owned.
- Reopen after close needs a bounded, measured, reported wait, because
the root LOCK outlives StoreEngine::drop. Diagnosed since as fork/exec
inheritance of the lock file description; the fix belongs in the lock
primitive, and this wait is removed when it lands.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh 100 cycles,
recovery_failures=0, acknowledged_loss=0, torn_transactions=0,
repeated_adoptions=0, bundle=schema-valid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Wave A review recorded that the crash matrix structurally could not
express its own first blocker: no failpoint corrupts a frame inside a
sealed segment, and none moves a journal between shards of one root. A
green matrix on a defect it cannot represent is the same trap as a helper
nothing calls, so the finding was carried forward rather than closed.
Add the two physical crash-image generators as a `damage` subcommand on
the crash driver, and two tests that drive them through the same
production `reconcile` path the matrix and the recovery script use.
- sealed-frame-corruption flips one payload byte in the first frame of
a segment the drive actually sealed and installed. The footer stays
structurally valid, so only frame verification can reject it.
- cross-shard-journal-movement relocates shard 1's active journal under
shard 0 of the same root. Root UUID validation cannot see this; the
journal header's shard index must be bound to the directory being
opened.
Both generators mutate a production-written image and fence the mutation;
neither synthesizes a footer, frame, manifest, or checksum, so a passing
test cannot be an artifact of the harness agreeing with itself. Each
source image is required to be unambiguous — exactly one segment or
journal — so the result does not depend on directory iteration order.
The assertions pin the refusal to its own cause rather than to any
non-zero exit: the frame case must fail on the digest recomputation and
the movement case on the shard binding, and neither may publish a partial
adoption result.
crash_matrix: 26 passed. scripts/check-phase1.sh green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the Wave A interfaces, frame format, durability ordering, and
crash/fault fixtures per doc/phase1-storage-spine-scope.md section 5. Wave B
may now start against these.
D0 lands crates/levcs-store: the sealed public API, the file-ownership split,
the single durability syscall funnel with its counters and fault hooks, the
17-entry failpoint registry in compiler-enforced correspondence with
oracle::AppendFailpoint, and the journal-level drive seam. Wave A lands the
frame codec and journal/segment lifecycle (A1), the recovery index and
checkpoints (A2), and the durable-ingest benchmark and crash harness (A3).
The adversarial review found five defects behind a green gate, three of them
blockers, all closed here. Two were the same shape: drive::reopen_through_
recovery had reimplemented a simplified recovery and called none of recovery.rs
-- so it adopted segment footer sequences without validating frame bytes,
journal_id, or root_uuid, and it double-adopted interrupted seals. The seam
between two packages was untested precisely because each package's own tests
passed. reopen_through_recovery now delegates rather than decides, and
DriveRecovery carries the recovery report verbatim so tests assert the
disposition and not merely its effect: a double adoption and a correct replay
produce the same adopted set, which is how the defect stayed invisible.
Also closed: store-bench now schema-validates its own emitted artifact with a
five-mutation negative control instead of matching JSON substrings; the ACK
reconciler distinguishes duplicates and regressions from forward gaps;
checkpoint writes and a journal truncation are routed through the durability
funnel, whose guard now covers writes and truncations rather than only sync,
rename, and unlink.
bench/result-schema.json is amended (contract review 2026-07-24-B, second and
third amendments): per-gate latency ceilings conditional on outcome so a failed
run is representable, and the verification claims split per gate so a storage
run cannot certify an object graph it never touches. Not-applicable claims are
forbidden rather than falsified; applicable-but-not-performed report false.
Every relaxation is re-pinned in the else branch and asserted member by member,
after an edit in this series silently un-pinned all eleven validation flags and
was caught only by revalidating against constructed bundles.
Arming the fault registry now requires a FaultSerial token, so the invariant is
a compile error rather than a comment. The file where this was diagnosed
carried a header saying it was deliberately the only test in it, and a second
test had been added under that comment anyway -- an 8-in-40 failure rate that
read as flakiness.
Evidence at this commit: check-phase1.sh GATE_EXIT=0 across all four feature
configurations, 124 test binaries, zero failures; verify-store-recovery.sh
--cycles 100 with recovery_failures=0, acknowledged_loss=0,
torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio
40/40 at four test threads; golden corpus byte-stable; fmt clean.
Carry-forwards, explicitly not Wave A blockers and recorded in scope section 5:
extend the crash matrix to generate sealed-frame corruption and cross-shard
journal movement, since it structurally cannot express the class the first
blocker belonged to; and wire GroupBuilder through B1's production path, since
deliverable 4-A1.2 is presently asserted only over a type nothing calls.
Charter item 9 -- ask every package what of its work is correct but uncalled --
is accepted for every subsequent wave. It, not the review, found the class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG
Levi Neuwirth