Scope 6.4 deliverable 1, startup state 1. StoreEngine::open now creates a
root rather than refusing to, and every harness that measured a store
production did not build is re-pointed at it. States 3 and 4 still refuse.
The four startup states are distinguished by one read-only classification
that creates nothing. A FORMAT entry means state 2 on the strength of the
name alone, because a FORMAT that does not decode is still FORMAT and
treating an unreadable marker as "no marker, therefore empty" would
authorize building a fresh tree over a populated root. A root holding only
LOCK is empty, since lock_root creates that file as a side effect of asking
whether the root is busy. The signer check moved above every root access,
because once an absent root is initialized rather than refused, a signerless
configuration would otherwise create a tree, write FORMAT and fsync the
parent before failing -- a configuration error must not leave a root behind.
Three defects found in review, all fixed here rather than deferred.
A deeply absent path was created with create_dir_all and only its immediate
parent fenced, so open could report success over ancestors a power loss
could take. Ancestors are now created one at a time -- so what this call
created is exactly what it fences, and a racing creator surfaces as
AlreadyExists rather than being absorbed -- and fenced deepest-first, since
a directory entry lives in the parent that names it and the reverse order
can leave a fenced parent naming an unfenced child.
An interrupted initialization was unrecoverable: any tree residue classified
the root as non-empty-without-FORMAT and it was refused forever, with a
message about legacy layouts that had nothing to do with what happened. A
root being built now carries an INITIALIZING marker, installed by rename as
the first durable act and removed as the last, so a durable partial tree
always has a durable marker beside it and every crash window is resumable.
Recognizing it requires both a byte-exact marker this crate alone writes and
every entry in the root drawn from a closed set of names this store
invented, so a foreign layout cannot be mistaken for abandoned
initialization and overwritten -- the direction that matters, since refusing
a resumable root costs an operator time and overwriting a real one costs
their data. Sibling staging with atomic installation was the alternative and
is structurally blocked: LOCK lives inside the root, so the root must exist
before any mutation can be serialized, and renaming a tree onto a directory
containing LOCK fails ENOTEMPTY.
The classifier then ignored INITIALIZING.tmp by name regardless of type or
contents, and initialization opened that name with create plus truncate. An
operator's file there was destroyed silently, and a symlink there truncated
a file outside the root to 27 bytes and then removed the link -- destroying
data the store never owned and erasing the evidence, while open returned Ok
and reported a working store. The justification for ignoring the name was
that only this path could have written it, which is circular: that is the
claim the classifier runs in order to establish. Every entry is now judged
by lstat type before anything opens it, the temporary marker is validated as
an exact regular marker or refused, and installation is create-new rather
than create-truncate. Contract review 2026-07-28-D records the two no-follow
open primitives this added to the frozen sys.rs, and the four further
symlink hazards in segment.rs that are recorded rather than fixed -- the
first of which lets two processes believe they hold one root lock.
A fifo at that name made the pre-fix open block forever: one mkfifo in a
configured root was an unbounded startup hang, not only a data hazard.
The engine and the drive seam are now asserted to recover one crash image
identically, closing a gap that was true by construction and untested.
Charter item 8 applied to the harness: the in-crate test helper no longer
calls segment::initialize_root, so every writer test builds its root through
open; the ROOT_SEEDED_BY_NON_PRODUCTION_PATH disclosure is retired; and the
fixture's root_seeded_by becomes a stable token matched by exact equality,
with the history moved to an adjacent reason field -- a substring match
passes on a value that has drifted to mean something else.
The d0 contract test asserting open returns NotImplemented for any valid
configuration is obsoleted by this deliverable and replaced with the
stronger property: a signerless configuration is refused and leaves no root
behind. It moves off a fixed /tmp path, which under the old check ordering
would have created a real store root on every gate run on every machine.
scripts/check-phase1.sh GATE_EXIT=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.6 deliverables 1-3, partial. This is evidence, not a freeze: no
Wave B deliverable set is complete and the throughput figure below is not
a publishable bundle.
The revert-and-observe-red acceptance for cd37f8b's two generators was
never actually performed -- the tests asserted it in their doc comments,
which is a claim. It is performed now, in a throwaway copy, one mutation
at a time, and each reddens exactly one test for the right reason. With
segment frame validation reverted to pushing footer offsets straight into
the adopted set, the store still opens and recovery still completes: it
reports recovery_ok=true and adopts the corrupted frame as authority, so
the red is exit 0 where 65 is required rather than a store that failed to
open. With the shard-index check removed from journal binding, the moved
journal is adopted whole and root_uuid alone does not catch it -- the Wave
A blocker reproduced.
The eight Wave B failpoint rows now drive StoreEngine::submit in-process
with the full six-field expectation asserted against the frozen oracle,
field by field rather than by struct comparison. No pending-wave-b row
remains. The four fields Wave A could not reach come from four independent
observations: the store naming itself poisoned, the two-root status read,
a receipt obtainable at all, and a different transaction submitted to the
same shard before any reopen. The last two are not the same question -- a
shard can be unpoisoned and still refuse a later append because its writer
thread died, which is exactly what phase-aware panic ownership fixed and
what this now checks from outside. AfterRootCasBeforeWaiterWake is
observed as a genuinely hung submit whose receipt is still retrievable,
so a waiter that never wakes is proved to be a hung request rather than an
absent transaction. Every row also asserts its group's fence count against
the public durability snapshot.
Two flake campaigns were measured rather than rerun: 4 failures in 40, then
7 in 40, from two distinct causes. One is a finding -- publishing a group
adds an index delta layer and none are sealed, so submit refuses after
exactly max_index_runs publications for the life of an engine. Both fixed
structurally; 200/200 and 40/40 after.
store-bench emit-skeleton now defaults to the submit path, with the journal
seam retained under --path drive for comparison. The signer is real, the
ref CAS is evaluated by the sequencer, and objects_new is summed from the
store's own receipts rather than multiplied out of the transaction count.
Explicit blockers, retained rather than worked around:
- StoreEngine::open still refuses startup state 1, so the benchmark seeds
its root by a non-production path. Seeding a store off the production
path in order to measure the production path is the charter item 8
smell; the disclosure is recorded in the fixture, a const doc, and the
module docs, and a test asserts open still refuses so it cannot go
stale in the safe direction.
- P2 is blocked three ways -- checkpointing disabled, no steady state
under the index-run ceiling, and no warmup/repetition/trim protocol.
The rate emitted is a debug build on tmpfs, marked preliminary.
- The 100 SIGKILL cycles still drive the journal seam, so kill -9 never
lands inside a real publication.
- Four schema claims became earnable and are requested, not emitted;
bench/result-schema.json is lead-owned.
- Reopen after close needs a bounded, measured, reported wait, because
the root LOCK outlives StoreEngine::drop. Diagnosed since as fork/exec
inheritance of the lock file description; the fix belongs in the lock
primitive, and this wait is removed when it lands.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh 100 cycles,
recovery_failures=0, acknowledged_loss=0, torn_transactions=0,
repeated_adoptions=0, bundle=schema-valid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the Wave A interfaces, frame format, durability ordering, and
crash/fault fixtures per doc/phase1-storage-spine-scope.md section 5. Wave B
may now start against these.
D0 lands crates/levcs-store: the sealed public API, the file-ownership split,
the single durability syscall funnel with its counters and fault hooks, the
17-entry failpoint registry in compiler-enforced correspondence with
oracle::AppendFailpoint, and the journal-level drive seam. Wave A lands the
frame codec and journal/segment lifecycle (A1), the recovery index and
checkpoints (A2), and the durable-ingest benchmark and crash harness (A3).
The adversarial review found five defects behind a green gate, three of them
blockers, all closed here. Two were the same shape: drive::reopen_through_
recovery had reimplemented a simplified recovery and called none of recovery.rs
-- so it adopted segment footer sequences without validating frame bytes,
journal_id, or root_uuid, and it double-adopted interrupted seals. The seam
between two packages was untested precisely because each package's own tests
passed. reopen_through_recovery now delegates rather than decides, and
DriveRecovery carries the recovery report verbatim so tests assert the
disposition and not merely its effect: a double adoption and a correct replay
produce the same adopted set, which is how the defect stayed invisible.
Also closed: store-bench now schema-validates its own emitted artifact with a
five-mutation negative control instead of matching JSON substrings; the ACK
reconciler distinguishes duplicates and regressions from forward gaps;
checkpoint writes and a journal truncation are routed through the durability
funnel, whose guard now covers writes and truncations rather than only sync,
rename, and unlink.
bench/result-schema.json is amended (contract review 2026-07-24-B, second and
third amendments): per-gate latency ceilings conditional on outcome so a failed
run is representable, and the verification claims split per gate so a storage
run cannot certify an object graph it never touches. Not-applicable claims are
forbidden rather than falsified; applicable-but-not-performed report false.
Every relaxation is re-pinned in the else branch and asserted member by member,
after an edit in this series silently un-pinned all eleven validation flags and
was caught only by revalidating against constructed bundles.
Arming the fault registry now requires a FaultSerial token, so the invariant is
a compile error rather than a comment. The file where this was diagnosed
carried a header saying it was deliberately the only test in it, and a second
test had been added under that comment anyway -- an 8-in-40 failure rate that
read as flakiness.
Evidence at this commit: check-phase1.sh GATE_EXIT=0 across all four feature
configurations, 124 test binaries, zero failures; verify-store-recovery.sh
--cycles 100 with recovery_failures=0, acknowledged_loss=0,
torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio
40/40 at four test threads; golden corpus byte-stable; fmt clean.
Carry-forwards, explicitly not Wave A blockers and recorded in scope section 5:
extend the crash matrix to generate sealed-frame corruption and cross-shard
journal movement, since it structurally cannot express the class the first
blocker belonged to; and wire GroupBuilder through B1's production path, since
deliverable 4-A1.2 is presently asserted only over a type nothing calls.
Charter item 9 -- ask every package what of its work is correct but uncalled --
is accepted for every subsequent wave. It, not the review, found the class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG
Levi Neuwirth