`adopt_projection` reached `build` and stopped there. Wiring it through
`submit` is one change because there is no safe partial: the moment the
builder stops refusing, a submit that ignores the adoption drops a live
pin with no outcome, which is the leak scope 6.5 declares a bug.
The pin cannot live inside the transaction. `Prepared` retains an
`Arc<ValidatedTransaction>` so a group can be re-sequenced when the
pre-mark deadline recheck drops a member, and an `Arc` has no move-out.
It travels beside it in a one-shot `AdoptionSlot` that settles itself
from an append phase rather than from a guess: the phase advances at the
first frame write, not at the fence, so torn and unfenced bytes stay
recovery-owned. That makes the unenumerated routes safe by construction
-- queue rejection, pre-append errors and panics, deadline reforming,
writer unwinding -- instead of correct only where someone remembered.
Two orderings are now statements rather than drop-order accidents. Both
the poison window and every pre-append refusal settle the pin before
resolving the waiter; otherwise a submit could return while staging still
believed the pin was live.
Membership could not travel as an index delta. Adopted generations sit in
a reserved band `1 << 63` away from journal generations, and an index run
packs `segment_generation` as a 16-bit delta from a per-namespace section
base, so one section cannot hold both domains -- the adoption committed
and the next checkpoint poisoned the shard. B3 now materializes
staging-owned runs post-fence, deterministically and idempotently, and
`resolve_committed` only opens, verifies, and pins them. Recovery replays
the same materialization, which is what makes a crash between the fence
and materialization recoverable rather than ambiguous. Runs partition one
per adoption, splitting every 65,536 ordinals, because two adoptions'
sequences differ by more than a section can span.
The run count is therefore knowable before anything is appended, so the
ceiling check is a refusal at revalidation rather than a poison after the
fence.
Payload kind is a matrix axis: all eight Wave B rows drive inline and
staged-projection, and the adoption outcome is read from staging's
counters rather than derived, so the fixture and the run remain two
independent derivations.
Contract review 2026-08-09-A records the interface changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGdH5V43XWnj1PdHqiPktQ
A delta layer's `through_shard_sequence` states that every frame from its
shard through that sequence is accounted for in the index. Skipping the
layer when the delta was empty left a group that introduced no objects
advancing the committed sequence without advancing coverage, so the next
checkpoint refused: it read the highest layer stamp as the reach of the
index and concluded that pruning would drop objects that never existed.
Any transaction introducing no objects reached it. An empty layer is the
smallest honest way to record that the sequence happened and carried
nothing, so `with_subtree` now installs one whenever the publication
advances the shard's committed sequence. A maintenance publication
appends no frame and still owes nothing.
`coverable_through` had the same gap from the other side: it stopped at
the first layer whose entries did not resolve as a segment or an active
tail, which excluded adopted projection artifacts. Those are the most
durable entries in the index -- not in the journal at all, and carried
forward by every successor generation -- and were read as the least.
The empty layer counts toward layer fan-out and may seal into a
zero-entry run. That is the bounded-run model working as designed and is
the failure-safe direction: a stamped empty layer costs a run slot, an
unstamped one costs the ability to checkpoint at all.
Asserted with a plain object-less transaction rather than through the
adopting path that found it, so the regression names what actually broke.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGdH5V43XWnj1PdHqiPktQ
D3 adopts a staged projection through the same eight Wave B locations an
inline transaction reaches, and it fails differently at each. Giving adoption
its own failpoint rows would produce a parallel list that drifts from this one
the first time a location is added to either, so payload kind becomes a third
axis alongside the location the failpoint names and the action the driver
chooses - the same independence `actions` already documents.
The adoption expectation gets the two-derivation treatment the recovery
outcome has. The fixture states it, `PhysicalStateClass::adoption_outcome`
derives it, and the matrix requires them to agree. The rule is physical and
nothing else: whether a frame binding the artifacts exists on the device. No
bytes means the artifacts are unreferenced and reclaiming them is correct; a
fenced frame binds the manifest, so the pin outlives the process; fenced and
published means the writer already settled `Adopted` against a root that
references them.
Note the asymmetry against `required_outcome`. `PartialFrame` and
`WholeFrameUnfenced` are `AbsentRetriable` there and `TransferredToRecovery`
here, because bytes are on the device and only recovery may say what they mean.
A transaction that will not commit and a pin whose artifacts may be referenced
are different questions, and collapsing them is how a retriable refusal would
come to delete content.
Both derivations run on every submit row now, before any row lists
`staged_projection`. That is deliberate: turning the kind on becomes a matter of
listing it rather than of also getting the expectation right in the same commit.
Verified by stating `DefinitivePreAppendFailure` on `DuringCommittedRootBuild`,
which the class derives as `TransferredToRecovery` - the assertion names the row
and says what settling it that way would license.
No row lists `staged_projection` yet and nothing drives one, so this commit adds
no coverage claim. D3 does not merge without the adoption cases running.
`inline` is required on every submit row rather than merely allowed: it is the
payload the eight locations were characterized with, and a row that dropped it
while adding the new kind would move coverage sideways while reading as having
added some.
check-phase1.sh reports GATE_EXIT=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.4 deliverable 1, startup state 1. StoreEngine::open now creates a
root rather than refusing to, and every harness that measured a store
production did not build is re-pointed at it. States 3 and 4 still refuse.
The four startup states are distinguished by one read-only classification
that creates nothing. A FORMAT entry means state 2 on the strength of the
name alone, because a FORMAT that does not decode is still FORMAT and
treating an unreadable marker as "no marker, therefore empty" would
authorize building a fresh tree over a populated root. A root holding only
LOCK is empty, since lock_root creates that file as a side effect of asking
whether the root is busy. The signer check moved above every root access,
because once an absent root is initialized rather than refused, a signerless
configuration would otherwise create a tree, write FORMAT and fsync the
parent before failing -- a configuration error must not leave a root behind.
Three defects found in review, all fixed here rather than deferred.
A deeply absent path was created with create_dir_all and only its immediate
parent fenced, so open could report success over ancestors a power loss
could take. Ancestors are now created one at a time -- so what this call
created is exactly what it fences, and a racing creator surfaces as
AlreadyExists rather than being absorbed -- and fenced deepest-first, since
a directory entry lives in the parent that names it and the reverse order
can leave a fenced parent naming an unfenced child.
An interrupted initialization was unrecoverable: any tree residue classified
the root as non-empty-without-FORMAT and it was refused forever, with a
message about legacy layouts that had nothing to do with what happened. A
root being built now carries an INITIALIZING marker, installed by rename as
the first durable act and removed as the last, so a durable partial tree
always has a durable marker beside it and every crash window is resumable.
Recognizing it requires both a byte-exact marker this crate alone writes and
every entry in the root drawn from a closed set of names this store
invented, so a foreign layout cannot be mistaken for abandoned
initialization and overwritten -- the direction that matters, since refusing
a resumable root costs an operator time and overwriting a real one costs
their data. Sibling staging with atomic installation was the alternative and
is structurally blocked: LOCK lives inside the root, so the root must exist
before any mutation can be serialized, and renaming a tree onto a directory
containing LOCK fails ENOTEMPTY.
The classifier then ignored INITIALIZING.tmp by name regardless of type or
contents, and initialization opened that name with create plus truncate. An
operator's file there was destroyed silently, and a symlink there truncated
a file outside the root to 27 bytes and then removed the link -- destroying
data the store never owned and erasing the evidence, while open returned Ok
and reported a working store. The justification for ignoring the name was
that only this path could have written it, which is circular: that is the
claim the classifier runs in order to establish. Every entry is now judged
by lstat type before anything opens it, the temporary marker is validated as
an exact regular marker or refused, and installation is create-new rather
than create-truncate. Contract review 2026-07-28-D records the two no-follow
open primitives this added to the frozen sys.rs, and the four further
symlink hazards in segment.rs that are recorded rather than fixed -- the
first of which lets two processes believe they hold one root lock.
A fifo at that name made the pre-fix open block forever: one mkfifo in a
configured root was an unbounded startup hang, not only a data hazard.
The engine and the drive seam are now asserted to recover one crash image
identically, closing a gap that was true by construction and untested.
Charter item 8 applied to the harness: the in-crate test helper no longer
calls segment::initialize_root, so every writer test builds its root through
open; the ROOT_SEEDED_BY_NON_PRODUCTION_PATH disclosure is retired; and the
fixture's root_seeded_by becomes a stable token matched by exact equality,
with the history moved to an adjacent reason field -- a substring match
passes on a value that has drifted to mean something else.
The d0 contract test asserting open returns NotImplemented for any valid
configuration is obsoleted by this deliverable and replaced with the
stronger property: a signerless configuration is refused and leaves no root
behind. It moves off a fixed /tmp path, which under the old check ordering
would have created a real store root on every gate run on every machine.
scripts/check-phase1.sh GATE_EXIT=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.6 deliverables 1-3, partial. This is evidence, not a freeze: no
Wave B deliverable set is complete and the throughput figure below is not
a publishable bundle.
The revert-and-observe-red acceptance for cd37f8b's two generators was
never actually performed -- the tests asserted it in their doc comments,
which is a claim. It is performed now, in a throwaway copy, one mutation
at a time, and each reddens exactly one test for the right reason. With
segment frame validation reverted to pushing footer offsets straight into
the adopted set, the store still opens and recovery still completes: it
reports recovery_ok=true and adopts the corrupted frame as authority, so
the red is exit 0 where 65 is required rather than a store that failed to
open. With the shard-index check removed from journal binding, the moved
journal is adopted whole and root_uuid alone does not catch it -- the Wave
A blocker reproduced.
The eight Wave B failpoint rows now drive StoreEngine::submit in-process
with the full six-field expectation asserted against the frozen oracle,
field by field rather than by struct comparison. No pending-wave-b row
remains. The four fields Wave A could not reach come from four independent
observations: the store naming itself poisoned, the two-root status read,
a receipt obtainable at all, and a different transaction submitted to the
same shard before any reopen. The last two are not the same question -- a
shard can be unpoisoned and still refuse a later append because its writer
thread died, which is exactly what phase-aware panic ownership fixed and
what this now checks from outside. AfterRootCasBeforeWaiterWake is
observed as a genuinely hung submit whose receipt is still retrievable,
so a waiter that never wakes is proved to be a hung request rather than an
absent transaction. Every row also asserts its group's fence count against
the public durability snapshot.
Two flake campaigns were measured rather than rerun: 4 failures in 40, then
7 in 40, from two distinct causes. One is a finding -- publishing a group
adds an index delta layer and none are sealed, so submit refuses after
exactly max_index_runs publications for the life of an engine. Both fixed
structurally; 200/200 and 40/40 after.
store-bench emit-skeleton now defaults to the submit path, with the journal
seam retained under --path drive for comparison. The signer is real, the
ref CAS is evaluated by the sequencer, and objects_new is summed from the
store's own receipts rather than multiplied out of the transaction count.
Explicit blockers, retained rather than worked around:
- StoreEngine::open still refuses startup state 1, so the benchmark seeds
its root by a non-production path. Seeding a store off the production
path in order to measure the production path is the charter item 8
smell; the disclosure is recorded in the fixture, a const doc, and the
module docs, and a test asserts open still refuses so it cannot go
stale in the safe direction.
- P2 is blocked three ways -- checkpointing disabled, no steady state
under the index-run ceiling, and no warmup/repetition/trim protocol.
The rate emitted is a debug build on tmpfs, marked preliminary.
- The 100 SIGKILL cycles still drive the journal seam, so kill -9 never
lands inside a real publication.
- Four schema claims became earnable and are requested, not emitted;
bench/result-schema.json is lead-owned.
- Reopen after close needs a bounded, measured, reported wait, because
the root LOCK outlives StoreEngine::drop. Diagnosed since as fork/exec
inheritance of the lock file description; the fix belongs in the lock
primitive, and this wait is removed when it lands.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh 100 cycles,
recovery_failures=0, acknowledged_loss=0, torn_transactions=0,
repeated_adoptions=0, bundle=schema-valid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the Wave A interfaces, frame format, durability ordering, and
crash/fault fixtures per doc/phase1-storage-spine-scope.md section 5. Wave B
may now start against these.
D0 lands crates/levcs-store: the sealed public API, the file-ownership split,
the single durability syscall funnel with its counters and fault hooks, the
17-entry failpoint registry in compiler-enforced correspondence with
oracle::AppendFailpoint, and the journal-level drive seam. Wave A lands the
frame codec and journal/segment lifecycle (A1), the recovery index and
checkpoints (A2), and the durable-ingest benchmark and crash harness (A3).
The adversarial review found five defects behind a green gate, three of them
blockers, all closed here. Two were the same shape: drive::reopen_through_
recovery had reimplemented a simplified recovery and called none of recovery.rs
-- so it adopted segment footer sequences without validating frame bytes,
journal_id, or root_uuid, and it double-adopted interrupted seals. The seam
between two packages was untested precisely because each package's own tests
passed. reopen_through_recovery now delegates rather than decides, and
DriveRecovery carries the recovery report verbatim so tests assert the
disposition and not merely its effect: a double adoption and a correct replay
produce the same adopted set, which is how the defect stayed invisible.
Also closed: store-bench now schema-validates its own emitted artifact with a
five-mutation negative control instead of matching JSON substrings; the ACK
reconciler distinguishes duplicates and regressions from forward gaps;
checkpoint writes and a journal truncation are routed through the durability
funnel, whose guard now covers writes and truncations rather than only sync,
rename, and unlink.
bench/result-schema.json is amended (contract review 2026-07-24-B, second and
third amendments): per-gate latency ceilings conditional on outcome so a failed
run is representable, and the verification claims split per gate so a storage
run cannot certify an object graph it never touches. Not-applicable claims are
forbidden rather than falsified; applicable-but-not-performed report false.
Every relaxation is re-pinned in the else branch and asserted member by member,
after an edit in this series silently un-pinned all eleven validation flags and
was caught only by revalidating against constructed bundles.
Arming the fault registry now requires a FaultSerial token, so the invariant is
a compile error rather than a comment. The file where this was diagnosed
carried a header saying it was deliberately the only test in it, and a second
test had been added under that comment anyway -- an 8-in-40 failure rate that
read as flakiness.
Evidence at this commit: check-phase1.sh GATE_EXIT=0 across all four feature
configurations, 124 test binaries, zero failures; verify-store-recovery.sh
--cycles 100 with recovery_failures=0, acknowledged_loss=0,
torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio
40/40 at four test threads; golden corpus byte-stable; fmt clean.
Carry-forwards, explicitly not Wave A blockers and recorded in scope section 5:
extend the crash matrix to generate sealed-frame corruption and cross-shard
journal movement, since it structurally cannot express the class the first
blocker belonged to; and wire GroupBuilder through B1's production path, since
deliverable 4-A1.2 is presently asserted only over a type nothing calls.
Charter item 9 -- ask every package what of its work is correct but uncalled --
is accepted for every subsequent wave. It, not the review, found the class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG
Levi Neuwirth