LeVCS/doc/phase1-storage-spine-scope.md

168 KiB
Raw Blame History

Phase 1 scope — storage engine spine

Companion to doc/instance-throughput-rewrite-plan.md §12 "Phase 1 — storage engine spine". The rewrite plan remains authoritative. Where this document is more specific, it is the lead's Phase 1 realization of the plan; where it appears to contradict the plan, the plan wins and this document is defective.

Status: Wave A frozen 2026-07-26 at 5ee9c6b; Wave B scoped, D0-B frozen 2026-07-27 at 5111d65. Sections 9.1-9.6 were resolved on 2026-07-24 and are recorded there with their conditions; 9.7 and 9.8 were ruled on 2026-07-26 and are recorded in §6.9, and contract review 2026-07-26-A resolved the EvidenceHandoffFailure classification (§6.3). D0-B is frozen at 5111d65, its frozen-surface implementation amendments are recorded by contract review 2026-07-27-A, and B1/B3/B4 dispatch is unblocked; no open decision remains. Contract review 2026-07-24-B (sections 9.1 and 9.2) has been applied to bench/result-schema.json and bench/reference-hardware.toml and recorded in the plan document. crates/levcs-store exists with the frozen API, the ownership split, the durability funnel, the failpoint registry, and the drive seam; the gate is scripts/check-phase1.sh.

What D0 shipped, against section 2:

  • The public API of 2.2 compiling and returning StoreError::NotImplemented, with AppliedRef re-exported from the protocol crate and PrivilegedConstruction gated on the store-privileged cargo feature rather than reachable from an engine.
  • sys.rs as the sole durability funnel, with counters and the fault set of 4-A3 deliverable 4; failpoints.rs in compiler-enforced correspondence with the frozen oracle, including the wave partition pinned by name.
  • format.rs and drive.rs with lead-authored types and signatures and unimplemented!() bodies, so A1 fills them while A2 and A3 work against stable shapes.
  • options.rs refusing every invalid configuration, delegating the replay-retention relation to the frozen protocol function so the store cannot disagree with it.
  • Eight D0 contract tests, of which two are grep-based structural guards that were verified by injecting real violations and observing them fail, not by assuming they work.

Scope review 2026-07-24-C

The first draft was reviewed before D0. The review found one structural gap (A3 had no way to drive anything in Wave A), one internal inconsistency (the CURRENT install could not be expressed by the sys.rs funnel, and recovery's manifest fallback had nothing to read), and a needed amendment to the §9.1 resolution. All are folded in here:

  • 2.1/2.3/4-A1/4-A3: a feature-gated journal-level drive API that A1 implements and A3 consumes, so the crash matrix and the P1-micro benchmark are runnable in Wave A rather than blocked on B1's submit.
  • 4-A3: the Wave A / Wave B failpoint partition is now given by name, per row, with the one boundary row called out; the earlier draft gave a count, and the count was wrong.
  • 3.5/3.8: durable versioned manifests/<generation>.manifest alongside CURRENT, and rename_replace added to the syscall funnel. CURRENT alone is a single mutable name; "select the newest checksum-valid CURRENT" and A2's "corrupt CURRENT with a valid predecessor" both presupposed a history the layout never kept.
  • 3.3: flags and every padding region are now pinned by completeness conditions. Without them, 2^32 flag values produce a "complete" frame the writer never wrote — precisely the byte sequence the section 5 charter directs the reviewer to construct.
  • 9.1: the flag split is per-flag, not blanket. durability_fence_before_response and typed_ref_cas remain true at P2; declaring the fence flag false would gut the meaning of the bundle that exists to certify the fence.
  • 9.3/4-A3: group_failpoint_expectation returns a contiguous adopted prefix, not a per-frame classification, so the matrix can distinguish a correct stop-at-first-hole from an incorrect adopt-past-a-hole.
  • 1: the dependency claim is restated as "no direct dependency and no calls into" levcs-identity; the link edge is transitive through levcs-protocol and the flat claim was false.
  • 8.1/8.2/9.4: fence-rate, free-space, and payload_digest cost figures corrected.

A second pass folded in four residuals — the charter's stale condition count (5), root_uuid's file list (3.1), the two new manifest-referent recovery cases (4-A2), and the overstated preallocation claim (3.2) — and recorded the rulings on 9.19.5 and the WriterPanicAfterFence wave assignment.

0. Standing and preconditions

Phase 0 is complete and amended by contract review 2026-07-24-A. Phase 1 binds to the amended artifacts:

  • levcs-protocol::v2CommittedTransactionV1, TransactionEvidenceV1, SignedCommittedTransactionV1, CommitReceiptV1, TransactionStatusV1, AppliedRefV1, TypedRefCas, RefTarget, SourceKindV1, StagedProjectionInstallV1, receipt_visible_until, status_tombstone_until.
  • levcs-protocol::oracleAppendFailpoint, append_publication_expectation, RecoveryOutcome, resolve_ambiguous_tail, two_root_status_read, public_status_read, coalescing_decision, deadline_expectation, recovered_receipt_visibility, ExternalAckJournal, verify_exact_restore.
  • bench/workloads/small-commit.toml, bench/result-schema.json, bench/reference-hardware.toml.

Phase 1 may not change any of these without a recorded contract review. Section 9.1 identifies one place where it appears a review is already required.

1. Scope boundary

In scope

One new workspace crate, crates/levcs-store, containing the durable transaction spine: frame format, journal append and group fence, segment sealing and rotation, recovery, object/ref/receipt index and checkpoints, namespace/transaction/snapshot engine, bounded projection staging, and the store-level benchmark and crash harness.

Out of scope for Phase 1

  • Any edit to crates/levcs-instance, crates/levcs-client, crates/levcs-cli. The v1 instance keeps working, untouched, on the v1 layout for the whole phase.
  • HTTP, request authentication, replay guard, admission, Pack streaming, graph/identity/ policy validation. Those are Phase 2; the store consumes an already-ValidatedTransaction.
  • compaction.rs and backup.rs (Phase 4), migration.rs (Phase 2 InstanceMigrator). Phase 1 must leave explicit, tested seams for them — see 3.5 and 6.1 — but implements neither.
  • Federation trust. The store takes an injected CommitEvidenceSigner and never decides who may sign.

Dependency direction

levcs-store depends on levcs-core (object IDs, raw framing) and on levcs-protocol (the frozen logical contracts it must construct and embed). It takes no direct dependency on and makes no call into levcs-identity, and does not depend on levcs-merge or levcs-instance at all.

levcs-identity is nonetheless present in the build graph: levcs-protocol depends on it. The invariant Phase 1 enforces is therefore about calls, not about the link edge — no levcs_identity:: path may appear anywhere in levcs-store, enforced by a D0 test that greps the crate source. The plan forbids the store from making identity-role decisions; it does not and could not forbid a transitive edge.

This is a deliberate reading of §5.1's "depends on levcs-core … but contains no HTTP, identity-role, merge-policy, or federation decisions". The store cannot avoid levcs-protocol: §5.2 requires the shard sequencer itself to assign repo_sequence and previous_event_digest, compute the event and state digests, and place a signed CommittedTransactionV1 in the frame. It can and must avoid interpreting evidence beyond source_kind and its digest, and avoid all signature verification.

2. Lead deliverable D0 — the sealed skeleton

D0 blocks all three Wave A packages. It is a single lead commit that compiles, passes cargo test --workspace, and contains no agent-owned implementation.

2.1 File ownership matrix

No two agents may edit the same file. A package that needs a change in another package's file files a lead-arbitrated interface change request; it does not edit.

Path Owner Phase 1 state
crates/levcs-store/Cargo.toml Lead D0
crates/levcs-store/src/lib.rs Lead D0
crates/levcs-store/src/types.rs Lead D0 (ids, StoreError, CommitReceipt, status)
crates/levcs-store/src/options.rs Lead D0 (StoreOptions, validation)
crates/levcs-store/src/sys.rs Lead D0 (durability syscall shim + counters + fault hooks)
crates/levcs-store/src/failpoints.rs Lead D0 (registry, 1:1 with oracle::AppendFailpoint)
crates/levcs-store/src/format.rs Lead types / A1 bodies D0 signatures, A1 implementation
crates/levcs-store/src/journal.rs A1 JournalWriter Wave A
crates/levcs-store/src/segment.rs A1 JournalWriter Wave A
crates/levcs-store/src/drive.rs Lead signatures / A1 bodies D0 signatures, A1 implementation
crates/levcs-store/src/index.rs A2 RecoveryIndex Wave A
crates/levcs-store/src/checkpoint.rs A2 RecoveryIndex Wave A
crates/levcs-store/src/recovery.rs A2 RecoveryIndex Wave A
crates/levcs-store/benches/durable_ingest.rs A3 StoreHarness Wave A
crates/levcs-store/src/bin/store-bench.rs A3 StoreHarness Wave A
crates/levcs-store/src/bin/store-crash-driver.rs A3 StoreHarness Wave A
crates/levcs-store/tests/support/** A3 StoreHarness Wave A
crates/levcs-store/tests/frame_golden.rs A1 Wave A
crates/levcs-store/tests/fixtures/phase1-frames.json A1 Wave A
crates/levcs-store/examples/phase1_frame_golden.rs A1 Wave A
crates/levcs-store/tests/recovery_*.rs A2 Wave A
crates/levcs-store/tests/crash_matrix.rs A3 Wave A
crates/levcs-store/tests/fixtures/phase1-failpoints.json A3 Wave A
crates/levcs-store/src/engine.rs B1 NamespaceTxn Wave B
crates/levcs-store/src/transaction.rs B1 NamespaceTxn Wave B
crates/levcs-store/src/snapshot.rs B1 NamespaceTxn Wave B
crates/levcs-store/src/staging.rs B3 StagingSessions Wave B
crates/levcs-store/src/roots.rs, completion.rs Lead D0-B
scripts/verify-store-recovery.sh A3 Wave A
scripts/check-phase1.sh Lead Wave A freeze gate
Cargo.toml, Cargo.lock Lead D0

format.rs and drive.rs are the two shared files. The lead writes their type definitions, constants, domain strings, and function signatures in D0 with unimplemented!() bodies; A1 fills the bodies; A2 consumes format.rs read-only and A3 consumes drive.rs read-only. This is what keeps the three Wave A packages genuinely parallel.

drive.rs exists because of a gap in the first draft: A3's store-crash-driver and store-bench are separate binary crates that see only the public API, and the public API's submit returns NotImplemented until B1 writes engine.rs in Wave B. As first written, neither of A3's Wave A acceptance criteria was reachable. drive.rs is a #[doc(hidden)], store-internals-feature-gated journal-level API — open a shard journal, append a pre-encoded frame group, fence, seal, and reopen through recovery — with no engine, status root, sequencer, or signer. It is a test and benchmark seam, never compiled into a release binary, and B1 does not build on it.

2.2 Frozen public API

D0 lands §5.1's API verbatim, compiling, with every method returning Err(StoreError::NotImplemented). Freezing signatures before implementation is what lets A3 write the benchmark and crash harness against the real API in Wave A rather than retrofitting it in Wave B.

pub struct NamespaceId(pub [u8; 32]);      // == repo_id
pub struct OperationId(pub [u8; 16]);

pub struct StoreEngine { /* private */ }
pub struct RepoSnapshot { /* immutable logical state */ }
pub struct ValidatedTransaction { /* sealed */ }
pub struct CheckpointLease { /* private */ }

pub struct CommitReceipt {
    pub operation_id: OperationId,
    pub repo_sequence: u64,
    pub current_authority: ObjectId,
    pub refs: Vec<AppliedRef>,
    pub objects_new: u64,
}

pub enum PendingPhase { Receiving, Validating, Queued, Sequenced }

pub enum TransactionStatus {
    Committed(CommitReceipt),
    Pending   { operation_digest: ObjectId, retry_until_micros: i64, phase: PendingPhase },
    Resolving { operation_digest: ObjectId, retry_until_micros: i64, shard_sequence: Option<u64> },
    Expired   { operation_digest: ObjectId, retry_until_micros: i64, tombstone_until_micros: i64 },
    Unknown,
}

impl StoreEngine {
    pub fn open(options: StoreOptions) -> Result<Self, StoreError>;
    pub fn snapshot(&self, repo: NamespaceId) -> Result<RepoSnapshot, StoreError>;
    pub async fn submit(&self, txn: ValidatedTransaction) -> Result<CommitReceipt, StoreError>;
    pub fn transaction_status(&self, repo: NamespaceId, operation: OperationId)
        -> Result<TransactionStatus, StoreError>;
    pub fn checkpoint(&self) -> Result<CheckpointLease, StoreError>;
}

Also frozen in D0:

/// Injected by instance composition. The store never decides who may sign.
pub trait CommitEvidenceSigner: Send + Sync {
    fn key_epoch(&self) -> u64;
    fn public_key(&self) -> [u8; 32];
    /// `SignedCommittedTransactionV1::signing_digest`, not the event digest.
    fn sign_event(&self, signing_digest: &ObjectId) -> Result<[u8; 64], SignerError>;
}

/// Privileged construction. Sealed so instance validation, recovery, and
/// (later) migration can build one and nothing else can.
pub struct ValidatedTransactionBuilder { /* private */ }
impl ValidatedTransaction {
    pub fn builder(_: PrivilegedConstruction) -> ValidatedTransactionBuilder;
}
pub struct PrivilegedConstruction(());

The parameter name is normative, and contract review 2026-07-28-A amended it from event_digest. What the signature must cover is SignedCommittedTransactionV1::signing_digest(transaction, source_key_epoch, durability_result), because that is what the frozen verify recomputes. A signature over the bare event digest omits the key epoch and the durability result and verifies nowhere — including at a mirror on another instance, after the transaction is already durable. The event digest is the chain identity previous_event_digest links; it is not the signature message.

PrivilegedConstruction must not be obtainable from a &StoreEngine. A StoreEngine::privileged() method would make the seal decorative: anything holding an engine — which is everything that can call submit — could mint one. D0 gates it instead on the store-privileged cargo feature, which only levcs-instance (validated ingest), the crate's own recovery.rs, and the Phase 2 migrator enable. A D0 test asserts that without that feature there is no safe path to a ValidatedTransaction.

CommitReceipt.refs is Vec<AppliedRef> in §5.1 while the frozen protocol type is AppliedRefV1. D0 must pick one and state why. Recommendation: re-export levcs_protocol::v2::AppliedRefV1 as AppliedRef. The receipt crosses the wire in CommitReceiptV1; a parallel store-local type would need a conversion that could silently disagree with a frozen contract, which is the failure mode contract review 2026-07-24-A was about.

StoreError taxonomy (D0, extended only by the lead): NotImplemented, Corruption, RecoveryRequired, ShardPoisoned, ReadOnly, FormatMismatch, LegacyLayout { migrate_command: String }, UnrecognizedLayout, AlreadyLocked, Io, NoSpace, LimitExceeded, Conflict, NotReady, Signer.

StoreError is never used for a normal lifecycle state; that is TransactionStatus (§5.1). D0 adds a compile-visible doc-test asserting this split.

2.3 Frozen internal interfaces

sys.rs (lead-owned) is the single funnel for every durability-relevant syscall. Nothing else in the crate may call File::sync_all, sync_data, rename, unlink, or open a directory fd.

pub(crate) fn write_vectored_all(fd: &File, offset: u64, bufs: &mut [IoSlice]) -> io::Result<()>;
pub(crate) fn fdatasync(fd: &File) -> io::Result<()>;
pub(crate) fn fsync_dir(path: &Path) -> io::Result<()>;
/// renameat2(RENAME_NOREPLACE). For names that must never overwrite: new
/// manifests, segments, checkpoints, index runs, migration/restore siblings.
pub(crate) fn rename_noreplace(from: &Path, to: &Path) -> io::Result<()>;
/// Plain atomic replacing rename. Required for exactly one thing: the
/// `CURRENT.tmp` -> `CURRENT` pointer install of 3.5, which by definition
/// replaces. Every other call site must use `rename_noreplace`.
pub(crate) fn rename_replace(from: &Path, to: &Path) -> io::Result<()>;
pub(crate) fn pread_exact(fd: &File, offset: u64, buf: &mut [u8]) -> io::Result<()>;
pub(crate) fn preallocate(fd: &File, len: u64) -> io::Result<()>;

/// Per-shard, monotonic, readable from tests.
pub(crate) struct DurabilityCounters {
    pub fdatasync: AtomicU64, pub fsync_dir: AtomicU64,
    pub write_vectored: AtomicU64, pub short_writes: AtomicU64,
    pub bytes_written: AtomicU64,
}

The counters are how "no per-object fsync" (Phase 1 exit) becomes a mechanical assertion rather than a claim: a run of N transactions carrying 3N objects in G groups must show fdatasync == G and fdatasync < N, never ≈ 3N.

failpoints.rs mirrors oracle::AppendFailpoint one-for-one, behind a failpoints cargo feature that is off in release:

#[cfg(feature = "failpoints")]
pub(crate) fn hit(point: AppendFailpoint) -> FailpointAction; // Continue | Fail | Panic | HardExit
#[cfg(not(feature = "failpoints"))]
#[inline(always)] pub(crate) fn hit(_: AppendFailpoint) -> FailpointAction { FailpointAction::Continue }

A D0 test asserts the store's failpoint enum is exhaustively convertible to and from oracle::APPEND_PUBLICATION_FAILPOINTS — a new failpoint in either place breaks the build.

2.4 Workspace dependency additions

Added to [workspace.dependencies] and to levcs-store in D0. Versions are pinned when D0 lands; Cargo.lock is a reviewed artifact per §10.

Crate Why Alternative rejected
arc-swap §5.1 mandates ArcSwap<CommittedRoot>, explicitly not RwLock<Arc<_>>
crossbeam-channel §7 "bounded crossbeam-style channels" to shard threads std::sync::mpsc is unbounded/single-consumer
im §5.3 requires structurally shared hot publication maps; decision 9.7 applies it to repositories, receipts/tombstones, typed refs, and transient statuses Arc<BTreeMap> clone-on-write is O(entries); a hand-rolled HAMT puts correctness risk in every captured read root
rustix renameat2(RENAME_NOREPLACE), O_TMPFILE, fdatasync, fallocate, pwritev, flock without hand-rolled libc unsafe raw libc (more unsafe surface), nix (heavier)
memmap2 §5.3 memory-mapped immutable index runs read-based runs; revisit if mmap SIGBUS-on-truncate handling proves worse
hdrhistogram §10 HDR-style histograms in the result bundle

Not taken: any third-party Bloom filter (the filter is part of a frozen on-disk format; index.rs implements a documented, seeded, versioned filter in-crate), any serde-based on-disk encoding (the format is hand-written canonical binary, as in Phase 0).

tokio is a dependency only for submit's async signature and oneshot completion; the store starts no runtime and does no I/O on a Tokio thread.

2.5 Decisions made in D0

Shard routing. shard_index = u16::from_le_bytes([repo_id[0], repo_id[1]]) % shard_count. repo_id is a BLAKE3 digest, so this is uniform. shard_count is written into FORMAT at initialization and is immutable for the life of the root; opening with a different configured shard_count is a hard FormatMismatch refusal, never a silent reroute. This is load-bearing: repo_sequence and the per-repository event chain are owned by exactly one shard thread with no cross-shard coordination, and that is only sound while a repository's shard assignment never moves. Resharding is a future offline tool.

Shard poisoning is per-shard. A poisoned shard 0 refuses new mutations for its repositories; shards 1..N keep committing. Availability is therefore partial, not total, and the harness must prove it (5.3).

Frames are byte-identical between journal and sealed segment. Sealing appends a footer and renames; it never re-encodes. One golden frame corpus therefore covers both readers.

Also settled in D0, from scope review 2026-07-24-C: the drive.rs signature set (2.1); PrivilegedConstruction gated on a cargo feature rather than reachable from &StoreEngine (2.2); CommitReceipt.refs element type (2.2); the WriterPanicAfterFence wave assignment (4-A3); and flags/padding pinning plus header_digest coverage (3.2, 3.3), all of which are frozen-byte or frozen-surface decisions that cannot be deferred into Wave A.

3. Physical format specification

This section is the lead's proposal. A1 implements it and owns the golden vectors; the Wave A freeze gate (section 5) is where it becomes binding. §13 keeps the physical format internal — nothing outside levcs-store may bind to these bytes.

3.1 Root layout, FORMAT, LOCK

Exactly §5.2's tree. Additions Phase 1 needs and the plan permits:

<root>/
  FORMAT
  LOCK
  quarantine/<journal_id>-<offset>.tail      # discarded torn tails, forensics only
  staging/<shard>/<session-id>/...
  shards/NN/{active,segments,indexes,checkpoints,manifests,CURRENT}

manifests/<generation>.manifest is the addition. §5.2 names only CURRENT, but CURRENT is a single mutable name, and both §5.2's own "select only a checksum-valid CURRENT" and A2's "corrupt CURRENT with a valid predecessor" case require a predecessor to fall back to. Manifests are therefore durable, immutable, and versioned: each is written under a new <generation> name with rename_noreplace, at least manifest_retain generations are kept, and CURRENT becomes the atomically replaced pointer naming the active generation. Recovery reads CURRENT; if it is corrupt, or its referenced files fail validation, it falls back to the newest valid manifests/<generation>.manifest. This is additive to the plan's layout and changes no plan-visible behavior.

FORMAT is { magic, format_version, storage_version, shard_count, root_uuid, created_at_micros, checksum }, written then directory-synced. root_uuid binds every CURRENT, manifest, journal, segment, index run, and checkpoint in the root; a file whose root_uuid differs is rejected, which catches a file copied in from another instance.

LOCK is held with flock(LOCK_EX|LOCK_NB) for process lifetime. Failure is AlreadyLocked, never a wait.

What the root lock does and does not exclude (contract review 2026-07-29-A, second finding). flock locks an inode. The name <root>/LOCK resolves to an inode once, at open, and nothing binds the name to that inode afterwards — so exclusion holds exactly as long as the name keeps resolving to the object the holder locked.

Two cases follow, and only one of them is a defect the store can close:

The same rule now holds for every other name the store invents beneath a root — FORMAT, the .tmp names a fenced write installs from, and every directory in the tree (contract review 2026-07-29-B). The root itself and its ancestors are excluded on purpose: an operator who configures a root behind a symlink has said where the store goes.

  1. The name already resolves elsewhere when a process arrives. A symlink, a fifo, a directory, a socket, or a device at LOCK — left by an operator, a restored backup, a symlink farm, or a previous tenant of the directory. A follow-through open takes the lock on a foreign inode, or on no inode the store owns, while the root itself stays unlocked. In scope and closed: segment::lock_root opens through sys::open_or_create_regular_nofollow and refuses anything that is not a regular file at exactly that name, before flock.
  2. The name is replaced while a holder holds it. unlink plus create, or a rename over it, gives the next arrival a fresh unlocked inode and a second holder — and this is true of any regular file at any name, so the type check does not address it and no check at this layer can. Out of scope, and stated here as an assumption rather than left implied: scope 3.1 exclusion assumes no noncooperating mutation of the root directory's entries while the root is held. The assumption is sound for the deployment this store is built for — anything able to replace LOCK can equally unlink a journal or a manifest, so advisory locking was never the boundary that would stop it — but it is an assumption, not a property, and a test in segment.rs pins it so it cannot quietly be believed to be stronger than it is.

The strongest available strengthening, if case 2 is ever brought into scope, is to hold the lock on the root directory rather than on a file inside it: rename over a non-empty directory fails ENOTEMPTY, so an adversary must move the whole root aside, after which every subsequent path resolves into a different tree and the failure stops resembling successful exclusion. It is not free — it changes a frozen surface, and it removes the "directory holding only LOCK is empty" special case that startup state 1 currently needs, which is a simplification but a behavioral change to classify_root and its tests. Not scheduled here.

Startup states (§5.2), decided in this order and with no inference:

  1. Path absent, or present and empty → initialize v2. Create the tree, write FORMAT, fsync every new directory and the root's parent.
  2. Valid FORMAT → open through production recovery (3.8).
  3. No FORMAT, but the directory matches the legacy signature (≥1 entry of the form <64-hex>/.levcs/) → LegacyLayout carrying the exact levcs-instance migrate-store --source <root> --destination <dest> string. No writes.
  4. Anything else non-empty → UnrecognizedLayout. No writes, no probing.

3.2 Journal file

shards/NN/active/<first_shard_sequence>.journal.

File header (512 bytes, zero-padded, checksummed): magic "LVCSJRN\0", storage_version u16, header_len u16, shard_index u16, pad, root_uuid [16], journal_id [16] (fresh random per file), first_shard_sequence u64, preallocated_len u64, created_at_micros i64, header_digest [32].

header_digest = BLAKE3("levcs-journal-header/v1\0" || header_bytes[0..512]) with the header_digest field itself zeroed for the computation. Every byte of the 512 that is not a named field is padding and must be zero; a non-zero padding byte rejects the header. Leaving reserved space unconstrained is how a format acquires undocumented variants.

The journal is preallocated to preallocated_len at creation (fallocate, without FALLOC_FL_KEEP_SIZE) and directory-synced before its first append. Appends are then overwrites within an already-sized file, so fdatasync never has to persist a file-size change, and the region past the write cursor reads as zeros.

That is a reduction in metadata work, not an elimination of it: the first write into each preallocated extent converts it from unwritten to written, which is metadata that fdatasync must and does persist. The benefit is that the conversion happens once per extent rather than once per append, and that no i_size update is ever needed. Under copy-on-write (9.2) even this is optimistic — btrfs allocates fresh extents per write regardless of preallocation. Rotation happens when the cursor reaches preallocated_len or the configured segment size, whichever is first.

journal_id is in every frame. It is the defense against a frame from a previous incarnation of a recycled or non-zeroing extent being accepted as a live frame, and it also catches a segment or journal misfiled under the wrong name in CURRENT.

3.3 Transaction frame

Header, 176 bytes, all integers little-endian:

Off Size Field
0 8 magic "LVCSFRM\0"
8 2 storage_version (1)
10 2 header_len (176)
12 4 flagsmust be 0 in storage version 1; any set bit rejects
16 8 total_len — header + payload + padding + trailer
24 16 journal_id
40 8 shard_sequence
48 8 repo_sequence
56 32 namespace (repo_id)
88 16 operation_id
104 32 operation_digest
136 8 payload_len
144 32 payload_digest = `BLAKE3("levcs-frame-payload/v1\0"

Payload — canonical TransactionFramePayloadV1, all of §5.2's frame contents: optional repository-create metadata; either inline new objects (embedded type, ObjectId, raw_len, raw bytes each) or exactly one StagedProjectionInstallV1 descriptor; the complete typed ref CAS set; expected/new current authority; canonical TransactionEvidenceV1; canonical CommittedTransactionV1 plus source key epoch, source signature, and resulting state digest; deterministic receipt fields.

Padding: zero bytes so that header_len + payload_len + pad is a multiple of 8.

Trailer, 48 bytes: magic "LVCSEND\0", total_len repeated (u64), then frame_digest [32] = BLAKE3("levcs-frame/v1\0" || bytes[0 .. total_len-32]).

A frame is complete iff, and only iff, all of the following hold — this is the exact definition recovery uses and the one an adversarial reviewer should attack first:

  1. total_len ≥ 224, total_len % 8 == 0, and offset + total_len is within the journal's preallocated length.
  2. Header magic and storage_version match; header_len == 176.
  3. journal_id equals the containing file's journal_id.
  4. Trailer magic matches and the trailer's repeated total_len equals the header's.
  5. frame_digest recomputes.
  6. payload_digest recomputes.
  7. flags == 0.
  8. Every padding byte between the payload and the trailer is zero.

Conditions 7 and 8 are not decoration. Without 7, any of 2^32 flags values yields a frame satisfying every other condition that the writer never produced — exactly the byte sequence the section 5 charter directs the reviewer to construct, so a format that does not pin it hands the reviewer a finding by construction. Without 8, the padding is a covert region that frame_digest authenticates but nothing constrains. A future storage version that wants flags allocates them explicitly and bumps storage_version.

Evaluation order is normative: cheap structural checks before digests. The numbering above is a list, not a sequence. verify_complete evaluates the length, magic, header-len, journal_id, flags, padding, and trailer conditions — all header or fixed-offset reads — before recomputing frame_digest or payload_digest, so a malformed frame is rejected without hashing it. The set of complete frames is identical under any order, so this is a determinism rule rather than a correctness one, but it has a testing consequence: a negative test that asserts a specific FrameError variant must violate exactly one condition. An input violating two conditions may legitimately report either. A2's independently written reference verifier followed the numbering and A1's follows cost, which is how the ambiguity surfaced; both accept and reject exactly the same frames.

The frame payload is canonical TransactionFramePayloadV1, and format.rs must expose at least previous_event_digest, event_digest, and source_kind from it. Recovery step 9 verifies the per-repository event chain and cannot do so against an opaque Vec<u8>.

The trailer exists so that a torn write cannot leave a self-consistent frame: the last bytes written are the ones that certify the whole.

Considered and rejected: a physical previous_frame_digest chain. shard_sequence contiguity plus the per-repository previous_event_digest in CommittedTransactionV1 already make deletion and reordering detectable, and internal storage is not defended against an attacker with write access to the root.

3.4 Sealing and segments

Sealing converts a full journal into segments/<generation>-<first>-<last>.seg without re-encoding a single frame byte:

  1. Stop appending; validate the whole file forward.
  2. Append a segment footer: { magic "LVCSSEG\0", storage_version, root_uuid, journal_id, generation, first/last shard_sequence, frame_count, offset table (shard_sequence → offset,len), footer_digest }.
  3. fdatasync the file.
  4. link the file into segments/ under its segment name — not rename — and fsync_dir on segments/. The file is now reachable under both names.
  5. Install the manifest generation referencing the segment name and swap CURRENT (3.5).
  6. Only now unlink the active/ name and fsync_dir on active/.

Step 4 must link rather than rename, and this is a correctness requirement, not a preference. A rename makes the file reachable under exactly one name at every instant. A crash after the rename and before the manifest install would then leave a valid segment that no manifest references and no active journal to scan — and 3.8 step 2 trusts only manifest-referenced files, so a fenced, acknowledged prefix would be silently lost. That is a violation of plan §4 transaction invariant 3. Linking keeps the prefix reachable through the active/ name across the entire window; every crash point in 16 leaves either the journal still scannable or a manifest that references the segment, and never neither.

The first draft of this document specified a rename here and additionally told step 5 to "unlink the old journal name", which a rename has already consumed. A1 found both while implementing and reported them rather than working around them.

Recovery consequence: after step 4 and before step 6 both names exist, so recovery may find an active/ journal whose journal_id is already covered by a manifest-referenced segment. That is a completed seal whose final unlink was lost, not a second copy of the data. Recovery unlinks the stale active name and proceeds from the manifest; it must not replay those frames twice.

Footer presence distinguishes sealed from active. segment.rs reads a sealed segment by offset table plus bounded pread, with an LRU open-fd cache under a configured ceiling. It can also read an unsealed journal by forward scan, which is what recovery uses.

3.5 CURRENT manifest

{ magic, storage_version, root_uuid, generation, base_generation, retained_tail_ranges[(generation, first_seq, last_seq, filename)], index_runs[(generation, filename)], checkpoints[(shard_sequence, filename)], committed_shard_sequence, digest }.

Install, extending §5.2's sequence with the durable manifest history of 3.1:

1  write manifests/<generation>.manifest.tmp   (new generation, never an existing name)
2  fdatasync it
3  rename_noreplace -> manifests/<generation>.manifest
4  fsync_dir manifests/
5  write CURRENT.tmp naming <generation>; fdatasync it
6  rename_replace  -> CURRENT           <-- the one replacing rename in the crate
7  fsync_dir the shard directory
8  only now unlink superseded names, then fsync_dir their directories
9  retain >= manifest_retain generations in manifests/ before unlinking older ones

Step 6 is the visibility boundary for the manifest. Steps 14 are what give recovery something to fall back to when CURRENT or its referenced files do not validate; the first draft's single-CURRENT layout had no such predecessor, which made both §5.2's "select only a checksum-valid CURRENT" and A2's corrupt-CURRENT acceptance case unsatisfiable.

Compaction (Phase 4) reuses this install path unchanged — the manifest already carries base_generation so that a BaselineStateV1 base plus retained tails is representable in Phase 1's format without a format change. Phase 1 always writes base_generation = 0 and no baseline; leaving the field is the seam.

3.6 Checkpoints and index runs

checkpoints/<shard_sequence>.checkpoint is derived state through that sequence: namespace catalog (repo_id, genesis authority, lifecycle, storage mode), per-repository repo_sequence + previous_event_digest + typed ref state + current/genesis authority, shard committed sequence, the active journal_id and journal offset the checkpoint resumes replay from, and the retained operation/receipt table with first_receipt_visibility_micros, retry_until_micros, receipt_visible_until.

The resume point is load-bearing and was missing from the first draft. §5.3 bounds "replay bytes/frames since the newest durable checkpoint" and 3.8 step 5 scans "from the checkpoint offset"; without the journal identity and offset recorded here, replay restarts at the journal header on every open, which is exactly the unbounded scan §5.3 forbids. The journal_id accompanies the offset because an offset alone is meaningless once the journal has rotated.

first_receipt_visibility_micros is optional on the wire with an explicit presence byte, never a sentinel value. A sentinel would make "not yet captured" indistinguishable from a legal timestamp and would silently break the step 10 promotion rule. Checksummed, versioned, root_uuid-bound. At least two independently validated generations are retained; if both fail validation, startup enters explicit offline rebuild mode rather than an unbounded scan (§5.3).

indexes/<generation>.idx is an immutable sorted run over (namespace, ObjectId) -> (segment_generation, offset, len, ObjectType, shard_sequence), with a versioned seeded Bloom filter in its header. Runs are mmapped. The active in-memory delta has hard entry and byte ceilings from StoreOptions; crossing the soft watermark backpressures, and the shard synchronously seals and checkpoints at the hard ceiling.

Entry-size budget, which §13 requires be reported: namespace is factored out per run section, leaving 32 (ObjectId) + 15 (packed location) = 47 bytes per object. See 8.2.

Measured (A2, Wave A): 47.00 bytes/entry packed, matching the budget exactly; 48.29 bytes/entry for a whole run including header, filter, section table, and trailer. Every packed field width is a LimitExceeded refusal at build time rather than a truncation. Bloom filter at 10 bits/entry and 7 hashes measures a 0.0080 false-positive rate; lookup fan-out over 8 runs is 1 searched / 7 filtered on a hit in the oldest run and 0 / 8 on a miss. These are the numbers §13 requires the result bundle to report.

Resident cost in the in-memory delta is ~93 bytes/entry, materially above the packed figure. At the default max_active_index_entries = 4_000_000 that is ~370 MB per shard and ~1.5 GB across four shards — against 60 GiB of cgroup limit on both frozen profiles, so it is comfortably inside §3's 80% ceiling and the default stands. The number worth watching is seal frequency rather than bytes: at 75k commits/s over four shards each shard admits ~56k index entries/s, so a 4M-entry delta seals about every 71 seconds. Across a 900-second P2 repetition that is ~13 runs per shard, under the max_index_runs = 64 ceiling. Across a two-hour P5 soak it would be ~100 runs, which exceeds it — and that is precisely the regime where Phase 4 compaction exists to merge runs. The defaults are therefore correct for Phase 1 and P2, and P5 depends on compaction landing rather than on retuning them.

3.7 Durability ordering — normative

The append path, per group, on the owning shard thread only:

 1  form group           (≤ max_group_transactions, ≤ max_group_bytes, ≤ max_idle_delay)
 2  recheck each signed retry deadline                     [oracle::deadline_expectation]
 3  mark every group operation Resolving in the status root (Release ordering)
 4  encode frames into iovecs at the current write cursor
 5  write_vectored_all  -> loop on short writes; verify final cursor == expected
 6  fdatasync(journal)   <-- THE fence. Exactly one per group.
 7  build the immutable shard subtree (index delta, snapshots, receipts, sequences)
 8  ArcSwap CAS the committed root; on contention, re-merge against the newer root
 9  remove the group's Resolving entries (Release, after step 8 is visible)
10  wake waiters with receipts

Normative rules attached to this sequence:

  • Steps 410 are the poison window. Any error, cancellation, or panic anywhere in it poisons the shard: read-only, every affected operation Resolving, no further append, publication, or acknowledgment until close-and-reopen through recovery (§5.2).
  • fdatasync failure is terminal for the shard and is never retried. On Linux an fsync/fdatasync error may be reported exactly once, after which the kernel has already dropped the dirty pages; a second call can return success while the data is permanently gone. Therefore a failed fence proves nothing about durability in either direction — which is precisely why contract review 2026-07-24-A moved FenceFailed to EitherWhole. Recovery must reopen the file with a fresh descriptor and re-read from the device before classifying.
  • A short write is not an error. It leaves a durable prefix. write_vectored_all loops, and the group's outcome is decided per frame by completeness (3.3), not by the return value of any single call.
  • Only the shard owner names, adopts, renames, or unlinks anything under its shard. Maintenance workers write uniquely named unreferenced artifacts and hand over descriptors (§4 resource invariants, §7 maintenance workers).

3.8 Recovery — normative

 1  acquire LOCK; validate FORMAT and root_uuid
 2  read CURRENT -> manifests/<generation>.manifest. If CURRENT is missing/corrupt,
    select the highest valid finalized manifest only when no higher finalized manifest is
    invalid. Never move to a shorter committed tail because a newer authoritative segment
    is missing/corrupt; that is RecoveryRequired, not fallback. Derived checkpoint failure
    is handled by step 3 and does not invalidate the transaction-authority manifest.
 3  from the selected manifest's retained checkpoint rows, load the newest of >=2
    validated checkpoints. Three outcomes, kept distinct:
      no checkpoint directory entries at all -> a fresh root; replay from the start
      at least one validates                 -> load the newest
      entries exist and all fail validation  -> explicit offline rebuild mode
 4  replay sealed segments above the checkpoint, then the active journal, in order
 5  scan the active journal forward from the checkpoint offset:
      at each offset, test completeness (3.3)
        complete   -> adopt, advance, continue
        incomplete -> STOP. This is the tail.
 6  everything at or after the stop offset is discarded, even if a later region
    contains a syntactically complete frame
 7  quarantine the discarded tail bytes to quarantine/<journal_id>-<offset>.tail
 8  seal the validated prefix into a segment and open a fresh journal; never rewrite
    a journal in place
 9  independently verify: shard_sequence contiguous, no duplicate; per repository
    repo_sequence contiguous, no duplicate, previous_event_digest chains
10  for each recovered operation whose first_receipt_visibility_micros was not in a
    durable checkpoint, set it to recovery publication time (extends, never shortens)
11  publish the recovered committed root and receipt table exactly once
12  readiness true only after replay and catalog/genesis validation complete

The three-way split in step 3 matters and the first draft collapsed it to two. A fresh root has zero checkpoint generations and is perfectly healthy; treating that as "all generations failed" would license a replay from sequence zero on a corrupt store, which is the exact failure the rule exists to prevent. Emptiness and total corruption must be distinguishable by the type recovery returns, not by a comment.

Contract review 2026-07-27-A narrows the original step-2 phrase "any referenced file." It was unsafe as written. If manifest 2 authorizes sealed frames 45 and checkpoint 5 is corrupt, falling back to manifest 1 may authorize only frames 03 while the active journal correctly begins at 6. Treating the shorter state as recovered manufactures acknowledged loss. Checkpoints are derived: keep manifest 2's transaction tail, load retained checkpoint 3 from manifest 2, then replay sealed frames 45 and active frame 6. If every retained checkpoint fails, enter explicit offline rebuild.

The same asymmetry is stricter for authority bytes. A missing or corrupt segment cannot be made safe by selecting an older manifest that omits it; recovery must refuse readiness. A corrupt or missing CURRENT may still select the highest valid immutable manifest in the directory, including one installed just before a crash prevented the pointer swap, but it may not skip a higher invalid finalized manifest and guess that the lower committed closure is complete.

Two rules in there carry most of the risk and must be individually asserted, not asserted in aggregate:

  • Step 6. Page writeback may persist a later page before an earlier one. After a crash it is entirely possible to find frame k torn and frame k+1 complete and checksum-valid. Adopting k+1 would publish a hole and break shard_sequence contiguity, multi-ref atomicity, and the "wholly present or wholly absent" invariant. Recovery stops at the first incomplete frame, unconditionally.

  • Step 5 must handle both a zero/stale tail and EIO. The frozen reference filesystem is btrfs with data checksums. Copy-on-write means an ordinary crash normally leaves the tail region reading as zeros or stale preallocated content, not as a torn block — CoW does not expose a half-updated extent. EIO is the other case: the device acknowledged a write it did not retain, which is live here because the frozen profile has write_cache = enabled and power_loss_protection = false. Recovery must therefore treat both a non-magic tail and an EIO in the tail region as "the tail ends here", never as a fatal store error; getting the latter wrong turns an ordinary crash into an unopenable store. Under nodatacow (9.2) the checksums are gone and a torn block reads back as garbage, which condition 5 of 3.3 catches. All three paths must be tested.

    The residual risk is real and must be named: treating EIO as end-of-tail will silently drop a frame that was fenced and that the device then lost. Recovery cannot distinguish that from an unfenced tail by inspection — nothing on the device says so. The detector is the external ACK journal reconciliation (A3 deliverable 5), which is run after every crash precisely because it is the only thing that can catch a device that lied about a flush. acknowledged_loss becoming non-zero there is a hardware finding, not a store bug, and it invalidates the run rather than being tolerated.

4. Wave A work packages

All three start together after D0. Each ends with its own tests green and its deliverables reviewable in isolation.

A1 — JournalWriter

Owns format.rs bodies, journal.rs, segment.rs.

Deliverables:

  1. Frame, journal-header, segment-footer, and CURRENT codecs implementing 3.23.5, with checked readers in the Phase 0 style: bounded with_capacity, trailing-byte rejection, decode-side revalidation, no serde.
  2. Group append with write_vectored, short-write looping, cursor verification, and exactly one fdatasync per group. Group formation bounded by transactions, bytes, and idle delay.
  3. shard_sequence assignment and the crash-safe rotation/seal sequence of 3.4.
  4. Every step in 3.7 instrumented with a failpoints::hit call, one per AppendFailpoint variant reachable at this layer (named in A3 deliverable 2).
  5. drive.rs bodies — the store-internals journal-level API A3 drives in Wave A: open/create a shard journal, append a caller-supplied pre-encoded frame group, fence, seal, install a manifest, and reopen through recovery.rs. It exposes DurabilityCounters and the failpoint registry. It has no engine, status root, sequencer, or signer, and B1 does not build on it. This is A1's contract with A3 and is part of what the Wave A freeze gate freezes.
  6. Golden frame vectorstests/fixtures/phase1-frames.json, regenerated only by cargo run -p levcs-store --example phase1_frame_golden, covering: minimum frame, repository-create frame, multi-object frame, multi-ref Set+Delete frame, authority transition frame, staged-projection-install frame, one frame per SourceKindV1, and the maximum-shard_sequence/repo_sequence edge frames.
  7. Negative decode coverage: every one of the eight completeness conditions in 3.3 violated independently — including a frame whose only defect is a non-zero flags and one whose only defect is a non-zero padding byte — plus wrong journal_id, wrong root_uuid, non-zero journal-header padding, total_len overflowing the file, total_len not 8-aligned, truncation at every byte offset of a small frame, and single-bit mutation asserting the frame never validates.

Acceptance: goldens byte-stable across two regenerations; fdatasync counter equals group count for a 10k-transaction run; a proptest showing encode/decode round-trips and that truncation at any offset is rejected without panic.

A2 — RecoveryIndex

Owns index.rs, checkpoint.rs, recovery.rs.

Deliverables:

  1. (namespace, ObjectId) index: bounded in-memory delta, immutable mmapped sorted runs with the versioned Bloom filter, run count/fan-out ceilings, and a namespace catalog. Namespace isolation is a property of the key, and there is a test that a byte-identical object inserted under namespace A is not found under namespace B (Phase 1 exit).
  2. Checkpoint construction, atomic installation, ≥2 validated generations, and explicit offline-rebuild mode when all validated generations fail.
  3. Recovery implementing 3.8 exactly, including the two rules called out there.
  4. Independent shard-sequence and repo-sequence/event-chain verification with distinct error variants — a shard gap and a repository gap must not be reported as the same fault.
  5. Receipt-table reconstruction, including first_receipt_visibility_micros promotion (step 10) asserted against oracle::recovered_receipt_visibility.

Acceptance: for each of the following, a dedicated test, not a shared one — torn final frame; complete frame after a torn frame (must be discarded); zeroed tail; stale preallocated content in the tail; EIO on tail read; corrupt checkpoint (one generation, then both); corrupt segment referenced by the active manifest; corrupt CURRENT with a highest valid immutable manifest and no higher invalid finalized manifest; valid CURRENT naming a missing manifest generation; valid CURRENT naming a manifest that fails its own checksum; wrong journal_id; wrong root_uuid; shard-sequence duplicate; repo-sequence gap; previous_event_digest mismatch.

The three manifest cases are distinct branches of recovery step 2 and may not be collapsed. A corrupt pointer may select the highest valid immutable manifest only when no higher invalid finalized manifest makes the committed closure ambiguous. A valid pointer naming a missing or corrupt manifest, or an authoritative manifest naming a missing or corrupt segment, must refuse readiness rather than silently shorten the acknowledged transaction tail. Checkpoint corruption is different because checkpoints are derived: recovery keeps the selected manifest, falls back among that manifest's retained checkpoint rows, and replays its remaining authoritative segments. These are the safe-fallback constraints of contract review 2026-07-27-A.

A3 — StoreHarness

Owns the benchmark and crash harness, against the D0 public API.

Deliverables:

  1. Deterministic crash driverstore-crash-driver child process; parent sets a failpoint by name, child runs a scripted workload through A1's drive.rs in Wave A and through StoreEngine::submit from Wave B onward, the failpoint fires (Fail | Panic | HardExit via _exit(3) so no destructor or buffer flush runs), parent reopens through production recovery and classifies the outcome. The two drive paths share one classifier; only the submission call differs.

  2. The crash matrixtests/crash_matrix.rs plus tests/fixtures/phase1-failpoints.json, one row per AppendFailpoint. Each row records the physical state class the failpoint produces and the recovery outcome observed, and the test asserts the observed outcome equals both the class-derived expectation and oracle::append_publication_expectation(point).recovery_outcome. Two independent derivations that must agree; no catch-all match arm anywhere in the file.

    Failpoint Physical state class Required outcome
    BeforeAppend NoBytes AbsentRetriable
    AfterMarkedResolving NoBytes AbsentRetriable
    EvidenceHandoffFailure NoBytes AbsentRetriable
    DuringFrameWriteTorn PartialFrame AbsentRetriable
    AfterFrameWrite WholeFrameUnfenced EitherWhole
    BeforeFence WholeFrameUnfenced EitherWhole
    FenceFailed WholeFrameUnfenced EitherWhole
    FenceAmbiguous WholeFrameUnfenced EitherWhole
    WriterPanicBeforeFence WholeFrameUnfenced EitherWhole
    AfterSuccessfulFence WholeFrameFenced Committed
    DuringCommittedRootBuild WholeFrameFenced Committed
    AllocationFailureBeforePublication WholeFrameFenced Committed
    BeforeRootCas WholeFrameFenced Committed
    DuringRootCasRetry WholeFrameFenced Committed
    WriterPanicAfterFence WholeFrameFenced Committed
    AfterRootCasBeforeWaiterWake WholeFrameFenced + published Committed, ack allowed
    BeforeResponse WholeFrameFenced + published Committed, ack allowed

    Each fixture row carries an explicit wave field. The partition is given by name, never by count — the first draft said "the six publication-side rows" and the count was wrong, which is the same class of error as an unasserted field.

    Wave A, drivable through drive.rs (nine): BeforeAppend, DuringFrameWriteTorn, AfterFrameWrite, BeforeFence, FenceFailed, FenceAmbiguous, WriterPanicBeforeFence, AfterSuccessfulFence, WriterPanicAfterFence.

    Wave B, requires B1 (eight): AfterMarkedResolving (needs the status root), EvidenceHandoffFailure (needs the sequencer's signer handoff), DuringCommittedRootBuild, AllocationFailureBeforePublication, BeforeRootCas, DuringRootCasRetry, AfterRootCasBeforeWaiterWake, BeforeResponse.

    WriterPanicAfterFence is ruled into Wave A, read as "the writer thread panics once fdatasync has returned". The claim it certifies — a fenced frame survives a panic-unwind death of the writer — is a journal-layer claim, and drive.rs contains the panic site. Its whole differential against AfterSuccessfulFence is unwind versus clean error return, and unwind is where a Drop impl that truncates, rewinds the cursor, or flushes a buffer would corrupt the fenced prefix. That is a format-freeze concern, and the driver's HardExit path deliberately cannot catch it because _exit(3) runs no destructors. The halves this row cannot assert in Wave A — poison, Resolving, no-ack — are equally unassertable for all nine Wave A rows, so they do not distinguish it.

    Two conditions attach. (a) In Wave A the fixture records that only the physical-state-class and recovery_outcome halves are asserted; Wave B re-asserts every row's complete FailpointExpectationshard_poisoned, immediate_status, acknowledgment_allowed, later_append_allowed_before_recovery — through submit. (b) The Wave B matrix must exercise the Panic action on the publication-side rows DuringCommittedRootBuild, BeforeRootCas, and DuringRootCasRetry, so the panic-in-publication coverage this assignment vacates is not lost. The failpoint enum names a location; the driver chooses the action, and the two axes are independent.

    Wave B rows are recorded as pending-wave-b with their reason, exactly as Phase 0's not-exercised adversarial rows are, and the fixture test asserts the pending set is empty at Phase 1 exit.

  3. Group-aware failpoint model. oracle::append_publication_expectation is a single-transaction contract; a real group has up to 512 frames and a failpoint has a victim index. A3 adds group_failpoint_expectation(point, victim_index, group_len) in levcs-store and proves it degenerates to the oracle at group_len == 1. This layers on top of the frozen oracle; it does not change it (see 9.3).

    It must return a contiguous adopted prefix, not a per-frame classification. Each individual unfenced frame is independently either-whole via page writeback, so a per-frame answer would permit "frame 3 absent, frame 4 committed" — which recovery step 6 forbids and which is exactly the bug the matrix exists to catch. The contract is: there exists some p ≤ victim_index such that frames 0..p are committed and frames p..group_len are absent; frames at or after the victim are never committed. Stated that way the matrix can fail an implementation that adopts past a hole, which a per-frame formulation cannot.

  4. Fault injection through sys.rs — short write, ENOSPC, EIO on fdatasync, EIO on pread, unexpected cursor. dm-flakey and power-cut belong to the reviewed root-only scripts of §10 and are Phase 4/5; Phase 1 gets the userspace layer so the matrix runs unprivileged in CI.

  5. External ACK reconciliation — reuse oracle::ExternalAckJournal. Every acknowledged operation is durably journaled before it is counted; after each crash the harness proves zero acknowledged loss and zero torn transactions against the recovered store.

  6. Durable-ingest benchmarkbenches/durable_ingest.rs (criterion, over drive.rs, for the P1-micro diagnostic gate, runnable in Wave A) and src/bin/store-bench.rs (long-run, for P2: 5 min warmup + 15 min measured × 3, open-loop, coordinated-omission corrected, HDR histograms, emitting a bench/result-schema.json-conformant bundle with gate = "storage_primitive"). store-bench goes through the public submit and so produces a real bundle only from Wave B; in Wave A it must still compile, and its bundle emitter and free-space precheck (8.2) must be exercised against a short drive.rs run so the schema question in 9.1 is settled long before the gate depends on it. Real Ed25519 evidence signing is on — §5.2 requires P2 to measure signing cost.

  7. scripts/verify-store-recovery.sh — run the matrix, run N randomized SIGKILL cycles, reconcile against the ACK journal, print a machine-readable summary.

Acceptance splits by wave, because A3 cannot reach submit until B1 exists:

Wave A — the nine drive.rs-reachable matrix rows run green on their physical-class and recovery_outcome halves, with the eight Wave B rows explicitly pending; verify-store-recovery.sh completes 100 SIGKILL cycles through drive.rs with zero acknowledged loss; durable_ingest produces a P1-micro number; store-bench compiles and emits a schema-valid skeleton bundle.

Phase 1 exit — zero pending rows, every row asserting its complete FailpointExpectation through submit, and the Panic action exercised on the three publication-side rows named above; store-bench produces a full P2 bundle over submit.

5. Wave A freeze gate and adversarial review charter

Wave A is frozen at commit 5ee9c6b78b6e5f77f988e99e477656cfdc7db352 (2026-07-26). That commit is the frozen state: the levcs-store public API, the frame format and golden corpus, the durability ordering, the failpoint registry, and the crash/fault fixtures. Changing any of them now requires a contract review recorded in doc/instance-throughput-rewrite-plan.md, not an edit.

Verified at that commit: check-phase1.sh GATE_EXIT=0 across all four feature configurations, 124 test binaries, zero failures; verify-store-recovery.sh --cycles 100 with recovery_failures=0, acknowledged_loss=0, torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio 40/40 at four test threads; golden corpus byte-stable.

Two carry-forwards are explicitly not covered by the freeze and are B-wave work: the crash-matrix extension and GroupBuilder's production wiring, both below. doc/swarm-fabric-roadmap-exploration.md (commit e6a058d) is deliberately outside this freeze and remains non-binding.

Per §12 as amended: Wave B starts only after Wave A's interfaces, golden frame vectors, crash/fault fixtures, and durability ordering have passed review and are frozen. Compilation alone is not the dependency gate.

The gate is scripts/check-phase1.sh (lead-owned), modelled on check-phase0.sh: cargo fmt --all -- --check, cargo test --workspace, the whitespace check, three levcs-store feature configurations (default; failpoints,store-internals,store-privileged; and store-privileged alone, so a leak into the default build is not masked by the combined run), a check that no test or privileged feature is enabled by default, a golden-drift check that phase1-frames.json matches a fresh regeneration, and a check that the crash matrix carries no pending-wave-b rows once engine.rs is implemented. It supersedes check-phase0.sh by running it, so the Phase 0 freeze stays enforced. The last two checks are inert until A1 and B1 land respectively, and say so rather than passing silently.

A green gate is necessary and not sufficient. The review is adversarial and precedes the freeze. The charter, derived from the plan's Phase 1 paragraph:

  1. Refute, do not confirm. For each durability claim, construct the physical state that would falsify it. A review that reproduces the author's reasoning has not reviewed it.
  2. Attack the completeness definition (3.3) first. Find a byte sequence that satisfies every completeness condition in 3.3 but is not a frame the writer produced, or a frame the writer produced that fails one condition. The charter says "every condition in 3.3", never a number: a stale count here would send the reviewer past exactly the conditions most recently added to close an attack, which is how this instruction fails silently.
  3. Attack step 6 of recovery. Produce a crash image with a torn frame followed by a complete one and prove the store discards the later frame.
  4. Attack the fence. FenceFailed and FenceAmbiguous must never resolve deterministically before recovery re-reads from the device with a fresh descriptor.
  5. Attack the seal/rotate/CURRENT sequence at every ordering point: what is on disk if the process dies between each pair of adjacent steps in 3.4 and 3.5?
  6. Every frozen outcome individually asserted. No catch-all match arm in any Phase 1 expectation test. The reviewer greps for one and fails the gate if it exists — this is the specific defect that let contract review 2026-07-24-A's unsound classification ship.
  7. Counters, not claims. "One fence per group" and "no per-object fsync" are checked against DurabilityCounters, not read out of the code.
  8. Assert against the path that runs, not the helper. For every safety property, find the entry point production actually calls and check that it enforces the property. A helper that is correct, thoroughly tested, and never called is not a safeguard — it is a decoy that makes a review feel finished.

Item 8 is not hypothetical. It is what the Wave A review found (review record below), and it is the same defect class as contract review 2026-07-24-A one level up: there, an assertion covered every neighboring field except the wrong one; here, a test suite covered every helper except the caller. Both produce a green gate over an unsound system, and neither is visible to a reviewer who only reads the tests that exist.

The mechanical form of item 8: for each package, list the entry points its consumers call, and confirm each safety property is asserted through one of those, not only through an internal function. Where two packages meet, that list is the seam, and the seam is what nobody owns by default.

  1. Ask every package what of its work is correct but uncalled. Put the question to each author directly, as a required part of their report. The author of a helper is the only party who reliably knows whether anything calls it; a reviewer reading the tests sees coverage either way. In the Wave A review this single question turned two located symptoms into the whole class — it found that an entire module was uncalled by the path under test, which no amount of reading the passing tests would have surfaced.

Reviewer output is a written finding list. Freeze happens after findings are resolved, not after they are filed.

Wave A review record 2026-07-24-D

The adversarial review ran against a green gate — 1,478 passing test executions, all three feature configurations, 100 SIGKILL cycles reporting zero acknowledged loss — and found five defects, three of them blockers. That gap between "green" and "sound" is the record's main point.

P1 — recovery trusted segment footers without validating anything. drive::reopen_through_recovery pushed every sequence from a manifest-referenced segment's footer offset table straight into the adopted set. It never validated frame bytes, never checked journal_id or root_uuid, never decoded payloads, and never ran either sequence verifier. SegmentReader validates the footer, not the frames it indexes. Reproduced: a corrupted frame magic inside a referenced segment recovered "successfully" with acknowledged_loss=0, and a journal moved from shard 1 into shard 0 was adopted whole.

P1 — interrupted seals were double-adopted. The same function adopted the manifest's segment sequences and then unconditionally scanned the active journal, never calling recovery::classify_active_journal or recovery::complete_interrupted_seal. A crash between manifest install and active-name unlink produced adopted_sequences=0,1,2,3,0,1,2,3 reported as contiguous and untorn.

P1 — store-bench emitted a schema-invalid bundle and its test could not tell, because the test matched JSON substrings rather than validating against the schema. The gate built and ran the binary but never schema-validated either.

P2 — a latency-gate failure was still unrepresentable. Contract review 2026-07-24-B made the one-minute-window rule conditional on outcome but left the per-gate latency ceilings unconditional — the same defect one block over, missed the same way, for the third time in this document's history. Now conditional.

P2 — checkpoint writes bypassed the durability funnel, so checkpoint bytes and short writes were invisible to DurabilityCounters and no fault campaign could reach checkpoint installation. The D0 guard missed it because it scanned for sync, rename, and unlink but not for the writes being made durable. Extending the guard immediately surfaced a second bypass nobody had reported, a set_len truncation in journal.rs.

The cause, which matters more than the five items. A2 built classify_active_journal, complete_interrupted_seal, and both sequence verifiers, and tested them thoroughly. A1's drive.rs reimplemented a simplified recovery and called none of them. Each package passed its own tests; the seam between them was untested, and the seam is what the crash matrix, the SIGKILL campaign, and the ACK reconciliation all actually drove.

No individual agent was negligent. The ownership matrix that keeps agents from colliding is the same thing that leaves the join between them unowned — and the freeze gate cannot see the difference between a property enforced and a property enforced somewhere. Charter item 8 above is the standing correction.

Wider than the review found. Asked directly whether anything else was correct-but- uncalled, A2 disclosed that drive.rs contains no reference to recovery:: at all: every public entry point in recovery.rs is uncalled by the path the crash matrix drives, not merely the two the review named. And checkpoint::install and checkpoint::prune have no production caller anywhere in the crate, so no crash image the matrix produces has a checkpoint directory — meaning recovery step 3's checkpoint selection, its one-generation- corrupt fallback, and its all-corrupt offline-rebuild decision have never run against a real crash image, only against directories a test assembled by hand.

Some of that is genuine sequencing: engine.rs is NotImplemented until B1, so there is no production caller yet for anything. But drive.rs exists precisely so Wave A can be exercised before B1, and a drive seam that skips half the production path defeats its own purpose. The drive API therefore gains a checkpoint install, and the routing must consult a checkpoint when one is present.

The disclosure is worth noting as a method result, not just a finding. The review located two symptoms; asking each package "what of yours is correct but uncalled?" located the class. That question belongs in the charter for every wave, because the agent that wrote a helper is the only party who reliably knows whether anyone calls it.

Closure (2026-07-26). All five findings are closed and the wider class with them.

drive::reopen_through_recovery no longer contains a recovery decision of its own. It delegates in order to resolve_manifest, load_checkpoint/checkpoint_for_recovery, per-segment validate_journal_binding and SegmentReader::read_frame, classify_active_journal, then either complete_interrupted_seal or recover_journal_tail and never both, then both sequence verifiers and promote_receipt_visibility. Every recovery.rs entry point is now called except two, declared in a table with reasons rather than left silent. ShardDrive::checkpoint() gives checkpoint::install/prune their first production caller, so checkpointed crash images exist and recovery step 3 runs against images the drive produced rather than directories a test assembled. store-bench now validates its emitted artifact against the frozen schema with a five-mutation negative control, and the ACK reconciler distinguishes duplicates and regressions from forward gaps instead of detecting only the last.

Two structural corrections outlast the findings:

  • DriveRecovery now carries the recovery report verbatim. The seam previously kept recovery's effects and discarded its decision, so a test could ask what was adopted but not why — and a double adoption and a correct replay produce different dispositions and the same adoption set. That is precisely how the missing classify_active_journal call stayed invisible while its own unit tests passed. Tests now assert the disposition directly, including that an ordinary replay is classified Replay rather than merely producing a plausible adopted set, and that a shard with no active journal has no disposition — a third state, not a Replay. The suite is mutation-verified: forcing the classifier to Replay fails one test, forcing it to AlreadySealed fails eight. Before this change, forcing either failed none, because nothing asked.
  • The durability funnel guard now covers writes and truncations, not just sync, rename, and unlink. It found one bypass the review had not reported.

The crash matrix could not have caught finding 1, and still cannot. Re-running it after the fix produced identical results — 24/24 rows, 100/100 SIGKILL cycles, zero acknowledged loss, nothing that previously passed now failing. That is not a vindication of the matrix and was not reported as one: it has no failpoint that corrupts a frame inside a sealed segment and none that moves a journal between shards of one root, so the workload never produced the state finding 1 describes. Both properties are now covered by tests driving the production entry point, so the property is enforced; what is missing is the randomized campaign's ability to find that class unaided. Recorded as a B-wave matrix extension, owned by whoever next touches the crash driver. A matrix that returns green on a defect it structurally cannot express is the same trap as a helper nothing calls.

Carry-forward, not closed. GroupBuilder has no production caller: drive.rs takes pre-formed groups, so deliverable 4-A1.2 — group formation bounded by transactions, bytes, and idle delay — is asserted only in unit tests over a type nothing calls. This is legitimately B1's engine.rs by sequencing, but it is the same class as the findings above, and a deliverable exercised only in isolation is not met in the sense the freeze gate is meant to assert. It is recorded as a Wave A carry-forward with B1 owning closure, not as satisfied. index::{catalog_mut, delta, delta_mut, run_count, set_lifecycle} and format::{repo_id, resulting_state_digest} are Wave-B-only for the same reason; sequencing, recorded, no action.

A third finding surfaced during closure and is recorded because its failure mode is subtle: tests/recovery_eio.rs was intrinsically flaky from intra-suite parallelism against the process-global fault registry. Measured rather than estimated: 8 failures in 40 runs of that binary alone with the guard removed, 0 in 50 with it. A fault-injection suite that fails intermittently is the worst case of all — it gets rerun until green, which is exactly when it stops being able to tell you anything, because a real regression and the flake are indistinguishable from outside.

The mechanism is worth stating because the obvious fix is the wrong one. The trigger was not two tests arming at once; it was a test performing a clean, unarmed journal scan before arming anything. That scan is itself a funnel read, so it consumed the fault another test had armed. A lock held only across arm would not have helped — it must cover the whole test body.

arm therefore takes a token, and the compiler enforces it. drive::faults::serial() returns a FaultSerial; faults::arm/disarm and points::arm/disarm all require a reference to it. One token covers both registries: they are separate globals, but both steer the same drive operation, so a failpoint armed during another test's fault campaign interferes exactly as badly as sharing one registry would.

A2 asked only for one shared definition of the lock, so it would have a home instead of existing in four copies. That was correctly diagnosed but insufficient: a lock a test may take is a lock a new test will not take. This exact file carried a header stating it was deliberately the only test in it, and a second test had been added under that comment anyway. A comment cannot fail a build; a parameter can. Acquiring the token also clears any residual armed fault, since a test that panics cannot run its own cleanup and would otherwise deliver its fault to whichever test ran next.

This is the same correction as charter item 8 in a different register. There, a property was enforced somewhere and not on the path that runs. Here, an invariant was documented rather than required. In both cases the gate reported green because nothing made the omission expressible as a failure.

5.1 Resource exhaustion is indistinguishable from a concurrency flake by symptom

Recorded from a Wave B incident, because the wrong diagnosis was already written down before the evidence arrived and only measurement caught it.

Verifying a commit in a second worktree put an 11 GiB target directory on this machine's /tmp, which is tmpfs. The store's tests build roots under TMPDIR, and the engine preallocates per shard, so the filesystem filled. Seventeen engine::tests failures appeared at once. Every one passed in isolation and failed under the full suite — the exact signature §5's recovery_eio record describes, and the reason is structural rather than coincidental: whichever tests run last are the ones that find the filesystem full, and which tests run last depends on scheduling. Re-running the main tree reproduced it 5 of 5, which looked like confirmation of an intrinsic flake in newly added startup tests. It was not. The panic carried Io(Os { code: 122, kind: QuotaExceeded }), and after the worktree was removed the same suite passed 5 of 5 unchanged.

The rule this yields. Before classifying clustered failures as a concurrency flake, preserve and read the errno, and capture free blocks, free inodes, and any quota state for the filesystem the test roots live on. ENOSPC, EDQUOT, and EMFILE all present as unrelated-looking failures that vanish in isolation, and all three are cheap to rule out and expensive to misdiagnose: the flake conclusion sends someone hunting a race that does not exist, and — worse — it invites the rerun-until-green habit §5 exists to forbid, which would have "resolved" this incident while leaving the disk full.

The corollary for harnesses: an I/O error must reach a report with its errno intact. A path that folds one into a boolean, a count, or a generic message destroys the only evidence that distinguishes these two diagnoses. That is the same requirement as the emitter's incomplete-accounting refusal in §6.6 — a failure the harness cannot account for must be reported as itself, not compressed into a symptom.

6. Wave B work packages

6.0 Preconditions

Wave A is frozen at 5ee9c6b78b6e5f77f988e99e477656cfdc7db352. Every file it owns is a frozen surface: format.rs, journal.rs, segment.rs, index.rs, checkpoint.rs, recovery.rs, drive.rs, sys.rs, failpoints.rs, types.rs, options.rs, and the golden corpus. No Wave B package may edit any of them. A change any package believes it needs is an interface request to the lead, arbitrated and — if granted — recorded as a contract review in doc/instance-throughput-rewrite-plan.md. D0-B already exercises this: nine of its eleven items amend a frozen or signature-frozen file, and each is a recorded amendment rather than an edit.

Contract review 2026-07-28-A is the second exercise of that rule, this time driven by B2's findings against the B1/B3 slice rather than by the lead's own integration. Five interface requests were granted and landed on the frozen surfaces ahead of the packages' fix passes: CommitEvidenceSigner::sign_event's parameter renamed to signing_digest (§2.2); a protocol-owned TransactionEvidenceV1::actor(); format::object_type_code made pub(crate) with recovery.rs's twin removed; typed RefRecord::target()/from_target() on the checkpoint record; and the projection ceilings above. Four of the five exist to delete a restated frozen table — B1 filed each of them instead of quietly restating it a third time, which is exactly what §6.0 asks for and what Wave A did not get.

The frozen surface is the library. Wave A's harness — src/bin/store-crash-driver.rs, src/bin/store-bench.rs, tests/crash_matrix.rs, and scripts/verify-store-recovery.sh — is not frozen; ownership transfers to B4, whose whole purpose is to extend it (§6.6). Test files belonging to a frozen library module stay with that module and are amended the same way it is. Wave A shipped one blocker because a package quietly reimplemented a neighbour's logic instead of asking; the cost of asking is a message, and the cost of not asking was three findings.

What Wave A already delivers, so no package rebuilds it:

Need Provided by Note
Transaction frame payload codec format::TransactionFramePayloadV1 Carries create, objects, ref CAS, both authorities, evidence, the signed CommittedTransactionV1, and receipt fields.
Group formation journal::GroupBuilder Bounded by transactions, bytes, and idle delay. Has no production caller — carry-forward, B1 deliverable 4.
Append + fence journal::append_group_and_fence Returns assigned shard_sequencees. One fence per group.
Object index index::{IndexDelta, IndexRunBuilder, IndexRun} Namespace-scoped keys, Bloom-filtered sealed runs.
Namespace catalog index::NamespaceCatalog bind, advance, set_lifecycle.
Durable ref/receipt tables checkpoint::{RefRecord, ReceiptRecord, Checkpoint} Install and prune have production callers as of the freeze.
Recovery recovery::* Wave A supplies parsing, validation, and logical replay. The production entry point returning a RecoveredShard is D0-B item 5; the physically complete step-8 repair and manifest-authority corrections are D0-B item 11. ShardDrive::reopen_through_recovery is a store-internals test seam, not the engine path.
Retention arithmetic oracle::{retained_terminal_status, recovered_receipt_visibility} Frozen; compute from these, never restate.

6.1 Ownership matrix

File Owner State at Wave B start
roots.rs (new) Lead (D0-B) Created and frozen before any package starts.
completion.rs (new) Lead (D0-B) Created and frozen before any package starts.
engine.rs, transaction.rs, snapshot.rs B1 NamespaceTxn D0 signatures frozen; bodies are B1's.
staging.rs B3 StagingSessions Empty.
src/bin/store-crash-driver.rs, src/bin/store-bench.rs, tests/crash_matrix.rs, scripts/verify-store-recovery.sh B4 StoreHarnessB Wave A versions; B4 extends.
Everything else Frozen (Wave A) Read-only to all packages.
bench/result-schema.json, bench/reference-hardware.toml Lead Amendment requires a contract review.

B2 StorageReviewer owns no file. Its output is findings, not edits — the property that made the Wave A review able to contradict the packages it reviewed.

6.2 D0-B — the sealed publication interfaces

The lead defines and freezes D0-B before B1, B3, and B4 begin. This is the same D0 pattern Wave A used, for the same reason: CommittedRoot and OperationStatusRoot are named in plan §5.1 and §5.3 as the visibility boundary and the linearizable-read substrate, they are consumed later by Phase 2 SnapshotReads and Phase 4 MirrorFeed, and B2 has to review them. A structure invented by the package that also implements against it has no independent contract to review, which is precisely the arrangement charter item 8 warns about.

D0-B is not two new files. The first draft of this section said roots.rs and completion.rs and was wrong: that set does not produce a compiling frozen interface, because the modules are undeclared, the dependencies are absent, two of the decisions below require configuration and error variants that do not exist, and the B1/B3 seam names a builder method that was never frozen. The complete amendment set is below. Every row is lead work, and every row touching a frozen Wave A file is a contract amendment recorded in doc/instance-throughput-rewrite-plan.md, not an edit.

# Change File Frozen?
1 CommittedRoot, RepoState, ShardSubtree, OperationStatusRoot, StatusEntry, merge roots.rs (new) no
2 The completion primitive behind async fn submit completion.rs (new) no
3 pub mod roots; and pub mod completion; plus re-exports lib.rs yes
4 im dependency (decision 9.7). No dependency for item 2 — see below Cargo.toml, workspace Cargo.toml, Cargo.lock yes
5 RecoverySession, RecoveredShard, and a non-feature-gated production recovery entry point recovery.rs yes
6 ShardDrive::reopen_through_recovery re-pointed onto item 5 drive.rs yes
7 Staging and status-root limits (§6.5, decision 9.8) options.rs yes
8 Cloneable exact completion outcomes: StoreError: Clone, Io(Arc<std::io::Error>) with lossless From, and Overloaded { limit, retry_after_micros } types.rs (plus mechanical construction call sites) yes
9 The adoption seam: ProjectionAdoption handle (resolve, pin, three-way outcome, reference proof), ValidatedTransactionBuilder::adopt_projection(StagedProjectionInstallV1, ProjectionAdoption), and recovery resolution of committed staged artifacts into the same index/generation ownership model transaction.rs, staging.rs, roots.rs, recovery.rs, index.rs yes
10 Retain exact CommitReceipt.refs in checkpoint receipt rows, populated from the canonical frame by production recovery and the drive seam checkpoint.rs, recovery.rs, drive.rs yes
11 Complete normative recovery step 8 and manifest authority: deterministically preserve and seal the validated active prefix, install a fresh active journal before readiness, validate authoritative manifest tuples, select only its retained checkpoints, allocate generations above rejected immutable artifacts, and publish drive checkpoints through the manifest journal.rs, segment.rs, recovery.rs, drive.rs, recovery tests yes

Item 10 is a D0-B integration finding, not B1 work. A replayed canonical frame carries the exact applied refs in committed.transaction.refs, but Wave A's ReceiptRecord omitted them. Once a checkpoint moves the replay horizon past that frame, the current ref table cannot recover old values, deleted refs, force, or which refs belonged to that transaction. Returning an empty projection would make a committed retry differ across reopen.

New checkpoints therefore set an authenticated checkpoint capability flag and encode the complete bounded applied-ref vector in every retained receipt row. A storage-version-1 checkpoint without that flag remains readable when it has no retained receipts. If it has any, it is rejected as ReceiptRefsUnavailable and follows the existing explicit offline rebuild path; inventing empty receipt refs is forbidden. Readers predating the amendment already reject the non-zero flag, so compatibility fails closed in both directions without a global storage-version bump. Contract review 2026-07-27-A records the amendment.

Item 5 is a blocker discovered in review and is the reason this section was rewritten. The first draft required StoreEngine::open to recover through ShardDrive::reopen_through_recovery. It cannot: drive.rs is compiled only under the store-internals feature, its own header states that B1 does not build on it, and DriveRecovery returns diagnostics and an adopted sequence set — not the catalog, refs, receipts, and index the engine needs to construct a CommittedRoot. Requiring the engine to call it would have forced B1 to write a second recovery path, which is exactly the defect that produced two of Wave A's three blockers.

The fix keeps one path with two callers. recovery.rs gains a production entry point returning a RecoveredShard, and drive.rs's seam is re-pointed onto it so the harness and the engine recover identically by construction rather than by review. If the two ever diverge again, it must be because somebody changed the shared function, not because a caller quietly grew its own.

LOCK is root-wide, so the production entry point is owned by a RecoverySession that holds it continuously while every shard is recovered and then moves into StoreEngine for the engine's lifetime. A free one-shard wrapper exists for the drive seam and is implemented by creating that same session. Acquiring and dropping one lock per shard would leave a second process able to enter between shards or immediately after recovery but before readiness, invalidating the recovered root before its first read.

RecoveredShard must carry everything CommittedRoot needs, which is more than the replayed tail. A first draft listed "an index delta" and that is a defect: an object whose only index entry lives in a sealed IndexRun or a checkpointed generation would be present before the crash and absent after open, and the segments holding its bytes would not be pinned against reclamation. The contents are therefore:

  • the namespace catalog, ref state, and receipt table;
  • the complete layered index — the replayed delta over the ordered set of sealed IndexRun references the selected manifest and checkpoint generation retain, in the lookup order CommittedRoot will use, not the delta alone. For a committed StagedProjectionInstallV1, the staging-owned recovery resolver must supply its exact namespace-scoped membership and live artifact/index pins before readiness; a notification without incorporating those objects is not complete recovery;
  • every retained generation reference — segments, index runs, and checkpoint generations — so constructing the root transfers ownership of the things that keep those files alive rather than merely naming them;
  • per-shard and per-repository sequences, and the ShardRecoveryReport.

Ownership is the subtle half: recovery opens these artifacts, and if RecoveredShard hands over names instead of live references there is a window in which nothing holds them and reclamation is legal.

The drive seam keeps the recovered state, not just a projection of it. DriveRecovery is diagnostic-only, and projecting RecoveredShard straight back down to it would make the engine/drive equivalence test impossible — the test could compare only the adopted sequence set, which is precisely the weakness that let Wave A's double-adoption hide. DriveRecovery therefore gains the RecoveredShard alongside its existing summary fields, exactly as it gained report after the Wave A review, and for the same reason. The equivalence test compares the recovered state itself: catalog, refs, receipts, index layering, and sequences.

roots.rs (lead-owned).

  • CommittedRoot — immutable, Arc-shared, published only through ArcSwap. Holds per-namespace RepoState, the layered object index (newest delta layers over sealed IndexRun references), the receipt table, per-shard committed sequences, and the generation/tail references that keep segments alive. Every read API captures exactly one of these (plan §5.3).
  • RepoStaterepo_sequence, current_authority, genesis_authority, the typed ref map, lifecycle, and previous_event_digest. Per-repository and Arc-shared, so a group touching three repositories replaces three of these and shares the rest.
  • ShardSubtree — what one fenced group produces, before merge: its index delta, its affected RepoStates, its receipts, its shard committed sequence, and its generation references. Plan §5.3 requires this be built as one immutable value and then merged; building it incrementally into the live root is the defect this type exists to prevent.
  • CommittedRoot::merge(&self, subtree: &ShardSubtree) -> CommittedRoot — pure, allocating a new root sharing all untouched substructure. The CAS loop calls this again against the newer root on contention, so it must be idempotent in effect and free of interior mutation: a merge that mutated anything reachable from self would corrupt the root a concurrent reader already captured.
  • OperationStatusRoot — bounded map of in-flight (NamespaceId, OperationId) to StatusEntry { operation_digest, retry_until_micros, phase, shard_sequence }. See decision 9.8 for what "bounded" does at the bound.

completion.rs (lead-owned). The runtime-agnostic completion primitive behind async fn submit. levcs-store takes no runtime dependency — plan §5.1 confines the crate to storage decisions, and §7 reserves Tokio for socket and timer work that performs no filesystem calls.

It is one-to-many, not one-to-one. A first draft specified a single Mutex<Option<Waker>> and was wrong: deliverable 7 coalesces same-ID/same-digest callers onto one leader, so several futures await one outcome. A cloned crossbeam receiver does not fix this — a channel distributes each value to exactly one receiver, which is work distribution, not broadcast, and would deliver the receipt to one arbitrary waiter and hang the rest.

StoreError is not Clone, so the shape is a real type decision and D0-B freezes it: shared result state holding one outcome, with a collection of registered wakers. The alternative — one slot per waiter with fan-out from the in-flight entry — was rejected because the fan-out has to duplicate the error anyway, so it does not avoid the problem, it relocates it.

"Read by reference or by a cloneable projection" was not a decision and is replaced. Every submit caller must receive an owned Result<CommitReceipt, StoreError> — that signature is frozen in D0 and printed in plan §5.1 — so the stored outcome must be convertible into an owned result once per waiter, losslessly. D0-B makes StoreError itself Clone, and stores the outcome as exactly Result<CommitReceipt, StoreError>.

That is a one-variant change. CommitReceipt already derives Clone, and every StoreError variant is already cloneable except one: Io(std::io::Error). D0-B amends it to Io(Arc<std::io::Error>) (types.rs, already amended for item 8), with a hand-written From<std::io::Error> so ? keeps working at the several hundred existing call sites.

This is lossless by construction rather than by reconstruction, which is why it is preferred to the obvious alternative. A cloneable mirror error — capturing (ErrorKind, String, raw_os_error) and rebuilding an io::Error per waiter — is lossy in a way that is easy to miss and impossible to detect later: the rebuilt error is a different object, and any downstream source() chain or downcast is gone. Sharing the original through an Arc gives every waiter the same error, which is also the truth — one transaction failed once, for one reason.

The conversion must be total. B2 checks that nothing in the completion path introduces a _ => arm mapping unexpected variants onto a generic failure; a waiter receiving a different, vaguer error than the leader is a silent divergence between callers that the coalescing contract says are indistinguishable.

The synchronization algorithm is part of the frozen contract, because the obvious implementation is racy. Receiving on a channel and then independently storing a waker has the classic lost-wakeup window: the sender can complete between the check and the registration, so the waker is stored after the only wake that would ever fire. Registration and the post-registration recheck must therefore share one synchronization point with the sender — one mutex covering {outcome, wakers}, where poll locks once, checks the outcome, and registers only while still holding the lock, and completion locks once, stores the outcome, and takes the waker list to wake after releasing. A waker registered under the lock is either seen by a completion that has not yet run, or made unnecessary by an outcome already present when it checked.

That single mutex is why no dependency is needed and why AtomicWaker would not have helped: AtomicWaker solves single-waiter registration, not the multi-waiter set, and the state this guards is shared anyway. It is contended by a handful of parties, held for the duration of a move, and sits on a path dominated by an fdatasync several orders of magnitude larger.

Hand-rolled futures fail by losing wakeups, and a lost wakeup here hangs a request that has already been durably committed — indistinguishable from a store that never returns. The contract is therefore stated as explicit traces, and is B2 charter material:

  1. Completion after first poll. poll returns Pending and registers a waker under the lock; completion occurs; the waker is woken; the next poll returns Ready.
  2. Completion before first poll. Completion occurs with no waker registered; the first poll returns Ready immediately. The value must not be lost for want of a waiter, and no wake is required for a poll that has not happened yet.
  3. Exactly-once delivery per waiter. Each waiter observes Ready exactly once. A future must not be polled again after Ready; the implementation makes that a clean panic or a documented unreachable, never a second half-formed value.
  4. Multiple attached waiters. Every waiter attached to one in-flight entry observes the same outcome. Not "an equivalent outcome" — the same one, since only one transaction was sequenced.
  5. A waiter dropping. One waiter dropping mid-flight must not disturb the others: the remaining waiters still observe the outcome, and the dropped waker is removed rather than woken.
  6. Dropped receiver, all of them. Every waiter dropping must not leak the state and must not panic the shard thread when it later signals. A disconnected receiver is a dropped request, not a publication failure — plan §7 stage 11 makes a fenced transaction committed regardless of whether anyone is still waiting for the receipt.
  7. Exactly one execution. N attached waiters cause exactly one signer handoff and exactly one append. This is the property coalescing exists to provide, and it is asserted by counting executions, not by observing that the receipts match — identical receipts are what a double execution would produce if it were idempotent, and it is not.

6.3 Publication ordering (normative)

Extends §3's durability ordering across the publication half. Wave A's poison window ended at the fence. The full poison window is steps 4 through 8 below — from marking Resolving through committed-root publication, inclusive (plan §7 stages 1011). Steps 13 are pre-append and definitively absent on failure; steps 910 are post-publication and cannot poison, because the transaction is already committed.

  1. Sequencer checks mutable preconditions against speculative state containing every earlier accepted transaction in the pending group. Rejection here is a 409 and commits nothing.
  2. Assign shard_sequence and repo_sequence, chain previous_event_digest, compute event and state digests, and hand off to CommitEvidenceSigner on its own bounded pool, returning in repository-sequence order. (EvidenceHandoffFailure — reclassified by contract review 2026-07-26-A; see below.)
  3. Re-check the signed deadline immediately before marking Resolving. Plan §7 stage 9: no operation may first append after its signed deadline.
  4. Mark every operation in the group Resolving in the status root. (AfterMarkedResolving)
  5. journal::append_group_and_fence — exact reserved frames, one fence.
  6. Build the ShardSubtree from the fenced group. (DuringCommittedRootBuild, AllocationFailureBeforePublication)
  7. CAS-publish: load the current root, merge, compare-and-swap; on contention reload and merge again against the newer root. (BeforeRootCas, DuringRootCasRetry)
  8. Remove the group's status-root entries, with release/acquire ordering that makes the step-7 publication visible to any subsequent mandatory B load.
  9. Wake waiters. (AfterRootCasBeforeWaiterWake)
  10. Return receipts. (BeforeResponse)

Two orderings are load-bearing and must be asserted, not assumed:

  • Step 8 after step 7, never before. Removing a status entry before the receipt is visible manufactures Unknown for a committed transaction — the exact race the mandatory committed-root B read exists to close, reintroduced from the write side where no read can fix it.
  • Failure at step 9 or 10 is not a failure. A fence that succeeded and a root that published is committed. AfterRootCasBeforeWaiterWake and BeforeResponse must both leave a queryable receipt; a waiter that never wakes is a hung request, not an absent transaction. Plan §5.1: "failure to wake a waiter afterward cannot hide the receipt." The frozen oracle agrees — both rows are shard_poisoned: false, immediate_status: Committed.

EvidenceHandoffFailure — resolved by contract review 2026-07-26-A

The frozen oracle classified this failpoint as poisoning the shard with immediate_status: Resolving, alongside AfterMarkedResolving. It fires at step 2, before the group is marked Resolving and before any byte is written, so the physical state is NoBytes and the outcome is unambiguous. The store-side crash-matrix fixture already said as much in its own rationale, which is how the disagreement surfaced: the fixture and the oracle described the same row differently.

Ruled and applied. EvidenceHandoffFailure takes the exact BeforeAppend shape — shard_poisoned: false, immediate_status: DefinitiveAbsent, recovery_outcome: AbsentRetriable, acknowledgment_allowed: false, later_append_allowed_before_recovery: true. The physical state class remains NoBytes and the fixture is unchanged. The failpoint stays in Wave B, because only B1's sequencer can exercise a signer handoff at all.

The deciding argument was operational rather than formal. Under the old classification a routine SignerError::Unavailable — a restarting signer, no storage fault — would poison the shard and admit no mutation until recovery ran, trading a real availability property for a safety property that was never at risk.

Recorded in doc/instance-throughput-rewrite-plan.md. The same edit replaced the _ catch-all in phase0_oracles.rs with an exhaustive list of the thirteen poisoning failpoints: a catch-all in a test that pins a frozen classification silently absorbs any newly added row into "poisoned", which is the mechanism by which a wrong classification ships past its own test.

6.4 B1 — NamespaceTxn

Owns engine.rs, transaction.rs, snapshot.rs. Deliverables 18, with acceptance criteria.

  1. StoreEngine::open with plan §5.2's four startup states handled explicitly and in order, no inference: absent-or-empty initializes; a valid FORMAT opens through the D0-B production recovery entry point (§6.2 item 5) — the same function the drive seam calls, never a second path; a recognized non-empty legacy layout without FORMAT returns LegacyLayout carrying the exact migrate-store command; every other non-empty unrecognized layout is refused without modification. Per-shard threads and readiness. Accept: a test per startup state, including that an unrecognized layout is byte-identical after the refusal, and a test that the engine and the drive seam produce the same recovered state from one crash image.

  2. Repository creation binding repo_id and genesis authority permanently in the catalog (plan §4 identity invariant 2). Accept: a second bind of the same repo_id with a different genesis is refused, and the refusal survives reopen.

  3. ValidatedTransaction and its sealed builder, plus the privileged constructors for recovery, plus adopt_projection (§6.2 item 9). Accept: the D0 contract test already asserts the seal; extend it to prove the builder rejects an incomplete transaction rather than producing a partial one.

  4. The shard sequencer, wiring GroupBuilder — this closes the Wave A carry-forward. Speculative state contains every earlier accepted transaction in the pending group. It checks only mutable preconditions: operation ID/digest, repository lifecycle, snapshot/config/policy epoch, typed ref CAS, expected authority, and precomputed force/ancestry facts. It never re-parses, re-hashes, re-verifies signatures, or re-evaluates policy while holding the mutation lane. Includes releasing or revalidating the speculative suffix when a member of a forming group fails before append. Accept: group formation bounded by transactions, bytes, and idle delay asserted against the sequencer, not against GroupBuilder in isolation — the carry-forward is closed by the caller existing, not by the unit tests that already pass.

  5. Sequence assignment and the signer handoff per §6.3 step 2, on its own bounded pool, returning in repository-sequence order. Accept: out-of-order signer completion still produces repository-sequence-ordered frames, and EvidenceHandoffFailure matches the amended oracle — definitively absent, shard not poisoned, later append allowed.

    Plus the suffix-repair case, which is where this gets hard: fail signing at the first, middle, and last position of a forming group, in each case with later signer results already available. Prove the failed transaction appends nothing, and that every retained suffix transaction is re-sequenced, re-chained, and re-signed as necessary, leaving no sequence gap. Signing covers SignedCommittedTransactionV1::signing_digest, which commits to the whole committed transaction and therefore to the chained previous_event_digest, so dropping a member from the middle of a group invalidates every signature after it — the already-returned results for the suffix are now signatures over a chain that no longer exists. A partial repair here produces a durable, correctly-fenced frame carrying a signature that verifies against nothing, which no amount of recovery can detect later.

  6. The two roots and the CAS merge loop, with the linearizable A → status → B read. Accept: asserted against oracle::two_root_status_read over every input combination; a concurrent-publication test proving contention causes a re-merge against the newer root and never a lost update; and a duplicate-subtree test proving merge is idempotent in effect — merging the same ShardSubtree twice must not double-count receipts, sequences, or index entries. Concurrency coverage alone does not test idempotence, and the CAS retry path depends on it.

  7. Receipts, idempotency, coalescing, terminal retention. Same-ID/same-digest attaches; same-ID/different-digest rejects. Accept: asserted against oracle::coalescing_decision and oracle::retained_terminal_status; receipt_visible_until and status_tombstone_until computed from the frozen protocol functions, never restated. Ordering against the 9.8 capacity check is normative — see the ruling.

  8. RepoSnapshot sharing persistent substructures. Accept: a measured assertion that taking a snapshot allocates no index copy — a count or a byte figure, not a comment. §5.3 makes this a correctness property, so it needs a test that fails if someone clones.

B1 must not touch the crash harness (B4), staging (B3), or any frozen Wave A file.

Carry-forward: index sealing does not yet bound what a reopen rebuilds

Recorded with the index-maintenance slice (contract review 2026-07-29-C), because it bounds what that slice can claim and it is the reason checkpointing follows it immediately.

Sealing moves index entries out of the delta layers and into a durable run. It moves no frame out of active/. A manifest's committed_shard_sequence advances only when a checkpoint or a segment rotation makes a prefix durable somewhere else, and neither exists yet — so recovery replays every frame the shard has ever written, into a single IndexDelta bounded by max_active_index_entries.

Two consequences, both live until StoreEngine::checkpoint lands:

  1. The writer refuses admission once the replayable set reaches that ceiling. Without it a shard would keep accepting work and produce a store that fails to open with LimitExceeded — writing what it cannot read back. The hole pre-dates index maintenance: the placeholder refusal capped delta layers, which never bounded the summed entries behind them.
  2. A seal triggered by entry pressure therefore lands exactly on that ceiling and the next admission is refused; only a seal triggered by the fan-out ceiling (max_index_runs unsealed layers) leaves the shard able to continue. Entry-pressure sealing is correct but cannot relieve what it is meant to relieve until the committed prefix can advance.

This is a scope consequence and not a defect: nothing here is unsound, and both behaviours are asserted by tests. It is written down so that "the shard seals under pressure" is not read as "the shard can run indefinitely under pressure", which is what checkpointing will make true.

Closed by contract review 2026-07-30-C. StoreEngine::checkpoint advances the committed prefix, so the replay set is what lies above the newest checkpoint rather than everything the shard ever wrote. The ordering that makes it safe is recorded there: the checkpoint format holds no object index, so every delta layer through the committed sequence is sealed into a run its manifest names, and discarded, before the sequence moves.

Closed: a run may cover frames the active journal still holds

Recorded as a carry-forward with the index-maintenance slice and closed by contract review 2026-07-30-A, which is why the restriction it describes is no longer in the code.

The constraint was that an IndexLocation names a logical generation, and only a segment's was stable: the active tail's came from max(manifest, .seg, .idx) + 1, so it moved whenever any artifact appeared — an index run's own manifest was enough — and recovery then sealed the tail into a segment under the moved value rather than the identity the frames already had.

recovery_generation was two numbers wearing one name. Split, each answers its own question: the sealed segment inherits the tail's logical generation, so moving frames from active/ to segments/ changes where they are and not what they are called; the recovery manifest takes the next free manifest generation, so an index run's manifest cannot collide with a recovery's. Sealing therefore covers current frames, in the session that wrote them.

Residual, one case wide, and it refuses rather than proceeds. If a .seg already occupies the tail's identity — an orphan from an interrupted seal, or a segment no manifest references — the seal cannot use that name and the frames must be renamed. Contract review 2026-07-30-B splits what that costs:

  • Nothing published names the displaced identity. Recovery falls back to a free generation and succeeds. An orphan segment is a documented state in which the active journal stays the authority, and refusing here would turn a recoverable root into an outage.
  • An index run the manifest names holds locations against it, and against nothing else. Recovery reclaims it: the row leaves the manifest it installs, the run leaves the retained generation and the layered index, and the open succeeds. Sound because the displaced identity is the active journal's — one past the last retained tail range, and nothing else can carry it — so its frames are above every checkpoint horizon and replay rebuilds each entry the run held. The .idx stays on the device and ShardRecoveryReport::reclaimed_index_runs records what happened.
  • The run also holds locations against some other identity. Recovery still refuses the open with Corruption, naming the run. Its other entries may name a segment below the checkpoint horizon, which replay does not touch, so discarding the run would delete the only index those objects have. A diagnosable outage on a root that has lost nothing is the right outcome; publishing a manifest whose run is authoritative and resolves to nothing is not — the replay delta above it hides that from every lookup until the first consumer that reads runs directly, which is the checkpointer.

The test is what the recovery names its frames, not how it got there: an interrupted recovery resuming a fallback an earlier session chose strands the same run, and takes the same treatment. A resumed seal at the identity the frames already carry displaces nothing and opens normally.

The fallback applies only to frames being sealed. A journal that survives recovery keeps the identity the manifest implies even when a .seg occupies that name, because nothing is being written to segments/ and there is no collision yet. This is what keeps the active tail's identity derivable from the manifest across every open — the property the split rests on. Giving a surviving journal a fallback identity moved a number nothing recorded, and the next open, deriving the old one, searched published runs for a generation the frames no longer carried.

Closed by contract review 2026-07-30-G. Recovery discards an index run whose covered identity was not preserved, which turned the refusal into successful reclamation for every run it can prove redundant. The refusal survives only where the proof does not hold — a run naming more than the displaced identity — and it is no longer a placeholder for missing capability but the correct answer for a run whose other entries replay would not rebuild.

6.5 B3 — StagingSessions

Owns staging.rs. Deliverable 9: bounded invisible projection staging.

The store/instance split, stated first because the first draft of this section got the size wrong by ignoring it. Plan §8's staging contract is long, but most of it is not the store's. §5.1 forbids levcs-store from making identity-role, merge-policy, or federation decisions, so ProjectionCore validation, identity and authority proofs, policy evaluation, source-snapshot and ForkProofV2 checks, authenticated source-kind and actor/key-epoch binding, session authentication, the v2 HTTP routes, and the remote source's export lease all belong to Phase 2's levcs-instance/staging.rs and ProjectionCore. B3 owns the storage mechanism and its bounds: what a session is on disk, what it costs, when it dies, and the guarantee that none of it is visible until a transaction adopts it. B3 must make every one of the instance-layer checks possible — by binding and exposing the fields they key on — without performing any of them.

B3's deliverables:

  1. Session lifecycle — begin, idempotent numbered chunk-put, read-only resolver, seal to StagedProjectionInstallV1, abort, expiry, cleanup. Restartable by session ID and chunk digest; no renewal beyond the advertised maximum.

  2. Canonical session binding. Creation binds the session ID to destination repo/genesis and expected state, projection mode, total object/byte/chunk counts, ordered manifest digest, final operation ID/stable digest, and expiry. B3 stores and enforces the binding; it does not evaluate the identity or policy fields it carries.

  3. Same-device validation. <root>/staging and the target shard must be on the same st_dev, checked at session creation. Cross-device adoption and copy fallback are forbidden, because adoption is a link, and a link across devices is not a rename — the same physical constraint as scope 3.4's seal.

  4. Bounds, all configured and all enforced before pinning (§6.2 item 7): per-session, per-principal, and global session counts; staged bytes, objects, and files; session age; and compaction debt, accounted independently of ordinary receive spools. Plus the feasibility check: configured maximum projection size, maximum session age, and minimum supported transfer rate must make one complete transfer possible, or creation rejects before pinning anything. A session that cannot finish is a session that only consumes budget.

    Contract review 2026-07-28-A adds the representability half of the same argument. max_projection_objects and max_projection_chunks are capped at codec::MAX_CANONICAL_ITEMS, and max_projection_bytes at max_projection_chunks * codec::MAX_CANONICAL_BYTES, because the sealed manifest is one flat canonical vector and the chunks are one canonical encoding each. Startup refuses a configuration above those caps rather than admitting sessions whose manifest could never encode. max_projection_objects now defaults to MAX_CANONICAL_ITEMS; the previous default of 100,000,000 advertised a capacity the format does not have.

  5. Artifacts are written by maintenance workers, synced, uniquely named, and unreferenced. They never enter namespace membership, object-existence answers, snapshots, refs, receipts, event feeds, dedupe state, or CURRENT.

  6. Finalize and the adoption pin. The state machine is Open → Sealed → Finalizing → {Sealed, Adopted}, and all four states are durable and reconstructable — decided by an artifact on disk, never by a flag one process remembers. Amended from Open → Finalizing by contract review 2026-07-31-A, which found two of the three transitions in the original wording unimplementable as written:

    • The pin is taken from Sealed, not Open. The sealed manifest is what an adopter revalidates against, so a pin on a session without one names state that does not exist.
    • A definitive pre-append failure returns the session to Sealed, not Open. Reconstruction reads the sealed manifest's presence as the seal's own commit point and would hand back Sealed on the next restart regardless; returning a session to a state no restart can reproduce is the defect the private SessionBusy enum exists to avoid. Sealed is still finalizable, which is the property the rule is about. If the session is no longer live the marker still goes and expiry reclaims it, which is "otherwise cleanup aborts it".
    • Adopted is durable, and is the transition the original wording had no name for. Adopted artifacts stay in staging/ with a committed root pointing into them, so dropping the session record would make the next reconstruction read the directory as an abandoned materialization and reclaim committed content; keeping the record with no state re-charges the whole reservation on every restart. Adopted releases the reservation — those bytes are store content, not staging occupancy — and leaves the directory to cleanup's reference proof.

    The pin becomes durable before it is issued. A handle backed by an in-memory flag is an adoption capability with nothing behind it.

    Only the sole bound operation/digest may finalize, which the binding enforces by carrying it from begin; a different one cannot reach a session at all. Coalescing identical concurrent finalizers is B1's, not B3's — also amended by 2026-07-31-A. A ProjectionAdoption is an unforgeable capability with exactly one terminal outcome and a Drop that reports its absence, so there is no second copy to hand a second caller: at this layer the guarantee is one pin, and a second finalizer is refused by name. Request-level coalescing across retries of one operation belongs to whoever owns the request.

    Expiry prevents a new finalizer but must not delete artifacts held by an already-admitted one — the expiry/finalize race is a named acceptance case, not an incidental detail.

  7. Cleanup proves absence of reference. It removes only artifacts carrying a valid session marker, and only after proving no committed manifest references them, then syncs the affected directories. Accept: a test that cleanup declines to remove an artifact a committed manifest still references.

    Two clarifications from contract review 2026-07-31-B, neither a relaxation:

    • The supplied root must have reached the adoption. A root captured before a session was adopted references none of its artifacts because it predates them, so absence measured against it is a date and not an absence; acting on it deletes a directory the current root points into. Each adopted session records the committed shard sequence its adoption frame reached — carried on ProjectionAdoptionOutcome::Adopted and made durable in the adoption marker — and is skipped unless the supplied root has reached at least that far. The root must say so explicitly: a root recording no sequence for that shard is silent, not a witness that the shard has committed through zero, and zero is a valid sequence. Contract review 2026-07-31-C.
    • The candidate set is the adopted sessions, and only those. Absence of reference is necessary and is not sufficient: a Sealed session a client is still finalizing, and a Finalizing one whose adoption frame may be mid-append, are both referenced by nothing, and a cleanup that proved absence and stopped there would delete them. Open and Sealed belong to expiry and abort; Finalizing has no absence to prove yet.
    • The unit of removal is the session directory, not the individual artifact. A session's chunks are not independent: the manifest names all of them and reconstruction refuses a sealed session missing any ordinal, so removing the unreferenced half of a directory trades a bounded leak for a root that fails to open. A session is reclaimed only when nothing in it is referenced. The per-artifact rule this clause is really about still applies inside that: reclamation removes only files carrying the session's own marker and refuses the whole directory if it finds anything else.
  8. Recovery treats synced-but-unreferenced artifacts as invisible garbage, and a complete final frame as authoritative adoption. It can never expose a partial chunk set.

The security property is the whole point of the package: sealing cannot publish membership. Only submit may adopt a sealed descriptor, so possession of a session ID never authorizes publication (plan §4 identity invariant 7, §8). Accept: a test that seals a session and then proves the objects are invisible to RepoSnapshot::locate until a submit adopts the descriptor — asserted through the reader API, not by inspecting staging's own state.

Validated against validate_projection_stage_binding and validate_projection_stage_finalize from the frozen protocol.

The B1/B3 seam is a lifecycle, and D0-B freezes all of it. An earlier draft called it "exactly one call" — adopt_projection(StagedProjectionInstallV1) — and that is not expressible. A bare wire descriptor cannot hold an adoption pin, cannot give B1 access to session and artifact state for the revalidation deliverable 6 requires, cannot tell B3 that a definitive pre-append failure released the session, and cannot let cleanup prove the committed root does not reference an artifact. With only the descriptor, either expiry may reclaim artifacts while a submit is already admitted, or B1 must read B3's private on-disk representation directly — and a package reading another's internals is how Wave A's first blocker happened.

D0-B therefore freezes an opaque adoption handle obtained from B3 and consumed by submit together with the wire descriptor. The handle is the pin: holding it is what keeps expiry and cleanup off the artifacts, so the pin cannot be forgotten separately from the adoption. Its interface covers exactly four things, and no more, so it does not become a general back door into staging:

  1. Resolution — read-only access to the sealed manifest, chunk digests, and artifact paths that B1 needs in order to recheck session, operation, digest, manifest, and artifact hashes. Read-only is load-bearing: the adopting side revalidates, and a handle that could mutate would let adoption repair what it was meant to reject.

  2. Pin lifetime — held from admission through committed-root publication. Compaction and GC may not reclaim a pinned session, and expiry may not abort it (plan §8).

  3. Outcome notification, all three ways — adopted, definitively failed before append, or transferred to recovery. On definitive pre-append failure the session returns to Open if it is still live, and is otherwise aborted by cleanup.

    The third outcome is the one an earlier draft missed, and its absence was a leak. A failure anywhere in §6.3 steps 48 poisons the shard and leaves the outcome unresolved until recovery runs — the transaction is neither adopted nor definitively absent, and nothing in-process can say which. Without a terminal outcome for that case, closing the poisoned engine either drops the handle without an outcome, which §6.5 declares a bug, or strands a Finalizing session that no expiry may collect, because expiry must not touch a pinned session. TransferredToRecovery is therefore a legitimate terminal state for the handle, and the pin outlives the process.

    The durable authority for the transferred pin is the frame itself, not a separate pin record. Once the final frame is appended it binds the manifest and membership root, and B3 deliverable 8 already makes a complete final frame authoritative adoption. So the window analysis closes cleanly: a crash before append leaves no frame, the artifacts are unreferenced, and cleanup reclaiming them is correct because the transaction is absent; a crash after append leaves a frame that recovery resolves. Recovery then notifies staging of the resolution — committed, so the artifacts are referenced and the session completes; or proved absent, so the session returns to Open if still live and is otherwise aborted. That notification is part of this seam, not an internal detail of recovery.

    The ordering constraint this creates is worth stating plainly: cleanup may not reclaim artifacts for a session whose adoption frame might be durable, so it must run after recovery has resolved the shard, never against a store that is still poisoned.

  4. Reference proof for cleanup — the query by which B3 establishes that no committed manifest references an artifact, answered against a CommittedRoot, not against B3's own bookkeeping. Cleanup asking itself whether something is referenced is not a proof.

Dropping the handle without an outcome is a bug, not a state: it must be observable, because a silently dropped pin is a leaked session that no expiry will collect.

Both packages write a test driving the full lifecycle — pin, adopt, publish, unpin — and B2 checks the two agree. A seam with tests on only one side is the Wave A finding restated.

Carry-forwards after the first B3 slice — recorded, not closed

Two properties are implemented and not operated. Both are recorded here rather than only in an ignored test, because a carry-forward that lives in a test attribute is invisible to anyone reading the scope to decide whether a deliverable is met, which is the mechanism by which Wave A's GroupBuilder gap nearly shipped as satisfied.

  1. Production staging use is incomplete. The ownership seam is closed — StoreEngine::open constructs the single ProjectionStaging under the held RecoverySession and passes it as recovery's ProjectionRecoveryResolver — but the use seam is not. No production path calls begin, finalize, or adopt_projection; they are reachable only from B3's own tests. Deliverable 5's security acceptance — sealed objects invisible to RepoSnapshot::locate until a submit adopts them — therefore remains unasserted, and its test stays ignored with both blockers named. Having a production caller for construction is not the same as having one for the mechanism, and only the second discharges charter item 8.

  2. ProjectionStaging::expire has no scheduler. Expiry is implemented and unit-tested, and nothing in production drives it. Session age is therefore a bound that is enforced when asked and never asked — a limit no deployment currently applies. The bound is not met until something operates it.

The StoreEngine staging accessor B3 requested is a pending interface amendment, and it must not be a bare Arc<ProjectionStaging>. A cloned Arc can outlive EngineShared, survive the release of the root LOCK, and keep serving staging operations against a root this process no longer holds — which reintroduces the exact defect the root-lock-proof constructor was added to make inexpressible, and reintroduces it from the reader side where the constructor cannot see it. The eventual façade or handle must retain engine and root-lock authority for the lifetime of every session it creates. Until that is designed and frozen, the accessor does not exist and the acceptance test above stays blocked.

6.6 B4 — StoreHarnessB

Owns the crash driver, the benchmark, the matrix, and the recovery script.

  1. The Wave A matrix carry-forward. Add fault generators for sealed-segment frame corruption and cross-shard journal movement. The Wave A record states plainly that the matrix could not have caught its own first blocker; until these exist, that is still true. Accept: both generators produce a crash image that the production recovery path rejects, demonstrated by the matrix failing when A1's validation is reverted.
  2. The eight Wave B failpoint rowsAfterMarkedResolving, EvidenceHandoffFailure, DuringCommittedRootBuild, AllocationFailureBeforePublication, BeforeRootCas, DuringRootCasRetry, AfterRootCasBeforeWaiterWake, BeforeResponse — each with its expected outcome asserted against oracle::append_publication_expectation, and Panic coverage for the three publication-side rows. The pending-wave-b rows must be gone from the matrix at Phase 1 exit; check-phase1.sh already enforces that.
  3. P2 against a real StoreEngine::submit. The Wave A benchmark drove the journal seam. Re-point it, and emit the bundle with the gate's honest outcome. Expect different numbers; a change is the measurement working.
  4. The storage_primitive verification claims may now be earnable. With a real engine, commits_in_recovered_closure and the object-graph flags forbidden at this gate should be re-examined. If any becomes genuinely earnable, that is a schema amendment and a contract review — request it, do not emit it. bench/result-schema.json is lead-owned. Requested, reviewed, and granted as contract review 2026-07-28-C; the schema and its contract tests have landed and item 5 below is what B4 must emit against them.

Consequence of contract review 2026-07-28-C: the emitter contract

bench/result-schema.json now requires a run_conditions block and a named resources.configured_ceilings.max_index_runs, and requires two verification claims on the submit path. The emitter does not produce any of them, so store-bench emit-skeleton currently writes a bundle that fails validation on exactly two fields:

[]: 'run_conditions' is a required property
['resources', 'configured_ceilings']: 'max_index_runs' is a required property

scripts/verify-store-recovery.sh --cycles 2 reports bundle=schema-invalid, VERIFY_EXIT=1 with everything else green (matrix=pass, acknowledged_loss=0, torn_transactions=0). scripts/check-phase1.sh is red as well, which is not what landing a schema alone would normally do: the gate runs store-bench's unit tests, and three of them validate the emitted bundle against the schema rather than against substrings, so schema conformance is inside the gate. That is the property review 2026-07-24-B was after, working. All three failures are in store-bench.rs and none is a defect in the store; they are expected collateral of landing the contract first and they close with item 5.

  1. Emit the run conditions, the index-run ceiling, and the two earned claims. All of it in store-bench.rs; no other file is involved. Nothing here may be a constant this file restates — every value must come from what the run configured or observed.

    a. resources.configured_ceilings.max_index_runs — the u32 from the StoreOptions the store was opened with, not ENGINE_MAX_INDEX_RUNS written out a second time. Emit it as an integer.

    b. A run_conditions object with all ten members. The truthful values today are:

    member submit path drive path
    initialization_path store_engine_open (see f) shard_drive_create
    mutation_path store_engine_submit journal_drive
    checkpointing unimplemented unimplemented
    index_maintenance deltas_retained_in_memory no_index_in_path
    index_run_ceiling raised_because_index_sealing_unimplemented store_default
    receipt_reconciliation acceptance_of_any_committed_status no_receipts_in_path
    objects_new_source summed_from_receipts derived_from_transaction_count
    commit_id_uniqueness checked_globally_across_ack_records (after d) not_checked
    build_profile from cfg!(debug_assertions) same
    environment_fidelity diagnostic unless every reference condition holds same

    These are declarations of what the run did, not configuration. Each must be derived from the MeasuredRun and the options rather than hardcoded per path where a derivation exists: index_run_ceiling follows from comparing the configured ceiling to the store default, objects_new_source is the existing MeasuredRun::objects_new_counted, build_profile follows from cfg!, and environment_fidelity follows from the profile the run verified. A hardcoded run_conditions block is the prose caveat with a different syntax.

    c. Both earned claims on the submit path, and neither on the drive path. objects_new_equals_three_per_commit may be emitted only when the summed receipt.objects_new total equals a separately counted 3 × counted_commits — count the commits, do not reuse the summed total to produce the expected value, and refuse to emit the bundle when they disagree rather than emitting the flag as false, which the schema does not permit and which would be a different untrue statement.

    d. A global uniqueness check over every recovered ACK record, across all records at once: collect every blob_ids, tree_ids, and commit_ids entry from ExternalAckJournal::recover into one set per kind and require no repeat. Per-record checking and any argument from the distinctness of the generator seed domains are both explicitly insufficient, and commit_id_uniqueness has named values for both so a harness that did either has something truthful to record.

    e. Do not emit operation_receipts_reconciled. The schema forbids it while receipt_reconciliation is acceptance_of_any_committed_status, which is the honest declaration for store-bench.rs:2033 — it accepts any Committed(_) without comparing the payload, and the AckRecord.receipt_digest it writes is blake3(operation_id) rather than a digest of the receipt. Earning it is separate work: reconcile the exact receipt, or freeze a canonical receipt digest and reconcile that, then declare the matching value and assert the claim. The schema will then require the claim rather than permit it.

    f. Build every measured root through StoreEngine::open. Added after B1 landed startup state 1. The benchmark previously seeded its root with segment::initialize_root and disclosed the fact in three places, because seeding a store off the production path in order to measure the production path is the charter item 8 smell and a disclosure is not a fix. With state 1 implemented the smell is closable rather than merely recordable, so it is closed: the seeding helpers and the ROOT_SEEDED_BY_NON_PRODUCTION_PATH constant are retired, and initialization_path is observed — the FORMAT marker is absent before the call and present after — rather than asserted. Deriving it from an observation is what keeps the declaration honest if the seeding ever regresses.

    Accept: bash scripts/verify-store-recovery.sh --cycles 2 reports bundle=schema-valid on both --path submit and --path drive; the emitter's own the_emitted_bundle_validates_against_the_frozen_schema passes; and the_schema_check_can_actually_fail gains a mutation for each newly required field, since a required field the negative control never removes is a field the suite cannot notice the loss of.

Carry-forward: the SIGKILL cycles still drive the journal seam

Recorded here because until now it existed only as a comment in the harness, and a harness comment is not scope evidence. It is a limitation on a §7 exit criterion, so it belongs where the exit criteria are read.

scripts/verify-store-recovery.sh's 100 randomized SIGKILL cycles drive the journal seam (drive.rs), not StoreEngine::submit. So kill -9 and the driver's _exit(3) never land inside a real publication: they land inside a frame append and fence, which is one step of a publication and not the step where the status root, the sequencer, the acknowledgment, and the checkpoint install are at risk. Every ordering hazard that only exists between those is untested by this script, at any cycle count.

Moving the cycles onto production submit was blocked on StoreEngine::open startup state 1, because the child process could not create a store root through the production entry point. That block is gone: B1 landed state 1, and every root B4 measures — the benchmark's and the Wave B rows' — is now built by StoreEngine::open, so engine_matrix.rs's ROOT_SEEDED_BY_NON_PRODUCTION_PATH disclosure and its segment::initialize_root seeding are retired.

The move is therefore deferred, not blocked — a separate B4 assignment that has not been made rather than one that cannot be done. That distinction matters here: a blocked item waits for someone else, and a deferred one waits only for a decision, so this is the entry that should be picked up first when the acknowledged-crash-recovery criterion is next worked.

Consequences, stated so no later reader has to reconstruct them:

  • The acknowledged-crash-recovery exit criterion (§7) is only partly earned. What is earned is that the journal survives abrupt termination and recovery adopts exactly the fenced prefix. What is not earned is that an acknowledgment survives a kill inside the publication that produced it.
  • The figures the script reports — recovery_failures=0, acknowledged_loss=0, torn_transactions=0, repeated_adoptions=0 — mean what Wave A meant by them, over the seam Wave A had. They are not a real-engine soak result and must not be quoted as one.
  • Closing this is B4 work once startup state 1 lands, and it is a re-point of an existing script rather than a new harness. Expect the numbers to change; a change is the measurement working.

Consequence of contract review 2026-07-28-B: delete the reopen wait

segment::lock_root now returns a RootLock guard that releases the root lock with an explicit LOCK_UN before closing the descriptor, which fixes the spurious AlreadyLocked B4 measured after StoreEngine is dropped (the lock lived on the open file description, and a concurrently forked child kept it alive past the close). The full record is in doc/instance-throughput-rewrite-plan.md.

B4's bounded retry at tests/support/engine_matrix.rs:516 (reopen_after_close) must therefore be replaced by a one-attempt immediate-reopen assertion: a single StoreEngine::open that must succeed, with no budget, no sleep, and no attempts field to publish. The wait was correct while the defect stood and is documented as compensating for it; it exists only for that reason and must not outlive it. Leaving it in place would hide a recurrence of exactly this defect behind a wait that succeeds on the second try — and §3.1 says AlreadyLocked is a refusal and never a wait, so a harness that waits on it is asserting something the store does not promise.

6.7 B2 — StorageReviewer

Read-only durability, concurrency, and security review of the whole crate. Same charter as §5, extended to the publication half: the status-root/committed-root race, the CAS merge loop under concurrent shard publication, the completion.rs wakeup contract in §6.2, the step-7/step-8 ordering, staging-session isolation, and the ValidatedTransaction seal.

Charter items 8 and 9 apply in full. Item 9 is now a required field in every package's report, asked at dispatch rather than after completion — every disclosure it produced in Wave A arrived only because the question was asked late, and each one was a defect the gate had already reported green.

6.8 Wave B freeze gate

scripts/check-phase1.sh unchanged, plus: no pending-wave-b rows, all seventeen failpoints exercised, and the §7 exit criteria measured rather than asserted. The gate is necessary and not sufficient — Wave A's gate was fully green while three blockers stood, and nothing about Wave B makes that less likely. The adversarial review, not the gate, is the freeze condition.

6.9 Decisions 9.7 and 9.8 — ruled 2026-07-26

9.7 — the persistent-map mechanism. Ruled: take the im dependency; no hand-rolled HAMT.

Arc<BTreeMap> with clone-on-write is O(entries) per group commit and fails §5.3's no-clone requirement at scale. A hand-rolled HAMT is a correctness risk in the one structure every read captures. im provides a HAMT-based HashMap and a structurally shared B-tree OrdMap, and its structures are Arc-backed and thread-safe, which is what publishing through ArcSwap requires.

Scope: every hot publication map, not only the namespace map. Fixing the namespace map alone still permits O(receipts), O(status entries), or O(refs-per-repository) cloning on the publication path, and any one of those defeats the requirement by itself. The choice applies to:

  • per-namespace RepoState in CommittedRoot
  • the receipt table and its tombstones
  • the typed ref map inside each RepoState
  • OperationStatusRoot

Use OrdMap only where canonical iteration order is required — checkpoint encoding, manifest construction, and anywhere a digest is computed over an iteration — and HashMap everywhere else. An ordered map used by default costs comparison work on every lookup on the hottest path in the crate; an unordered map used where a digest is computed produces a digest that depends on insertion history, which is a correctness bug that only appears under rehashing.

B2's charter covers the dependency explicitly.

9.8 — status-root overflow. Ruled: reject with a typed overload error; never evict.

Eviction silently converts a Pending operation into Unknown, which is a correctness change wearing a capacity policy's clothes. The bound is enforced by refusal.

The ordering is normative, because a capacity check placed too early breaks idempotency guarantees that have nothing to do with capacity. Before the capacity check, in order:

  1. the durable lookup (a committed or expired operation answers from the committed root and never touches the status root);
  2. same-ID/same-digest attachment to an existing entry (attaching adds no entry, so it cannot exceed a bound);
  3. same-ID/different-digest detection, which must still return OperationIdMismatch rather than an overload error — a client sending a conflicting digest gets the same answer under load as it does idle.

Only a new distinct reservation may receive the overload error. Insertion must enforce the bound atomically with the insert; a check-then-insert race admits entries past the ceiling under exactly the concurrency the ceiling exists for.

Resolving entries are never subject to the bound once created: they remain until publication or recovery resolves them. Discarding a Resolving entry would discard the only record that an append is in flight.

Requires, in D0-B: a configured max_status_entries (§6.2 item 7), observable occupancy and rejection counters, and StoreError::Overloaded { limit, retry_after_micros } carrying retry guidance (§6.2 item 8). Plan §5.1 reserves StoreError for inability to answer, and a store at capacity genuinely cannot answer — this is the one overload condition the store itself originates, so it must be typed rather than folded into Conflict or NotReady.

7. Phase 1 exit criteria and evidence

§12: "Exit: P2 ≥75k, acknowledged crash recovery, namespace isolation, exact same-ref winner, multi-ref atomicity, no per-object fsync."

Exit criterion Evidence artifact Owner
P2 ≥75k canonical commits/s, p99 ≤50 ms, 3 repetitions gate="storage_primitive" bundle, schema-valid, ACK-reconciled A3
Acknowledged crash recovery full 17-row crash matrix, zero pending; 100 SIGKILL cycles; acknowledged_loss = 0, torn_transactions = 0 A3 + A2
Namespace isolation identical bytes in A not readable through B; index key test A2
Exact same-ref winner concurrent same-ref CAS: exactly one Committed, others typed conflict, no partial state B1
Multi-ref atomicity multi-ref transaction crash-tested at every failpoint: all-old or all-new, never mixed B1 + A3
No per-object fsync DurabilityCounters.fdatasync == group_count and < transaction_count over a ≥10k-transaction run A1
Recovery oracle passes every assertion bound to levcs-protocol::oracle, no catch-all arms A3

Also required before the phase closes, from §13's stop conditions: the result bundle must report index bytes/object and checkpoint lookup fan-out, and the P2 runs must not have been achieved with checkpointing disabled.

Both of those last clauses are now mechanical rather than prose (contract review 2026-07-28-C). A gate="storage_primitive" bundle with outcome="pass" must declare run_conditions.checkpointing = "exercised" and run_conditions.index_maintenance = "runs_sealed", alongside initialization_path = "store_engine_open" and mutation_path = "store_engine_submit". A reader no longer has to take the P2 row of the table above on trust: the conditions the number was obtained under travel inside the bundle as values, and a bundle that met none of them cannot encode a pass.

The harness as it stands satisfies two of the four: since B1 landed startup state 1, the submit path both creates its root through StoreEngine::open and mutates it through StoreEngine::submit. It satisfies neither checkpointing nor index_maintenance, and those are the two that decide whether a P2 figure describes a steady state or a burst — a run holding every index delta in memory, with a lookup fan-out that grows for its whole duration and no checkpoint ever taken, is measuring a system that has not yet reached the condition the number is supposed to characterize. Two of four is the accurate reading of how much of the P2 exit criterion is currently earned, and the remaining two are the expensive ones — the same disclosure the SIGKILL carry-forward above makes about the crash-recovery row.

8. Capacity analysis for P2 on the frozen reference hardware

bench/reference-hardware.toml freezes both profiles as a Ryzen 7 9800X3D (8 physical cores), 64 GiB, a Samsung 990 PRO 1 TB with write_cache = enabled and power_loss_protection = false, on btrfs with compress=zstd:3, data_profile=single, metadata_profile=dup, barriers=enabled. Three consequences bind Phase 1.

8.1 Write bandwidth and the fence rate

A canonical commit is a 1,024-byte blob + one-file tree + signed commit ≈ 1.5 KB raw; the frame adds header, evidence, signed CommittedTransactionV1, and receipt fields for roughly 2.53 KB per single-commit transaction. At 75k/s that is ~200 MB/s sustained journal write, comfortably inside a 990 PRO's direct-to-TLC rate.

The fence rate is set by the idle delay, not by the group ceiling. At the 512-transaction ceiling 75k/s would be only ~147 groups/s, but that is the floor under maximal batching and is not the operating point: with 4 shards and a 1 ms maximum idle delay, each shard closes a group at least every millisecond, giving ~4,000 fences/s in groups of ~19. That is the number that must inform device-flush budgeting — 4k flushes/s at ~50100 µs is ~2040% of one queue's service time on this device, comfortable but not free, and it is the parameter P5 tuning will be tempted to relax. The 50 ms p99 budget remains dominated by the idle delay plus queueing, not by the flush.

8.2 Capacity, not bandwidth, is the binding constraint

15 measured minutes at 75k/s writes ~170 GB of journal, plus ~9.5 GB of index runs (225k objects/s × 47 bytes/entry × 900 s). Three repetitions × four topologies would be ~2.2 TB on a 1 TB drive, and Phase 1 has no compaction to reclaim with.

Therefore the P2 procedure must be: each measured repetition starts from a freshly initialized store root, and store-bench refuses to start unless free space is at least

((warmup_seconds + measured_seconds) × target_rate × frame_bytes + index_run_estimate) × 1.25

The warmup writes too — 300 s of it, another ~56 GB — and the index runs, checkpoints, and btrfs metadata_profile = dup overhead are all outside the journal figure. A precheck over measured_seconds alone with a 25% margin covers only about 225 s of the 300 s warmup, so it can pass and the run can still hit ENOSPC inside the measured window, which destroys the repetition rather than failing it cleanly. Between repetitions the harness must delete the root, let discard=async settle, and record the settle interval in the bundle, so that repetition 3 does not run against a drive in a different garbage-collection state than repetition 1. A3 owns this; it is not optional bookkeeping, it is the difference between three comparable repetitions and three different experiments.

8.3 btrfs is copy-on-write, checksummed, and compressing

All three interact with a hand-rolled WAL:

  • CoW means the preallocate-then-overwrite design of 3.2 does not actually overwrite in place; btrfs allocates new extents and updates metadata per write. The fdatasync-has-no- metadata-to-persist benefit is lost.
  • compress=zstd:3 will attempt to compress every journal write. Frame contents are BLAKE3 digests, Ed25519 signatures, and pseudo-random blob bytes — incompressible. That is pure CPU burned at 200 MB/s on an 8-core box that also needs ~1.4 cores for Ed25519 evidence signing at 75k signatures/s.
  • Data checksums turn a torn block into EIO on read rather than garbage, which is why 3.8 step 5 must handle EIO as end-of-tail.

The standard mitigation is chattr +C (nodatacow) on shards/*/active and segments, which restores in-place overwrite, disables compression, and disables data checksums — making our own frame digest the sole integrity check, which it already is. See 9.2: this is a decision, not a detail, and it changes which recovery behavior in 3.8 is live.

8.4 CPU budget

75k Ed25519 signatures/s at ~18 µs each is ~1.4 cores of the 8 available, before BLAKE3 (~0.2 core at 200 MB/s), index insertion at 225k entries/s, and four shard threads. P2 has no validation pool, so this fits — but it means P2's headroom over 75k is not large, and any per-transaction signature beyond the one CommittedTransactionV1 signature would consume it. The bundle must report signing cost separately (§5.2: "P2/P3 measure this ordering, signing cost, and backpressure").

9. Decisions — 9.1-9.6 resolved 2026-07-24 (Wave A)

Every item in this section was ruled on before D0. Each subsection states the question, the recommendation as originally written, and the ruling with any conditions attached to it. Nothing in 9.1-9.6 is open; the subsections are retained because the reasoning is the record.

Two Wave B decisions are stated in §6.9, not here, because they gate D0-B rather than D0: the persistent-map mechanism for CommittedRoot (9.7) and the status-root overflow behaviour (9.8). Both were ruled on 2026-07-26 and are recorded there with their scope and conditions, on the same rule §11 applied to Wave A — a package must not discover a decision of this size while implementing against it.

9.1 and 9.2 together constitute contract review 2026-07-24-B, amending two frozen Phase 0 benchmark artifacts (bench/result-schema.json and bench/reference-hardware.toml). Per the Phase 0 freeze rules it must be recorded in doc/instance-throughput-rewrite-plan.md the way contract review 2026-07-24-A was, before A3's bundle emitter is written.

9.1 result-schema.json cannot validate a storage-primitive bundle — contract review 2026-07-24-B

bench/result-schema.json is frozen and its gate enum already includes storage_primitive. But the schema requires, unconditionally for every gate:

  • workload.validation_flags.* all const: true, including request_signature, replay, pack_hash_and_framing, complete_graph, authority_and_role, instance_policy, and repository_policy.
  • deployment.proxy, deployment.tls, deployment.systemd, deployment.cgroup with minLength: 1.

A P2 run is, by §3's own definition, against the "Production levcs-store API" — a layer that §5.1 explicitly forbids from making identity-role, policy, or federation decisions, and that runs in-process with no proxy or TLS. Reporting those flags as true would be false; reporting them as false fails the frozen schema. The deployment strings can be answered honestly ("none (in-process P2)" satisfies minLength), but validation_flags cannot.

Recommended resolution, as contract review 2026-07-24-B: make validation_flags conditional on gate, per flag rather than as a block, and add a required promotable: false. A blanket const: false for gate = "storage_primitive" would be wrong in the other direction: it would force durability_fence_before_response: false on the one bundle whose entire purpose is to certify that the fence precedes acknowledgment, and would deny typed_ref_cas, which the shard sequencer genuinely enforces.

For gate = "storage_primitive":

Flag Required value Why
durability_fence_before_response true The store is the fence. This is the claim under test.
typed_ref_cas true The sequencer performs the typed CAS against speculative state.
request_signature false No envelope at this layer.
replay false Replay guard is Phase 2 admission.
pack_hash_and_framing false No Pack; objects arrive already parsed.
outer_embedded_type_match false Parse-once overlay is Phase 2.
complete_graph false §5.1 forbids the store from traversing the graph.
authority_and_role false §5.1 forbids identity-role decisions.
instance_policy false Policy evaluation is Phase 2.
repository_policy false Policy evaluation is Phase 2.
fast_forward false The store consumes precomputed ancestry facts (§7 stage 9); it does not derive them.

This encodes §3's rule — "A storage primitive result can never be promoted to an instance throughput claim" — in the schema instead of in prose, and does so more precisely than a blanket denial: the bundle states exactly which checks the measured system performed. Note that $defs.validation_flags and the top-level object both set additionalProperties: false, so promotable must be added explicitly wherever it lands, not merely permitted.

Ruling: approved as written, including fast_forward in the false column — a validation flag must state what the measured system performed, and the sequencer consumes ancestry facts rather than deriving them. Two implementation constraints so that 2026-07-24-B does not weaken what it amends:

(a) Relaxing is not un-pinning. Making the nine flags conditional means removing their const: true from $defs.validation_flags. Every other gate's if/then branch — in_process_protocol, deployed_30k, deployed_60k, and the remaining gate values — must then re-pin all eleven flags to true explicitly. Otherwise the amendment silently un-pins validation for the P3/P4/P5 bundles, which is a strictly worse defect than the one it fixes: it would let a deployed-node result declare complete_graph: false and still validate. A schema test must assert that for every gate value other than storage_primitive, a bundle with any flag false fails validation.

(b) promotable is required at top level for every gate, const: false under storage_primitive and const: true otherwise — not a field that only storage-primitive bundles carry. An evaluator's refusal to promote then becomes a mechanical schema check rather than a lookup in prose, which is the entire point of moving the rule out of §3's text.

9.2 btrfs nodatacow for journal and segment directories

Decide before A1 and A2 write recovery, because it determines whether a torn tail read returns EIO (CoW, checksums on) or garbage (nodatacow, checksums off), and 3.8 step 5 must handle whichever is live. Recommendation: run nodatacow, and still handle EIO. Nodatacow makes preallocate-and-overwrite real, removes futile zstd work on incompressible data, and our frame digest already provides end-to-end integrity that is strictly stronger than a per-block csum for our purposes. Handling EIO anyway costs nothing and keeps the store correct on a default-mounted root, and it remains necessary under either choice because power_loss_protection = false means the device can lose acknowledged writes regardless of filesystem.

The same ruling must answer a second question the first draft left open: does chattr +C on shards/*/active and shards/*/segments amend the frozen hardware profile, or is it per-bundle metadata? reference-hardware.toml freezes mount_options including compress=zstd:3, and per-directory nodatacow overrides compression and checksums for exactly the files that carry the throughput. Recommendation: treat it as an amendment to bench/reference-hardware.toml — add a [profile.filesystem].store_directory_attributes field naming the attribute and the directories it applies to — folded into contract review 2026-07-24-B alongside 9.1, since both are benchmark-contract changes and both must land before A3's emitter. Per-bundle recording is then additionally required but is not sufficient: a profile that silently permits two different on-disk configurations is not a frozen profile.

Ruling: approved, including the profile amendment. Run nodatacow, handle EIO under either configuration, and add [profile.filesystem].store_directory_attributes to bench/reference-hardware.toml as part of 2026-07-24-B, recorded in the plan.

One condition: store-bench must verify the effective attributes at startup and refuse to run on mismatch, not merely record them. It reads the inode flags back for each configured store directory and compares them against the profile. Recording alone is insufficient for the same reason the profile amendment is necessary — a run on a silently copy-on-write-mounted root would otherwise emit a bundle claiming nodatacow, and the resulting number would be incomparable to every other P2 result while looking identical. The check belongs with the free-space precheck of 8.2 in the same refuse-to-start path.

9.3 Group-aware failpoint expectations

Confirm the reading in A3 deliverable 3: the frozen single-transaction oracle stays unchanged and levcs-store layers group_failpoint_expectation on top, proving degeneracy at group_len == 1. The alternative — extending oracle.rs — would require a Phase 0 contract review and regolden, for a concept that is a property of the storage engine rather than of the protocol. Recommendation: layer, do not extend.

Confirm also that the layered function returns the contiguous-prefix contract of A3 deliverable 3 rather than a per-frame classification. This is the substance of the decision, not a detail of it: a per-frame formulation is satisfied by an implementation that adopts a valid frame sitting after a hole, which is the single most consequential recovery bug this phase can ship.

Ruling: approved. Layer, do not extend the frozen oracle. The contiguous-prefix contract — ∃ pvictim_index splitting committed from absent, with nothing at or after the victim ever committed — is the operative content of the function.

One condition: the degeneracy proof at group_len == 1 must hold for every physical state classNoBytes, PartialFrame, WholeFrameUnfenced, and WholeFrameFenced — not only for the torn one. A degeneracy proof covering a single class does not establish that the layered function generalizes the oracle; it establishes that the two agree on one input.

9.4 payload_digest redundancy

3.3 carries both payload_digest (over the payload) and frame_digest (over everything). The cost is 32 bytes per frame and a second full pass over the payload — since the payload is nearly the whole frame, this roughly doubles store-side BLAKE3 work, from about 0.2 to about 0.4 of a core at 200 MB/s. That is still cheap on an 8-core box, and it buys the ability to validate a payload read by pread from the index without re-reading the trailer. Recommendation: keep it; revisit at Phase 5 if profiles say otherwise. Confirm in D0 because it is a frozen-byte decision. (The first draft priced this as "a few percent of BLAKE3 time", which described the 32 extra bytes hashed rather than the extra pass.)

Ruling: approved. Keep payload_digest at the corrected ~0.4-core price; freeze the field in D0.

9.5 P2 measurement without an instance

store-bench must generate the canonical workload's object graph itself (deterministic 1,024-byte blobs from the frozen seed, real Ed25519 commits, real authority) and construct ValidatedTransaction through the privileged constructor. Confirm this is acceptable and is not "a benchmark-only shortcut" in the Phase 0 exit sense: it is not, because the store is genuinely the system under test and the bundle declares validation_flags false and promotable: false under 9.1. But it should be stated explicitly rather than assumed.

Ruling: approved, with one tightening that makes the claim checkable instead of declared: the bundle's workload.seed and workload.generator must equal the frozen values in bench/workloads/small-commit.tomlseed = 126394451485337 and generator = "blake3-xof(seed || repo_ordinal_le || ref_ordinal_le || commit_ordinal_le)". The generator is already pinned in the frozen workload, so an evaluator can recompute every 1,024-byte blob from the bundle alone and verify it against the recovered store. That turns "the harness generated the canonical workload" from an assertion into a reproduction, which is what separates this from the benchmark-only shortcuts the Phase 0 exit criteria forbid. workload.generator is not currently a schema field; adding it is part of 2026-07-24-B.

9.6 Deferred to Phase 2: no public canonical codec for typed ref CAS

TypedRefCas, RefTarget, and RefMutation are frozen Phase 0 types with no public CanonicalCodec implementation, so levcs-store had to define its own physical encoding for the "complete typed ref CAS set" that scope 3.3 requires in every frame. That is a drift surface: two encodings of the same logical content, only one of which is frozen.

It is not Phase 1's to close. v2.rs is a frozen artifact and exposing public codecs for those types belongs to Phase 2 ProtocolV2, which owns protocol codecs. Phase 1 keeps its physical encoding and delegates ref-name validation to levcs_core::refs:: validate_ref_name — the same function the protocol uses — so the two cannot disagree about what a legal ref name is. When ProtocolV2 exposes the codec, the store's encoding must be validated against it. Recorded here so the obligation is not lost between phases.

10. Risks

  • Custom journal correctness (§13). If the crash matrix cannot be made deterministic, or recovery requires guessing, stop performance work and fix the format. The gate for this is section 5's charter, not the benchmark.
  • Index memory at sustained rates (§13). 8.2's arithmetic must be validated, not assumed; if entries cost materially more than 47 bytes, or run fan-out grows unbounded, redesign before tuning.
  • Wave A/Wave B seam. Eight failpoints, named in 4-A3, are unreachable until B1 exists, and all nine Wave A rows can assert only two of the five FailpointExpectation fields until then. The risk is that either set is quietly forgotten, and the first draft of this document demonstrated the failure mode by miscounting them. Mitigation: each Wave B row is a named pending-wave-b entry carrying a wave field, and the Wave A rows record which halves they assert; the fixture test requires both the pending set to be empty and every row to assert its complete expectation at Phase 1 exit — the same mechanism Phase 0 used for not-exercised adversarial rows. No count anywhere substitutes for the names.
  • drive.rs divergence. A test-only seam that appends frames without the sequencer can drift from the real append path, at which point the Wave A matrix certifies something the production writer does not do. Mitigation: drive.rs must call the same journal.rs group-append and fence functions engine.rs calls, adding only frame construction; a Wave B test asserts a group submitted through submit and the same group driven through drive.rs produce byte-identical journal contents.
  • The 8-core budget. 8.4 shows P2 headroom over 75k is modest. If P2 lands close to the bar, resist tuning the fence or the group ceiling to reach it; §13 forbids raising tuning parameters to hide overload, and P3 will only add load to the same cores.
  • Format churn after freeze. Every post-freeze frame change invalidates the goldens and the crash fixtures. The mitigation is the completeness of section 3 before Wave A starts, which is what section 9 is for.

11. Sequencing

D0  lead skeleton, frozen API, sys/failpoint shims, deps, decisions 9.1-9.5
     |
     +-- A1 JournalWriter  ---+
     +-- A2 RecoveryIndex  ---+--> Wave A freeze gate: check-phase1.sh + adversarial review
     +-- A3 StoreHarness   ---+          |
                                         v
                              Wave A frozen 2026-07-26 at 5ee9c6b
                                         |
D0-B  lead: roots.rs, completion.rs, RecoveredShard entry point, adoption handle,
      lib/deps/options/error/recovery amendments (11 items, 9 touching frozen files)
                                         |
                              D0-B frozen 2026-07-27 at 5111d65
                                         |
     +-- B1 NamespaceTxn   ---+
     +-- B3 StagingSessions ---+--> Wave B freeze gate + adversarial review
     +-- B4 StoreHarnessB  ---+          |
              B2 StorageReviewer ||      v
                              Phase 1 exit: P2 x3, full crash matrix, section 7 table

Wave B sequencing. D0-B gates everything, exactly as D0 gated Wave A, and for the reason the Wave A review demonstrated: CommittedRoot is the structure B1 publishes, B3 becomes visible through, B4 measures, and B2 reviews. Whoever defines it first defines it for all four, and if that is a package rather than the lead then there is no independent contract to review.

The three implementing packages are not equally unblocked. B1's committed-root publication path and StoreEngine::submit gate B4's re-pointed benchmark and all eight Wave B failpoint rows, so B1 should land the publication path before receipts and snapshots. B3 is the most independent — it needs the adoption seam from D0-B, meaning the ProjectionAdoption handle with its three-way outcome and reference-proof query, not merely a call signature — and once that is frozen it should start immediately rather than waiting on B1. B4's first deliverable, the two missing crash-matrix fault generators, depends on nothing in Wave B at all: it is Wave A work that Wave A did not do, and it should be finished before B1 has anything to test, so that B1 is tested by a matrix that can express the class of defect Wave A shipped.

Sections 9.1-9.6 are resolved, so Wave A was unblocked once D0 landed; 9.7 and 9.8 were ruled on 2026-07-26 in §6.9, and contract review 2026-07-26-A closed the EvidenceHandoffFailure conflict. D0-B is frozen at 5111d65 after a GATE_EXIT=0 run, so B1, B3, and B4 dispatch is unblocked. The item that was sequenced ahead of A3 — contract review 2026-07-24-B, the result-schema.json per-flag conditional with its re-pin requirement and the top-level promotable, the added workload.generator field, and reference-hardware.toml's store_directory_attributeswas applied and recorded in doc/instance-throughput-rewrite-plan.md before A3 wrote the bundle emitter, and was amended twice more during Wave A. It was never a Wave A deliverable; it was lead work alongside D0. Nothing is now sequenced ahead of any package.

Within Wave A the three packages are not equally unblocked. A1's format.rs and drive.rs bodies gate A3's crash driver and benchmark, and A1's frame codec gates A2's recovery scanner. A1 should therefore land format.rs and drive.rs first and in that order, before journal.rs internals, so A2 and A3 are not idle. A3's Wave A output is a working matrix over eight or nine rows and a P1-micro number, not a P2 result; the P2 bundle is a Phase 1 exit artifact that depends on B1.