B3 deliverables 6, 7 and 8. Each was an explicit NotImplemented stub whose
recorded blocker had lapsed; together they take a sealed session from a pin
nobody could hold to one recovery can resolve after the process that held it
is gone.
Finalize takes an adoption pin, and the state machine the deliverable
described could not be built as written. The pin is taken from Sealed rather
than Open, because the sealed manifest is what an adopter revalidates against.
A definitive pre-append failure returns the session to Sealed rather than
Open, because reconstruction reads that manifest's presence as the seal's own
commit point and would hand back Sealed on the next restart regardless -
returning a session to a state no restart can reproduce is the defect the
private busy/state split exists to prevent. Adopted is durable and is a state
the wording had no name for: adopted artifacts stay in staging/ with a
committed root pointing into them, so dropping the session record would make
the next reconstruction read the directory as an abandoned materialization and
reclaim committed content. The pin is made durable before it is issued; a
handle backed by an in-memory flag is an adoption capability with nothing
behind it.
Cleanup proves absence of reference, and absence is not enough on its own. A
sealed session a client is still finalizing and a pinned one whose frame may
be mid-append are both referenced by nothing, so the candidate set is the
adopted sessions alone. The unit of removal is the session directory rather
than the artifact: a manifest names every chunk and reconstruction refuses a
sealed session missing any ordinal, so removing the unreferenced half of a
directory trades a bounded leak for a root that fails to open. The supplied
root must have reached the adoption - a root captured earlier references none
of these artifacts because it predates them, and acting on that deletes a
directory the current root points into.
Recovery resolves a committed descriptor against the bytes on disk, not
against staging's cached idea of them. Every declared ordinal is read back and
its digest bound to the sealed record, the manifest's ordered list, and the
frame's own artifact-set digest, so a valid replacement chunk of identical
shape is refused rather than indexed. It resolves whole or not at all. A
location names the whole certified record, so every object in a chunk shares
that chunk's location and a reader validates the artifact before extracting
from its decoded vector.
Three frozen seams move, each recorded as a contract review:
ProjectionAdoptionOutcome::Adopted and resolve_committed both gain the
adoption's committed shard sequence, from whichever side made the frame
authoritative; and checkpoint-style proof-bounded replacement earns a third
caller for sys::rename_replace.
Contract reviews 2026-07-31-A through -D. Scope 6.5 and plan 8 are amended
where they specified behaviour that could not be built.
Deliverables 6-8 are implemented and not yet operated: no production path
calls begin, finalize, or adopt_projection, and expire still has no scheduler.
Both carry-forwards stand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKzM69CHmBuDcA3qN1jFdh
Lead fix, folded in. The previous repair accepted any `#[cfg(test)]` item whose
first line ended in `{`, which is the same latch in another spelling: a
semicolon-terminated `static` or `const` can open a block initializer there and
close with `};`, so the exemption ran past it into the next function. Verified
by reverting the condition — the scanner returned no offender at all for a
`std::fs::write` in the function following a `LazyLock` initializer.
Only `mod` and `impl` are accepted as braced shapes, being the two the crate
actually uses. Everything else fails the guard by name rather than being
bounded by a brace that may not be the item's own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKzM69CHmBuDcA3qN1jFdh
Two findings against the previous commit, both upheld.
The funnel guard's test exemption could still latch. Ending it at the next
column-zero `}` is right for a braced item and wrong for every other shape:
after `#[cfg(test)] use crate::test_support;` the first such brace belongs to
the *next* function, so all of it went unscanned. The scanner now reads the
attributed item's shape — braced items are exempt to their closing brace,
semicolon-terminated items exempt only themselves, and any third shape,
including an item header rustfmt split across lines, fails the guard. A shape
it cannot bound is not a shape it may assume is harmless.
It is now a function over `&str` with synthetic tests, which is the more
important half. Mutating real sources only probes the shapes those sources
happen to contain: no file in this crate has a semicolon-terminated
`#[cfg(test)]` item followed by production code, so no mutation of a real
file could have produced this defect. Charter item 8's analogue for tooling.
The two-owner claim was overstated. The regression arranges its wrong-typed
name by replacing `LOCK` under a live holder — and replacing it with a fresh
*regular* file succeeds just as well, since both opens are then of a regular
file at the right name with nothing to tell them apart. The type check closes
"the name already resolves to the wrong kind of object", the operator-error
and stale-state case; it does not close "the name is replaced under a
holder", and no check at this layer can.
So scope 3.1 now separates the two, says which is in scope, and states the
replacement case as an explicit deployment assumption rather than leaving it
implied: anything able to replace `LOCK` can equally unlink a journal, so
advisory locking was never the boundary that would stop it. The assumption is
pinned by a test asserting the current behavior on purpose — if a stable
locking object is ever adopted, that test is meant to fail, and the failure
is the signal that the documented assumption changed. §3.1 records locking
the root directory as the candidate and what it would cost.
Also exact rather than caveated: a Unix socket fails `open(2)` with `ENXIO`
before any `fstat`, so it surfaced as `Io` while the documentation promised
`UnrecognizedLayout`. `ENXIO` and `EISDIR` both now mean "not a regular
file", and the socket is one of four occupants the test loop covers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKzM69CHmBuDcA3qN1jFdh
`flock` locks the inode a descriptor reached, not the name that was asked
for, and `lock_root` opened `<root>/LOCK` with `create(true)` — a
follow-through open. A symlink at that name therefore put the root lock on
a foreign inode and left the root's own lock file unlocked, so a second
process arriving at the same root took the lock as well: two owners, each
believing it held one root exclusively. Nothing is destroyed and everything
downstream is permitted to race, which is why this is scheduled on its own
rather than folded into the remaining symlink work.
`classify_root` already refused a non-regular `LOCK`, but only on the
`StoreEngine::open` path. `RecoverySession::open` and `drive.rs` reach
`lock_root` directly with no classification ahead of them, so the guarantee
had to move down to the open itself.
`sys.rs` gains a third no-follow primitive for the shape the other two do
not cover — a name the store must own and keep, adopting an existing regular
file and creating an absent one, never truncating. The type is established
by `fstat` on the descriptor already held, before `flock` is attempted, so a
refused name is never locked even momentarily. A non-regular occupant is
`UnrecognizedLayout`, not `AlreadyLocked`: the root is malformed, not busy.
Measured on a reverted copy, three distinct failures rather than one:
- two owners of one root, with the first lock still held;
- a dangling link at `LOCK` created a file outside the root;
- a fifo at `LOCK` returned `Ok(RootLock)`, the store reporting that it
held the root lock on a pipe. That one was found by writing the test
for the type check, not predicted.
The funnel guard needed amending to accept these tests, and the reason it
did is a defect in the guard: it exempted test code by matching the literal
name `mod tests`, so the two modules named otherwise were scanned as
production code while a file could have evaded the guard entirely by naming
a module `tests`. It now keys on the `#[cfg(test)]` attribute and, unlike
before, the exemption ends at the module's closing brace — code appended
after a test module used to be unscanned. Both directions mutation-checked.
Disclosed, not closed: a device node at `LOCK` still receives one
`O_NONBLOCK` open before the `fstat` refuses it. The three remaining
symlink hazards in `segment.rs` stand unfixed; contract review 2026-07-29-A
records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XKzM69CHmBuDcA3qN1jFdh
Scope 6.4 deliverable 1, startup state 1. StoreEngine::open now creates a
root rather than refusing to, and every harness that measured a store
production did not build is re-pointed at it. States 3 and 4 still refuse.
The four startup states are distinguished by one read-only classification
that creates nothing. A FORMAT entry means state 2 on the strength of the
name alone, because a FORMAT that does not decode is still FORMAT and
treating an unreadable marker as "no marker, therefore empty" would
authorize building a fresh tree over a populated root. A root holding only
LOCK is empty, since lock_root creates that file as a side effect of asking
whether the root is busy. The signer check moved above every root access,
because once an absent root is initialized rather than refused, a signerless
configuration would otherwise create a tree, write FORMAT and fsync the
parent before failing -- a configuration error must not leave a root behind.
Three defects found in review, all fixed here rather than deferred.
A deeply absent path was created with create_dir_all and only its immediate
parent fenced, so open could report success over ancestors a power loss
could take. Ancestors are now created one at a time -- so what this call
created is exactly what it fences, and a racing creator surfaces as
AlreadyExists rather than being absorbed -- and fenced deepest-first, since
a directory entry lives in the parent that names it and the reverse order
can leave a fenced parent naming an unfenced child.
An interrupted initialization was unrecoverable: any tree residue classified
the root as non-empty-without-FORMAT and it was refused forever, with a
message about legacy layouts that had nothing to do with what happened. A
root being built now carries an INITIALIZING marker, installed by rename as
the first durable act and removed as the last, so a durable partial tree
always has a durable marker beside it and every crash window is resumable.
Recognizing it requires both a byte-exact marker this crate alone writes and
every entry in the root drawn from a closed set of names this store
invented, so a foreign layout cannot be mistaken for abandoned
initialization and overwritten -- the direction that matters, since refusing
a resumable root costs an operator time and overwriting a real one costs
their data. Sibling staging with atomic installation was the alternative and
is structurally blocked: LOCK lives inside the root, so the root must exist
before any mutation can be serialized, and renaming a tree onto a directory
containing LOCK fails ENOTEMPTY.
The classifier then ignored INITIALIZING.tmp by name regardless of type or
contents, and initialization opened that name with create plus truncate. An
operator's file there was destroyed silently, and a symlink there truncated
a file outside the root to 27 bytes and then removed the link -- destroying
data the store never owned and erasing the evidence, while open returned Ok
and reported a working store. The justification for ignoring the name was
that only this path could have written it, which is circular: that is the
claim the classifier runs in order to establish. Every entry is now judged
by lstat type before anything opens it, the temporary marker is validated as
an exact regular marker or refused, and installation is create-new rather
than create-truncate. Contract review 2026-07-28-D records the two no-follow
open primitives this added to the frozen sys.rs, and the four further
symlink hazards in segment.rs that are recorded rather than fixed -- the
first of which lets two processes believe they hold one root lock.
A fifo at that name made the pre-fix open block forever: one mkfifo in a
configured root was an unbounded startup hang, not only a data hazard.
The engine and the drive seam are now asserted to recover one crash image
identically, closing a gap that was true by construction and untested.
Charter item 8 applied to the harness: the in-crate test helper no longer
calls segment::initialize_root, so every writer test builds its root through
open; the ROOT_SEEDED_BY_NON_PRODUCTION_PATH disclosure is retired; and the
fixture's root_seeded_by becomes a stable token matched by exact equality,
with the history moved to an adjacent reason field -- a substring match
passes on a value that has drifted to mean something else.
The d0 contract test asserting open returns NotImplemented for any valid
configuration is obsoleted by this deliverable and replaced with the
stronger property: a signerless configuration is refused and leaves no root
behind. It moves off a fixed /tmp path, which under the old check ordering
would have created a real store root on every gate run on every machine.
scripts/check-phase1.sh GATE_EXIT=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.6 deliverables 1-3, partial. This is evidence, not a freeze: no
Wave B deliverable set is complete and the throughput figure below is not
a publishable bundle.
The revert-and-observe-red acceptance for cd37f8b's two generators was
never actually performed -- the tests asserted it in their doc comments,
which is a claim. It is performed now, in a throwaway copy, one mutation
at a time, and each reddens exactly one test for the right reason. With
segment frame validation reverted to pushing footer offsets straight into
the adopted set, the store still opens and recovery still completes: it
reports recovery_ok=true and adopts the corrupted frame as authority, so
the red is exit 0 where 65 is required rather than a store that failed to
open. With the shard-index check removed from journal binding, the moved
journal is adopted whole and root_uuid alone does not catch it -- the Wave
A blocker reproduced.
The eight Wave B failpoint rows now drive StoreEngine::submit in-process
with the full six-field expectation asserted against the frozen oracle,
field by field rather than by struct comparison. No pending-wave-b row
remains. The four fields Wave A could not reach come from four independent
observations: the store naming itself poisoned, the two-root status read,
a receipt obtainable at all, and a different transaction submitted to the
same shard before any reopen. The last two are not the same question -- a
shard can be unpoisoned and still refuse a later append because its writer
thread died, which is exactly what phase-aware panic ownership fixed and
what this now checks from outside. AfterRootCasBeforeWaiterWake is
observed as a genuinely hung submit whose receipt is still retrievable,
so a waiter that never wakes is proved to be a hung request rather than an
absent transaction. Every row also asserts its group's fence count against
the public durability snapshot.
Two flake campaigns were measured rather than rerun: 4 failures in 40, then
7 in 40, from two distinct causes. One is a finding -- publishing a group
adds an index delta layer and none are sealed, so submit refuses after
exactly max_index_runs publications for the life of an engine. Both fixed
structurally; 200/200 and 40/40 after.
store-bench emit-skeleton now defaults to the submit path, with the journal
seam retained under --path drive for comparison. The signer is real, the
ref CAS is evaluated by the sequencer, and objects_new is summed from the
store's own receipts rather than multiplied out of the transaction count.
Explicit blockers, retained rather than worked around:
- StoreEngine::open still refuses startup state 1, so the benchmark seeds
its root by a non-production path. Seeding a store off the production
path in order to measure the production path is the charter item 8
smell; the disclosure is recorded in the fixture, a const doc, and the
module docs, and a test asserts open still refuses so it cannot go
stale in the safe direction.
- P2 is blocked three ways -- checkpointing disabled, no steady state
under the index-run ceiling, and no warmup/repetition/trim protocol.
The rate emitted is a debug build on tmpfs, marked preliminary.
- The 100 SIGKILL cycles still drive the journal seam, so kill -9 never
lands inside a real publication.
- Four schema claims became earnable and are requested, not emitted;
bench/result-schema.json is lead-owned.
- Reopen after close needs a bounded, measured, reported wait, because
the root LOCK outlives StoreEngine::drop. Diagnosed since as fork/exec
inheritance of the lock file description; the fix belongs in the lock
primitive, and this wait is removed when it lands.
scripts/check-phase1.sh GATE_EXIT=0; verify-store-recovery.sh 100 cycles,
recovery_failures=0, acknowledged_loss=0, torn_transactions=0,
repeated_adoptions=0, bundle=schema-valid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope 6.5 deliverables 1-5: what a staging session is on disk, what it
costs, when it dies. The instance-layer half of plan §8 — identity,
policy, ProjectionCore, the v2 routes — is deliberately absent; B3 binds
and exposes the fields those checks key on and evaluates none of them.
Two findings from the B2 review shaped the result more than the original
deliverables did.
Restart was in-memory handle reuse. open() built a fresh registry and
never read the filesystem, so after a real reopen an existing session ID
was admitted as new -- and if materialization then found the old
directory, error cleanup could unlink a durable session. That is data loss
reachable from an ordinary restart plus one error. open() now scans,
validates, and reconstructs sessions, chunk indexes, and the whole
occupancy account before returning, so a reconstructed ID is occupied and
refused as a conflict long before the error path; and that path no longer
calls remove_dir_all. Either change alone closes the loss.
A session is final iff its directory holds a valid session record for its
own ID, installed by rename_noreplace over fenced, digest-checked bytes,
so the final name only ever appears atomically over complete data. A
directory without one is abandoned materialization -- a crash between
mkdir and that rename, unaccounted and unreferenceable. The two states
share no code path, and reclaimed abandonments count on their own counter
so they can never be read as aborts or expiries.
Global quotas were per-handle. Every open() built an independent registry
outside the root LOCK, so two handles admitted twice the global limit and
the atomic-insertion work bought nothing across them. Construction now
requires proof of the held root lock and refuses a second in-process
instance, making two accountants on one root inexpressible rather than
discouraged.
Also: bounds are enforced atomically with insertion under one mutex with
no read-then-decide path, refused as typed LimitExceeded or Overloaded and
never by eviction; the directory-sync test pinned two syncs when the first
session in a shard needs three, a counter assertion that encoded the bug;
cleanup now validates a whole directory before unlinking anything, rather
than discovering a surprise midway through destroying a live session; and
artifact I/O moved off the registry mutex onto maintenance workers, with
the calling thread asserted to hold no guard rather than documented not to.
Deliverables 6-8 are explicit NotImplemented naming themselves.
StagedSessionState omits Finalizing, so deliverable 6 will fail to compile
at exactly the expiry and abort sites that must learn about a pin.
Carry-forwards recorded in §6.5, not closed: no production path begins a
session, so the sealed-invisibility acceptance stays ignored with both
blockers named; and expire() has no scheduler, so session age is a bound
enforced when asked and never asked.
116 library tests, 27 staging tests, 1 intentionally ignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Wave A review recorded that the crash matrix structurally could not
express its own first blocker: no failpoint corrupts a frame inside a
sealed segment, and none moves a journal between shards of one root. A
green matrix on a defect it cannot represent is the same trap as a helper
nothing calls, so the finding was carried forward rather than closed.
Add the two physical crash-image generators as a `damage` subcommand on
the crash driver, and two tests that drive them through the same
production `reconcile` path the matrix and the recovery script use.
- sealed-frame-corruption flips one payload byte in the first frame of
a segment the drive actually sealed and installed. The footer stays
structurally valid, so only frame verification can reject it.
- cross-shard-journal-movement relocates shard 1's active journal under
shard 0 of the same root. Root UUID validation cannot see this; the
journal header's shard index must be bound to the directory being
opened.
Both generators mutate a production-written image and fence the mutation;
neither synthesizes a footer, frame, manifest, or checksum, so a passing
test cannot be an artifact of the harness agreeing with itself. Each
source image is required to be unambiguous — exactly one segment or
journal — so the result does not depend on directory iteration order.
The assertions pin the refusal to its own cause rather than to any
non-zero exit: the frame case must fail on the digest recomputation and
the movement case on the shard binding, and neither may publish a partial
adoption result.
crash_matrix: 26 passed. scripts/check-phase1.sh green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Freezes the Wave A interfaces, frame format, durability ordering, and
crash/fault fixtures per doc/phase1-storage-spine-scope.md section 5. Wave B
may now start against these.
D0 lands crates/levcs-store: the sealed public API, the file-ownership split,
the single durability syscall funnel with its counters and fault hooks, the
17-entry failpoint registry in compiler-enforced correspondence with
oracle::AppendFailpoint, and the journal-level drive seam. Wave A lands the
frame codec and journal/segment lifecycle (A1), the recovery index and
checkpoints (A2), and the durable-ingest benchmark and crash harness (A3).
The adversarial review found five defects behind a green gate, three of them
blockers, all closed here. Two were the same shape: drive::reopen_through_
recovery had reimplemented a simplified recovery and called none of recovery.rs
-- so it adopted segment footer sequences without validating frame bytes,
journal_id, or root_uuid, and it double-adopted interrupted seals. The seam
between two packages was untested precisely because each package's own tests
passed. reopen_through_recovery now delegates rather than decides, and
DriveRecovery carries the recovery report verbatim so tests assert the
disposition and not merely its effect: a double adoption and a correct replay
produce the same adopted set, which is how the defect stayed invisible.
Also closed: store-bench now schema-validates its own emitted artifact with a
five-mutation negative control instead of matching JSON substrings; the ACK
reconciler distinguishes duplicates and regressions from forward gaps;
checkpoint writes and a journal truncation are routed through the durability
funnel, whose guard now covers writes and truncations rather than only sync,
rename, and unlink.
bench/result-schema.json is amended (contract review 2026-07-24-B, second and
third amendments): per-gate latency ceilings conditional on outcome so a failed
run is representable, and the verification claims split per gate so a storage
run cannot certify an object graph it never touches. Not-applicable claims are
forbidden rather than falsified; applicable-but-not-performed report false.
Every relaxation is re-pinned in the else branch and asserted member by member,
after an edit in this series silently un-pinned all eleven validation flags and
was caught only by revalidating against constructed bundles.
Arming the fault registry now requires a FaultSerial token, so the invariant is
a compile error rather than a comment. The file where this was diagnosed
carried a header saying it was deliberately the only test in it, and a second
test had been added under that comment anyway -- an 8-in-40 failure rate that
read as flakiness.
Evidence at this commit: check-phase1.sh GATE_EXIT=0 across all four feature
configurations, 124 test binaries, zero failures; verify-store-recovery.sh
--cycles 100 with recovery_failures=0, acknowledged_loss=0,
torn_transactions=0, repeated_adoptions=0, bundle=schema-valid; recovery_eio
40/40 at four test threads; golden corpus byte-stable; fmt clean.
Carry-forwards, explicitly not Wave A blockers and recorded in scope section 5:
extend the crash matrix to generate sealed-frame corruption and cross-shard
journal movement, since it structurally cannot express the class the first
blocker belonged to; and wire GroupBuilder through B1's production path, since
deliverable 4-A1.2 is presently asserted only over a type nothing calls.
Charter item 9 -- ask every package what of its work is correct but uncalled --
is accepted for every subsequent wave. It, not the review, found the class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG
Levi Neuwirth