LeVCS/doc/instance-throughput-rewrite...

2605 lines
245 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# LeVCS instance throughput rewrite plan
**Status:** Approved for implementation; Phase 0 contracts frozen
**Date:** 2026-07-20
**Owner:** Main implementation lead
**Target:** 30,000 commits/s minimum and 60,000 commits/s release target, durably acknowledged end to end
No production code should change for this plan until this document is approved. The rewrite is deliberately scheduled before the broader instance workflow surface so that review, CI, mirroring, hooks, and future workflow objects are built on the measured transaction and event model rather than around the current loose-file behavior.
### Normative language
Architecture, protocol, acceptance, risk, and definition-of-done sections specify contracts: **MUST**, **MUST NOT**, **SHOULD**, and **MAY** carry their RFC 2119 meanings; uncapitalized “must” and “may not” are equally normative in prose, tables, and invariants. Section 12 is intentionally imperative because it is the execution procedure for the lead and subagents. Explanatory and historical passages describe rationale rather than adding hidden requirements.
## 1. Executive decision
Build a new node-wide, namespace-isolated storage and ingestion subsystem for `levcs-instance`.
The core design is:
1. A unified append-only transaction journal carries new object bytes, repository creation, every ref compare-and-swap, explicit authority movement, typed accepted-source evidence, transaction identity, and the durable receipt.
2. Complete transaction frames are batched across repositories and covered by one durability fence. A journal becomes an immutable segment when sealed; there is no second hot-path WAL write.
3. Repository state is sharded by `repo_id`. Each shard has one short mutation/sequencing lane; parsing, hashing, graph validation, signatures, policy checks, and most ancestry work run concurrently outside it.
4. A success response is sent only after the transaction frame is durable and the committed snapshot is published.
5. Push and init move to a signed binary v2 envelope. The envelope precedes an otherwise unchanged Pack v1 stream and binds repository identity, a stable operation ID, freshness, authority CAS, typed ref CAS operations, and the exact pack length/hash.
6. Every newly reachable object is parsed and validated through one staged overlay. Authority chains are pinned to the repository's stored genesis/current authority. Pack type mismatches, missing closure, unreachable extras, stale authority, and policy read/parse failures reject the whole transaction.
7. Push, init, mirror apply, instance migration, and future instance-side imports use the same transaction service. Byte-exact offline restore is the sole mutation-path exception: it installs an already verified exported generation without creating transactions or events. No other instance component may write objects or refs.
### Scope boundary
This rewrite replaces instance storage. Local working repositories continue using the current `levcs-core::ObjectStore`, `Refs`, and `.levcs/` loose layout. They have different latency, migration, worktree-index, and GC requirements and do not benefit from cross-repository group commit. The CLI and client will change where required for v2 federation, correct CAS construction, and receive-side verification, but local on-disk migration is not part of this program.
Existing instance roots receive one offline, validated loose-to-segment migrator. The runtime will not dual-read or dual-write old and new formats.
### Explicit non-goals
- PRs, review threads, issues, search, CI orchestration, and the web UI.
- Changing Blob, Tree, Commit, Release, Authority, merge-record, signature, or BLAKE3 object bytes.
- Replacing Pack v1 for object exchange in this rewrite.
- Synchronous federation quorum durability. Source acknowledgment remains local-node durable; replication remains asynchronous.
- Optimizing one contended branch beyond its required linear CAS semantics.
- Cross-repository physical deduplication in v2. Namespace isolation takes priority.
- A compatibility v1 POST ingestion handler.
## 2. Why the current path must be replaced
The measured current path is dominated by one file sync per loose object:
- One 1 KiB object write on persistent Btrfs/NVMe: approximately 4.88 ms.
- One small three-object commit in one repository: approximately 66.7 commits/s.
- The same workload across 16 repositories: approximately 396 commits/s.
- A single append log with one sync per batch measured approximately 6,395 records/s at batch 32, 24,163/s at batch 128, and 79,005/s at batch 512.
- Ed25519 verification measured approximately 39,789 verifications/s/core.
The implementation explains those results:
- `crates/levcs-core/src/store.rs::ObjectStore::write_at` hashes, creates, writes, calls `sync_all`, and renames every new object. It does not fsync the containing directory after rename.
- `crates/levcs-core/src/refs.rs::atomic_write` writes and renames one ref at a time with neither file nor directory sync.
- `crates/levcs-instance/src/lib.rs::handle_push` holds a blocking per-repository mutex while writing every object, validating, walking history, and updating refs sequentially.
- Object writes occur before semantic validation, so a rejected push leaves durable junk.
- Only declared ref tips receive full identity verification. Intermediate commits, ordinary tree/blob closure, unreachable entries, and declared-versus-embedded object types are not comprehensively validated.
- Manifest authority is client-selected and is not pinned to the repository's initialized genesis/current authority. A stale or foreign valid authority chain can be considered.
- Multiple refs and `refs/authority/current` are separately visible and separately fallible.
- `handle_init` creates a skeleton, genesis object, and two refs as separate operations without a catalog transaction.
- `sync_mirror` bypasses the push lock and independently writes objects and refs.
- The process-wide nonce mutex has a 600-second live set and periodically scans the entire map.
- Axum and the blocking client materialize whole packs and duplicate request bodies.
The rewrite is therefore a correctness rewrite with a throughput consequence. Removing `fsync` or weakening verification is not an acceptable path to the target.
## 3. Performance claim and canonical workload
All headline rates mean **new, valid, durably acknowledged commits per second**. They never mean objects/s, requests/s, offered load, duplicates, rejected traffic, or a storage primitive rate.
### Canonical small commit
Each counted commit must:
- Be signed by a real authority member with the required role.
- Add a unique deterministic 1,024-byte pseudo-random Blob.
- Add a canonical one-file Tree referencing that Blob.
- Add a signed Commit referencing the Tree, current authority, and previous tip of its assigned ref.
- Be reachable from a successfully committed typed branch update.
- Pass request/envelope signature, replay, hash, framing, full graph, authority, role, instance policy, repository policy, CAS, and fast-forward checks.
- Reach a response only after the storage transaction's durable sequence is fenced.
Record actual raw, Pack-compressed, application, and wire bytes in every result.
### Headline topologies
- `many-ref`: one repository, 1,024 active agent refs, 64 persistent clients; uniform and Zipf(0.9) ref selection.
- `many-repo`: 256 repositories × 16 refs, 64 persistent clients; uniform and Zipf(0.9) repository selection.
Packs may contain 1, 8, 32, or at most 64 commits. The 60k claim may use any batch up to 64, but batch size and request rate must be published. P4 minimum must independently pass ≥30k commits/s with batch=1 in both headline topologies. P4 release must pass ≥60k commits/s at a published batch≤64 and must also repeat the ≥30k batch=1 gate. Batch-1 and batched latency/request-rate results are separate verdict inputs and may not substitute for one another.
The two batch dimensions are independent. The protocol cap is at most 64 commits in one client push. The storage writer may group up to 512 already validated transactions from many clients into one durability fence. A batch-1 swarm therefore still benefits from cross-request group commit; the 512-transaction append-log result does not authorize a 512-commit client pack.
A single hot ref is also measured but is not a headline topology. It is necessarily ordered and CAS-conflict-prone.
### Release gates
| Gate | Environment | Required result |
|---|---|---|
| P0 correctness | PR suite, properties, fault fixtures | Zero invariant failures; rejected operations expose nothing |
| P1 micro | Codec, hash, signature, journal framing | No unexplained >10% same-host regression; diagnostic only |
| P2 storage transaction | Production `levcs-store` API; 5 min warmup + 15 min measured ×3 | ≥75k canonical commits/s; transaction payload≤64 commits, writer fence group≤512 transactions; p99≤50 ms; durable recovery oracle passes |
| P3 in-process protocol | Real router and all checks; 5 min warmup + 15 min measured ×3 | ≥66k commits/s batched and ≥33k batch=1 in both headline topologies; p50≤20 ms, p95≤50 ms, p99≤100 ms |
| P4 minimum deployed node | Release binary, systemd, Caddy/TLS; 10 min warmup + 30 min measured ×3 | ≥30k commits/s at batch=1 in both headline topologies with the same latency rules |
| P4 release deployed node | Canonical release hardware, systemd, Caddy/TLS; same duration ×3 | ≥60k commits/s at published batch≤64 plus ≥30k batch=1 in both topologies; every run passes; Caddy degradation ≤10% from direct |
| P5 minimum soak | Minimum node, each topology, batch=1 at ≥30k for two hours | Throughput/latency windows and all numeric resource ceilings pass |
| P5 release soak | Release node, each topology, ≥60k for two hours | Crosses at least two checkpoint, compaction, and receipt-expiry cycles; all release and resource gates pass |
| P5 resilience | Deterministic failpoints, random kills, power loss, overload, combined background work | Zero acknowledged loss/torn state; recovery, overload, compaction, federation gates pass |
The release hardware profiles and their complete CPU, RAM, NVMe, filesystem, cache/barrier, NIC, proxy, and kernel configuration must be frozen before P4. A storage primitive result can never be promoted to an instance throughput claim.
At P4/P5, ≥95% of one-minute windows must meet the target and no window may fall below 90%. Aggregate CPU must remain ≤85% and sustained storage utilization ≤90%. `memory.current` must remain ≤80% of the deployed cgroup limit with no swap or OOM event; open FDs must remain ≤80% of `LimitNOFILE`; every queue byte/count cap must hold; compaction debt must return below its low watermark each cycle. Equal-load beginning/end windows may show no positive growth in replay entries, expired receipts, leases, FDs, or non-live index state. Archive every configured ceiling, peak, and time series.
## 4. Non-negotiable invariants
### Identity and object invariants
1. Raw serialized objects and their BLAKE3 IDs remain byte-for-byte unchanged across local repositories, clients, instance segments, mirrors, migration, compaction, backup, and restore.
2. Repository initialization permanently binds `repo_id` and the genesis authority hash in the instance catalog.
3. Every native authority chain terminates at the repository's exact genesis and follows its current authority or one explicit valid successor CAS. The sole exception is a typed `ForeignForkBoundary`: the destination fork Commit is native and cites the destination genesis, while its one source-parent subgraph is independently authenticated to the explicitly bound source repo/genesis and never participates in destination authority CAS.
4. Every newly introduced reachable Commit, Release, and Authority has all required signatures and roles checked. Release verification must require the declared release signer semantics used by `sign_release`.
5. Validation input and Full projection require every Tree/object edge with the expected embedded type. Reduced projections may omit only the explicitly enumerated, already-validated boundary edges carried in signed projection evidence; omitted IDs are never namespace members or advertised as readable. Pack entry type must always equal parsed object type.
6. Instance and repository merge policy are evaluated fail-closed for every newly exposed commit.
7. Unreachable extra pack objects are rejected. Normal push never grants membership to pre-uploaded bytes. Large initial mirror, full-fork, and network-migration projections use only the bounded, invisible projection-staging protocol in §8; it is not an ingestion loophole and cannot expose objects before one final atomic publication.
### Transaction invariants
1. Repository creation, new object visibility, all ref CAS operations, current-authority movement, authenticated typed source evidence, and receipt are one logical transaction.
2. Multi-ref updates are entirely old or entirely new, including after crashes.
3. A success response means every reachable object and ref in the receipt survives immediate power loss under the documented storage stack.
4. An unacknowledged transaction may recover wholly present or wholly absent; it may never be torn.
5. Same-ref CAS has exactly one winner. Disjoint refs may validate concurrently and group into the same durability fence.
6. A failed request exposes no staged object, ref, transaction event, or dedupe hit as committed state.
7. Operation idempotency and server-enforced ID uniqueness are window-scoped, but an accepted operation may not age out while its outcome is unresolved. While an in-flight, resolving, or retained terminal record exists through the signed `retry_until` deadline, same ID/same stable digest attaches or returns the original receipt and same ID/different digest rejects. A request still before append when that deadline passes is definitively rejected as `ReceiptExpired`, appends nothing, and removes its transient reservation; this rejection need not create durable status state because the signed envelope itself is permanently expired. Once append begins, the ID/digest remains reserved and status remains `Resolving` until recovery determines committed versus absent, even beyond the signed deadline. For a committed outcome compute, with checked arithmetic, `receipt_visible_until = max(retry_until, first_receipt_visibility_time + terminal_status_grace)`. The first visibility time lives in committed/checkpoint state; if a crash occurs before it is durably checkpointed, recovery resets it to the later recovery-publication time and therefore only extends retention. Compaction may transition the receipt to `Expired` only after a checkpoint/baseline durably records that deadline; with another checked addition, the tombstone then remains through `receipt_visible_until + status_tombstone_grace`. If recovery proves that no complete frame exists, it releases the reservation and returns definitive `Unknown`/absent-retriable while the signed deadline is live; after the signed deadline the envelope itself returns 410. Because a power loss may leave no durable bytes naming that absent attempt, server-enforced mismatch detection ends with that proved-absent reservation; clients nevertheless MUST retry an ambiguous attempt only with the same ID/digest. Other definitive pre-append validation/policy/conflict failures likewise create no durable ID reservation. Durable committed receipt/digest/retention state survives checkpoint, compaction, backup, and restore. A retry envelope presented after its own signed deadline returns non-retriable `ReceiptExpired`/HTTP 410, but that does not hide a still-resolving or retained committed outcome from authenticated status lookup. Only after the committed tombstone ends does status-by-ID return 404 and permit reuse of that committed ID. Beyond that point the server does not preserve lifetime uniqueness: a fresh, newly signed operation may reuse the ID and is explicitly a new operation. Clients MUST generate random IDs and MUST NOT otherwise reuse them. The invariant does not claim that `operation_id` alone distinguishes historical operations after expiry.
8. Physical `shard_sequence`, segment offsets, checkpoint generations, and shard topology are internal. Logical receipts/events and federation cursors expose per-repository `repo_sequence`; the two sequence domains are never interchangeable.
### Resource invariants
- Request wire/decoded bytes, missing-object IDs, receive-spool memory/disk bytes and FDs, object/ref/graph/authority counts, validation jobs, queued transaction bytes, in-flight/status entries, replay entries/horizon, receipt/terminal/tombstone retention, open segments, snapshot base/request pins, projection-staging sessions/artifacts, and backup leases are bounded and observable.
- Blocking temporary/storage filesystem work and CPU validation never run on Tokio I/O workers; only the receive-spool pool performs receive-spool I/O. Maintenance workers may write and sync only uniquely named, unreferenced checkpoint/compaction/projection-staging artifacts. The shard owner alone assigns manifest-visible names, adopts those artifacts, updates `CURRENT`, publishes committed roots, and unlinks referenced or superseded shard files.
- Overload rejects early with documented retry guidance. It does not turn memory into a queue.
- Namespace membership, not global object existence, controls reads. Identical bytes in private repository A do not make an object readable through repository B.
## 5. Target architecture
```mermaid
flowchart LR
A[HTTP v2 stream] --> B[Admission and signed envelope]
B --> C[Bounded Pack v1 stream]
C --> D[Parse-once staged object index]
D --> E[Parallel graph identity and policy validation]
E --> F[Validated transaction plan]
F --> G[Shard sequencer and final CAS]
G --> H[Group append]
H --> I[One durability fence]
I --> J[Publish snapshot and event]
J --> K[Durable receipt]
```
### 5.1 New `levcs-store` crate
Add a workspace crate at `crates/levcs-store`. It depends on `levcs-core` for object IDs/framing, but contains no HTTP, identity-role, merge-policy, or federation decisions. It also depends on `levcs-protocol`, because §5.2 requires the shard sequencer itself to assign `repo_sequence` and `previous_event_digest`, compute the event/state digests, and place a signed `CommittedTransactionV1` in the frame; it cannot treat those contracts as opaque bytes. It takes no direct dependency on and makes no call into `levcs-identity` — signing is injected through a `CommitEvidenceSigner` and a test enforces the absence of any `levcs_identity::` path in the crate. The rule is about decisions and calls, not about the transitive link edge that `levcs-protocol` brings in.
Proposed modules:
- `format.rs`: format marker, frame/segment/checkpoint codecs, versioning, checksums, golden vectors.
- `engine.rs`: `StoreEngine`, shard routing, startup/recovery, lifecycle.
- `transaction.rs`: namespace creation, object records, typed ref CAS, authority transition, operation evidence, receipts, error taxonomy.
- `journal.rs`: active append journal, group formation, `write_vectored`, sequence assignment, sync, rotation.
- `segment.rs`: immutable segment reader, bounded `pread`, open-FD cache, integrity checks.
- `index.rs`: `(namespace, ObjectId) -> location/type/transaction sequence`, namespace catalog, transaction receipt index.
- `snapshot.rs`: committed repository snapshot and ref view.
- `checkpoint.rs`: rebuildable object/ref/receipt checkpoints and atomic installation.
- `recovery.rs`: tail validation/truncation, replay, checkpoint fallback, readiness result.
- `compaction.rs`: safe-point, live-copy, generation install, reader/backup pinning, reclamation.
- `backup.rs`: named checkpoint export lease and restore verification.
- `staging.rs`: bounded invisible projection sessions/chunks, immutable manifest construction, adoption descriptors, expiry, and cleanup.
- `migration.rs`: validated offline import from the current instance loose layout.
Public API shape to freeze before implementation:
```rust
pub struct NamespaceId(pub [u8; 32]);
pub struct OperationId(pub [u8; 16]);
pub struct StoreEngine { /* private */ }
pub struct RepoSnapshot { /* immutable logical state */ }
pub struct ValidatedTransaction { /* constructible only by instance validator/importer */ }
pub struct CommitReceipt {
pub operation_id: OperationId,
pub repo_sequence: u64,
pub current_authority: ObjectId,
pub refs: Vec<AppliedRef>,
pub objects_new: u64, // inline or atomically adopted staged-projection membership
}
pub enum PendingPhase { Receiving, Validating, Queued, Sequenced }
pub enum TransactionStatus {
Committed(CommitReceipt),
Pending { operation_digest: ObjectId, retry_until_micros: i64,
phase: PendingPhase },
Resolving { operation_digest: ObjectId, retry_until_micros: i64,
shard_sequence: Option<u64> },
Expired { operation_digest: ObjectId, retry_until_micros: i64,
tombstone_until_micros: i64 },
Unknown,
}
impl StoreEngine {
pub fn open(options: StoreOptions) -> Result<Self, StoreError>;
pub fn snapshot(&self, repo: NamespaceId) -> Result<RepoSnapshot, StoreError>;
pub async fn submit(&self, txn: ValidatedTransaction)
-> Result<CommitReceipt, StoreError>;
pub fn transaction_status(
&self,
repo: NamespaceId,
operation: OperationId,
) -> Result<TransactionStatus, StoreError>;
pub fn checkpoint(&self) -> Result<CheckpointLease, StoreError>;
}
```
`objects_new` is a per-transaction field, not a lifetime counter. Inline client/init transactions remain bounded by `max_objects_per_transaction < u32::MAX`; an atomic staged-projection install is instead bounded by separately configured `max_projection_objects`/`max_projection_bytes` and reports its complete adopted membership count as `u64`. Process-lifetime, benchmark, storage, and accounting totals use `u64` or `u128` accumulators.
`StoreEngine` owns one `ArcSwap<CommittedRoot>` and one bounded `ArcSwap<OperationStatusRoot>`; neither uses `RwLock<Arc<_>>`. `snapshot()` acquires one committed root. `transaction_status()` is lock-free but linearizable across the two roots: it loads committed root A and returns any receipt/expired tombstone; otherwise it loads the status root and may return an existing pending/resolving entry; if no status entry exists, it loads committed root B, returns any terminal entry found there, and otherwise returns `Unknown`. The mandatory B read closes the old-committed/new-empty-status race without requiring A and B to have the same global generation, so unrelated high-rate shard publications cannot starve a status lookup. `RepoSnapshot` retains the committed generation and shared persistent substructures, so reads do not clone the full index/catalog. After a shard fence, its writer merges the immutable shard subtree—including receipts—into the currently loaded committed root and publishes with an atomic compare-and-swap loop; a concurrent shard publication causes a merge against the newer root, never a lost update. The committed-root swap is the visibility boundary. Pending/resolving status-root entries are removed only after that swap with release/acquire ordering that makes the prior committed-root publication visible to the mandatory B load; failure to wake a waiter afterward cannot hide the receipt. Old roots and segment generations remain alive through `Arc`/lease ownership, and status reads cannot block group commit.
`ValidatedTransaction` is sealed against arbitrary callers, but migration/recovery have explicit privileged constructors. `ProjectionStageSession` is likewise opaque and exposes only bounded begin, idempotent chunk-put, read-only resolver, seal-to-`StagedProjectionInstallV1`, and abort operations; sealing cannot publish membership, and only `submit(ValidatedTransaction)` may adopt its descriptor. Reads operate on a `RepoSnapshot` plus a committed sequence. An instance verification overlay resolves request-staged or projection-session objects first and snapshot objects second.
`TransactionStatus` represents normal lifecycle states, never generic errors. `Committed`, `Pending`, `Resolving`, `Expired`, and `Unknown` map to HTTP 200, 202 `OperationPending`, 202 `OperationResolving`, 410 `ReceiptExpired`, and 404 respectively. The public `IngestService` status path gives durable store state precedence over its pre-store in-flight registry: query the store first; only on `Unknown` inspect the registry; then query the store again before returning registry state or `Unknown`. Any non-`Unknown` store result wins, and a registry result may be returned only when the second store query is still `Unknown`. Thus a stale registry entry cannot mask a published receipt and an old committed-root read plus a new empty status-root read cannot manufacture `Unknown`. The same public enum is used at both layers. `StoreError` is reserved for inability to answer, such as corruption or unavailable recovery state.
### 5.2 Physical format and durability
Instance root v2:
```text
<root>/
FORMAT
LOCK
staging/<shard>/<session-id>/... # unreferenced same-device artifacts only
shards/
00/
active/<first_shard_sequence>.journal
segments/<generation>-<first_shard_sequence>-<last_shard_sequence>.seg
indexes/<generation>.idx
checkpoints/<shard_sequence>.checkpoint
CURRENT
01/...
```
`FORMAT` is checksummed, versioned, and directory-synced. Startup handles four states explicitly: a configured absent or empty new root is initialized as v2 with crash-safe directory/`FORMAT` sync; a valid v2 `FORMAT` opens through production recovery; a recognized non-empty legacy instance layout without `FORMAT` is refused with the exact `migrate-store` command; and every other non-empty unrecognized layout is refused without modification. It never infers legacy state merely from a missing marker.
A transaction frame contains:
- Magic, storage version, total length, `shard_sequence`, `repo_sequence`, namespace, operation ID, and operation digest.
- Repository-create metadata when applicable.
- New raw objects: embedded type, ObjectId, exact raw length, exact raw bytes; or, only for a finalized projection-staging import, one checksummed `StagedProjectionInstallV1` descriptor naming an already synced immutable object/index manifest whose complete IDs, types, lengths, byte digests, projection digest, and destination membership root are bound into the transaction/state digest.
- Complete typed ref CAS set.
- Explicit expected/new current authority.
- Canonical typed `TransactionEvidenceV1` appropriate to the mutation source.
- Canonical `CommittedTransactionV1`, source key epoch, source signature, and resulting state digest.
- Deterministic receipt fields.
- A BLAKE3 checksum over frame metadata and payload plus a commit trailer.
`TransactionEvidenceV1` freezes one canonical discriminated union:
```rust
pub enum TransactionEvidenceV1 {
ClientV2 { signed_envelope: SignedClientOperationV2 },
MirrorEventV1 {
source_instance: PublicKey, source_key_epoch: u64,
source_snapshot_digest: ObjectId,
source_event: SignedCommittedTransactionV1,
},
MirrorSnapshotV1 {
source_instance: PublicKey, source_key_epoch: u64,
source_snapshot: SignedRepoSnapshotV1,
destination_projection: ProjectionMode,
projected_manifest_digest: ObjectId,
projected_object_count: u64, projected_object_bytes: u64,
},
LegacyMigrationV1 {
actor: PublicKey, actor_key_epoch: u64, migration_id: [u8; 16],
repo_id: ObjectId, chunk_ordinal: u32, chunk_count: u32,
chunk_digest: ObjectId, manifest_digest: ObjectId,
source_layout_digest: ObjectId, signature: [u8; 64],
},
ProjectionAdminV1 {
actor: PublicKey, actor_key_epoch: u64, command_digest: ObjectId,
previous_projection: ProjectionMode, new_projection: ProjectionMode,
signature: [u8; 64],
},
AdministrativeV1 {
actor: PublicKey, actor_key_epoch: u64, command_digest: ObjectId,
signature: [u8; 64],
},
}
```
`SignedClientOperationV2` is itself a frozen enum over `SignedInitOperationV2` and `SignedPushOperationV2`; its discriminant is covered by the evidence digest.
Every variant has canonical bytes, an authenticated actor/key epoch, and a domain-separated digest. Administrative signatures cover `BLAKE3("levcs-evidence/<variant>/v1\0" || canonical_unsigned_evidence)`, including the variant discriminant, destination repo, operation ID, actor, key epoch, and every displayed field. `MirrorEventV1` retains the complete signed source event; `MirrorSnapshotV1` retains the complete deterministic signed source generation checkpoint plus the exact destination projection/manifest/counts rather than only its digest. It never contains the server-local snapshot lease token or lease record. Mirror-event destination IDs derive from trusted source instance/repository/`repo_sequence`; snapshot IDs derive from source instance/repository/snapshot generation digest, destination projection, and projected manifest digest. Each migration transaction derives a unique ID from `BLAKE3("levcs-migration-chunk/v1\0" || migration_id || repo_id || chunk_ordinal || chunk_count || chunk_digest)` and rejects ordinal/count/digest reuse. A frame does not require fictional client-v2 evidence for non-client work. Offline restore has no variant: it reproduces existing frames and evidence byte-for-byte and emits no new event.
The engine accepts a generic bounded `CommitEvidenceSigner` registered by instance composition; storage does not interpret federation trust. Before append, the shard sequencer assigns repository sequence, previous event digest, and `sequenced_at_micros`, computes the event/state digests from reserved speculative state, obtains the source signature, and places all of it in the frame. The event digest excludes its signature, so later reservations can chain deterministically. No receipt or event is exposed unless that signed frame passes the same durability fence. Signer failure occurs before journal append and fails/revalidates the affected speculative suffix; it cannot produce an unsigned committed frame. P2/P3 measure this ordering, signing cost, and backpressure.
The sequencer assigns two independent values. `shard_sequence` is contiguous within one physical shard and orders journal append, recovery, segment ranges, checkpoints, and durability vectors. `repo_sequence` is contiguous within one repository, drives its event hash chain and federation cursor, and is the `CommitReceipt` value. Recovery rejects a shard gap/duplicate and independently rejects a repository gap, duplicate, or previous-event-digest mismatch.
The active journal is the WAL. The shard sequencer checks only mutable commit preconditions against speculative state containing every earlier accepted transaction in the pending group: operation ID/digest, repository lifecycle, snapshot/config/policy epoch, typed ref CAS, expected authority, and precomputed force/ancestry facts. It does not repeat pack parsing, graph traversal, signature verification, or policy evaluation. Winners reserve ref/authority/operation-ID results, append the resulting complete frames, and share one `sync_data`. New requests queue while the device flushes and form the next group. Default initial tuning is four storage shards, 512 transactions or 8 MiB maximum per group, and a 1 ms maximum idle batching delay; all values remain measured internals until P2 determines safe operator defaults.
The shard enters a recovery-required poisoned state for every failure, cancellation, or panic after frame append may have begun and before the corresponding committed root and receipt table are published, including short/failed append, unexpected file position, failed/ambiguous fence, evidence handoff failure, atomic-root compare-and-swap loop failure, allocation failure, or writer-thread panic. It becomes read-only, completes every affected operation as `Resolving`, performs no later append/publication/acknowledgment, and must close/reopen through tail recovery before mutation resumes. If the fence is known successful, recovery MUST publish the complete durable prefix; if append/fence outcome is ambiguous, recovery resolves each complete checksum-valid frame as committed and every absent/torn frame as absent/retriable. The shard never rolls back speculative state and continues, and never continues past a durable but invisible prefix.
Rotation is a crash-safe manifest operation. Sync the new segment, index, checkpoint, and initial next journal; sync every directory receiving a new name; write and sync a checksummed/versioned `CURRENT.tmp` naming the complete base generation and every retained tail range; rename it to `CURRENT`; sync its parent; only then unlink superseded names and sync their source directories. Initial active-journal creation is directory-durable before its first acknowledgment. Recovery selects only a checksum-valid `CURRENT` whose referenced files validate, accepts complete checksum-valid tail frames, truncates/quarantines a torn final tail, and replays in sequence. Recovered valid frames may include an operation whose response was lost. If its first receipt-visibility timestamp was not yet captured by a durable checkpoint/baseline, recovery sets that timestamp to recovery publication time; reclamation cannot use it until a later durable checkpoint records it. Readiness remains false until replay and catalog/genesis validation complete.
### 5.3 Index and snapshots
Index keys include namespace and ObjectId. V2 does not physically deduplicate across namespaces.
Active entries are in an in-memory delta with hard byte/entry ceilings. Sealed segments receive immutable memory-mapped index runs with Bloom filters. Configuration bounds delta size, index-run count/fan-out, open runs, and replay bytes/frames since the newest durable checkpoint. Admission backpressures before a bound is exceeded; the shard synchronously seals/checkpoints when necessary. Retain at least two independently validated checkpoint generations. If both are corrupt, startup enters explicit offline rebuild mode rather than performing an unbounded normal-readiness scan. Ref state and namespace metadata are derived/checkpointed; journal/segment frames remain authoritative.
After a successful fence, build one immutable shard subtree containing the object-index delta, every affected repository snapshot, per-repository sequence/cursor, shard committed sequence, receipt visibility/retention metadata, and generation/tail references. Merge it into the immutable node committed-state root and publish through the `ArcSwap` compare-and-swap loop specified in §5.1. Every read API captures exactly one root. Operations remain `Resolving` from append start until successful root/receipt publication. Only after publication may logical events become visible and request futures complete with receipts; any intervening failure follows the poison/recovery rule in §5.2.
Readers never observe an index entry newer than their captured root, and `/info`, `/refs`, pack closure, backup, and mirror export use that one snapshot.
### 5.4 Compaction and GC
Compaction starts from a committed safe point S and explicit `ProjectionCore` roots. In addition to unchanged raw live objects, the new base contains an authoritative checksummed `BaselineStateV1` record through S: namespace catalog, membership, typed refs, current/genesis authority, per-repository sequence/event digest, shard sequence, retained operation/receipt state including first-visibility and `receipt_visible_until`, event low watermark, and projection/state digests. Checkpoints and indexes remain derived from this baseline plus later frames.
Before installation, the shard sequencer rotates the active journal so every transaction above S is in an immutable retained tail, then atomically installs a `CURRENT` manifest referencing the new base and every tail range above S. Recovery can rebuild bounded state from `BaselineStateV1` plus the retained tail even when both derived checkpoints are corrupt. It loads the base and replays the tail; no post-S object/ref/evidence state disappears. A corrupt baseline is an explicit store-corruption/offline-restore failure, not guessed around.
Compaction retains every event record above the advertised per-repository event floor together with every projection-specific object and proof needed to independently validate and transactionally apply that event from the authenticated snapshot at the floor. An object introduced or subsequently referenced by any retained replayable event remains pinned even if no current ref reaches it. Reclamation may instead atomically install a new authenticated snapshot, advance the event floor past every affected event, and reclaim the event/object set in the same `CURRENT` publication; consumers below the new floor receive `CursorExpired` and must resnapshot. There is no state in which an event is advertised as replayable while one of its required objects is unavailable. Operation-ID/digest/receipt/resolution retention remains independently bounded by §4. Grace uses committed insertion sequence/time, never file mtime. Missing/corrupt retained or reachable edges fail compaction. Old segments are reclaimed only after read, mirror, snapshot base/request, finalized-staging-adoption, and backup pins unpin them.
Release cache files are not imported into instance v2. They are non-authoritative.
### 5.5 Future instance software contract
Future review, CI, webhook, indexing, and workflow components consume a frozen `CommittedTransactionV1` contract:
```text
{ repo_id, repo_sequence, previous_event_digest, event_digest,
sequenced_at_micros, source_kind, actor, operation_id, operation_digest,
signed_evidence_digest, old/new authority, complete typed old/new refs,
bounded object/commit ids, resulting_state_digest }
```
`repo_sequence` is monotonic per repository; the hash chain and resulting-state digest make gaps detectable. `source_kind` is derived from `TransactionEvidenceV1` and distinguishes client, mirror, legacy migration, projection administration, and other authenticated administration. Restore is not a source kind because exact restore creates no transaction or event. Pages are evaluated against one fixed upper `repo_sequence` and `after` is exclusive. Each snapshot advertises the minimum retained event sequence. A cursor below it returns typed `CursorExpired` with the authenticated snapshot/cursor needed to rebuild; it never returns a partial feed. Signed checkpoints/events and all projection-specific objects/proofs required to replay them are retained together through active consumer cursors, or the floor advances atomically and the consumer is forced through resnapshot.
This logical contract freezes in Phase 0 before protocol work. Physical segment offsets remain private. Secondary indexes consume with durable cursors and must demonstrate dispatch/recovery for every source kind, including falling behind retention, before P3 exits.
## 6. Protocol v2 and client changes
### 6.1 Signed ingestion envelope
Add deterministic manual binary codecs in `levcs-protocol`; do not sign serde JSON. Object signatures remain unchanged.
```rust
pub struct PushOperationV2 {
pub operation_id: [u8; 16],
pub repo_id: ObjectId,
pub issued_at_micros: i64,
pub retry_until_micros: i64,
pub nonce: [u8; 16],
pub expected_authority: ObjectId,
pub authority_update: Option<ObjectId>, // direct successor ID
pub updates: Vec<TypedRefCas>,
pub pack_len: u64,
pub pack_hash: ObjectId,
pub snapshot_generation_digest: ObjectId,
pub projection_stage: Option<ProjectionStageRefV1>,
pub kind: PushKindV2,
}
pub struct ProjectionStageRefV1 {
pub session_id: [u8; 16],
pub manifest_digest: ObjectId,
pub object_count: u64,
pub object_bytes: u64,
}
pub enum RefTarget {
Branch(String),
Release(String),
}
pub enum RefMutation {
Set(ObjectId),
Delete,
}
pub struct TypedRefCas {
pub target: RefTarget,
pub expected: Option<ObjectId>, // None is create-only; Delete requires Some
pub mutation: RefMutation,
pub force: bool,
}
pub struct SignedPushOperationV2 {
pub operation: PushOperationV2,
pub signer: PublicKey,
pub signature: [u8; 64],
}
pub struct InitOperationV2 {
pub operation_id: [u8; 16],
pub repo_id: ObjectId,
pub issued_at_micros: i64,
pub retry_until_micros: i64,
pub nonce: [u8; 16],
pub genesis_len: u64,
pub genesis_hash: ObjectId,
}
pub struct SignedInitOperationV2 {
pub operation: InitOperationV2,
pub signer: PublicKey,
pub signature: [u8; 64],
}
```
```rust
pub enum PushKindV2 {
Normal,
Fork(ForkProofV2),
}
pub struct ForkProofV2 {
pub source_repo_id: ObjectId,
pub source_genesis: ObjectId,
pub source_tip: ObjectId,
pub source_authority: ObjectId,
}
```
The push signature input is `BLAKE3("levcs-push/v2\0" || canonical_operation)`. Its stable digest uses a separate `levcs-push-digest/v2\0` domain and covers operation ID, repo ID, signer, retry deadline, push-kind discriminant and fork fields, expected/new authority, ordered typed Set/Delete mutations including per-ref force, snapshot generation digest, the optional projection-stage discriminant/session/manifest/counts, and pack length/hash; it excludes only retry timestamp, nonce, and signature. `projection_stage` MUST be absent for `Normal` and is allowed for `Fork` only when the staged manifest is bound to the same signer, Fork proof, source generation, destination genesis, and final operation ID/digest. For `Normal`, `authority_update` is the new Authority ID and `expected_authority` is the sole pre-state. A non-FORK transition transaction requires exactly one newly exposed `MODIFIES_AUTHORITY` boundary Commit whose `commit.authority == expected_authority`, whose `.levcs/authority` entry is the direct valid successor, and whose updated ref exposes that boundary. The successor Authority, boundary Commit, and exact Tree path nodes needed to bind `.levcs/authority` are the only newly introduced authority-transition objects allowed to reference the successor. Every other newly exposed Commit/Release in that transaction MUST cite and be authorized by `expected_authority`; none may cite the successor, even on another ref. The authority CAS publishes only with the boundary. Successor-authority Commits/Releases are accepted first in a later transaction whose `expected_authority` is the now-published successor. Reject multiple boundaries, successor-signed side branches, unrelated successor references, a modifying non-FORK commit without the matching CAS, or a CAS without the boundary commit.
Init uses `BLAKE3("levcs-init/v2\0" || canonical_init)` and a separate `levcs-init-digest/v2\0` stable digest. The stable digest covers operation ID, repo ID, signer, retry deadline, genesis length, and genesis hash, and excludes only issued-at, nonce, and signature. Its body is `envelope_len || signed_init_envelope || exactly genesis_len raw genesis bytes || EOF`. Before create, parse/hash the genesis, derive and compare repo ID, verify the Owner self-signature, and require the envelope signer to be a genesis Owner. Push and init codecs, discriminants, errors, and golden bytes—including original/retry and same-ID/different-genesis vectors—freeze in Phase 0.
`Fork` is a separate validation path, not an authority successor. It is accepted only as the first create-only branch publication into an otherwise empty initialized destination whose current authority is its genesis; `authority_update` MUST be absent. Exactly one newly exposed Commit has `FORK|MODIFIES_AUTHORITY`, exactly one parent equal to `source_tip`, `commit.authority == destination genesis`, and `.levcs/authority == destination genesis`. The envelope signer MUST be a destination Owner and the Commit signer. The verifier hashes/parses the source parent, requires its cited authority to equal `source_authority`, verifies the source authority chain and source history closure to `source_genesis`, derives and matches `source_repo_id`, rejects destination/source repo equality, and enforces the source authority's read policy for the fork signer. Native destination validation resumes at the fork Commit; the parent edge becomes a typed `ForeignForkBoundary`, not a destination authority successor. If the required Full foreign closure exceeds normal push limits, the same signer first creates a projection-staging session bound to the exact `ForkProofV2`, source snapshot generation, destination genesis, final push operation ID/digest, and manifest digest; the final Fork transaction may adopt only that completely verified staged manifest.
`ProjectionCore` represents that boundary explicitly. Full projection retains and independently verifies the foreign parent/history/object closure under `{source_repo_id, source_genesis, source_authority}`. Release and metadata projections may omit only their already-validated foreign subgraph and must carry source-signed projection evidence binding the fork Commit, source tip, source repo/genesis/authority, omitted-edge digest, and destination event. Omitted foreign IDs are not advertised as locally readable. Golden fixtures preserve current fork Commit/Tree/Authority bytes while proving valid public/private forks, foreign-chain substitution rejection, stale/malformed proof rejection, and destination-genesis isolation.
The push HTTP body is `envelope_len || signed_push_envelope || exactly pack_len Pack v1 bytes || EOF`. The server authenticates operation intent and gross limits before decompression, then streams while checking `pack_hash`. There is no redundant whole-body request signature plus manifest signature.
A retry creates a fresh timestamp/nonce/signature but retains operation ID, retry deadline, and stable digest. Before replay reservation or body allocation, the server validates with checked arithmetic that `issued_at_micros` is within configured clock skew and `issued_at_micros <= retry_until_micros <= issued_at_micros + max_retry_window_micros`; past deadlines, overflow, negative windows, and values beyond the configured authoritative maximum reject. The server never clamps a signed deadline. The effective maximum is advertised in capabilities and archived in benchmark configuration. The sequencer performs a final deadline check immediately before append: expiry before append is definitive and appends nothing, while append start freezes the operation in `Resolving` until terminal recovery. For a committed outcome it computes/persists the checked `receipt_visible_until` rule in §4; committed receipt retention, retry-envelope expiry, and the later `Expired` tombstone are distinct.
Return structured `PushReceiptV2`/`InitReceiptV2`, not an empty success. Add transaction-status lookup by operation ID for ambiguous network outcomes through the advertised retry/terminal-retention window.
### 6.2 HTTP surface
Use `/levcs/v2`. Remove v1 POST init/push on cutover; do not add a downgrade path.
Add:
- `GET /repos/{repo_id}/snapshot`: a response pair `{ signed_generation, lease_token }`. `signed_generation` is one deterministic source-signed checkpoint containing proved genesis/current authority, effective storage mode, typed refs, transaction low/high `repo_sequence` and digest, federation key epoch, and generation-frozen capabilities/config epoch. Any covered capability change publishes a new generation digest before it is served. The bounded random server-local `lease_token` MAC-binds that generation but is excluded from `SignedRepoSnapshotV1`, the generation digest/signature, federation evidence, and destination operation digests.
- `GET /repos/{repo_id}/transactions?after={cursor}`: source-signed, hash-chained committed transactions over a fixed page upper bound; old cursors return `CursorExpired`.
- `GET /repos/{repo_id}/transactions/{operation_id}`: HTTP 200 with the original receipt while a committed terminal record is retained; typed 202 `OperationPending` before append; typed 202 `OperationResolving` from append through recovery/publication, even past `retry_until`; `ReceiptExpired` after committed receipt retention through tombstone grace; then 404 `Unknown`. A pre-append deadline rejection or recovery-proved absent outcome removes transient status and may therefore be 404 by ID, while replaying its expired signed envelope remains 410. The linearizable store/registry precedence rules in §5.1 apply.
- Existing object and Pack exchange semantics require `LeVCS-Snapshot: <token>` and are served only from the generation bound by that token.
- `DELETE /repos/{repo_id}/snapshot-lease`: idempotently closes the lease named by `LeVCS-Snapshot` to new requests. Expiry does the same. The base token pin is released immediately when no admitted request pin remains; already admitted request pins drain under their own bounds.
- Read authorization enforces the proved current Authority policy from the same `RepoSnapshot`; missing/malformed policy fails closed. Private snapshot tokens are bound to the authenticated current Reader key.
- Readiness is distinct from liveness and storage recovery status.
- `POST /repos/{repo_id}/objects/missing`: bounded canonical missing-object negotiation against the generation in `LeVCS-Snapshot`.
- `POST /repos/{repo_id}/push` requires the same token used for CAS construction/missing negotiation; its generation digest must equal the signed `snapshot_generation_digest`, and a private token's Reader key must equal the signed envelope key. Live-token validation atomically acquires a separately accounted request pin. That pin survives concurrent token expiry/release and remains through definitive pre-append rejection or append start; once append begins, the complete frame/speculative reservation is self-contained and no longer depends on the snapshot generation. The ingestion envelope authenticates push; there is no second whole-body request signature.
- Bounded projection-staging routes create an authenticated session, upload canonical numbered chunks, inspect status, finalize through the ordinary transaction service, and abort. They are available only for initial mirror/resnapshot, oversized Full fork closure, and network migration; normal push cannot reference them. Every route binds destination repo/projection, source snapshot or Fork proof, final operation ID/digest, total bytes/objects/chunks, manifest digest, expiry, and authenticated source kind.
Pack v1 gets bounded streaming reader/writer APIs while retaining golden wire bytes. Total wire/decoded bytes, entries, compression expansion, and delta depth are limited before allocation.
The snapshot token is base64url of canonical binary `{ version, mac_key_epoch, random_128_bit_lease_id, repo_id, projection, generation_digest, high_repo_sequence, high_event_digest, issued_at_micros, expires_at_micros, reader_key_digest_or_zero } || keyed_BLAKE3_MAC`. It is an ephemeral lease grant paired with, but never part of, the deterministic signed generation. The rotated server-local MAC key authenticates but does not replace the bounded in-memory lease record keyed by the random ID; that record owns the base generation `Arc`/segment lease. Under the lease-record synchronization, each request verifies that the token is live/unreleased and atomically clones a separately counted generation/request pin before work begins. Token acquisition and request-pin admission reserve configured per-principal/global lease-count, pinned-byte, maximum-duration, and compaction-debt budgets or return 429/503 before pinning. Expiry or DELETE prevents new request pins and releases the base pin when the admitted-request count reaches zero; it never invalidates a pin already acquired by a bounded request. Read/Pack pins last through response completion or cancellation; exceeding request-pin maximum duration actively aborts/cancels the request and then releases the pin, never silently unpins a live reader. Push pins follow §6.2. Tokens cannot be renewed beyond the advertised maximum; clients acquire a new snapshot. Restart rotates the key epoch, rejects new use of old tokens, and drops each base/request pin only after its in-process owner is gone; old tokens return typed HTTP 410 `SnapshotExpired`. Missing, malformed, wrong-repo/projection/reader, released, expired, unknown-key-epoch, or unknown-record tokens return the same error without falling back to current state. Clients release promptly; on `SnapshotExpired` they discard partial negotiation results, fetch a new signed snapshot/token, and restart missing-object/Pack negotiation. Object and Pack responses include the bound generation digest and high `repo_sequence`; tests hold compaction between snapshot, missing negotiation, and Pack fetch and force expiry/DELETE during active reads and pushes to prove one-generation results and bounded pin release.
`MissingObjectsRequestV1` is canonical binary `{ version, generation_digest, count_u32, sorted_unique_object_ids[count] }`. `MissingObjectsResponseV1` is `{ version, generation_digest, count_u32, sorted_unique_missing_ids[count] }`. The request generation must match the snapshot token; the response is computed only from that captured namespace root. Strict request/response byte and ID-count limits apply before allocation, duplicates or noncanonical ordering reject, and no unbounded JSON/list endpoint exists. A live token atomically yields the request pin that holds membership through the associated push's pre-append validation/reservation; `SnapshotExpired` before pin acquisition forces renegotiation rather than optimistic fallback.
Private snapshot acquisition and every private GET, missing-object POST, Pack request, and lease DELETE use `LeVCS-Key`, `LeVCS-Timestamp`, `LeVCS-Nonce`, and `LeVCS-Signature`. The signature is Ed25519 over `BLAKE3("levcs-read/v2\0" || canonical_read_request)`, where the manual binary `canonical_read_request` contains method/route discriminants, repo ID, typed canonical query/body hash, `BLAKE3(snapshot_token)` or zero for acquisition, timestamp, and nonce. It never signs proxy-rewritten text or unordered query strings. Timestamp acceptance uses configured skew; `(public_key, nonce)` is atomically reserved before work and shares the replay horizon below. Snapshot acquisition binds the verified Reader key digest into the token; all later read/negotiation/release requests require the same key, while push binds that key through its signed envelope as specified above. Golden vectors cover every route, public/zero-token and private/token-bound requests, query ordering, body tampering, stale/future timestamps, nonce replay, and proxy path normalization.
Projection-staging create/chunk/status/finalize/abort commands use separate domain-separated canonical codecs and sign the route discriminant, complete session binding, chunk/body hash where applicable, timestamp, and nonce. A Fork session requires the same destination Owner/Commit signer as the final `SignedPushOperationV2`; mirror sessions require the configured source peer identity; network migration requires the configured administrative migration identity. Every command rechecks the session actor/source kind and uses the same replay guard. The final transaction evidence and optional `ProjectionStageRefV1` independently bind the adopted manifest, so possession of a session ID or snapshot read token alone never authorizes publication.
### 6.3 Client and CLI
`levcs-client` gains `PreparedPush` and v2 methods that stream a prepared bounded body/spool rather than duplicating whole body vectors. The blocking compatibility client remains acceptable for ordinary CLI use; the permanent load generator uses an async client emitting identical public wire bytes.
Prepared client operations include `PushKindV2`; the fork command emits `ForkProofV2` from the existing fork Commit and source authority data rather than pretending the fork is a destination authority successor.
`crates/levcs-cli/src/fed_cmds.rs::push` must:
1. Fetch one remote snapshot.
2. Populate exact current ref and authority expectations.
3. Walk one union closure with every error propagated.
4. Parse embedded object types rather than indexing raw byte offsets.
5. Send only missing objects or use an explicit missing-object negotiation.
6. Use deterministic dependency-friendly ordering.
7. Preserve operation ID across retry and verify the receipt.
Pull, fork, dial, mirror, and network migration must pin repo identity and verify received tips/closure before publishing local refs. Fork preserves existing object bytes but uses the explicit foreign-boundary proof above. Network `migrate` uses `ProjectionStageSessionV1` for the source snapshot projection, applies a bounded final delta/fence through the same authenticated final transaction, and returns a durable destination receipt; it remains distinct from instance on-disk migration.
## 7. Instance ingestion pipeline
Split the monolithic instance module into explicit components:
```text
crates/levcs-instance/src/
lib.rs router/composition only
config.rs strict config and limits
state.rs shared services
auth.rs v2 request auth and replay guard
admission.rs bounded bytes/jobs/fairness
ingest/
mod.rs
receive.rs envelope + streaming Pack
overlay.rs parse-once staged object index
validate.rs graph/identity/type/closure
policy.rs merge/repository/instance policy
service.rs init/push orchestration
read.rs snapshot/object/pack endpoints
federation.rs transaction feed/snapshot types
staging.rs authenticated projection session/chunk/finalize service
mirror.rs verified transactional mirror apply
metrics.rs counters/histograms/readiness
```
### Executor and queue model
The implementation MUST use explicit executors rather than discover a threading model while tuning:
- **Tokio I/O runtime:** socket accept, bounded body streaming, timers, admission waits, and response delivery only. It performs no filesystem calls, decompression, hashing, signatures, graph traversal, or policy evaluation.
- **Receive-spool I/O pool:** dedicated fixed-size OS workers own all temporary-file create/write/read/unlink operations and stream bounded buffers between Tokio and validation. Admission reserves per-request/client/global spool bytes, queued-buffer bytes, file descriptors, and minimum-free-space budget before reading. Small requests may remain in the same bounded buffer budget; larger requests use `O_TMPFILE|O_CLOEXEC` under configured `<root>/spool`, with random `O_EXCL` create-and-unlink fallback on supported production filesystems. Spools are non-durable, never call fsync, never enter namespace membership, and are removed on success, rejection, disconnect, cancellation, panic, and startup scavenging. Validation reads spool bytes only through this pool. Crossing a quota or storage low-watermark backpressures or returns 429/507 before exhausting the store device.
- **Validation pool:** one dedicated fixed-size Rayon pool for Pack decompression, parse/hash, signature and authority verification, graph traversal, ancestry facts, and policy evaluation. Jobs reserve both job-count and byte permits before submission.
- **Evidence-signing pool:** a separate bounded fixed-size pool for `CommittedTransactionV1` signatures so slow keys/signers cannot consume validation capacity. Results return in repository sequence order.
- **Storage shard threads:** one dedicated OS thread per shard owns speculative state, sequence assignment, append position, group formation, `write_vectored`, `sync_data`, rotation, manifest-visible naming/adoption, `CURRENT`, unlink, and committed-root publication. No other thread mutates an active or referenced shard file or namespace.
- **Maintenance workers:** bounded dedicated workers perform checkpoint construction, compaction, projection-staging construction, backup copying, and offline rebuild. They write only uniquely named unreferenced artifacts under independent I/O/debt budgets, sync and validate them, and hand immutable adoption descriptors to the shard owner. The shard owner revalidates the descriptor and alone installs or discards those artifacts; maintenance never updates `CURRENT`, renames over a referenced name, or unlinks a referenced file.
- **Completion path:** bounded crossbeam-style channels carry prepared transactions to shard threads; Tokio `oneshot` futures carry receipts/errors back to request tasks.
Every executor crossing has explicit job and byte ceilings, queue latency metrics, cancellation behavior, and overload status. The steady-state path MUST NOT use Tokio's shared `spawn_blocking` pool; it is reserved for incidental startup/administrative work.
### Ordered stages
1. **Admission:** enforce content length, configured wire/decoded/object/update limits, per-client and per-repository fairness, and global in-flight bytes. Overload returns 429 with retry guidance before expensive work.
2. **Envelope authentication and operation reservation:** canonical decode, timestamp/deadline/ref syntax/repo checks, Ed25519 verification, and nonce replay reservation occur before body allocation. The server checks the durable receipt/tombstone table, then atomically reserves `(repo_id, operation_id)` in a bounded in-flight registry keyed with the stable digest before expensive receive/validation work.
3. **Replay guard:** a sharded map plus timing wheel avoids a process-wide mutex and O(total entries) request-path sweep. For every accepted timestamp, nonce expiry is computed with checked arithmetic as at least `issued_at_micros + clock_skew_micros + timer_resolution_micros`; eviction before that instant is forbidden. An insertion-relative implementation therefore requires configured replay retention `>= 2 × clock_skew + timer_resolution`, and startup rejects overflow, negative values, or a shorter horizon. Initial hosted defaults are ±60 seconds skew, 1-second timer resolution, and at least 121 seconds retention.
4. **Streaming receive:** after spool-byte/FD admission, Tokio hands bounded chunks to the receive-spool I/O pool, which hashes and stores exactly the signed Pack length and supplies bounded read buffers to decompression/validation. No Tokio, validation, or shard thread performs temporary filesystem I/O.
5. **Parse-once overlay:** hash and parse each entry once; compare outer/embedded type; reject conflicting duplicates, unknown types, malformed canonical bodies, quota violations, and unreachable extras.
6. **Anchored graph verification:** use `TrustAnchor { repo_id, genesis, current }`; require complete typed closure according to `ProjectionCore`. Full mode follows every edge; reduced modes may terminate only at their signed, already-validated predecessor/parent/content boundaries and never grant membership to omitted IDs. Verify each visited node and authority edge once; cache only fully proved authority chains keyed by repo/genesis/authority.
7. **Authorization and policy:** envelope signer authorization, force/Delete, protected-target policy, and final CAS use the expected current pre-state Authority. Each Commit/Release signer/declarer and object-level role/policy use that object's cited Authority, proven on the pinned chain; target-ref contextual role checks run for every ref exposing it. In an authority-transition transaction, all newly exposed Commit/Release objects remain old-authority objects and only the single boundary Commit may expose the successor Authority; successor-authority objects wait for a later transaction after the CAS is published. Cache cryptographic facts, not context-dependent authorization decisions. Evaluate all merge records fail-closed.
8. **Prepare:** produce immutable `ValidatedTransaction` containing exact new bytes, complete ordered typed Set/Delete refs, authority CAS, operation evidence/digest/deadline, and snapshot/config epochs. No handler writes storage.
9. **Sequence and reserve:** only the elected in-flight leader reaches the target-shard sequencer. It checks the operation reservation, signed retry deadline, refs, authority, repository lifecycle, and policy/config epoch against speculative state containing all earlier assigned but not-yet-published transactions. A mutable-state conflict returns 409 and commits nothing. Deadline expiry here is a definitive pre-append `ReceiptExpired`. A winner reserves state plus `shard_sequence`/`repo_sequence` before joining the group; the writer checks the deadline once more immediately before marking `Resolving`/starting append so no operation first appends after its signed deadline.
10. **Group append, fence, and resolving boundary:** mark every group operation `Resolving` before append, append exact reserved frames, and perform one durability fence. Any failure or panic from append start through committed-root/receipt publication poisons the shard and transfers resolution to recovery.
11. **Atomic publish and acknowledge:** publish one committed-state root and receipt table, expose signed logical events, then return durable receipts. A known-successful fence followed by publication failure is recovered as committed, never retried as absent.
`ValidatedTransaction` is the immutable output of stages 48. The sequencer in stage 9 rechecks only mutable preconditions and consumes precomputed verified facts. If its snapshot, policy, or configuration epoch is stale, it rejects with a typed revalidation/conflict result; it never runs semantic validation while holding the mutation lane.
Concurrent retry behavior is exact while an in-flight or durable operation record exists. Same repository/operation ID and same stable digest attaches to the leader's shared completion future and never submits a second `ValidatedTransaction`; after authenticating the envelope, HTTP/1.1 drains the already bounded body or closes the connection, while HTTP/2/3 resets that request stream. Same ID/different digest returns typed 409 `OperationIdMismatch`. A definitive validation, deadline expiry, sequencing, signing, or other failure before append begins removes the transient in-flight entry, releases or revalidates its speculative suffix, and wakes every waiter with the same error; no durable status reservation is created, and a fresh valid retry may become leader only if its signed deadline is still live. Client disconnect does not cancel a leader after its body is accepted. From append start through root/receipt publication the status is `Resolving`, and this state dominates `retry_until`. Any failure or panic in that interval poisons the shard; waiters receive typed 503 `OutcomeUnknown`, and status lookup remains `OperationResolving` until production recovery resolves a known-fenced frame to the original receipt or proves it absent/retriable. A committed outcome remains status-queryable through `receipt_visible_until` even if resolution occurred after the signed deadline; a proved-absent outcome releases its reservation and becomes `Unknown`, so a fresh same-ID/same-digest retry is safe only while the signed deadline remains live. A bounded request deadline cancels only before append; a poisoned shard admits no new mutations, so unresolved entries are bounded by the failed group.
The identity crate gains a repository-scoped `VerificationSession` over an object resolver/overlay. It returns verified facts and does not mutate storage. Fix release declarer/signature rules and invalid extra-signature behavior in the same canonical verifier.
## 8. Federation and storage modes
### One mutation path
`sync_mirror`, on-disk migration, and future online instance imports construct validated transaction plans and submit to `levcs-store`. Offline restore is deliberately different: it verifies a complete exported generation in a staging root, preserves its shard sequence vector, event/receipt evidence, and cursor history exactly, installs it with the crash-safe `CURRENT` protocol, opens it through production recovery, then atomically installs the destination. Restore does not resubmit transactions or assign new sequences. Mirror polling may not write storage directly.
Large initial projections use one bounded `ProjectionStageSessionV1`; `PushKindV2::Normal` never does. The initiator generates the random session ID before computing any final operation/stable digest that contains `ProjectionStageRefV1`. Session creation verifies that `<root>/staging` and its target shard are on the same `st_dev` and canonically binds that session ID, destination repo/genesis and expected empty/current state, projection, authenticated source kind and actor/key epoch, source snapshot generation or `ForkProofV2`, final operation ID/stable digest/evidence digest, total object/byte/chunk counts, ordered manifest digest, expiry, and hard per-session/principal/global staging budgets. Every numbered chunk has a canonical digest and bounded object/byte count; upload validates framing, embedded types, IDs, exact bytes, canonical order, declared manifest position, and same-session ordinal/digest idempotency before a maintenance worker writes and syncs a uniquely named unreferenced artifact. Chunks never enter namespace membership, object-existence answers, snapshots, refs, receipts, event feeds, dedupe state, or `CURRENT`; cross-device adoption and copy fallback are forbidden.
Finalize atomically moves the session from `Sealed` to `Finalizing` for its sole bound operation/digest and takes an adoption pin; every different operation/digest rejects. **Amended by contract review 2026-07-31-A** in two places, and the superseded wording is not reachable: the pin is taken from `Sealed` rather than `Open`, because the sealed manifest is what an adopter revalidates against and a pin on a session without one names state that does not exist; and *coalescing identical concurrent finalizers is the request owner's, not the store's* — an adoption pin is an unforgeable capability with exactly one terminal outcome, so there is no second copy to hand a second caller, and the store's guarantee is one pin with a second finalizer refused by name. Expiry prevents a new finalizer but cannot delete artifacts held by an admitted bounded finalizer. Finalize requires every chunk, reconstructs the exact ordered manifest, and runs the same complete `ProjectionCore`, identity, authority, policy, source-snapshot/Fork-proof, and destination precondition validation used by inline ingestion. It builds and syncs an immutable staged object/index generation and then submits one ordinary bounded transaction with the appropriate `ClientV2`, `MirrorSnapshotV1`, or authenticated network-migration administrative evidence plus a `StagedProjectionInstallV1` descriptor. The shard owner rechecks session/operation/digest, destination lifecycle/CAS, source cursor, config/policy epoch, manifest and artifact hashes; assigns manifest-visible names; directory-syncs them; and appends a small final frame that binds the complete manifest/membership root and resulting state digest. A definitive pre-append failure releases the adoption pin and returns the session to `Sealed` only if it remains live; otherwise cleanup aborts it. **Amended by 2026-07-31-A**: `Open` is not a state a restart can reproduce for a session whose sealed manifest is durable — reconstruction reads that manifest's presence as the seal's own commit point — and returning a session to an unreconstructable state is the defect the private busy/state split exists to prevent. `Sealed` is still finalizable, which is the property the rule is about. The full state machine is `Open → Sealed → Finalizing → {Sealed, Adopted}`, all four durable. Only the final frame's fence and committed-root swap atomically grant membership and publish refs/authority/cursor/receipt/event. Recovery treats synced-but-unreferenced artifacts as invisible garbage and a complete final frame as authoritative adoption; it can never expose a partial chunk set.
Sessions are restartable by ID and chunk digest, have no renewal beyond their advertised maximum, and are aborted on authentication mismatch, quota/debt/low-space breach, explicit cancellation, or expiry. A remote source reserves a corresponding bounded snapshot-export base/request lease for the signed generation; each source chunk is served from that generation, and source-lease expiry aborts the destination session rather than mixing generations. Configured maximum projection size/session age and the supported minimum transfer rate must make one complete transfer feasible; otherwise session creation rejects before pinning. Cleanup removes only artifacts carrying a valid session marker after proving that no committed manifest references them, then syncs affected directories. Compaction/GC pins a finalized session from shard adoption through root publication and otherwise may reclaim expired unreferenced sessions. Admission accounts staged bytes, objects, files, sessions, validation work, age, and compaction debt independently of ordinary receive spools. Full-fork and network-migration clients use the v2 staging routes; an in-process mirror uses the identical codecs/service API without loopback HTTP.
Mirrors consume a source-signed snapshot at one generation and a durable hash-chained transaction cursor. Separate unversioned `/info` and `/refs` reads are removed from mirror correctness. Ref Set/Delete and rewinds follow explicit configured policy and apply atomically.
Initial sync and `CursorExpired` recovery apply one bounded final `MirrorSnapshotV1` transaction containing the complete deterministic signed source generation checkpoint, validated projected membership-root manifest/refs/authority, and exact source cursor/state digest. If all projected bytes fit inline limits, the frame embeds them normally; otherwise the mirror first uses `ProjectionStageSessionV1` and the final transaction atomically adopts its complete manifest. Subsequent relevant source events apply with `MirrorEventV1`. A source event irrelevant to the destination's reduced `ProjectionCore` still produces a cursor-only `MirrorEventV1` transaction: it retains the complete signed event, advances the durable source cursor/hash-chain, changes no projected object/ref/authority state, sets `objects_new=0`, and is excluded from commit-throughput counts. Cursor-only and projected applications share ordering, idempotency, crash recovery, and destination event-signing rules, so a mirror never skips an authenticated source sequence silently.
### Exact projections
- **Full:** native authority chain, branch/release refs, and all objects reachable from those roots. A `ForeignForkBoundary` additionally retains the independently authenticated foreign parent/history/object closure and its source authority chain without treating either as destination-native authority.
- **Release:** release refs and signed Release envelopes, release tree/blob closure, native authority chain, parent-release lineage, and each immediate signed predecessor Commit envelope. Verify predecessor ID/type/signature/anchored authority and require `Commit.tree == Release.tree`; ordinary parent IDs are explicit non-projected boundary commitments. A fork predecessor carries the signed `ForeignForkBoundary` evidence defined in §6.1.
- **Metadata:** native authority chain, release envelopes/headers, source-signed/hash-chained v2 ref transaction evidence, and signed `ForeignForkBoundary` evidence where present; no commit/tree/blob content. A metadata peer explicitly trusts the configured source federation key because discarded content cannot be revalidated locally. Initial sync validates full transient native and foreign proofs before discarding non-projected bytes.
The same `ProjectionCore` governs ingest, mirror, reads, pack export, backup, and compaction, with golden vectors for every edge/boundary. Extra objects outside the projection never become visible. Mode strings are strictly validated.
Mode transition is a per-repository state machine: build the target projection at source cursor C, apply or block later deltas, then atomically publish `{mode, refs, namespace membership, cursor}`. A reduced mode hides bytes immediately at publication; an expanded mode advertises nothing new until closure is complete. Crash recovery selects the old or new generation wholly.
`writeback=true` is rejected at startup in this cutover. It is not restored until it actually authenticates and forwards to the source with defined CAS, error, and durability semantics.
### Signed federation evidence
Each instance has a configured federation signing identity with explicit trusted peer keys and key epochs. It signs the `CommittedTransactionV1` hash chain, durable receipt result, and snapshot checkpoints over source key epoch, repo/genesis, `repo_sequence`/previous digest, operation/evidence digest, complete old/new refs and authority, projection/state digest, and durability result. Client signatures remain intent evidence; typed mirror/migration/administrative evidence authenticates those non-client sources; none alone proves local acceptance. Full/release peers still validate every retained native or foreign object/proof required by the projection. Metadata peers rely on the configured source trust root after transient initial validation.
Compaction carries each replayable event together with its required projection objects/proofs above the advertised low watermark. Reclaiming either advances the authenticated snapshot and watermark atomically as specified in §5.4; a mirror below that watermark receives `CursorExpired`, resynchronizes at cursor C, and then consumes C+1 onward.
### Federation gates
Under target source load, test a full, release, and metadata peer across 1 ms/20 ms/80 ms RTT, loss, a 10-minute partition, source restart, and mirror restart during apply.
- Federation alone may degrade an otherwise matching source run by ≤10%, but the release profile behind Caddy/TLS must still sustain ≥60k durable commits/s under the combined required reads + full-peer federation + checkpoint/compaction campaign. The minimum profile repeats this combined gate at batch=1 and ≥30k.
- Same-rack full mirror ref lag p99 ≤5 seconds; WAN ≤30 seconds.
- Partition catch-up reaches zero lag within two minutes after reconnection.
- Final object/ref/identity digests match each projection exactly.
- Crash during apply yields one old or new complete destination generation.
## 9. Migration, backup, and operations
### Offline instance migration
Add `levcs-instance migrate-store --source <legacy-root> --destination <v2-root>`.
The service must be stopped or the source must be an immutable snapshot. The migrator:
1. Acquires an exclusive source migration lock and refuses a running service unless the source is an immutable snapshot.
2. Requires the final destination path to be absent. Under its existing destination parent, creates a uniquely named sibling `<destination>.migrate-<operation_id>.tmp` with an ownership marker, verifies parent and staging directory have the same `st_dev`, and fsyncs the parent. It never stages in `/tmp` or across a mount boundary.
3. Enumerates every `<repo_id>/.levcs` ref and loose object without granting membership.
4. Pins and validates genesis/current authority and derives the selected mode projection from validated refs.
5. Verifies filename hash, raw framing, exact bytes, typed reachable closure, ref syntax/type, genesis-derived repo ID, signatures, and policies.
6. Imports only the validated reachable projection through `LegacyMigrationV1` namespace transactions and one baseline snapshot per repository. Unreachable legacy loose objects—including possible rejected-push junk—are reported and quarantined/dropped without becoming namespace members.
7. Syncs every segment, checkpoint, manifest, format marker, staging directory, and the staging root, then fsyncs the shared destination parent.
8. Reopens the staging root through production recovery and compares deterministic digests over the validated projected object/ref/evidence set.
9. Rechecks same-device placement and destination absence, installs with Linux `renameat2(RENAME_NOREPLACE)` from sibling staging name to final name, and fsyncs the one shared parent before reporting success. No copy fallback is allowed.
10. On restart, removes only incomplete sibling roots carrying a valid migration marker for this destination/operation, then fsyncs the parent; unknown siblings are untouched. The source remains read-only for a bounded rollback window and is never consulted at runtime.
Migration is restartable/idempotent by operation ID and emits machine-readable results. Legacy rollback is permitted only before v2 writes reopen. After any v2 acknowledgment, rollback means a separately validated migration of those transactions; the old binary may not reopen the root.
`migrate-store` requires a configured administrative signing identity and destination federation signing identity. Each imported repository records canonical `LegacyMigrationV1` evidence binding migration operation ID, source layout digest, validated manifest/projection digest, tool/storage version, and administrator key epoch/signature; the resulting `CommittedTransactionV1` is signed by the destination identity. It never fabricates a client envelope. Exact restore instead preserves those existing bytes and signatures without invoking either key.
### Backup
Raw live `rsync` is not a supported consistency mechanism.
`levcs-instance backup create` coordinates a per-shard committed-sequence vector, fences and rotates every active journal through that vector (or copies a checksum-validated immutable prefix), and persists a checksummed export manifest. It pins every referenced segment, index, checkpoint, journal-prefix, and not-yet-compacted adopted-projection manifest/object/index artifact before export. The export manifest records that complete dependency closure. The lease has configurable maximum age, pinned bytes, and compaction-debt ceilings; a stalled/slow export is aborted, its pins released, and its incomplete output invalidated before the storage high watermark can be crossed. `restore --destination <absent-root>` requires the destination to be absent and stages only in a uniquely named sibling `<destination>.restore-<backup_id>.tmp` under the existing destination parent. It verifies the export manifest, all files/projections/evidence, exact shard/repository sequences and cursors, same `st_dev`, and production recovery before installation; syncs every restored file/directory and the shared parent; rechecks absence; installs with `renameat2(RENAME_NOREPLACE)`; and fsyncs the parent before success. Destination collision is a no-change error; replacement/exchange of an existing root is not supported in this phase. Crash cleanup removes only sibling roots with a valid restore marker binding destination and backup ID, then fsyncs the parent. Restore creates no transaction/event and never invokes the transaction service.
Atomic filesystem/block snapshots are supported only when the whole instance root is captured at one point. Otherwise operators use checkpoint export or stop the service.
### Configuration
Add strict `[storage]` and `[ingest]` sections. Initial safe keys:
- Storage root, spool root/worker count/per-request/per-client/global bytes and FDs/minimum-free-space, shard count, max group transactions/bytes/delay, segment size, checkpoint cadence, compaction high/low watermarks, authoritative `max_retry_window`, `terminal_status_grace`, `status_tombstone_grace`, receipt retention, event retention, snapshot-token MAC key/epoch and base/request-pin age/count/pinned-byte/debt ceilings, same-device projection-staging subdirectory plus session/principal/global age/count/object/byte/file/debt ceilings, minimum supported projection-transfer rate and finalize margin, backup lease age/pinned-byte/debt ceilings, administrative and federation signing identities/key epochs, and hard ceilings for active-index entries/bytes, index runs/fan-out, replay frames/bytes, and open runs.
- Max request wire/decoded/object bytes, inline object/update/graph/authority counts, `max_projection_objects`/`max_projection_bytes`/chunk limits, in-flight operation count/bytes, validation concurrency, per-repo queue share, clock skew, replay timer resolution, and replay retention. Startup enforces `replay_retention >= 2 × clock_skew + timer_resolution` and checked nonzero retention/staging formulas.
Unknown modes and invalid limits fail startup. Startup verifies projection staging is on the target shard's device and, with checked ceiling division/arithmetic, that `ceil_div(max_projection_bytes, minimum_projection_transfer_rate) + finalize_margin <= staging_session_max_age`; otherwise it refuses the feature/configuration. Every benchmark archives the effective config. Tuning parameters may not be raised to hide overload.
### Observability
Expose at least:
- Accepted/rejected operations and reason.
- Commits/objects/application bytes/wire bytes per second.
- Receive, decode, hash, signature, authority, policy, queue, fence, publish, and response latency.
- Queue bytes/depth, group size, sync latency, shard utilization and fairness.
- Receive-spool queued buffers/bytes, open files, disk bytes, free-space rejections, I/O latency, cleanup count, and startup scavenging.
- Transaction-status counts and age by pending/resolving phase; shard poison cause and last known fence outcome.
- Replay entries, authority-cache hits, receipt-table size.
- Segment/index/checkpoint size, compaction debt, write/read amplification.
- Recovery duration, last durable `shard_sequence`, per-repository `repo_sequence`, corruption/read-only/disk-full state.
- Snapshot-token base/request and projection-export/backup lease counts, pinned bytes, age, expiry/release reason, and compaction debt.
- Projection-staging session/chunk/object/file/byte counts, age, validation/cleanup/adoption outcomes, pins, and compaction debt.
- Mirror cursor/lag/bytes and projection.
Liveness remains cheap. Readiness is false during recovery, migration, storage failure, or unresolved corruption.
## 10. Permanent benchmark and fault infrastructure
Add:
```text
tools/levcs-loadgen/
Cargo.toml
src/main.rs generate/run/verify/recover
bench/workloads/
small-commit.toml
federation.toml
bench/result-schema.json
scripts/bench-instance.sh
scripts/verify-instance-recovery.sh
scripts/compare-bench-results.py
```
The load generator is open-loop, async, deterministic by seed, emits the exact public v2 protocol, corrects coordinated omission, records HDR-style histograms, and keeps an external durable acknowledgment journal on a fault-isolated host. It pre-generates when measuring server capacity and separately reports full client serialization/signing cost.
Before calculating any rate, the evaluator excludes setup traffic; persists each received receipt with operation digest and generated IDs; requires unique Blob/Tree/Commit IDs and `sum(objects_new) == 3 × counted_commits`; recomputes every deterministic 1,024-byte Blob; proves each counted Commit is in the recovered closure of its acknowledged ref; reconciles operation IDs against recovered receipts; and excludes offered, rejected, duplicate, or unacknowledged operations. It rejects tmpfs, overlay, remote, or non-persistent data mounts, disabled durability, missing production validation flags, batch>64, or incomplete metadata. These checks and counts precede every verdict in the signed result bundle.
Every result bundle records source revision/dirty hash, Cargo.lock, compiler/flags, binary/config hashes, workload/seed/corpus digest, CPU/NUMA/governor/microcode, RAM/swap, filesystem/mount, NVMe model/firmware/cache/barrier/scheduler/temperature, NIC/driver/link/MTU, proxy/TLS, systemd/cgroup, telemetry versions, raw metrics, and every configured resource ceiling.
### Correctness/fault matrix
- Outer/embedded type mismatch; malformed reachable and unreachable entries.
- Missing/wrong-type Blob, Tree, parent, predecessor, parent-release, or Authority.
- Invalid/extra signatures, wrong release declarer, stale/foreign/forked authority.
- Duplicate ref updates, invalid namespace, branch→Release, release→Commit, invalid Delete.
- Stale CAS, concurrent same-ref race, unauthorized/non-FF force cases.
- Policy read/parse failure and disallowed record in newly exposed history.
- Init/init, init/push, push/mirror, authority-successor races.
- Authority transition with an unrelated successor-signed side branch/Release, multiple boundaries, or successor use in the same transaction; the equivalent later transaction succeeds.
- Mirror initial/resnapshot with complete `MirrorSnapshotV1`, relevant `MirrorEventV1`, reduced-projection cursor-only events, crash/retry, and source sequence gaps.
- Administrative actor/key-epoch/signature-domain substitution; migration chunk ordinal/count/digest collision and multi-repository operation-ID uniqueness.
- Valid public/private fork boundaries; forged source repo/genesis/tip/authority, wrong destination genesis, non-empty destination, and missing foreign projection proof.
- Concurrent same-ID/same-digest coalescing, same-ID/different-digest rejection while a reservation/record exists, leader failure/retry, disconnect, ambiguous fence, recovery-proved absence/release, and client same-digest retry. Deterministic status interleavings pause a reader after committed-root A, publish/remove status, and require the B recheck to return the receipt; they also leave a stale pre-store registry entry after publication and require durable store state to win.
- Deadline passage during receive/validation/queue rejects before append; passage immediately after append start and during poisoned recovery remains `OperationResolving`, forbids ID reuse while unresolved, and eventually returns the retained committed receipt or definitive `Unknown`/proved-absent result. Crash/recovery before the first visibility timestamp is checkpointed may extend but never shorten receipt retention.
- Snapshot-token wrong repo/projection/reader, release/expiry/restart, resource ceilings, and compaction between negotiation steps. Two acquisitions of one generation produce byte-identical signed checkpoints/digests and distinct lease tokens; mirror evidence contains neither token. Expiry/DELETE during an admitted object/Pack/push request rejects new uses but preserves exactly the accounted request pin until its response/cancellation or append handoff, then releases it within bounds.
- Missing-object codec count/byte/order/generation bounds; private GET/POST/DELETE route/body/token/key tampering and replay.
- Projection-staging auth/source/projection/final-operation substitution; chunk order/count/digest/manifest mismatch; duplicate idempotent chunk versus same-ordinal conflict; concurrent same/different finalizers; expiry racing finalization; quota/debt/low-space/source-export-lease/session-expiry cleanup; crash before/after artifact sync, shard adoption, final append/fence/root publication; and proof that no partial or abandoned session affects membership, reads, dedupe, refs, receipts, or events. Include an initial mirror, resnapshot, oversized Full fork, and network migration whose projections exceed inline transaction/group limits.
- Future timestamp at the positive skew boundary retained through `issued_at + skew + timer_resolution`; invalid retention configuration refuses startup.
- Receive-spool byte/FD/free-space exhaustion, short I/O, disconnect/panic cleanup, startup scavenging, and proof that Tokio/validation/shard threads perform no spool filesystem calls.
- Force/delete followed by compaction with a lagging consumer above/below the atomically advanced event floor.
- Migration sibling/same-device enforcement, occupied destination race, crash before/after `RENAME_NOREPLACE`, and parent-directory durability.
- Short writes, fsync errors, ENOSPC, corrupt index, corrupt segment, torn tail.
- Append, fence, atomic-root publish, rotation/CURRENT install, checkpoint, compaction base/tail install, mirror, migration, and pre/post-response boundaries.
- Successful fence followed by fail/panic before root CAS, during CAS retry, or during pre-publication writer supervision poisons and recovers the durable prefix exactly once. A separate failpoint after the committed-root/receipt swap but before waiter wakeup remains committed, requires no reappend, and returns the original receipt on status/retry.
- Maintenance-artifact handoff tests prove workers write only unreferenced names and that only the shard owner adopts, updates `CURRENT`, or unlinks; crashes leave either cleanable unreferenced artifacts or one fully referenced generation.
- Restore cross-device staging, occupied destination, malformed marker, crash before/after `RENAME_NOREPLACE`, parent fsync, and proof that no transaction/event is added.
- Stalled/slow backup export reaching lease age, pinned-byte, or compaction-debt limits; it must abort, release every pin, invalidate partial output, and let debt return below the low watermark without acknowledged loss.
Every named injection point receives deterministic before/after-durability failpoint campaigns before random testing. The append-through-publication matrix includes a known-successful fence followed by every pre-publication failpoint and writer panic; the oracle requires poison, `Resolving`, recovery publication exactly once, and no subsequent append before recovery. Post-publication/pre-response failpoints instead require the receipt already visible and idempotently retrievable. Run one complete matrix in each headline topology at ≥30k; repeat a declared subset covering append, fence, publish, compaction install, and response at ≥60k. Then run 100 randomized `SIGKILL` cycles and at least 20 abrupt VM/physical power cuts at ≥30k through the frozen production block stack. The external ACK journal is durably updated before an operation is counted. Production recovery and complete graph/ref/receipt reconciliation run after every fault before the next cycle. `SIGKILL` is not a substitute for power loss.
All destructive or privileged setup, `dm-flakey`, block-device, cache/barrier, and power-cut steps must live in reviewed shell scripts that:
- Require root explicitly and print the exact device/config being touched.
- Refuse the root filesystem and any non-empty/non-whitelisted device.
- Default to dry-run.
- Capture before/after state and provide cleanup traps.
- Never embed an undocumented `sudo` sequence in benchmark prose.
Ordinary performance profiling remains unprivileged where the host permits it.
### Overload and interference
At 1.25× and 2× target offered load, every configured queue cap holds, useful admitted throughput remains ≥90% of target, excess work receives documented overload responses, and p99 returns ≤100 ms within 30 seconds after load normalizes.
The two-hour P5 soaks span at least two checkpoint, compaction, and idempotency/event-expiry cycles. On the release profile behind Caddy/TLS, a combined reads + full-peer federation + checkpoint/compaction run must sustain ≥60k durable commits/s in both headline topologies and meet P4 windows/latency; degradation versus matching no-background runs is also ≤15% and federation-only ≤10%. The minimum profile repeats the full two-hour soak and combined-background campaign at batch=1 and ≥30k in both topologies; no batched run may substitute. Backup is a separate campaign with pinned-export latency and primary impact reported. Every numeric CPU/storage/memory/FD/queue/debt/no-growth ceiling in §3 applies.
## 11. Exact source change map
### Workspace and new store
- `Cargo.toml`, `Cargo.lock`: add `levcs-store`, load generator, and measured concurrency/index/histogram dependencies.
- New `crates/levcs-store/**`: transaction journal, recovery, namespace index, snapshots, checkpoints, compaction, projection staging/adoption, backup, migration, tests, and durable-ingest benchmark.
### Protocol and identity
- `crates/levcs-protocol/src/wire.rs` or new `push_v2.rs`: canonical envelope, normal/fork push kind, typed mirror/admin evidence, dual sequences, typed transaction status/CAS, deterministic signed-generation versus ephemeral lease response, projection-staging session/chunk/manifest/finalize codecs, missing-object codec, transaction event, receipts, limits/errors.
- `crates/levcs-protocol/src/auth.rs`: domain-separated v2 ingestion/read signing, typed route canonicalization, checked retry-deadline/replay-horizon bounds, snapshot-token MAC, and corrected key encoding documentation/test vectors.
- `crates/levcs-protocol/src/pack.rs`: bounded streaming reader/writer preserving Pack v1 bytes.
- Protocol property/fuzz tests and pack benchmarks: v2 codec, real signed graphs, limits, golden vectors.
- `crates/levcs-identity/src/verify.rs`: `TrustAnchor`, `ForeignForkBoundary`, `VerificationSession`, shared authority-cache facts, and whole native/foreign reachable-graph result.
- `crates/levcs-identity/src/sign.rs` and release verification: align declarer/signature/role semantics.
### Instance
- Split `crates/levcs-instance/src/lib.rs` into the modules in §7, including the dedicated receive-spool executor.
- Remove `AppState::repo_dir`, `AppState::store`, `repo_locks`, direct `ObjectStore`/`Refs` construction, and the process-wide nonce cache.
- Replace `handle_init`/`handle_push` with `IngestService` and linearizable receipt-precedence operation status/coalescing, including deadline-versus-resolution retention.
- Replace path/ref enumeration in info/refs/object/pack endpoints with snapshot reads; add signed missing-object, private-read/lease, and projection-staging session/chunk/status/finalize/abort routes.
- Replace `collect_closure`, `is_ancestor`, and `find_merge_record` rereads with verified graph facts/index metadata.
- Rewrite `mirror.rs` around `MirrorSnapshotV1`, bounded projection-staging for oversized snapshots, projected/cursor-only `MirrorEventV1`, and destination transaction submission.
- `main.rs`: strict config validation, spool/shard supervision, storage recovery/readiness, subcommands for migrate/verify/backup/restore, and service composition.
- Instance tests: retain dogfood, force-push, policy, storage-mode, health, federation, and mirror behavior; add atomicity, identity substitution, idempotency/status, post-fence publication failure, spool, private reads, restore, limits, and concurrency suites.
### Client and CLI
- `crates/levcs-client/src/lib.rs`: v2 prepared streaming push/init, signed private reads, bounded missing negotiation, snapshot base/request-pin lifecycle, projection-staging sessions, typed transaction status/feed, structured receipts and errors.
- `crates/levcs-cli/src/fed_cmds.rs`: correct remote CAS, token-bound missing-object negotiation/push, deterministic complete closure, pending/resolving/terminal-retention receipt verification, receive-side identity verification, projection-staged snapshot/final-delta migration.
- `crates/levcs-cli` federation/P2P tests: preserve repo ID, exact object bytes, authority, release, merge-record, and force behavior.
### Deployment and documentation
- `deploy/instance.toml.example`: strict storage/ingest limits and corrected modes/writeback.
- `deploy/levcs-instance.service`: readiness, shutdown/checkpoint behavior, resource limits proven by P4.
- Caddy/nginx examples: measured body/streaming/timeouts and metrics/readiness routing.
- `deploy/README.md`, `README.md`, `doc/technical-report.md`: v2 protocol, durable receipt meaning, exact storage modes, migration, checkpoint backup, measured claims only.
- `scripts/bench.sh`: retain microbench purpose, register omitted storage/GC benches, emit machine-readable metadata; do not mix its results with instance gates.
## 12. Implementation phases and subagent execution
The lead owns contracts, shared types, workspace dependency changes, phase gates, cross-slice integration, and final verification. Agents never independently change a frozen shared interface. Each wave starts only after its prerequisites and ends with a review agent plus lead-run targeted gates.
### Phase 0 — freeze contracts and oracles
Lead deliverables:
- Approve this plan and freeze object invariants, v2 envelope/push-kind/fork fields, single-transaction authority ordering, checked retry/replay-horizon bounds, pre-append expiry versus append-start resolution and terminal-status retention, typed Set/Delete ref and authority-boundary CAS, durable append-through-publication ACK/idempotency/status/coalescing semantics, linearizable two-root/registry status precedence, exact storage projections, canonical workload, and result schema.
- Freeze `TransactionEvidenceV1` including mirror snapshot/event/cursor-only and administrative actor/chunk fields, `CommittedTransactionV1`, `ProjectionStageSessionV1`/chunk/manifest/`StagedProjectionInstallV1`, independent shard/repository sequences, per-repository hash chain, federation signing identity/key epochs, signed private-read/missing-object codecs, deterministic signed snapshot checkpoint versus ephemeral token and request-pin lifecycle, event/object low-watermark/expiry/resnapshot behavior, and mixed-version maintenance cutover.
- Add golden legacy/fork/mirror/admin evidence fixtures and deterministic snapshot/object/ref/evidence/event/status digests.
- Record reference hardware profiles.
Parallel agents after the freeze:
- **ProtocolFixtures:** v2 ingestion/private-read/missing/status/evidence golden vectors and malformed envelope/Pack properties; owns protocol tests only.
- **DurabilityOracle:** external acknowledgment journal, append-through-publication poison/recovery model, and restore fixtures; owns benchmark support only.
- **IdentityAdversary:** foreign/stale authority, same-transaction successor use, signature, release declarer, closure, and policy adversarial fixtures; owns identity/instance test fixtures only.
Exit: P0 tests demonstrate current failures where expected and define the new observable contract without benchmark-only shortcuts. Consumer fixtures cover inline and staged mirror snapshot, projected event, cursor-only event, every administrative source, cursor-expired resnapshot, signed-generation/token separation, deadline-crossing resolution, and typed status. Deterministic status races and failpoints cover every append-through-publication boundary.
#### Frozen Phase 0 artifacts
The Phase 0 freeze was completed on 2026-07-24 and amended the same day by the contract review recorded below. The authoritative artifacts are:
- `crates/levcs-protocol/src/codec.rs`, `v2.rs`, and `oracle.rs` for canonical codecs, signed logical contracts, projection/staging rules, deadline/status/coalescing behavior, external ACK journaling, restore rules, and append-through-publication failpoint outcomes.
- `crates/levcs-protocol/tests/fixtures/phase0-vectors.json` for byte-exact object, Pack v1, ingestion, retry, fork, private-read, snapshot, staging, evidence, event, and typed-status vectors/digests. `cargo run -p levcs-protocol --example phase0_golden` regenerates the candidate vector set for deliberate review.
- `crates/levcs-identity/tests/fixtures/phase0-adversarial.json` plus the Phase 0 identity tests for foreign/stale authority, same-transaction successor, signature, release-declarer, closure, and fail-closed policy expectations. Cases marked as legacy accepts are intentional proofs of the gap that Phase 2 must close, not authorization to preserve that behavior.
- `bench/workloads/small-commit.toml`, `bench/workloads/federation.toml`, `bench/result-schema.json`, and `bench/reference-hardware.toml` for the canonical workload, integrity metadata, independent verdicts, and minimum/release machine profiles.
The enforced Phase 0 gate is `scripts/check-phase0.sh`. It runs formatting, every workspace test (including golden, malformed-input, adversarial, consumer, concurrency/status, failpoint, ACK, restore, workload/schema, and hardware-profile tests), and the repository whitespace check. The whitespace check inspects every tracked-or-untracked, non-ignored, non-binary, non-Markdown file directly; `git diff --check` alone is not sufficient because it ignores untracked files and is a no-op on a clean checkout. Changing a frozen discriminant, canonical byte, domain, digest, workload rule, result field, or reference profile requires an explicit protocol/benchmark contract review and corresponding golden update.
##### Contract review 2026-07-24-A
The initial freeze was reviewed before Phase 1 scoping. The review found one unsound frozen contract, four contract defects, and a set of items the freeze claimed but did not pin. All were corrected and the goldens were regenerated once, deliberately, under this record. Phase 1 binds to the amended artifacts.
Corrected contracts (these changed frozen bytes or behavior):
- `oracle.rs` append-through-publication outcomes: `BeforeFence`, `FenceFailed`, and `WriterPanicBeforeFence` were frozen as deterministically `AbsentRetriable` while the physically identical `AfterFrameWrite` state was `EitherWhole`. A complete unfenced frame may still reach durable storage through page-cache writeback, and a failed `fsync` does not prove non-durability, so §5.2 tail recovery could legitimately resolve such a frame as committed. All five now map to `EitherWhole`; only states before any frame write remain deterministically absent.
- `RepoSnapshotV1` ref lists enforced strict sorting on the `(target, object)` pair, which admitted one branch at two different tips into a signed generation and its `ref_state_digest`. Ref targets are now unique, consistent with every other ref list in the file.
- `SnapshotLeaseClaimsV1` now carries the §6.2 field set: `version`, `mac_key_epoch`, `token_id`, `repo_id`, `projection`, `generation_digest`, `high_repo_sequence`, `high_event_digest`, `issued_at_micros`, `expires_at_micros`, and `reader_key_digest_or_zero` (replacing the raw reader key). `SNAPSHOT_LEASE_TOKEN_VERSION` is checked on both mint and verify.
- Mirror destination operation IDs are now derived and enforced, not assumed: `mirror_event_operation_id` binds source instance/repository/`repo_sequence`; `mirror_snapshot_operation_id` binds source instance/repository/generation digest/destination projection/projected manifest digest. `validate_mirror_application` checks both.
- `validate_projection_stage_finalize` rebinds a finalizing transaction to its session's exact `final_operation_id`, `final_operation_digest`, `final_evidence_digest`, actor, and Fork proof, and rejects a session past `expires_at_micros`. `validate_projection_stage_binding` proves only byte/manifest/install consistency and never bound the operation it was created for.
- Push and init HTTP body framing is now a codec with golden bytes: `envelope_len` is `u32` little-endian, followed by the signed envelope and exactly `pack_len`/`genesis_len` trailing bytes.
- `PushOperationV2::validate` rejects `authority_update == expected_authority`. Decode-side `Vec::with_capacity` is bounded by remaining input at every count-prefixed site.
Freeze coverage added (no behavior change):
- Golden vectors for the `SignedClientOperationV2::Init` discriminant, the init stable digest, `ProjectionStageSessionV1`/`StagedProjectionInstallV1` and the session digest, `SnapshotLeaseTokenV1`, `TransactionPageV1`, `MaintenanceCutoverV1`, both request-body framings, and one `SignedReadRequestV2` per HTTP method including a public zero-token acquisition. All six `SourceKindV1` values are pinned.
- Negative coverage the freeze named but lacked: lease-token MAC tamper, wrong MAC key, and wrong token version; clock-skew rejection in both directions; durable same-ID/different-digest `OperationIdMismatch`; the lease oracle's immediate-release path; and a `ReplayGuardOracle` proving `(public_key, nonce)` is held through `issued_at + clock_skew + timer_resolution` and never evicted early. The envelope-mutation fuzz test now asserts that mutated bytes never verify, which its name previously only claimed.
- Every failpoint's `recovery_outcome` is asserted individually; the prior catch-all left 14 of 17 unpinned, which is why the unsound classification above survived the original review.
- Benchmark contracts pin all four topology rows by name with `persistent_clients`, read the `[measurement]` table, and compare required-array names rather than lengths. The §3 one-minute-window rule (≥95% meeting target, none below 90%) and the per-gate latency ceilings are now encoded in both `small-commit.toml` and `result-schema.json`, the latter enforced conditionally on `gate`. `federation.toml` gained the §8 packet-loss dimension, and its lag/catch-up/degradation numbers are now asserted.
- `phase0-adversarial.json` rows are bound to live verifier outcomes where a Phase 0 code path exists, and the full table is pinned by `(name, class, legacy_observation, v2_expected)`. The three `not-exercised` rows now state why no Phase 0 call site exists and which later phase owns them; they are deferred coverage, not silent gaps.
The `verify.rs` release-declarer change (parse through `Release::from_signed`, reject duplicate signers) is the §7 fix aligning the canonical verifier with `sign_release`, and lands with these tests because the fixtures record post-fix behavior.
Known deferred: identity release-signer authorization still checks Authority membership only, not role. §11 assigns role-semantics alignment to Phase 2; no fixture class currently documents this gap, so the Phase 2 change will not trip a frozen expectation.
##### Contract review 2026-07-24-B
Phase 1 scoping found that no P2 result bundle could validate against the frozen `bench/result-schema.json`. The schema required all eleven `workload.validation_flags` to be `const: true` for every gate, including `storage_primitive`, but a P2 run measures the `levcs-store` API, which §5.1 forbids from making identity-role, merge-policy, or federation decisions and which runs in-process with no proxy or TLS. Reporting those flags as true would be false; reporting them false failed the schema. The gate was therefore unreachable as specified, not merely awkward.
Two frozen benchmark artifacts are amended. The regenerated schema and profile are the authoritative ones from this record forward.
`bench/result-schema.json`:
- `workload.validation_flags` values are now pinned per flag, conditionally on `gate`. For `storage_primitive`, `durability_fence_before_response` and `typed_ref_cas` are `true` — the fence is the claim the bundle exists to certify, and the shard sequencer genuinely performs the typed CAS against speculative state — and the other nine are `false`. `fast_forward` is among the nine: §7 stage 9 has the sequencer consuming precomputed ancestry facts rather than deriving them, and a validation flag must state what the measured system performed.
- Relaxing `const: true` in `$defs.validation_flags` is not un-pinning. Every other gate re-pins all eleven to `true` in the same conditional rule's `else` branch. Without that, the amendment would silently permit a deployed-node bundle declaring `complete_graph: false`, which is a strictly worse defect than the one being fixed. A test asserts both branches, and the schema was validated against constructed bundles proving that each cheat is rejected and each honest bundle accepted.
- `promotable` is required at top level for every gate: `const: false` under `storage_primitive`, `const: true` otherwise. This moves §3's rule that a storage primitive result can never be promoted to an instance throughput claim out of prose and into a mechanical check.
- `workload.generator` is required, so an evaluator can recompute every deterministic 1,024-byte Blob from the bundle alone rather than trusting the harness's claim to have generated the canonical workload.
- `deployment.store_directory_attributes` is required, recording the attributes the run verified at startup.
`bench/reference-hardware.toml`: both profiles gain `[profile.filesystem].store_directory_attributes = "nodatacow"` and the `store_directories` it applies to. The frozen filesystem is btrfs with `compress=zstd:3` and data checksums; per-directory `nodatacow` on the journal and segment directories restores in-place overwrite, stops zstd burning CPU on incompressible frames, and leaves the frame digest as the sole integrity check, which it already is. A profile that silently permits two different on-disk configurations for the files carrying the throughput is not frozen, so this belongs to the profile rather than to per-bundle metadata. Recording is additionally required but is not sufficient: the benchmark must read the effective attributes back at startup and refuse to run on mismatch, because a silently copy-on-write-mounted run would otherwise emit a bundle claiming `nodatacow` and produce a number incomparable to every other P2 result while looking identical.
**Third amendment (2026-07-26), from the Wave A review.** The first pass split `workload.validation_flags` per gate but left `$defs.verification` and `measurement.coordinated_omission_corrected` blanket `const: true`. That is the same defect one block over: a `storage_primitive` bundle had to certify `blobs_recomputed`, `metadata_complete`, `operation_receipts_reconciled`, `unique_blob_tree_commit_ids`, and `objects_new_equals_three_per_commit` for a layer §5.1 forbids from touching an object graph, and had to claim coordinated-omission correction that a closed-loop driver does not perform. The Wave A harness emitted all six as `true`, so the first honest thing the storage gate would have produced was six false certifications.
The amendment distinguishes two cases that the blanket `const: true` had collapsed, and the distinction is the substance of it:
- **Not applicable → omitted.** The five object-graph claims are *forbidden* at `storage_primitive` — a `not`/`anyOf` clause, not merely optional — and required-and-true at every other gate. `false` is not the honest encoding: it asserts the check applied and did not pass, which is its own untrue statement. Absence is the only encoding that says "there is no object graph here."
- **Applicable but not performed → `false`.** `coordinated_omission_corrected` relaxes to `type: boolean` in the base and is re-pinned `const: true` in the `else`. CO correction applies to any latency measurement; the Wave A driver simply does not do it, and must say so rather than omit the field.
`objects_new_equals_three_per_commit` additionally carries a new normative description: it must be counted independently of the transaction total. The harness derived `objects_new = transactions * 3` and then asserted the flag, making the schema's anti-batch-gaming check unfalsifiable — a flag that cannot fail is not a check. It is forbidden at `storage_primitive`, and B1 must count objects from what the store actually staged before it may be emitted at any gate.
The `else`-branch re-pin rule from the first amendment applies unchanged, and this pass proved why it needs a test rather than a convention: the edit that added these clauses **destroyed** the existing `else` that re-pinned `promotable` and all eleven validation flags, silently un-pinning every instance gate — the exact failure the first amendment was written to prevent, committed by the amendment fixing an honesty defect one field over. It was caught by re-validating against constructed bundles rather than by reading the diff. The contract test now asserts the `else` branch member by member and was proven against six injected regressions, including that one.
Under `nodatacow` a torn block reads back as garbage rather than as `EIO`; under copy-on-write an ordinary crash leaves zeros or stale preallocated content. Both, plus `EIO` from a device that lost an acknowledged write — live here because the profile has `write_cache = enabled` and `power_loss_protection = false` — must be handled as end-of-tail rather than as a fatal store error. Treating `EIO` that way can silently drop a frame that was fenced and then lost by the device; nothing on the device distinguishes that from an unfenced tail, so the external ACK-journal reconciliation is the detector, and a non-zero `acknowledged_loss` there is a hardware finding that invalidates the run.
##### Contract review 2026-07-26-A
Wave B scoping found that `oracle::append_publication_expectation` classified
`EvidenceHandoffFailure` as poisoning the shard, alongside `AfterMarkedResolving`, with
`immediate_status: Resolving`. That is physically wrong and operationally harmful.
The failpoint fires while the sequencer hands a transaction to the `CommitEvidenceSigner`
§7 stage 9, before the group is marked `Resolving` and before any byte is written. The
physical state is `NoBytes` and nothing about the outcome is ambiguous. The frozen store-side
fixture already said so in its own rationale ("signer failure occurs before journal append
and therefore writes nothing"), which is how the disagreement was found: the fixture and the
oracle described the same row differently.
The operational cost of the old classification is the decisive part. A routine
`SignerError::Unavailable` — a restarting or briefly overloaded signer, with no storage fault
of any kind — would poison the shard and admit no mutation until recovery ran. That trades a
real availability property for a safety property that was never at risk.
§7 is authoritative and says the opposite: a failure before append "removes the transient
in-flight entry, releases or revalidates its speculative suffix, and wakes every waiter with
the same error; no durable status reservation is created."
`EvidenceHandoffFailure` therefore takes the exact `BeforeAppend` shape: `shard_poisoned:
false`, `immediate_status: DefinitiveAbsent`, `recovery_outcome: AbsentRetriable`,
`acknowledgment_allowed: false`, `later_append_allowed_before_recovery: true`. The physical
state class remains `NoBytes` and the crash-matrix fixture is unchanged. The failpoint stays
assigned to **Wave B**, because only B1's sequencer can exercise a signer handoff at all.
Amended: `crates/levcs-protocol/src/oracle.rs`, and
`crates/levcs-protocol/tests/phase0_oracles.rs`, where the row moves beside `BeforeAppend`.
The same edit replaced that test's `_` catch-all arm with an exhaustive list of the thirteen
poisoning failpoints. A catch-all in a test that pins a frozen classification silently
absorbs any newly added row into "poisoned" — which is the mechanism by which a wrong
classification ships past its own test, the same defect as contract review 2026-07-24-A.
Adding a failpoint must now fail to compile until someone classifies it.
This is the second frozen Phase 0 classification found to be physically wrong. Both were
found by asking what state the device is actually in, rather than by checking the model
against itself.
##### Contract review 2026-07-27-A
Wave B's D0-B publication freeze requires nine amendments to Wave A frozen or
signature-frozen surfaces. They land as one reviewed interface change before B1, B3, or B4
is dispatched:
- `lib.rs` declares and re-exports the immutable publication roots and runtime-agnostic
completion primitive.
- The workspace and `levcs-store` manifests take `im`; `Cargo.lock` records the resolved
graph. Decision 9.7 applies structural sharing to repositories, terminal
receipts/tombstones, typed refs, and transient operation statuses. Canonically iterated
refs use `OrdMap`; unordered hot lookup maps use the HAMT-backed `HashMap`.
- `recovery.rs` exposes one production recovery path returning `RecoveredShard`, including
the complete layered index, catalog, refs, exact receipts, sequences, report, staging
resolutions, and live retained segment/index/checkpoint/tail ownership. A root-wide
`RecoverySession` holds `LOCK` continuously across all shard recoveries and is retained by
the engine; the one-shot drive wrapper uses that same session. `drive.rs` now retains the
recovered state instead of projecting it down to diagnostics.
- `options.rs` gains the status-root and projection-staging session/principal/global
count/object/byte/file/age/rate/debt ceilings. Startup checks all non-zero and nesting
constraints and, with checked ceiling arithmetic, refuses a configuration in which one
maximal projection cannot finish before the session horizon.
- `types.rs` makes the exact shared completion outcome cloneable. `StoreError::Io` carries
`Arc<std::io::Error>` and a handwritten `From<std::io::Error>` preserves existing `?`
call sites without reconstructing the error; `StoreError::Overloaded` is the typed
status-capacity refusal from decision 9.8.
- `transaction.rs` and `staging.rs` freeze the opaque projection-adoption handle. The handle
is the pin, exposes only read-only resolution and committed-root reference proof, and must
end as adopted, definitively failed before append, or transferred to recovery. Recovery's
committed/proved-absent notification is part of the same seam, and cleanup may run only
after recovery resolves the shard. A staging-owned recovery resolver turns every
authoritative committed descriptor into namespace-scoped index layers and live artifact
pins before readiness; notification alone is insufficient.
The resolver exposed one Wave A index assumption that inline-only tests could not exercise.
`IndexLocation` names the complete certified storage record, not naked object bytes. That
record is a transaction frame for inline objects and a canonical digest-bound stage chunk
for an adopted projection. The packed storage-version-1 fields do not change: the captured
committed root maps `segment_generation` to a retained source kind and selects the
corresponding decoder. Generation collisions across source kinds are corruption. This
amends the `index.rs` contract without changing its bytes.
Integration found one additional frozen-format defect. A canonical transaction frame
contains the exact applied refs returned in `CommitReceipt`, but Wave A's checkpoint
`ReceiptRecord` omitted them. After the replay horizon moved past the frame, current ref
state could not reconstruct old values, deletions, `force`, or transaction membership, so
the same committed retry could return a different receipt after reopen.
`checkpoint.rs` therefore adds a bounded applied-ref vector to each retained receipt and
sets authenticated checkpoint capability flag bit 0. Production recovery and the drive
seam populate it only from the canonical committed transaction. A storage-version-1
checkpoint without the flag remains readable if it has no retained receipts; if it has any,
the generation is rejected as `ReceiptRefsUnavailable` and the existing fallback/offline
rebuild policy applies. Older readers already reject the new non-zero flag. The derived
checkpoint format therefore fails closed in both directions without a global
`STORAGE_VERSION` bump; journal and segment authority bytes are unchanged.
Production-path integration also found that Wave A's recovery stopped after logical replay:
it did not perform normative step 8, so a damaged active journal could be reported ready
without sealing its validated prefix and installing a fresh active journal. Recovery now
preserves crash evidence, constructs the recovered segment and fresh journal through
deterministic resumable names, validates any pre-existing construction artifact byte for
byte, and publishes readiness only after the repaired physical state is complete.
The same amendment tightens manifest authority. Checkpoints are derived and may fall back
only among checkpoint rows retained by the selected authoritative manifest; their failure
does not authorize a shorter transaction tail. Missing or corrupt authoritative segment
bytes refuse readiness. If `CURRENT` is corrupt or missing, recovery may choose the highest
valid finalized manifest only when no higher invalid finalized manifest makes the closure
ambiguous. Manifest tuple validation includes segment generation/sequence coverage, index
generation, and checkpoint sequence. Generation allocation scans immutable names and moves
above rejected artifacts within a configured bound, so repair never reuses an ambiguous
name. The drive checkpoint path now seals and publishes its checkpoint reference in one
manifest installation, keeping the test seam subject to the production authority model.
The new `roots.rs` and `completion.rs` files are not amendments to Wave A, but their
interfaces freeze with this review. The completion state stores exactly
`Result<CommitReceipt, StoreError>` behind one mutex shared by outcome and the full waiter
set. Publication wakes outside the mutex, a dropped waiter is not a publication failure,
and every waiter receives one owned clone of the same result. The committed root uses pure,
idempotent subtree merge and carries addressable live pins for every retained artifact so a
captured read cannot race reclamation.
**D0-B was frozen on 2026-07-27 at commit
`5111d655da7689c73b3ad3189d9ecb3cd207f888`** after the complete Phase 1 gate returned
`GATE_EXIT=0`. B1 NamespaceTxn, B3 StagingSessions, and B4 StoreHarnessB may now dispatch
against that surface; changes to it require another contract review here.
##### Contract review 2026-07-28-A
B2's adversarial review of the B1/B3 slice returned five interface requests against frozen
surfaces. All five are granted, one with a tightened bound, and they land as a single
reviewed change before either package's fix pass — so that B1 and B3 consume a contract
rather than negotiate with it. Four of the five exist to delete a restated frozen table,
which is the Wave A finding in its interface form: a package that cannot reach a mapping
does not stop needing it, it writes a second copy that agrees today.
**1. `CommitEvidenceSigner::sign_event` signs the signing digest, and now says so.** The
parameter was named `event_digest`. What a correct implementation must sign is
`SignedCommittedTransactionV1::signing_digest(transaction, source_key_epoch,
durability_result)`, because that is the value the frozen `verify` recomputes. The two are
not interchangeable: the signing digest commits to the key epoch and the durability result
in addition to the canonical transaction, while the event digest is the chain identity that
`previous_event_digest` links. B1 inferred this correctly and passes the signing digest;
the interface, read literally, told it to do something else.
The weaker fix — a doc line on the trait, leaving the parameter named `event_digest` — was
rejected because of where the error surfaces. A signature over the wrong message is
produced by the signer, accepted by the store, fenced, and made durable, and is first
detected by a *verifier*, which may be a mirror on another instance. There is no point
between those two events at which anything can decline the transaction. An interface whose
misuse is undetectable until after durability has to state the requirement in the name a
caller types, not in prose beside it.
Amended: `types.rs`, and scope §2.2 where the frozen signature is printed. No
implementation changed; `engine.rs` already passed the correct value.
**2. `TransactionEvidenceV1::actor()`, which closes a P1 rather than adding a
convenience.** `validate_mirror_application` requires `destination_event.actor ==
source_instance` for all four mirror kinds, and the administrative signing digests bind
`actor` directly, so `actor` is a property of the evidence that the committed event
restates. B1 filled it from the destination signer's public key — the only value it could
reach — which is a durable event that signs, fences, and publishes and *then* fails
evidence verification at the mirror. That is the worst available failure ordering: the
transaction is committed and unwithdrawable before anything says it is malformed.
The accessor is exhaustive over the six variants and reads the client principal off the
signed envelope's signer. Deriving it in the store instead was rejected on the grounds that
a second derivation is precisely what produced the defect: whatever the store computes,
`validate_mirror_application` is the authority, and only the protocol crate can be wrong
about it in one place.
Amended: `v2.rs`. `phase0_consumers.rs`'s `destination_event_for` helper now takes the
actor from the accessor instead of its own match, and its `_` catch-all — which had been
supplying both the actor and a constant operation ID to four named variants — is replaced
with those four named arms. The same edit adds a test asserting the actor of every evidence
variant and proving that substituting the destination instance's own key makes mirror
validation fail. Charter item 8: the accessor's first consumer is a test that would have
caught the P1, not a helper beside it.
**3. One object-type table in the crate.** `format::object_type_code` and an identical
private copy in `recovery.rs` both existed, so `engine.rs` wrote a third. `format`'s is now
`pub(crate)` and `recovery.rs`'s twin is deleted; `recovery.rs` calls the shared one. The
half-measure — exposing `format`'s and leaving recovery's in place — would have left the
crate with two tables and a new way to reach a third, which is worse than either, because
the exposure makes it look resolved.
**4. Typed `RefRecord` conversion.** `checkpoint::RefRecord` carried `ref_kind: u8` with no
way to interpret it, so `engine.rs` restated the branch/release codes to read a record
`checkpoint.rs` had written. `RefRecord::from_target` and `RefRecord::target` now own that
conversion, with the code table stated once per direction in `checkpoint.rs` and
`read_applied_refs` rewired onto the decode half. `recovery.rs`'s replay map is keyed by
`RefTarget` rather than by the physical `(kind, name)` pair, which removes its copy too.
An unknown code is refused as `CheckpointError::RefKind(u8)`, carrying the byte. Neither a
default to `Branch` nor a skip is acceptable: a ref table is derived state that the next
checkpoint writes back, so a silently reclassified ref is laundered into the record within
one checkpoint interval and is indistinguishable from a ref that was always there. Naming
the byte is the difference between a torn write and a reader that predates a kind a newer
writer emits.
The refusal is at interpretation, not at checkpoint decode. Rejecting an unknown kind while
decoding the body would be stronger — it would let checkpoint fallback try another
generation instead of failing at first use — but it changes which generations are loadable,
which is a durability-visible change to a frozen reader and beyond what these five rulings
grant. It deserves its own ruling; it does not deserve to ride along beside four table
deletions.
`format.rs` keeps its own ref-kind table for the *frame* encoding, deliberately. That is
not the duplicate that mattered: the frame and the checkpoint are independently versioned
physical formats, each self-consistent, and a frame code is only ever compared to a frame
code. The copies worth removing were the ones interpreting a `RefRecord` produced
elsewhere.
**5. Projection ceilings, approved with a tightened bound.** `max_projection_objects`
defaulted to 100,000,000 while `ProjectionStageManifestV1.objects` is a flat canonical
vector capped at `MAX_CANONICAL_ITEMS` (1,000,000). The ceiling is now that cap and the
default is lowered to it; `max_projection_chunks` is capped identically against
`chunk_digests`; and `max_projection_bytes` is capped at `max_projection_chunks *
MAX_CANONICAL_BYTES`, since object bytes travel in per-chunk canonical encodings. The byte
bound is necessary and not sufficient — a chunk also pays framing — and it is stated that
way in the code, because a bound advertised as exact and enforced as approximate is worse
than one that admits what it is.
**The reasoning that belongs on the record: supporting a hundred million objects requires a
versioned chunked/indexed manifest design, not a larger hostile-decode ceiling.** Raising
`MAX_CANONICAL_ITEMS` would buy the object count by giving up the property the ceiling
exists for — that a reader can bound its work before it materializes an attacker-declared
vector. The right shape is a manifest a reader can traverse in pieces, and that is a
Phase 2+ format with its own version, not a constant edit.
Refusing at startup rather than at seal is the other half. The old configuration did not
fail early; it admitted sessions, pinned a principal's whole staging budget, accepted hours
of transfer at the supported floor, and refused at manifest encode. `options.rs` now
refuses the configuration, which is the only point where the operator learns this from the
configuration instead of from a stuck mirror.
Amended: `options.rs`, with tests asserting the boundary in both directions, that the
default *is* the format ceiling, and that the byte bound tracks the chunk allowance rather
than a fixed number. `staging.rs`'s per-session `MAX_CANONICAL_ITEMS` checks are now
redundant with startup validation; they are B3's to remove or to keep as a belt, and either
is defensible now that no admitted configuration can reach them.
**Expected collateral.** The tightened ceilings break exactly one test,
`staging_sessions.rs::a_declared_projection_over_a_configured_ceiling_is_refused_before_pinning`,
which sets `max_projection_chunks = 1` while leaving `max_projection_bytes` at the 1 TiB
default — now an unsatisfiable configuration that `StoreOptions::validate` refuses before
the store opens. The test's intent survives; its fixture has to vary the two together. It
is B3's file and folds into B3's fix pass. The gate is transiently red between the two
passes, which is accepted: landing the contract first is what keeps both packages from
implementing against a surface that is about to move.
##### Contract review 2026-07-28-B
B4's crash harness found that an immediate reopen after `StoreEngine` is dropped
intermittently fails `AlreadyLocked` — up to 29 retries over about 147 ms, seven failures in
forty uninstrumented runs, on a single-shard engine. It compensated with a bounded, measured
wait and filed the finding rather than fixing a file it does not own, which is what §6.0
asks for. The finding is granted, and the fix lands in the frozen root-lock primitive
(`segment.rs`, `sys.rs`) rather than in `StoreEngine::drop`.
**What was actually wrong.** Not `Drop` ordering. `StoreEngine::drop` closes every shard
channel, joins every writer, and leaves no extra `Arc<EngineShared>` behind; the refusal
comes from `segment::lock_root`'s filesystem lock on `<root>/LOCK`, not from B3's
process-wide staging guard, which returns `Conflict`. **An `flock` is held by the open file
description, not by the file descriptor.** A concurrently forked child inherits a descriptor
onto the parent's open descriptions, and `FD_CLOEXEC` closes it at `exec`, not at `fork`. So
for the whole fork-to-exec window — and for as long as any forked child runs without
exec'ing — the parent closing its `LOCK` descriptor releases nothing. Reproduced
deterministically: an explicit `LOCK_UN` before the close makes an immediate reopen succeed
*while the child still holds its inherited descriptor*.
**The amendment.** `segment::lock_root` returns `RootLock` instead of `File`. `RootLock`
issues `sys::unlock` — a new `LOCK_UN` beside the existing `try_lock_exclusive`, so the
syscall stays inside the durability funnel — in `Drop`, before its `File` field closes.
`Drop` is deliberately the *only* release path: an explicit `release()` returning the
`LOCK_UN` error would have been a second way to do the same thing with no caller, which
charter items 8 and 9 treat as a decoy rather than a safeguard. Both owners
hold the guard: `recovery::RecoverySession` (which `engine.rs` retains for the engine's
lifetime, so `StoreEngine` inherits the fix without an edit) and `drive::ShardDrive`. A
`LOCK_UN` that fails in `Drop` cannot be returned and must not be swallowed, so it is
counted in `RootLock::release_failures()` — charter item 7 applied to the one place where a
claim would otherwise be all there is.
**The weaker alternative, rejected: fix it only in `StoreEngine::drop`.** It would have
turned the harness green. It leaves every other `lock_root` caller — `ShardDrive`, the
one-shot `RecoverySession` in the drive's reopen path, and whatever Phase 2's migrator
opens — holding a lock whose release is still a *consequence of a descriptor lifetime that a
process outside this one can extend*. That is the property worth stating plainly: the lock
lives on the open file description, so releasing it has to be an explicit act, and no amount
of tightening the shutdown sequence changes who else holds a descriptor onto that
description. A drop-ordering fix would also have looked correct in every single-process
test, because the defect needs a concurrent fork to appear at all — which is exactly why it
presented as a flake at one run in six and not as a failure.
What would have gone wrong: a consumer that closes a store and reopens it — a recovery
drill, an in-place restart, the Phase 2 migrator, any of which may spawn a subprocess — gets
`AlreadyLocked`, which scope §3.1 defines as a refusal and never a wait. There is no defined
retry, so the correct consumer behaviour on that error is to give up. The store would have
been refusing itself, from a process that no longer exists, and reporting a state
indistinguishable from a genuine second owner.
**Regression evidence.** `segment.rs` gains a synchronized fork-inheritance test: the child
is forked while the lock is held, signals readiness over a pipe, and blocks until the parent
has completed its release *and its reopen*, so the inherited descriptor is provably open
across the whole window. Nothing in it is timed. It passed 40/40 as landed; with the
explicit unlock removed it failed 10/10 with `reopen was refused AlreadyLocked on its first
attempt`. §5's Wave A record is why this had to be synchronized rather than raced: an
intermittently failing fault test gets rerun until green, at which point a real regression
and a flake are indistinguishable.
**Measured, not asserted.** 200 close-and-immediately-reopen cycles through
`StoreEngine::open`, concurrent with four threads spawning subprocesses (72,936 fork/execs
during the run): `refusals=0`, `worst_tries=1`, `worst_elapsed=10.4ms`, against B4's
reported `tries=29 elapsed=146ms`. The same measurement on the pre-fix release behaviour
fails with `AlreadyLocked` within the first cycles under that load.
Amended: `segment.rs` (`RootLock`, `lock_root`'s return type, the fork regression test),
`sys.rs` (`unlock`), `recovery.rs` and `drive.rs` (both owners hold the guard). `engine.rs`
required no change and none was made: it holds a `RecoverySession`, never a `File`, so no
interface request against B1 arises from this. Scope §6.6 records the consequence B4 must
land: the bounded retry at `tests/support/engine_matrix.rs:516` is now compensating for a
defect that no longer exists and must become a one-attempt assertion.
##### Contract review 2026-07-28-D
**Two no-follow `open(2)` primitives added to the frozen `sys.rs`.** Granted, and the grant is
narrower than it looks: `sys.rs` gains `open_regular_nofollow` and `create_new_nofollow`, both
pure syscall wrappers — one `open`, one `fstat` on the descriptor already held, and errno-to-
variant mapping. Neither knows what a marker is, reads content, or decides a startup state.
The defect that forced it. `StoreEngine::open`'s classifier ignored `INITIALIZING.tmp` by name
regardless of type or contents, and initialization then opened that name with `create` plus
`truncate`. An operator's regular file at that name was destroyed silently; a **symlink** at
that name caused a file *outside the root* to be truncated to the marker's 27 bytes and the
link then removed, so the store destroyed data it never owned and erased the evidence. Measured
on a reverted copy: the victim went from 4096 bytes to 27, the link was gone, and `open`
returned `Ok` reporting a working store.
The justification for ignoring the name had been that only this code path could have written
it. That reasoning was circular — it is the claim the classifier runs in order to establish,
and before the store owns the root it has no standing to assume anything about the contents.
Why the primitives belong in `sys.rs` rather than in `engine.rs`. These are new `open(2)` sites
whose **flags are the safety property**. `sys.rs` is the sole funnel precisely so that the next
author looking for how this crate opens files finds every such site in one place; an
`O_NOFOLLOW`/`O_EXCL` open living alone in `engine.rs` is one the next author does not find,
and what they write instead is `create(true).truncate(true)` — the defect this closes. The
weaker alternative, a raw syscall at the call site with a comment, was rejected for that reason.
Neither wrapper increments a durability counter, because neither produces durable bytes; the
existing write and fence primitives still do the counting.
`O_NONBLOCK` is load-bearing rather than defensive. A fifo at `INITIALIZING.tmp` made the
pre-fix `O_WRONLY` open block forever: a single `mkfifo` in a configured root was an unbounded
startup hang, not merely a data hazard. That was found while writing the tests, not predicted.
**Four further symlink hazards are recorded and not fixed**, because they are in `segment.rs`
and are A1's surface. Ranked: `lock_root` follows a final symlink at `LOCK` and is reachable on
the state-2 path, where the classifier never runs — through a live link the process `flock`s a
foreign inode and believes it holds the root lock, so two processes can both own one root and
the exclusion scope §3.1 depends on is defeated. `write_fenced` repeats the `create`+`truncate`
defect for `FORMAT.tmp`, closed from `open` by the new classifier but open to `initialize_root`'s
direct callers. `initialize_root`'s `create_dir_all` follows a symlinked directory and builds a
shard tree outside the root. `read_format` follows a link at `FORMAT`, read-only and lowest
priority. The first is a correctness defect in the locking discipline and should be scheduled on
its own, not folded into a later pass.
##### Contract review 2026-07-29-A
**The first of 2026-07-28-D's four recorded hazards is closed at the frozen surface.** Granted and
landed by the lead, not by a package: `segment::lock_root` now takes `<root>/LOCK` through a third
`sys.rs` primitive, `open_or_create_regular_nofollow``O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC |
O_NONBLOCK`, no `O_TRUNC`, followed by an `fstat` on the descriptor already held. A name occupied by
anything that is not a regular file is `UnrecognizedLayout`, and the type is established **before**
`flock` is attempted, so a refused name is never locked even momentarily.
Why this one was scheduled alone rather than folded into a later pass. The other three hazards risk
a file. This one breaks a **safety property**: `flock` locks the inode a descriptor reached, not the
name that was asked for, so a link at `LOCK` puts the root lock on a foreign inode and leaves the
root's own lock file unlocked. Every single-writer argument above this layer — the shard writer, the
publication window of scope §6.3, the poison-window reasoning — rests on §3.1 exclusion holding.
Nothing is destroyed and everything downstream is permitted to race.
Why `classify_root` was not sufficient, which is the whole reason this could not stay a B1 fix.
B1's classifier does refuse a non-regular `LOCK`, but only on the `StoreEngine::open` path.
`RecoverySession::open` and `drive.rs` call `lock_root` directly, with no classification anywhere
ahead of them, and those are the callers a future `migrate-store` and every recovery tool reach
through. A guarantee that holds only for one of three entry points is not a guarantee.
Measured on a reverted copy, all three distinct failures observed rather than predicted:
- **A wrong-typed name is locked.** First lock taken and *still held*; `LOCK` then replaced with a
link to an unlocked file elsewhere; the second `lock_root` returned `Ok`. A state, not a race.
- **A dangling link at `LOCK` created a file outside the root**, because `create(true)` through an
unresolved link is a create at the target.
- **A fifo at `LOCK` returned `Ok(RootLock)`** — the store reported holding the root lock on a pipe.
This was not in the ranked hazard; it was found by writing the test for the `fstat` arm.
`FileType::RegularFile` is checked from the descriptor, never from a second path lookup, so the
answer is about the object the caller holds and cannot be changed underneath it. One `O_CREAT` open
rather than an `O_EXCL` create with a plain open on `EEXIST`: the single call leaves no window
between deciding a name is taken and opening what is there.
**Disclosed and not closed.** A device node at `LOCK` still receives one `open(2)` before the
`fstat` refuses it. `O_NONBLOCK` stops that open from hanging — the fifo case is otherwise a startup
denial of service — but a driver that acts on being opened has been opened. Closing it needs `O_PATH`
classification plus a re-open of the same inode, which is more machinery than a hazard gated behind
the privilege to `mknod` inside a configured store root warrants. Recorded, not silently accepted.
The remaining three hazards from 2026-07-28-D stand unchanged and unfixed: `write_fenced` for
`FORMAT.tmp`, `initialize_root`'s `create_dir_all`, and `read_format`. Their common shape is now
one primitive away from a fix, but each needs its own refusal semantics decided, and none of them
breaks an exclusion property.
###### Amended after review, 2026-07-29
Two findings against the above, both upheld.
**The two-owner claim was overstated, and the correction is a scope amendment rather than a code
change.** The regression arranges its wrong-typed name by *replacing* `LOCK` while the first holder
holds it — and replacing it with a fresh **regular** file succeeds just as well, since both opens
are then of a regular file at exactly the right name and nothing distinguishes the second from the
first. So the type check closes "the name already resolves to the wrong kind of object", which is
the operator-error and stale-state case, and does **not** close "the name is replaced under a
holder". Scope 3.1 now states the two cases apart, says which one is in scope, and states the
replacement case as an explicit deployment assumption: anything able to replace `LOCK` can equally
unlink a journal, so advisory locking was never the boundary that would stop it. The assumption is
pinned by `replacing_the_lock_file_still_admits_a_second_holder`, which asserts the *current*
behavior deliberately — if a stable locking object is ever adopted, that test is expected to fail,
and the failure is the signal that the documented assumption changed. §3.1 records locking the root
**directory** as the candidate, and what it would cost: a frozen-surface change, and the removal of
the "directory holding only `LOCK` is empty" special case that startup state 1 depends on.
**The funnel guard's exemption could still latch, and the fix is a scanner that refuses to guess.**
Bounding a `#[cfg(test)]` exemption at the next column-zero `}` is right for a recognized braced
item and wrong for every other shape: after `#[cfg(test)] use crate::test_support;` the first such
brace belongs to the *next* function, so all of it was skipped. A first repair still accepted
anything whose first item line ended in `{`; that was the same latch in another spelling, because a
semicolon-terminated `static` or `const` can begin a block initializer there and close with `};`.
The scanner now accepts only the two braced top-level shapes the crate actually uses (`mod` and
`impl`), exempts a one-line semicolon-terminated item only for that line, and **fails the guard** on
every other shape (including a block initializer or an item header rustfmt split across lines)
rather than assuming it is safe. A shape the scanner cannot bound is not a shape it may treat as
harmless.
It is also now a function over `&str` with synthetic tests, which is the load-bearing half. Mutating
real sources only ever probes the shapes those sources happen to contain: no file in the crate has a
semicolon-terminated `#[cfg(test)]` item followed by production code, so no mutation of a real file
could have produced this defect. That is a general lesson about source-scanning guards and belongs
with the charter's item 8 — assert against the path that runs — as its analogue for tooling.
**One promise made exact rather than caveated.** A Unix socket at the name fails `open(2)` with
`ENXIO` before any `fstat`, so it surfaced as `Io` while the documentation promised
`UnrecognizedLayout`. `ENXIO` and `EISDIR` are both mapped to "not a regular file", which is what
they mean here, and the socket case is now one of four occupants the test loop covers. Safety was
never affected; the type of the refusal was.
##### Contract review 2026-07-29-B
**The remaining three redirection hazards of 2026-07-28-D are closed.** One commit, one owner, one
invariant: no name the store *invents* beneath a root may be reached through a link or resolve to an
object of the wrong type. `sys.rs` gains the mechanics — the flags and the descriptor checks are the
safety property, so they belong in the funnel where the next author looking for how this crate opens
files will find them.
Refusal semantics, decided per name rather than uniformly, because the three names mean different
things:
- **`write_fenced`** (`FORMAT.tmp`, `<generation>.manifest.tmp`, `CURRENT.tmp`). An existing regular
file is **adopted and emptied**: it is residue from an interrupted attempt at this exact write, and
reusing the name is how a retry works. The open and the truncation had to be separated to make that
possible at all — no flag combination truncates only regular files, so the type check needs the
descriptor first, and `O_TRUNC` cannot be in the open. Any non-regular occupant is
`UnrecognizedLayout`. The truncation goes through the funnel's `truncate` and is counted; it is
skipped when the file is already empty, so the common create path moves no counter.
- **`read_format`**. Three answers kept apart, because callers act on them differently: absent stays
`Io(NotFound)` — startup states 1 and 3 depend on it — non-regular is `UnrecognizedLayout`, and a
corrupt *regular* marker keeps its existing decode error, which is a different finding from a
redirected one.
- **`initialize_root`**. The root and its ancestors stay the caller's path: an operator who configures
a root behind a symlink has said where the store goes, and resolving that is out of scope. Every
directory *beneath* it is store-invented — existing directories adopted, absent ones created,
symlinks and non-directory occupants `UnrecognizedLayout`. Two passes: every planned entry is
classified before any missing one is created, so a refusal cannot half-extend the tree it refused.
`O_DIRECTORY` is the type check and the kernel applies it before the descriptor exists, so there is
no window in which a non-directory is open; `mkdir(2)` never follows a final symlink, so the create
side needs no separate guard.
**Directory fences now go to descriptors already validated** (`sys::fsync_dir_fd`) rather than
re-resolving the name, which would hand the fence to whatever the name resolves to *now* instead of to
what was checked. The fence sequence is otherwise byte-for-byte the same, deliberately: `engine.rs`
asserts `initialize_root`'s `fsync_dir` count exactly, and that assertion is load-bearing evidence
about initialization, not incidental.
Each of the three protections was reverted independently and the witnesses observed, not predicted:
- `write_fenced` — a 4096-byte file outside the root truncated and rewritten through a live link; a
file *created* outside the root through a dangling one; and a fifo at the name **blocked the open
for the full ten-second deadline**, an unbounded startup hang from one `mkfifo`.
- `read_format` — a foreign `FORMAT` read in full and its `shard_count` and `root_uuid` returned as
this root's, so every file in the tree would then be validated against a marker the store never
wrote; and the same ten-second hang at `FORMAT` on the read side.
- `initialize_root` — returned **`Ok(FormatMarker)`**: a successful initialization reporting a working
store, with the shard tree built outside the root through a link at `shards/`. The two-pass
preflight has its own witness, `shards/00/active` existing after a refusal that named
`shards/00/segments`.
The witnesses sit on the `segment` entry points, not on `StoreEngine::open`. Its classifier refuses a
redirected root before any of these are reached, so a test entering that way passes whether or not the
protection exists — and `RecoverySession::open`, `drive.rs` and `store-bench` all arrive without it.
Coverage of the caller is not coverage of the callee.
**Disclosed.** The device-node residual from 2026-07-29-A now applies to `FORMAT.tmp` as well: one
`O_NONBLOCK` open lands before the `fstat` refuses it. Unchanged in judgement — it needs `mknod`
privilege inside a configured store root. `rename_noreplace` needed no change: `renameat2` with
`RENAME_NOREPLACE` fails `EEXIST` on an occupied target name whether or not it is a link, so the
`FORMAT.tmp``FORMAT` install could never have followed one.
##### Contract review 2026-07-29-C
**Index maintenance (scope 3.6, B1 deliverable 1) reaches production.** A shard now seals its
accumulated delta into a durable `IndexRun`, publishes it through the manifest, and discards exactly
the layers that run covers — in one committed-root CAS. Four frozen-surface amendments were needed;
all are granted and recorded here.
**Amendment 1, `segment.rs` (A1): `install_index_run` + `index_run_filename` + `SealedIndexRun`.**
The whole durability sequence of a seal — fenced temp, `rename_noreplace`, `fsync_dir`, then
`install_manifest` — lives here, so `engine.rs` performs no durability operation of its own. The
predecessor manifest is *read* here rather than reconstructed by the caller: an engine rebuilding a
`Manifest` from its in-memory generation pins would be re-deriving `retained_tail_ranges`,
`base_generation` and `committed_shard_sequence` from a lossy projection, and any field it got wrong
is a field recovery then trusts. `committed_shard_sequence` is carried forward unchanged — sealing an
index makes no journal frame more durable than it already was.
**Amendment 2, `index.rs` (A2): `delta_pressure` as a free function.** The writer's backlog is
several delta layers, not one `IndexDelta`, so it cannot ask `IndexDelta::pressure`. Restating
`>= max_entries || >= max_bytes` in `engine.rs` would be two implementations of one watermark, and
the copy in `engine.rs` is the one nobody would think to change. `IndexDelta::pressure` is now this
function over its own fields, so the rule has one implementation and gained no second definition.
**Amendment 3, `roots.rs` (lead): `CommittedRoot::merge` recognizes a maintenance publication.** The
idempotency guard discarded any subtree whose `shard_committed_sequence` was already published —
which is *every* index seal, because a seal appends no frame. Mutation-checked: with the guard
unamended the run reaches the device and the manifest, and the root never sees it. The snapshot reads
`(0 runs, 3 layers)` while `CURRENT` names the run: precisely the silent divergence the seal path
poisons to avoid, arriving quietly instead. `ShardSubtree::publishes_index_maintenance` is a question
about the payload rather than a flag the writer sets, because a flag can disagree with the fields and
the disagreement that matters — a seal marked as a group — drops the seal.
**Amendment 4, `recovery.rs` (A2): a manifest may have an empty retained tail when it commits through
zero.** The rule was "an empty shard has no manifest", true for as long as the only reason to write
one was to name a sealed segment. Index maintenance is a second reason, and a shard that has
published a run but never rotated its journal has a manifest, no retained tail, and a committed
prefix entirely in `active/`. `committed_shard_sequence == 0` is what distinguishes that from the case
the rule protected against — a manifest claiming committed frames while naming nothing that holds
them — and an empty range list is additionally required to carry a run or a checkpoint, so a wholly
empty manifest is still refused.
**A finding that bounds what this slice delivers, and it is not a defect in the code.** Sealing moves
entries out of the delta layers; it moves no *frame* out of `active/`. The manifest's committed prefix
advances only on a checkpoint or a segment rotation, neither of which exists yet — so recovery still
replays every frame the shard ever wrote, into one delta bounded by `max_active_index_entries`. Two
consequences:
- The writer now refuses when the replayable set reaches that ceiling, so a shard cannot write a store
it could not reopen. This hole **pre-dates** the change: the placeholder refusal capped delta
*layers* at `max_index_runs`, which never bounded the summed entries behind them. It is closed here
because removing the layer cap made it reachable in ordinary configurations.
- A seal triggered by **entry pressure** therefore lands the shard exactly on that ceiling and the next
admission is refused. Only a seal triggered by the **fan-out** ceiling leaves the shard able to
continue. Entry-pressure sealing becomes useful when `StoreEngine::checkpoint` can advance the
committed prefix; until then it converts an unreopenable store into an honest refusal, which is all
it can do. Recorded in scope §6.5, and it is the reason checkpointing is the next dispatch rather
than a later one.
`IndexMaintenanceSnapshot` is frozen as specified: two `u64`s, both read from one captured
`CommittedRoot`, no counter handle. Mutating the discard back to "retain every delta" makes it report
`(1 run, 3 layers)` where the test requires `(1, 0)`, so the snapshot is load-bearing rather than
decorative. `max_index_runs` and `max_open_index_runs` are enforced against the same numbers recovery
enforces, before anything durable happens, so a seal that passes can always be reopened.
`store-bench` is untouched. Its `index_maintenance` condition stays preliminary until B4 consumes this
after `checkpoint()` can flush the final below-watermark backlog.
##### Contract review 2026-07-29-D
Four findings against 2026-07-29-C, all upheld. Each fix has a regression that fails against the
landed code, and each was mutation-checked back to the landed behaviour.
**P1 — admission projected nothing.** The replay-ceiling check read the published root only, so it
decided about a transaction it had not counted. One object present, a two-object transaction against
a ceiling of two: committed, then failed to reopen. The open group was the same hole one step
further along, since its members are admitted and will publish. Admission now projects sealed runs,
unsealed layers, the open group, **and** the transaction being decided.
**P1 — the byte ceiling was unguarded.** Recovery rebuilds into one `IndexDelta::from_options`,
which refuses on either ceiling; the guard checked entries alone, so narrow frames spread over
namespaces passed admission and failed to reopen on `max_active_index_bytes`. Both are checked now,
and the projection counts namespaces because the encoding pays a section header per namespace.
`index::encoded_bytes_for` (amendment 2) is that arithmetic, extracted so `engine.rs` does not carry
a copy of the encoding's shape; `IndexDelta::encoded_bytes` is the same function over its own fields.
**P1 — a recovered run's locations did not resolve, and this is the finding that reshaped the
slice.** An `IndexLocation` names a logical generation, and a run is the first thing here that
persists one across a session. Only a segment's generation is stable: the active tail's is assigned
from `max(manifest, .seg, .idx) + 1`, which moves whenever any artifact appears — the run's *own*
manifest is enough — and recovery seals a journal holding frames into a segment at that counter's
value rather than at the generation the tail had. The landed reopen test could not see it because its
lookups were answered by the replay delta shadowing the run.
Coverage is now restricted to an oldest-first prefix of layers whose every entry is segment-backed.
This makes the broken run unwritable rather than merely untested, and the consequence — sealing lags
one session behind until frames leave `active/` — is recorded in scope §6.5.
The alternative, preserving the tail's logical generation across the seal, was attempted and
withdrawn: `recovery_generation` is simultaneously the new manifest's generation and the sealed
segment's logical generation, so the fix requires separating those two numbers in A2's recovery core.
That is the right change and it belongs with checkpointing, not inside a B1 integration commit; the
first attempt at it also targeted the wrong branch, since a journal holding frames is *replaced*
rather than kept, which is worth knowing before the next attempt. `active_tail_logical_generation` is
left extracted, called from the one path that already used that formula, so the two ways of numbering
an active tail are at least visible in one place.
**P2 — the run ceilings counted every shard.** Recovery enforces them against one shard's manifest,
so a four-shard root with `max_index_runs = 1` refused the second shard its first run. Counted per
shard now, from the shard's own retained generations.
##### Contract review 2026-07-30-A
**`recovery_generation` was two numbers wearing one name, and they are now separated.** Granted and
landed in `recovery.rs` (A2) as the prerequisite the checkpointing dispatch was to open with.
One `max(manifest, .seg, .idx) + 1` served as the logical generation of the segment recovery seals
*and* as the generation of the manifest recovery installs. Every index run publishes a manifest, so
every seal moved the number; the segment recovery later wrote took the moved value while the frames
inside it were already named by the old one. Persisted index locations dangled — `object_source`
returning `None` for a run the manifest still named.
Separated:
- **logical** — the identity of the frames, taken from `active_tail_logical_generation`, which is
derived from the manifest's committed prefix and moves only when a rotation seals a tail. Sealing a
journal into a segment now changes where the bytes are, not what they are called. An interrupted
recovery still resumes: a `.seg` footer naming this journal, or a `.recovery-<id>-<n>.prefix`,
fixes the identity that was already chosen.
- **manifest** — the next free manifest generation, so an index run's manifest and a recovery's
cannot collide.
Collision detection moved with them. The old check compared a resumable generation against a maximum
that mixed all three namespaces, which an index run could raise on its own; only a segment can
collide with a segment, so occupancy is tracked over `.seg` names alone — and recorded from the
parsed name even when the file does not open, because an unreadable `.seg` holds its name as firmly
as a readable one. Two recovery artifacts for the same journal in different generations remains
`Corruption`. An identity held by some *other* `.seg` is a different question, answered by contract
review 2026-07-30-B below and not by this one.
**What this closes.** The scope §6.5 carry-forward "a run may only cover frames a segment already
holds" is gone, and with it the restriction in `ShardWriter::coverable_through`: a run may now cover
locations naming the active tail, because the tail's identity survives being sealed away. Sealing
covers current frames in the session that wrote them rather than lagging one behind. Mutation-checked
by putting the segment's identity back on the manifest counter, which reproduces the original defect
exactly — a recovered run pointing at a logical generation nothing pins.
**One case is not closed, and it was found by a test rather than predicted.** An orphan `.seg` from
an interrupted seal occupies a logical generation whether or not it is a readable segment — and the
identity the tail wants may be exactly the one it holds. `frame_golden`'s
`an_orphan_segment_leaves_the_active_journal_the_authority` failed on the first attempt for that
reason, which is also the test that documents why recovery must not refuse in this state: the active
journal is still the authority and an outage here would be the wrong answer. The seal therefore falls
back to a free generation and renames the frames, exactly as before this review, and an index run
against the old identity dangles.
**That last state was disclosed here and rejected on review; 2026-07-30-B below is what landed.**
The disclosure conflated two cases under one orphan: an orphan alone, where the fallback is right and
recovery still succeeds, and an orphan holding an identity a *published* run names, where the
fallback opens a store whose manifest points at a run that resolves to nothing. This review's own
finding — that replay masks it — was the argument against leaving it, since `StoreEngine::checkpoint`
consumes runs directly.
The other §6.5 carry-forward stands: sealing still moves no frame out of `active/`, so the replay
ceiling still bounds admission and an entry-pressure seal still lands on it. That is what
`StoreEngine::checkpoint` is for, and it is now unblocked.
##### Contract review 2026-07-30-B
**A displaced identity is a refusal, not a disclosure.** Review of 2026-07-30-A declined the silent
state it disclosed and chose `Corruption` for orphan-plus-published-run until recovery can discard
such a run. Granted, with one frozen seam, and landed before the checkpointing dispatch it would
otherwise have poisoned.
**What the two states cost, which is why they part.** When a `.seg` occupies the logical generation
the active tail carries, the frames cannot keep their name whatever recovery does.
- *Orphan alone.* Nothing persisted names the displaced identity, so renaming the frames costs
nothing. Recovery falls back to a free generation and **succeeds**, which is what
`an_orphan_segment_leaves_the_active_journal_the_authority` requires: the active journal is still
the authority and an outage would be the wrong answer.
- *Orphan plus a run the manifest names.* Recovery would publish a manifest naming a run whose every
location resolves to nothing — authoritative and unreadable in one step. Recovery **refuses** with
`Corruption`, naming the run file rather than only the generation.
Refusing is not good; it is an outage on a root whose data is all present. It is chosen because the
alternative is a store that opens and lies, and because the masking is temporary in the worst way:
the replay delta sits above the run and answers every lookup that would otherwise expose it, so the
first consumer to read a run directly — the checkpointer — is also the first to find out.
**The frozen seam.** `IndexRun::references_segment_generation` in `index.rs` (A2), read-only, with
recovery as its one caller. Exact rather than a range test over section headers: entries pack a
16-bit delta from the section base, so `[base, base + u16::MAX]` says only what a section *could*
name, and answering `true` on that alone would refuse recoveries over a generation no entry
mentions. `index.rs` is otherwise untouched.
**`coverable_through` keeps its widened coverage.** The restriction 2026-07-30-A removed does not
come back: a run may cover locations naming the active tail, because the unsound state that
restriction existed to avoid is now refused at the one point it can arise rather than designed
around at every seal. Its doc comment described the pre-split world and is rewritten to this one.
**Amendment, on review of the first landing: resumption is not exempt.** The guard sat only on the
fresh fallback, and the resumable branch returned before it. An interrupted recovery that had already
fallen back to generation 2 resumed at 2 — correctly, since finishing an interrupted seal must not
orphan its artifact — and stranded the run naming generation 1 exactly as a fresh fallback would,
one crash later. Reproduced by review with a published run, an orphan at 1, and a
`.recovery-<journal>-2.prefix`; the store opened with `object_source(0, 1) == None`.
The guard is now one closure both paths call, keyed on **choosing anything other than the identity
the frames already carry** rather than on how the choice was reached. `preferred` is computed before
either path can return. A resumable artifact *at* `preferred` displaces nothing and the guard is a
no-op on it — pinned by `a_resumed_seal_at_the_frames_own_identity_still_opens`, so a future guard
that keyed on "a resumable artifact exists" would fail rather than quietly refuse every interrupted
recovery on a root that has ever sealed. The lesson generalizes past this fix: the check belongs on
the *outcome* — the frames are being renamed — not on the branch that produced it.
**Second amendment: the fallback was reaching a journal that seals nothing.** Review found the guard
searching for the wrong generation entirely. A recovery that keeps an *empty* journal was giving it
the sealing fallback's identity — but nothing was sealed, so no `TailRange` recorded the move, and
the next open derived the identity the manifest still implied. The session in between had appended
frames under the moved number and sealed a run over them. The guard looked for a run naming the
derived generation, found none, and opened a store whose run resolved to nothing. Confirmed by
review: empty tail plus an orphan at 1, reopen taking 2, frames and a run at 2, an orphan at 2, and
the next open choosing 3 with `object_source(0, 2) == None`.
`RecoveryGenerations` now carries **three** numbers, and the third is the point: `tail` is the
identity a surviving journal keeps and is always the tail's own. Segment-name occupancy is a fact
about `segments/`, and a journal that seals nothing does not go there — the fallback exists to avoid
a name collision that path never risks. `logical` remains the sealing identity and still falls back
under the guard. The invariant this restores is the one 2026-07-30-A was built on and did not fully
hold: **the active tail's identity is always derivable from the manifest**, so the guard and the
frames are always talking about the same number.
What the tail keeps can be a generation an orphan already occupies, and that is correct: the
collision is only real when frames are sealed under that name, and it is refused then, by the guard,
with the run in hand. An identity nothing persists is not an identity.
**Evidence.** `an_orphan_holding_a_published_runs_identity_refuses_the_open`,
`a_surviving_journal_keeps_an_identity_the_next_open_can_derive`,
`a_resumed_fallback_refuses_on_the_identity_it_resumes`, and
`an_orphan_holding_no_published_identity_still_opens` are the states, and every refusing test shares
one assertion helper so the paths cannot drift in the tests either. It asserts the damage rather than
an expectation, and it **reads the generation out of the published run** rather than assuming one —
hard-coding it is how a drifting identity would report the number the test expected instead of the
number the store chose. With any of the three changes reverted the open succeeds and the helper
reports the run's own generation with `None` pinned at it.
The helper reads that generation from a **named entry**, `IndexRun::get` on a key the test knows is
covered. Its first version probed generations `0..16` and had the original defect one level up: it
could not report an identity outside the range it guessed, so review's drift to generation 101 failed
as "no generation found" before any assertion about the identity ran, and a run spanning generations
would have reported the lowest. The regression now keeps a distant orphan precisely so the fallback
it must not take is 101 rather than a number that reads as an off-by-one. The exactness test is
`a_run_reports_only_the_segment_generations_its_entries_actually_name`, whose negative cases include
a generation inside a section's packed span that no entry uses. A first draft of the refusing test
passed for the wrong reason — its workload re-pushed the genesis object id as a blob, so the reopen
failed on a duplicate-object `Conflict` whether or not the guard existed; the mutation is what
exposed it.
**Still open, unchanged.** Recovery discarding a run whose covered identity was not preserved, which
turns this refusal into successful reclamation. Recorded in scope §6.5 with the checkpointing work
that will exercise it.
##### Contract review 2026-07-30-C
**`StoreEngine::checkpoint` is implemented, and the ordering inside it is the correctness.** The
frozen D0 signature returned `NotImplemented`; it now checkpoints every shard on its own writer
thread and returns a lease.
**The index run comes first, and completely.** The checkpoint format carries catalog, refs and
receipts and **no object index**. Once `committed_shard_sequence` advances past a frame, recovery
stops replaying it, so nothing rebuilds the entries that frame carried — if they live only in the
root's delta layers, the objects are on disk, named by a segment, and unreachable. Every layer
through the committed sequence is therefore sealed into a run the checkpoint's own manifest names,
and discarded from the root, before the sequence moves. Coverage short of the committed sequence is
refused rather than partially applied.
**Real pinning.** `CheckpointLease` holds the exact `Arc<RetainedGeneration>` each shard published,
so it pins segment and checkpoint descriptors and index-run mappings directly. Count-based retention
cannot stand in: two further checkpoints prune the generation, and a lease that stopped being true
after two unrelated operations would not be a lease.
**The group boundary is structural.** The checkpoint travels the same channel as submissions, and
the writer loop publishes any open group before running one. No lock, and no way to advance past
sequenced-but-unfenced frames.
**Four edge cases, each with a test.** A shard with no committed sequence is skipped rather than
checkpointed at sequence 0 — the second is a real frame, and writing `0.checkpoint` for an empty
shard would collide with the real one later. A repeat with no new work reuses rather than renaming
onto its own name. A finalized but unreferenced checkpoint left by a crash between the rename and
the manifest is validated through recovery's own reader and adopted, instead of wedging the shard on
a permanent `EEXIST` for a file that is correct and merely unpublished. A journal holding no frame
seals nothing: that is the state a reopen leaves when everything committed is already in segments,
and A1 rightly refuses to seal an empty journal — the checkpoint is then a checkpoint row and
nothing else.
**Evidence.** `a_checkpoint_keeps_every_object_resolvable_through_a_reopen` writes below seal
pressure, checkpoints, reopens, and resolves every object with the replay delta explicitly excluded
as the answer. Both mutations the review named fail it on reachability, not on bookkeeping: dropping
the run publication loses the genesis object, and sealing one sequence short loses the last push.
**`store-bench` found the empty-journal case before any test did**, by calling the entry point that
had always refused. It now also earns `index_maintenance` from a measurement: the emitter reads
`index_maintenance().unsealed_delta_layers` where it previously recorded `None` for "no such reading
exists". That was the interface request the guard itself had written down, and it is the one line of
B4 surface this commit touches — the emitter's remaining claims are unchanged.
**Next, and in this dispatch:** recovery discarding an index run whose covered identity was not
preserved, which turns 2026-07-30-B's refusal into reclamation.
##### Contract review 2026-07-31-D
B3 deliverable 8 — recovery treats synced-but-unreferenced artifacts as invisible garbage and a
complete final frame as authoritative adoption. With this, deliverables 6, 7 and 8 are complete.
**Frozen-seam amendment: `ProjectionRecoveryResolver::resolve_committed` gains
`adoption_shard_sequence`.** Recovery passes `frame.facts.shard_sequence` from the complete
replayed adoption frame. Required, never optional or inferred: every artifact's logical generation
derives from it, and staging cannot supply it — a transferred `Finalizing` session has no durable
position, which is exactly the state being resolved, and that frame is the sole authority that
creates one. This is the **recovery-direction counterpart** of the D0-B
`ProjectionAdoptionOutcome::Adopted` amendment in 2026-07-31-C: the same value, granted for the
same reason, flowing from whichever side made the frame authoritative. Both resolver test doubles
are updated, and recovery's asserts that it is handed the frame's own sequence rather than a
placeholder.
**The generation mapping** is `BAND | (adoption_sequence << 24) | ordinal`, injective in the pair
and confined to a reserved top-bit band because segments, tails and projections share one
generation space. Its boundary is one value wide and closed by a ceiling rather than a shift
round-trip; see 2026-07-31-C's successor note in `staging.rs`.
**No per-object offsets, and none were needed.** `IndexLocation` names the whole certified record
— its own doc says so and names adopted projections explicitly — so every object in a chunk shares
that chunk's location with `frame_offset: 0` and `frame_len` covering header, canonical payload and
digest. A reader validates the artifact and then selects from the decoded vector, exactly as
several objects share one journal frame. I had raised this as a suspected protocol gap; it was a
contract I had not read on the type I was populating. Pointing at object bytes would let a reader
return bytes from a record it never proved complete.
**Whole or not at all.** Every declared ordinal is read back, decoded, and checked against the
manifest's session and chunk count before any location is produced, and
`RecoveredProjectionArtifacts::new` then refuses unless the entry count equals the descriptor's
`object_count`. A directory holding some of its chunks resolves to an error, never to a smaller
projection — the failure a "resolve what is present" implementation reaches by being helpful.
**`notify_recovered`.** `Committed` ends the pin as an adoption at the position the resolution
recorded, and refuses a session this recovery never resolved rather than adopting at a guess —
without a position, cleanup could never prove absence of reference. `ProvedAbsent` means recovery
read the complete authoritative history and no frame names these artifacts, so the session returns
to a reclaimable state and its directory goes. Both are idempotent, including for a session that is
already gone, because recovery repeats every notification when a later one fails.
**Evidence.** A committed projection resolving to complete membership at the expected generation,
with every object in a chunk sharing that chunk's location and `frame_len` equal to the artifact's
size on disk; the same session refusing once a chunk is removed. Both notification outcomes driven
twice each to assert idempotence, with the `Committed` case then measured by cleanup against a root
short of the adoption to confirm the position it established is real. The integration suite's
"resolver seam is deferred" test is replaced by one asserting it answers.
##### Contract review 2026-07-31-C
P1 against 2026-07-31-B: cleanup could delete committed artifacts through a stale root. Found in
review, reproduced, repaired. Deliverable 8 was held until the authority boundary below was
decided, and the decision is recorded here.
**The defect.** `cleanup_unreferenced` selected candidates from current staging state and then
trusted whatever `CommittedRoot` the caller supplied. Capture `R0` before a projection is
published; publish `R1` referencing its artifacts; record the session `Adopted`; call cleanup with
`R0`. Every artifact is absent from `R0`'s pins — because `R0` predates them — so the absence
proof succeeds and the directory is deleted out from under a committed root that still points into
it. Reproduced exactly as reported: the session was reclaimed.
**My reasoning in B was inverted, and that is the part worth keeping.** I wrote that a racing
newer root "can only add references" and concluded the proof was safe. Adding references is
precisely the hazard: it means an older root *omits* references a newer one holds. I checked the
direction in which references change and never checked the direction in which the root travels.
**The authority boundary, and why there is no purely-B3 repair.** `ProjectionStaging` is
constructed *before* any committed root exists — it has to be, since it is recovery's
`ProjectionRecoveryResolver` — so there is no moment at which staging can observe a root position
on its own. Both candidate repairs need an authority minted outside B3.
**Decision: a durable minimum recorded at adoption.** `ProjectionAdoptionOutcome::Adopted` now
carries the committed shard sequence its frame reached — a **frozen D0-B amendment**, granted and
recorded here. B1 supplies it because only B1 knows it: the append that produced it has just
returned. Staging writes it into the adoption marker, which is already rewritten by that exact
transition and already carries a payload, so durability costs nothing new. Cleanup then skips any
adopted session whose recorded sequence exceeds the supplied root's, and a root at or past the
adoption necessarily includes its effects — which is what makes the absence real.
Engine-owned cleanup under the same synchronization as root publication was the alternative. It is
the stronger property but reaches for the `StoreEngine` staging façade that §6.5 still records as a
pending interface amendment, and it would not on its own cover a session adopted before a restart.
The two are not exclusive; if the façade lands, this check remains the restart half.
**`Some(0)` and `None` are different answers.** The first cut compared
`shard_committed_sequence(shard).unwrap_or(0)`, which conflates them — zero is a valid committed
sequence, so a root that says *nothing* about a shard became indistinguishable from one that has
committed through its first frame, and an adoption at sequence 0 was reclaimable through a root
that never mentioned the shard it lives on. The check now requires an explicit
`Some(through) if through >= adopted_at`, so silence is treated as no evidence. The conflation
disarmed the position check for precisely the adoption that needs it most, the earliest one.
**A session missing a position is not reclaimable at all.** An adopted session always records
where it was adopted, so the absence of one means the store cannot say when the reference it is
about to disprove came into existence. Skipping is the only answer that cannot lose data.
**Evidence.** The reviewer's R0/R1 sequence as a regression, which failed by deleting the session
before the repair. It now asserts three things rather than one: the stale root reclaims nothing,
the decline is attributed to the root's age rather than to some incidental refusal, and the *same*
session against a root that has reached the adoption is reclaimed — without that last clause the
test would pass against a cleanup that declined everything. Durability is asserted in the reopen
test: after a restart, a root short of the reconstructed adoption still declines and one at it
reclaims. A zero-boundary regression covers the other half against one adopted-at-zero session: a
root with no sequence for the shard declines, and the same session against an explicit `Some(0)`
reclaims — both halves together, because a test that only checked the absent case would pass
against a cleanup that declined every root.
Five mutations, each caught: trusting any supplied root, recording no position at adoption,
writing no position into the marker, restoring the `unwrap_or(0)` conflation, and tightening the
comparison to `>` so an exactly-reached root is refused. The last fails four tests rather than
one, which is the shape an over-strict boundary should have.
A test-fixture defect surfaced on the way and is worth recording because it would have hidden the
repair: the helper root populated a sequence for shard 0 only, while the fixture's destination
repository hashes to another shard. Every cleanup test would have measured a stale-root skip while
claiming to measure the reference proof. The helper now populates every shard, as a real root does.
##### Contract review 2026-07-31-B
B3 deliverable 7 — cleanup proves absence of reference. Deliverable 8 follows separately.
**Two clarifications to §6.5's wording**, both recorded in the scope and neither a relaxation.
The deliverable reads as a per-artifact rule over everything staging owns; implementing it that
way would have been wrong twice.
*Absence of reference is necessary and not sufficient.* A `Sealed` session a client is still
finalizing and a `Finalizing` one whose adoption frame may be mid-append are both referenced by
nothing at all. A cleanup that proved absence and stopped there would delete them. The candidate
set is therefore the **adopted** sessions: the one state where the artifacts are store content
and "does anything still point at them?" is both meaningful and answerable. `Open` and `Sealed`
belong to expiry and abort, which already own them.
*The unit of removal is the session, not the artifact.* A session's chunks are not independent
files — the manifest names all of them and reconstruction refuses a sealed session missing any
ordinal — so removing the unreferenced half of a directory trades a bounded leak for a root that
fails to open. A session is reclaimed only when nothing in it is referenced, and the per-artifact
rule the clause is really about still applies inside that: `reclaim_session_directory` removes
only files carrying the session's own marker and refuses the whole directory on anything else.
**Every file is checked, not only the chunks.** A root that pinned a session's manifest and
nothing else would still be holding that directory, and answering on chunks alone would delete
the file it holds.
**The supplied root must be new enough to be evidence — corrected by 2026-07-31-C.** The first
version of this argued that a racing newer root "can only add references" and concluded the proof
was safe. That is an argument *for* the hazard: adding references is exactly what makes an older
root omit them, so absence measured against a root captured before an adoption is a date rather
than an absence. See 2026-07-31-C for the defect and its repair.
**Evidence.** The deliverable's named acceptance case and its inverse in one test, against one
adopted session: a root holding a single chunk declines the whole directory and leaves every file
in place, then a root holding nothing reclaims it. Both halves together are what make the first
assertion mean "the reference proof declined it" rather than "cleanup does nothing here" — a
cleanup that never reclaimed anything would pass the decline half perfectly. A second test pins
that a sealed session and a pinned one both survive a cleanup against a root referencing nothing.
Mutation-checked: skipping the reference proof fails the first, and treating every state as a
candidate fails the second, each and only each.
The integration suite's `cleanup_is_deferred_and_names_its_deliverable` is replaced rather than
deleted — the distinction it stood for, that "nothing to do" and "cannot answer yet" are
different results, is now asserted the other way round at the public surface.
##### Contract review 2026-08-09-B
`bench/result-schema.json``run_conditions.index_run_ceiling`. Requested after index-delta
sealing landed, because the enum's only raised value names a cause that is no longer true and
that nothing ever verified. One value is added, none is removed.
**The declaration was never a claim about *why*.** `store-bench.rs` derives it by comparing the
configured ceiling with the store default; that comparison cannot know what motivated a raise.
It nevertheless emitted `raised_because_index_sealing_unimplemented`, so the bundle asserted a
cause on the strength of a subtraction. That was accurate while the only reason to raise the
ceiling was the unimplemented seal. It is not accurate now, and a future run raising the ceiling
for any other reason — deliberately measuring fan-out, say — would have emitted a false
explanation that the schema called valid.
**`raised_above_store_default` is added and is what the emitter now writes.** It states the fact
the comparison establishes and stops there.
**The old value is retained and deprecated, not removed.** `schema_version` is `const: 1`, so
there is no later version to move archived bundles to, and this repository has never held a
bundle to check against — the reference machine may hold ones that declare it. Removing it would
invalidate evidence already produced, which is a worse outcome than carrying a spelling nothing
emits. It is retained **solely so archived v1 bundles validate, and is not a current causal
claim**; the enum's description says so.
**Deprecated does not mean unchecked.** The `allOf` rule that floors a declared raise at 65
accepts either spelling, so a bundle using the old string is still cross-checked against
`resources.configured_ceilings.max_index_runs`.
`both_raised_spellings_are_accepted_and_bound_the_same_way` asserts both halves — that each
validates, and that each is refused when it records the store default. Verified by mutation:
narrowing the rule to the new spelling alone makes the deprecated one silently unchecked and
fails that test.
The `configured_ceilings.max_index_runs` description carried the same causal claim and now
records what is true — the run seals against that ceiling, and its fan-out trigger is what
imposes the steady state a P2 number must be measured in.
##### Contract review 2026-08-09-A
B1 deliverable 3 — `adopt_projection` through submit. Requested by B1 on starting the wiring,
because the public entry point the deliverable is named after could not be called. Three
visibility changes, two accessors, one lifetime correction, and one signature amendment —
`adopt_projection` takes the adoption as a single value, for the reason recorded below. No wire
format moves.
**`ValidatedTransactionBuilder::adopt_projection` was a public function no external caller
could invoke.** It takes a `ProjectionAdoption`, whose only issuer is
`ProjectionStageSession::finalize`, which was `pub(crate)` and returned the `pub(crate)`
`StagedProjectionAdoption`. The pin has no other constructor by design — that unforgeability is
the point — so the two together made the signature readable and uncallable. **`finalize` and
`StagedProjectionAdoption` become `pub`.** `StagedProjectionAdoption` sits in D0-B's frozen
adoption seam; this changes who may name it and nothing about what it is.
**`adopt_projection` now takes the whole `StagedProjectionAdoption`, and its fields stay
private.** Review found that publishing the two fields and keeping the two-argument signature
admits a pairing that is *wrong* rather than merely incomplete: finalize session A, take the
descriptor from session B, submit `(descriptor_B, handle_A)`. Every field of `descriptor_B` is
internally valid, so nothing looks damaged — the frame would simply name B's artifacts while
A's are the ones held against deletion. The frozen two-argument form existed so neither half
could be *forgotten* independently; taking one value keeps that and additionally makes the
mismatch unrepresentable. A pre-append revalidation against the pin's canonical resolution
still runs, because staging's state can change between `finalize` and submit — but it is now a
recheck of a pairing the type guarantees rather than the only thing standing between a caller
and a mispaired frame. `descriptor()` is added for inspection; it borrows, because taking the
descriptor out is what the type exists to prevent.
**`StoreEngine::staging()` is added**, returning `&Arc<ProjectionStaging>`. Since the P1-4 fix
moved staging inside the locked engine lifetime, no caller can construct one beside a
`StoreEngine` — holding two root locks is what that fix made unrepresentable — so without an
accessor there is no reachable way to stage the projection an adoption adopts. This closes the
interface request recorded in the ignored test at `tests/staging_sessions.rs`.
**`ProjectionStaging` now retains the `RecoverySession` lease, and the first version of this
accessor was unsound without it.** Its justification claimed that returning a reference kept
staging from outliving the lock. It does not: `Arc::clone` escapes the borrow, and so do a
`ProjectionStageSession` and an adoption pin, each of which owns an `Arc<ProjectionStaging>`.
With `LOCK` released when the engine dropped, any of those could still write into a root
another process had since opened — a second accountant for one root, which is the exact
condition staging's construction rules exist to prevent. **`ProjectionStaging::open` therefore
takes `Arc<RecoverySession>` and keeps it**, so the lock is released when the last holder is
dropped rather than when the engine is. `EngineShared` shares the same lease.
`a_retained_staging_holds_the_root_lock_past_the_engine` asserts it by taking the lock
directly, not by attempting a second engine open — staging's in-process root registry would
refuse that whether or not the lock were held, so only the direct form fails when the lease is
not retained. Verified by mutation: replacing the retained `Arc` with a `Weak` makes the test
fail with a second session holding the root.
**`RepoSnapshot` gains `object_source()` and carries its shard index.** `locate` answers *which
generation*, which stopped being sufficient the moment adoption existed: a logical generation
is a segment, an active tail, or a projection artifact, and an adopted object's bytes are not in
the frame that installed it. Before adoption every location a reader could hold was a frame in
this shard's journal, so the distinction was invisible; now a reader that cannot ask "what
kind" cannot act on the answer at all. The shard index is passed to `capture` rather than
derived because deriving it needs the shard count, which lives in `StoreOptions` and not in the
root. `RetainedObjectSource` is already re-exported, so `lib.rs` does not move.
**One finding recorded and not acted on.** `checkpoint.rs` has no notion of projection
artifacts: a retained generation's `projection_artifacts` are in-memory pins, and their
durability comes from the journal frame plus staging's adoption marker, with recovery
rebuilding them by replaying the frame through `resolve_committed`. The successor-generation
paths clone them forward correctly within a process. Whether a manifest that outlives the
frames it was derived from needs to record them is a B3/checkpoint question, not a B1 one, and
is raised here rather than answered.
##### Contract review 2026-08-07-A
B1 deliverable 8 — `RepoSnapshot` and `StoreEngine::snapshot`. Requested by B1 before any body
was written, because the frozen D0 signatures could not express the answer to a question the
deliverable cannot avoid. Two lead-owned files move; both are amendments, neither is a
redefinition.
**`StoreEngine::snapshot` had no way to say "that namespace is not bound".** The frozen
signature returns `Result<RepoSnapshot, StoreError>`, and no `StoreError` variant covered an
absent repository. The alternatives were all worse than an amendment: `Conflict`, `NotReady`
and `UnrecognizedLayout` each name a different condition, so reusing one would make the error
a false statement about what happened, and a caller could not distinguish it from a genuine
instance of that condition. **`StoreError::NoSuchRepository { namespace }` is added to
`types.rs`.** It carries the typed `NamespaceId` rather than a rendered string so a caller
matches on the identity it asked for instead of parsing a message.
**It is an inability to answer and not a lifecycle state**, which is the distinction the
taxonomy's own doc comment turns on. A namespace that was never bound has no `RepoState`, so
there is no `genesis_authority` to report and no sequence to report it at, and `snapshot`
cannot manufacture either without fabricating a trust root. A namespace that *is* bound and
retired is the opposite: `Deleted` is a lifecycle, it has a state, and it answers every other
question. Refusing both would erase a difference the store knows.
**`RepoSnapshot` therefore gains `lifecycle()` and `storage_mode()`.** `RepoState` already
carries both; without accessors a reader holding a snapshot could not tell an `Active`
repository from a `ReadOnly` or `Deleted` one, and §4 makes that distinction observable. Both
read straight off the pinned state and allocate nothing. `lib.rs` re-exports `ObjectLocation`
and the two namespace enums, which name the return types of the frozen `locate` and of these
two accessors — a caller cannot use the re-exported `RepoSnapshot` without them.
**Deliverable 8's acceptance is amended, and the reason is that the design already
succeeded.** "A test that fails if someone clones" assumes a clone is a copy; `CommittedRoot`
is built entirely from `im` persistent structures, so cloning it allocates zero bytes and a
byte figure cannot fail on a clone anywhere in this crate. The scope records the full finding.
The test carries a measured no-materialization figure *and* an `Arc::ptr_eq` structural proof,
with a live control, rather than the single measured assertion the wording asked for.
**One observation, recorded and not acted on.** `ObjectLocation::segment_generation` is
per-shard, not global, so the same `(generation, offset)` pair occurs in every shard's journal.
This is not ambiguity — a location is only ever read through a snapshot, and the snapshot's
namespace determines the shard — but it means any test comparing locations across shards
asserts nothing. `tests/namespace_snapshot.rs` co-locates its two repositories for that reason
and for the stronger one: only a shared shard makes the namespace component of the index key
load-bearing.
##### Contract review 2026-07-31-A
B3 deliverable 6 — finalize and the adoption pin. Deliverables 7 and 8 are **not** in this
commit; review found a blocker in 6 that had to close first. The dispatch is 6-8 only: the
ignored sealed-invisibility test stays ignored, and its blocker list is corrected below.
**Scope §6.5's state machine was unimplementable as written**, in two of three transitions, and
is amended to `Open → Sealed → Finalizing → {Sealed, Adopted}` with all four states durable and
reconstructable. The reasoning is recorded in the scope rather than repeated here; the short
version is that `Open` appears twice in the original wording where only `Sealed` can be
reconstructed, and `Adopted` is a state the wording needed and did not have. Reconstructability
is the membership rule for this enum — a value no restart can produce has no business on the
wire — and both amendments follow from applying it.
**Coalescing moves to B1.** §6.5 promised that "identical concurrent finalizers coalesce onto
one" at this layer. They cannot: `ProjectionAdoption` is an unforgeable capability with exactly
one terminal outcome and a `Drop` that reports its absence, so there is no second copy to hand a
second caller. B3's guarantee is *one pin*, and a second finalizer is refused by name.
Request-level coalescing across retries of one operation belongs to whoever owns the request. No
B3 coalescing regression is claimed.
**The blocker, found in review.** `finalize` published the pin marker through the shared artifact
writer, which publishes with `rename_noreplace` because every other staging artifact is
unique-by-name. Recording `Adopted` then wrote the same pathname through the same writer and
always failed `EEXIST`, wedging every adoption in `Finalizing` — silently, because nothing
inspected the error. The repair is **not** a replace flag on the shared writer: that would hand
overwrite permission to the chunk and manifest artifacts the no-replace rule exists to protect.
It is one marker-specific path whose licence to overwrite is bounded by proof — the marker on
disk must decode as a staging artifact of this session, carry the operation and digest the
session binds, and say `Finalizing` — and which publishes through the no-follow temporary-open
primitive, then a replacing rename, then a directory sync. Idempotent on an
already-`Adopted` marker, because a retried outcome is not a second event.
**`SESSION_FIXED_FILES` is 4, and it is a peak.** Deliverable 6 adds the marker and, for the
width of the replacement, its temporary. The reservation is charged against the peak because
that is the instant the directory is widest. Two things surfaced on the way: the constant's doc
claimed to be "the shared definition rather than a second opinion" while `options.rs` carried a
literal `+ 2`, and the **default `staging_max_files_per_session` was sized to the old layout
exactly**, so every root became an invalid configuration until it was raised. Both are
configuration-visible.
Four is the *true* peak, not a conservative one. I first recorded a general residual here —
that every artifact write peaks at `+1` because `write_artifact` also publishes through a
`<name>.tmp` — and review corrected it: those temporaries stand in for final names that are
absent, so each occupies the slot it is about to become rather than an extra one, and state
ordering keeps chunk writes from overlapping a seal or a finalize. `adoption.tmp` is the only
temporary that coexists with a final file already on disk, because its transition replaces a
marker rather than creating one. The claim was generalised from "there is a temporary" without
checking whether the final name was occupied.
**Recorded, not fixed.** `write_artifact` opens its temporary with `File::options()` and no
`O_NOFOLLOW` — the same redirection family closed in `checkpoint.rs` under 2026-07-30-D. The new
marker path uses the funnel; the shared writer still does not.
**Plan §8 is amended in place**, not merely superseded by this record. It still specified
`Open -> Finalizing`, store-level coalescing, and return-to-`Open`; a contradiction left standing
in the plan is a contradiction, and the later review being right does not make the earlier prose
unread.
**Evidence.** Six regressions, covering every transition of
`Open -> Sealed -> Finalizing -> {Sealed, Adopted}` against both the in-memory state and the
device. The adoption chain — pin taken, adoption recorded, reservation released, reopen
reconstructing `Adopted` without re-charging — plus one each for: a second finalizer refused
rather than issued a second pin; expiry declining a pinned session and leaving its artifacts in
place; a definitive pre-append failure removing the marker, returning `Sealed`, and *permitting
refinalization*; and a transferred pin whose marker survives, reconstructs as `Finalizing`, and
is offered to `transferred_sessions` for its own shard and no other. Each asserts the marker on
disk, not only the state in the registry — a test that checked the registry alone would pass
against a pin that never reached the device.
The expiry/finalize race needs **both** orderings and originally had one. The covered ordering
was expiry meeting an already-durable pin, where the *state* reads `Finalizing`. The other is the
admission window: the finalizer has been admitted under the registry lock and released it to
write the marker, so the session still reads `Sealed` with nothing on disk, and a sweep that
looked only at state would find an expired, sealed, unpinned session and delete the artifacts out
from under a pin about to be issued. What prevents it is the `busy` exclusion, so the regression
drives that shape directly rather than racing into it — a race reproducing one time in a thousand
is a test that passes for the wrong reason the other nine hundred and ninety-nine. Its second
half clears `busy` and sweeps again, which *does* reclaim: without that, "nothing was reclaimed"
proves only that something declined, not that the exclusion is what declined it.
Every one is mutation-checked: admitting a second finalizer, letting expiry reclaim a pinned
session, skipping the marker removal on release, and hiding `Finalizing` from
`transferred_sessions` each fail exactly one test and no others.
Mutating the admission-window exclusion produced a result worth recording. The sweep does change
behaviour and the regression fails — but on `Overloaded` from `reclaim_session`'s own `busy`
check rather than on a deletion, because the guard is two independent layers. Removing the second
one as well **does not compile**: that match is exhaustive, so dropping the pinned case from the
reclamation guard is a compile error rather than an omission. Reaching artifact loss from here
requires deliberately writing a `Finalizing => {}` arm, which is no longer something a future edit
does by accident.
And for the adoption chain, both halves:
routing the adoption write back through the shared writer reproduces
`Io(Os { code: 17, kind: AlreadyExists })` at the finish, and reconstructing an adopted session
with its reservation fails the accounting assertion. The fixture had to be built inside
`staging.rs`'s unit module: `finalize` is `pub(crate)` so its regressions cannot live in the
integration file where all the sealing machinery is.
**The ignored sealed-invisibility test remains blocked, and its stated reason is stale.** It
names `submit` as a blocker; `submit` has been production-reachable since the B-wave work. The
real blockers are `RepoSnapshot::locate`, `StoreEngine::snapshot`, and
`ValidatedTransactionBuilder::adopt_projection` — all three B1 deliverable 8 — plus the staging
accessor, which stays a pending interface amendment. Finishing 6-8 will not unblock it.
##### Contract review 2026-07-30-G
Recovery run-discard, which 2026-07-30-B deferred and named as the thing that would turn its refusal
into reclamation. It is a change to what recovery *reclaims*, not an addition to it, which is why it
is its own commit with its own mutation evidence.
**The state.** A `.seg` occupies the identity the active tail's frames already carry, so recovery
must seal them under a different generation. A published index run holding locations against the
displaced identity then stays authoritative through the manifest while resolving to nothing. B
refused every root in that state. Refusing is an outage on a root that has lost nothing, and B said
so at the time.
**What makes reclamation sound, and it is `preferred` itself.** `active_tail_logical_generation` is
one past the last retained tail range, so it is the identity of the **active journal** and of nothing
else. Frames at that identity sit above the manifest's committed prefix, therefore above any
checkpoint horizon, therefore replayed in full. A run holding *only* locations at `preferred` is
rebuilt entry for entry by the delta recovery is about to construct, at the generation the frames
actually receive. Discarding it loses nothing; keeping it publishes locations that resolve to
nothing.
**What is still refused, and why that is the good outcome.** A run may name more than one identity: a
reopen replays what the previous session sealed into a segment *and* what its journal still holds,
and a seal over that backlog covers both. Displacing the tail strands only the tail's entries — the
segment's may sit below a checkpoint horizon, which replay does not touch. Discarding that run would
delete the only index those objects have, trading a diagnosable outage for silent loss, which is the
trade B existed to prevent. So the refusal survives for mixed runs and now says why.
**Frozen-seam amendment to `index.rs` (A2).** `references_only_segment_generation`, the sibling of
B's `references_segment_generation` and the second question its one caller has to ask: the first says
a run is *affected*, this one says it is *reclaimable*. Read-only, exact rather than a range test,
and deliberately false for an empty run — emptiness is not something to reclaim on this evidence.
**A reclamation is three removals, not one.** The manifest row is what makes a run authoritative, so
dropping it governs the next open; but this session must also stop consulting the run and stop
pinning it, or a direct run consumer — `StoreEngine::checkpoint` is one — reads dangling locations
for as long as the process lives. A lookup will not reveal it, because the replay delta sits above
the runs and answers first. The `.idx` itself is **left on the device**: nothing but a manifest makes
a run authoritative, and unlinking during recovery would destroy the evidence for a state this store
has only just learned to handle. `ShardRecoveryReport::reclaimed_index_runs` reports it, because a
reclamation is otherwise invisible in the opened store.
**One placement that matters.** The reclamation is applied inside the seal, not where the generation
is chosen. `logical` is decided before the journal is scanned, so it can name a fallback for a
journal that turns out to hold no frame — and then nothing is renamed, the surviving tail keeps
`preferred` (2026-07-30-B amendment 2), and a run at `preferred` goes on resolving. Reclaiming there
would discard a run that was never stranded.
**Mutation evidence.** Four mutations, each caught: treating every run as reclaimable fails the mixed
-run refusal; treating none as reclaimable fails all three reclamation regressions; leaving the
manifest row fails them on the authority assertion; and leaving the run in this session's index and
pins fails them on the direct-consumer assertion. That last mutation **survived the first version of
these tests**, because the replay delta shadows the run for ordinary lookups — the assertion that
catches it was added after the mutation showed the gap rather than before.
##### Contract review 2026-07-30-F
Two P1s and a P2 against the checkpoint-equivalence half of 2026-07-30-E. The namespace and
run-baseline halves of E passed review and are untouched. Run-discard remains held.
**The mistake underneath both P1s.** E loosened the adoption predicate so the crash path could
succeed, and stopped there. It should have asked what *gets published* once the predicate passes.
Adoption publishes the bytes the crash left behind — an image written **before** the recovery that
the crash forced — so every field recovery is entitled to move was published at its pre-recovery
value. Erasing a field from a comparison is not the same as deciding it does not matter, and E
treated the two as one.
**P1 — adoption shortened receipt retention.** Plan invariant 7: recovery re-derives a receipt's
first-visibility and deadline and "therefore only extends retention". The stranded image carries the
pre-promotion pair, so publishing it puts an earlier `Some(first_visibility)` on the device — and
the next open anchors on that value instead of re-promoting, because a durable anchor is exactly
what suppresses promotion. Retention regresses across a repair. Confirmed by the reviewer against an
isolated `DirSyncEio` run: both timestamps advanced during recovery and then went backwards after
adoption.
**P1 — the resume pair could not be erased.** E claimed recovery independently verifies the offset.
It does not. When a checkpoint's `active_journal_id` matches the surviving journal, recovery trusts
the recorded offset outright. The reviewer changed only a valid orphan's resume pair to the
surviving journal's ID plus its preallocation end; adoption accepted it, a frame committed, and the
next open failed `Corruption("shard 0 recovered through sequence 2 but its active journal resumes at
3")`.
**The repair is replacement, not adoption.** The shard writes its own image and installs it over the
found name. Both hazards close with one stroke: what reaches the device is what this call built,
carrying the promoted deadlines and a resume point naming the journal this checkpoint is about to
seal. `durable_claims` survives as what it always was — a comparison aid — and `reconciles_with` now
states the whole safety argument as three tests:
1. **Same durable claims**, or the file describes state this shard cannot produce and overwriting it
would destroy the only evidence of however that happened. This is 2026-07-30-D's rule, unchanged.
2. **The resume point may differ only by naming a different journal.** The ordinary strand passes
freely — it names the journal the crash left behind, which recovery sealed away. A matching
identity at a disagreeing offset is refused, which is the reviewer's attack.
3. **The replacement may not shorten retention.** The candidate must be at or beyond the found image
on both visibility fields, which is the direction recovery guarantees. If it is not, something
other than the expected crash produced that file and the checkpoint refuses.
**Frozen-seam amendment to `checkpoint.rs` (A2), and a narrow exception to a funnel rule.**
`replace_reconciled` is added beside `install`, sharing one body. `sys::rename_noreplace`'s contract
lists checkpoints among the names that must never be overwritten; **a checkpoint name this shard has
proved out under `reconciles_with` is now the one exception**, and the rule stands everywhere else,
including for any name the shard has not proved out. The safety argument is that the two images
agree on every durable claim and differ only where the replacement is the better of the pair, and
that `rename` is atomic — so a manifest naming that file holds a valid referent at every instant,
before, during and after. An unlink-then-install repair would not have this property, which is why
it was not used. A published name is refused outright as a further guard: if the current manifest
already references the file, the shard is not recovering from a strand.
**Evidence.** The `DirSyncEio` regression now asserts what is on the device after the repair, not
only that the checkpoint succeeded: the published resume point must not name the journal recovery
replaced, no receipt may lose retention, and at least one must have *advanced* — that last one
exists so the retention assertions cannot pass vacuously. Reverting replacement to adoption fails
both halves independently. The three reconciliation rules also have direct unit coverage, including
the ordinary strand, which must keep reconciling.
**P2 — the contracts contradicted the code and each other.** The method documentation still required
the resume point to match; E claimed both exclusions were safe. Both are rewritten here against what
the code now does.
**P2, second round — an unproved overwrite was on the module's public surface.** The first cut put
`install_replacing` beside `install` as `pub`, while `reconciles_with` — the proof that licenses
overwriting anything — was private to `engine.rs`. Any caller reaching `checkpoint` could therefore
replace a checkpoint without the proof, which is precisely the operation the exception above was
granted for and only for. The predicate and its replacement are now **one operation**:
`durable_claims` and `reconciles_with` move to `checkpoint.rs`, which owns the type they reason
about; `install_replacing` is gone; and `pub(crate) replace_reconciled` performs the proof and the
rename together, so no path in the crate — let alone outside it — can overwrite a checkpoint without
first establishing that the occupant reconciles with what replaces it. `install` remains the only
public installer and remains no-replace. `sys::rename_replace`'s "exactly one legitimate call site"
sentence is amended to name both.
##### Contract review 2026-07-30-E
Two P1s against `bff8e8a`, one terminology correction, and the gap 2026-07-30-D disclosed. All
closed. Everything is in `engine.rs` except one addition to `RecoveredShard`: a new
`checkpoint_namespaces` field and the line that captures it. That is a **frozen-seam amendment to
`recovery.rs` (A2)**, directed by the lead in this dispatch, read-only in effect — it introduces no
rule and changes no existing value — and recorded here rather than assumed.
**Not a bug, and the distinction is the point.** The reviewer's first correction was aimed at me:
the two subtractions in `replayable_index` look like one idea and are not. The **entry** term is a
*duplicate-representation* baseline. Recovery both retains every run it finds and rebuilds every
replayed frame into the delta, so `current run entries pre-open run total + backlog` counts each
replayable entry exactly once, and a later seal merely moves entries from the backlog into runs
*above* that fixed baseline. Narrowing it to runs below the horizon — which I had proposed — would
double-count every post-checkpoint entry and refuse early. No entry accessor and no per-run horizon
seam is being added. What was wrong there was the name and the comment claiming a proof the term
does not make: it is now `run_entry_baseline`, documented as what it is. The reviewer's own
regression (four-entry ceiling, three replayed plus one incoming accepted, reopen succeeds, fifth
refused `observed: 5, allowed: 4`) pins the boundary from both sides.
**P1 — replayed namespaces were subtracted away.** The **namespace** term has no backlog term to
cancel against, so it is a genuine horizon claim, and it was reading `recovered.catalog` — the
checkpoint's catalog with every replayed frame already applied. A namespace created above the
horizon therefore subtracted itself out: admission measured a section that was not there, accepted
on that basis, and the next reopen refused the store it had just written. Recovery now carries
`RecoveredShard::checkpoint_namespaces`, captured before replay applies anything, and the writer
counts that. Mutation-checked both ways — with the old count the regression's submit is accepted
and the reopen fails `observed: 158, allowed: 111`, which is the reviewer's reproduction exactly.
**P1 — the advertised adoption path could not succeed.** Equality was the wrong predicate. The only
state that produces an unreferenced checkpoint is a crash, and the retry necessarily happens after
the recovery that follows it — so the stranded image and the rebuilt one always differed, and at the
replay ceiling no write could move the shard to another filename. The refusal was the original wedge
with a better error message. **Superseded in part by 2026-07-30-F**: the first attempt at this
loosened the predicate and then *adopted* the found bytes, which was wrong for reasons recorded
there. The repair is replacement, not adoption.
**The disclosed gap is closed.** `DirSyncEio` produces the state deterministically, and no new
seam was needed. `checkpoint::install` fences the directory *after* the rename, so a failure there
strands a finalized, validating, unreferenced checkpoint. Reaching it needs the checkpoint to skip
`seal_index`, which is why the test first drives an admission that seals the due backlog and is
*then* refused by the ceiling — the one state that is checkpointable, uncheckpointed, and carries no
layers. The test asserts that precondition rather than assuming it.
**One asymmetry left deliberately.** With no checkpoint at all both terms are zero, so a run's
entries are counted once in the run and again in the replayed backlog. That over-states pressure and
can only refuse early, never admit work a reopen cannot rebuild, so it is documented rather than
changed here.
##### Contract review 2026-07-30-D
Five review findings against `5462952`, all closed. The first two are the ones that mattered.
**P1 — adoption was content-blind.** The guard compared the path and the committed sequence, so a
*different* checkpoint at the right name was adopted and published. That is not a wedged shard, it is
silent logical deletion: the manifest suppresses replay of the frames the checkpoint covers, so an
empty catalog at the live sequence makes the namespace disappear at the next open with nothing
reporting a fault. Adoption now requires logical equivalence to the checkpoint the call would itself
have written, with `created_at_micros` the only field allowed to differ — **narrowed by
2026-07-30-E** and again by **2026-07-30-F**, which reaches the same place by replacing the stranded
artifact rather than adopting it.
Anything else at the name is
refused — and refused **without poisoning**, since nothing durable has moved. The test is the reopen:
catalog, ref and receipt all read back through the public surface after the refusal, and the shard
is still writable.
**P1 — checkpointing did not relieve the replay ceiling.** `replayable_index` summed every sealed-run
entry and every namespace regardless of the horizon, so a shard at the ceiling checkpointed
successfully and was then refused its next single-object transaction — the store telling an operator
that checkpointing did nothing. The writer now tracks what the newest checkpoint made unreplayable,
set at open from what recovery loaded and again by every checkpoint it takes, and subtracts it.
Subtracting rather than dropping the accounting: runs sealed *after* a checkpoint cover frames a
reopen does replay, and ignoring them would reopen the hole that let a store accept work it could not
read back. Mutation-checked — removing the subtraction reproduces `observed: 4, allowed: 3`.
**P1 — the installer wrote through symlinks.** `checkpoint::install` opened its temporary with
`create + truncate` and no `O_NOFOLLOW`, which was survivable while nothing production could reach
it and stopped being so the moment this dispatch gave it a caller. It now goes through the same
no-follow funnel `segment.rs` uses, and an occupied final name is type-checked before adoption.
Frozen-seam amendment to `checkpoint.rs` (A2), granted and recorded here.
**P2 — pins grew without bound.** The manifest trimmed to `checkpoint_retain` while the successor
cloned every prior pin and appended one more, so pruned inodes and their disk space stayed alive for
the life of the process. The successor now pins exactly the rows its manifest publishes; an older
lease's `Arc` still independently retains its own generation, which is what a lease is for.
**P2 — construction was not canonical.** Refs and receipts were appended straight from randomized
HAMT iteration. Both are sorted by their stable keys before encoding, which the equivalence guard
above also depends on.
**One gap, disclosed.** The adoption *success* path — a byte-equivalent stranded checkpoint being
adopted — is not covered by a test. Constructing an equivalent image from outside the engine needs
the body the engine builds, and the honest options were a new failpoint seam in the publication
window or a test-only accessor. Neither belongs in this commit. The refusal path, which is the one
that could lose data, is covered.
**Closed by 2026-07-30-E and 2026-07-30-F**, and the gap turned out to be hiding a defect rather
than only a missing test: writing the success path proved the equivalence predicate could never hold
across the recovery that separates a crash from its retry, and then that loosening it was not enough
either. Neither of the options considered here was needed — `DirSyncEio` reaches the state on its
own.
##### Contract review 2026-07-28-C
B4 re-pointed `store-bench` at a real `StoreEngine::submit` and found that four verification
claims `bench/result-schema.json` declares not-applicable at `storage_primitive` have become
genuinely earnable. It **requested rather than emitted** them, which is what §6.6 item 4 asks
for. All four are granted, three with conditions that are landed as schema constraints rather
than as prose, and one — `operation_receipts_reconciled` — is granted **in principle and is
not earned by the emitter as it stands.** The schema and its contract tests land here; the
emitter is B4's follow-up and is specified in scope §6.6.
**1. Machine-readable run conditions, ranked first because the other three depend on it.**
There was nowhere in a bundle to record that the root was seeded outside `StoreEngine::open`,
that `max_index_runs` was raised, or that no checkpoint was taken. Every one of those is
true of the run that produces today's numbers, and every one of them lived only in a harness
comment and a human report. A bundle whose caveats live outside it reads as unconditional to
everyone who receives it, and the people most likely to receive it without the report are the
ones furthest from the harness.
`run_conditions` is a new required top-level block with ten members: `initialization_path`,
`mutation_path`, `checkpointing`, `index_maintenance`, `index_run_ceiling`,
`receipt_reconciliation`, `objects_new_source`, `commit_id_uniqueness`, `build_profile`, and
`environment_fidelity`. Every member is a closed enumeration; there is no free-text member and
no catch-all value, and a contract test asserts both properties over the block rather than
over a list this file also wrote.
**A prose caveat field was rejected outright, and the reason is the whole design.** A string
is something a consumer reads; these are things a consumer checks. The block is not a place to
put disclosures beside the claims — it is what the claims are *conditioned on*.
`objects_new_equals_three_per_commit` is forbidden when `objects_new_source` says the count was
derived from the transaction total. `unique_blob_tree_commit_ids` is forbidden when
`commit_id_uniqueness` says the check was per-record or inferred. `operation_receipts_reconciled`
is forbidden when `receipt_reconciliation` says any `Committed` status was accepted, and
*required* when it says otherwise. A harness can only declare what it did, and the declaration
decides what it may claim. That is the difference between a caveat and a condition.
Two of scope §7's exit clauses were prose until now and are mechanical here: a passing
`storage_primitive` run must declare `checkpointing: "exercised"` — §7 requires the P2 runs not
to have been achieved with checkpointing disabled — and `index_maintenance: "runs_sealed"`,
because a run holding every index delta in memory with a lookup fan-out that grows for its
whole duration is not the steady state a P2 number describes. It must also declare
`initialization_path: "store_engine_open"` and `mutation_path: "store_engine_submit"`. The
harness satisfies **two of the four** — B1's startup state 1 landed, so the submit path both
creates its root and mutates it through the production entry points. It cannot satisfy
`checkpointing` or `index_maintenance`, so it still cannot emit a passing P2 bundle, which is
correct and was previously only an assertion in a comment.
That the count moved from zero to two is worth stating rather than silently editing: the four
conditions are not decoration on a number, they are the number's preconditions, and knowing
which remain unmet says exactly how far the P2 criterion is from being earned. The two that
remain are the two that make a P2 figure a steady-state measurement rather than a burst.
**2. `objects_new_equals_three_per_commit`, granted for the production submit path.** The
schema said it was absent at `storage_primitive` "which creates no objects". That stopped being
true when the measured path became `submit`: the path stages the canonical three objects per
commit and the harness sums `receipt.objects_new` from the store's own receipts. The condition
is that the claim compares an **independently summed receipt total against a separately counted
`3 × counted_commits`**. Both sides deriving from the transaction count is what the claim was
originally excluded for — an assertion that cannot fail is not a check — and
`objects_new_source` is what makes that exclusion survive the grant, at this gate and at every
other.
**3. `operation_receipts_reconciled`, granted in principle and not earned.**
`store-bench.rs:2033` accepts any `TransactionStatus::Committed(_)` without comparing its
payload, and the `receipt_digest` the harness journals is `blake3(operation_id)` — a digest of
the operation, not of the receipt. So the run reads back that *something* committed, which is
already what `acknowledged_sequences_reconciled` reports, and calling it receipt reconciliation
would be a second name for the same evidence. The claim is landed as expressible and correctly
constrained: an emitter that reconciles exact receipts, or a frozen canonical receipt digest,
declares it and **must** then assert the claim; an emitter that does not declares
`acceptance_of_any_committed_status` and **cannot**. Landing it now means closing it is an
emitter change rather than a second schema amendment, and the schema description says in as
many words that the current emitter does not earn it.
**4. `unique_blob_tree_commit_ids`, granted for the production submit path**, on condition that
uniqueness is established **globally across every recovered acknowledgment record**. Per-record
uniqueness is not uniqueness — two records may each be internally distinct and still share a
commit id — and distinct generator seed domains make a collision unlikely rather than absent,
which is an argument about probability rather than an observation. `inferred_from_seed_domains`
is therefore a *named* value of `commit_id_uniqueness` rather than something folded into the
passing one: a harness that reasoned that way has a truthful thing to record and is refused the
claim, which is a better outcome than having to choose between a lie and silence.
**The branch conditional, which is the part that is easy to get wrong.** These are not global
loosenings. A schema that merely *permitted* the three claims on both paths would hand the
Wave A journal seam a way to assert what nothing below `engine.rs` can observe — and that is a
worse defect than the one being fixed, because it arrives disguised as the fix. Two rules key
on `gate == "storage_primitive"` and `run_conditions.mutation_path`:
- **`journal_drive`** forbids all three claims outright *and* pins the four provenance
declarations to the only values the seam can truthfully make (`no_index_in_path`,
`no_receipts_in_path`, `derived_from_transaction_count`, `not_checked`). Forbidding the
claims alone would have left the same hole one field over: a drive-path bundle could declare
exact receipt reconciliation it has no receipts to perform, and nothing would have noticed.
- **`store_engine_submit`** requires `unique_blob_tree_commit_ids` and
`objects_new_equals_three_per_commit`, both `const true`. Omission is a missing result here,
not an inapplicable one.
`blobs_recomputed`, `metadata_complete`, and `commits_in_recovered_closure` stay unavailable on
**both** paths and stay in the gate-wide rule, because they need the graph traversal plan §5.1
forbids the store from performing. The gate-wide forbidden set is now exactly the claims no
path can earn, and a contract test asserts its size so a future claim cannot be quietly parked
there.
**`max_index_runs` is now a named required member of `resources.configured_ceilings`.** It was
reachable only through that block's free-form `additionalProperties`, so an emitter could omit
the one ceiling this workload actually reaches — the run seals against it — and
the bundle stayed valid. It must be the value the run configured, read back from the options the
store opened with. The schema cannot see the process's options, so the bite is a cross-check:
`index_run_ceiling: "store_default"` bounds the recorded value at 64 and
`"raised_because_index_sealing_unimplemented"` floors it at 65, and a bundle that declares one
while recording the other is invalid in both directions. A contract test asserts that
`StoreOptions::default`'s 64 is still 64, so the schema's bound and the library cannot drift in
silence. What remains open is a bundle that lies about both consistently, which no schema
closes, and it is stated here rather than left to be discovered.
**The truthful-environment ruling: yes, and here is why.** `deployment.persistent_data_mount`
and `deployment.tmpfs` were unconditional `const true` / `const false`, so B4's debug/tmpfs
diagnostic run at 8,052/s **could not be encoded at all**. It was not disqualified — it was
unrepresentable, which is a strictly worse state: the number existed, it was informative, and
the only places it could live were a console and a paragraph. That is the same failure the
`outcome` field was added to fix in review 2026-07-24-B, one block over, and the same argument
applies. A schema that can only express successful runs is not a record of what was measured.
The two fields relax to `type: boolean` and are **re-pinned** by
`environment_fidelity: "reference_profile"`, which additionally requires a `release` build and
a named hardware profile. `outcome: "pass"` requires `reference_profile` **at every gate**, so
nothing a claim used to cost has changed — a `gate="storage_primitive"` bundle claiming P2
still requires the real environment, by a rule that is one implication instead of two consts.
Going the other way, `"diagnostic"` is not merely a label: outcome is bounded to `fail` or
`preliminary` and no verdict may be `pass`. Recording a diagnostic run is worth nothing unless
the record also refuses to let it be read as a result.
What was **not** relaxed, deliberately: `overlay`, `remote_storage`, and `durability_enabled`
keep their unconditional consts. A run with durability disabled is not a slower measurement of
the same thing, it is a measurement of something else, and there is no diagnostic value in a
fence-free number that would justify making it expressible.
**Expected collateral: the emitter no longer produces a valid bundle, and the gate is red until
B4's follow-up.** Exactly two fields are missing, on both paths:
```
[]: 'run_conditions' is a required property
['resources', 'configured_ceilings']: 'max_index_runs' is a required property
```
`scripts/check-phase1.sh` does not run `scripts/verify-store-recovery.sh`, so the expectation
was that the gate would stay green while bundle emission broke. **It does not, and the reason
is worth recording:** the gate runs `store-bench`'s own unit tests, and since review
2026-07-24-B three of them validate the emitted bundle against `bench/result-schema.json`
rather than against a list of substrings. So the emitter's schema conformance is inside the
gate, which is exactly the property that review was after — the drift is reported by the gate
instead of by a script nobody ran. `the_emitted_bundle_validates_against_the_frozen_schema` and
`a_failing_run_is_representable_rather_than_suppressed` fail with the two errors above;
`the_schema_check_can_actually_fail` fails on its final assertion for the same reason and not a
second one, because it asserts that a coordinated-omission mutation leaves a *clean* bundle and
the base bundle is no longer clean. `scripts/verify-store-recovery.sh --cycles 2` was run
directly rather than assumed and reports `matrix=pass`, `cycles_completed=2`,
`acknowledged_loss=0`, `torn_transactions=0`, `repeated_adoptions=0`, `bundle=schema-invalid`,
`VERIFY_EXIT=1`.
All three failures are in `store-bench.rs`, which is B4's file, and the fix is scope §6.6 item
5 rather than an edit here. This is the sequencing of 2026-07-28-A repeated deliberately: the
gate is transiently red between the contract and the package's pass, and landing the contract
first is what keeps B4 from implementing against a surface that is about to move. It is
recorded rather than worked around, because a lead who edits the emitter to keep the gate green
has moved a package's work into a review and left no one able to see that it happened.
One measurement from that run belongs on the record, because it is what makes amendment 2 more
than an argument: the submit path reported `objects_new=861` against `transactions=287`, summed
from 287 independent receipts. Three per commit, counted rather than multiplied.
Amended: `bench/result-schema.json`, `crates/levcs-protocol/tests/phase0_benchmark_contracts.rs`,
and scope §6.6 and §7. `bench/reference-hardware.toml` required no change: the frozen profiles
describe the reference environment, and `environment_fidelity` records which runs met it —
putting a diagnostic profile in the frozen file would have made a non-comparable configuration
part of what "frozen" means.
### Phase 1 — storage engine spine
Lead first defines sealed transaction/frame/snapshot interfaces and file ownership. That deliverable (D0) landed on 2026-07-24 as `crates/levcs-store`: the frozen public API compiling against `StoreError::NotImplemented`, the file-ownership split, strict configuration validation, the single durability syscall funnel with its counters and fault hooks, the failpoint registry in enforced one-to-one correspondence with `oracle::AppendFailpoint`, and the journal-level drive seam that lets the crash harness run in Wave A. The enforced gate is `scripts/check-phase1.sh`, which runs `check-phase0.sh` first so the Phase 0 freeze stays enforced. That work is scoped in `doc/phase1-storage-spine-scope.md`, which realizes this section as a file-ownership matrix, a frozen `levcs-store` API, a physical format and durability/recovery specification, per-package deliverables and acceptance criteria, the Wave A adversarial review charter, and the capacity analysis for P2 on the frozen reference hardware. This plan remains authoritative; that document is the Phase 1 realization of it and lists the decisions that must be resolved before Wave A starts.
Parallel wave A:
- **JournalWriter:** `format.rs`, `journal.rs`, `segment.rs`; frame codec, append, fence, rotation.
- **RecoveryIndex:** `index.rs`, `checkpoint.rs`, `recovery.rs`; rebuild and torn-tail handling.
- **StoreHarness:** durable-ingest benchmark and deterministic crash harness against the public store API.
Wave B starts only after Wave A interfaces, golden frame vectors, crash/fault fixtures, and durability ordering have passed review and are frozen; compilation alone is not the dependency gate.
**Wave A was frozen on 2026-07-26 at commit `5ee9c6b78b6e5f77f988e99e477656cfdc7db352`**, after the adversarial review found five defects behind a fully green gate — three of them blockers — and all five were closed. Changing a frozen Wave A interface, the frame format, the durability ordering, or the golden corpus now requires a contract review recorded in this document. Two carry-forwards are explicitly outside the freeze and belong to Wave B: extending the crash matrix to generate sealed-frame corruption and cross-shard journal movement, and wiring `GroupBuilder` through B1's production path. Both are recorded in §5 of the scope document with their reasons.
That review is adversarial and precedes the freeze. Phase 0 shipped a physically unsound recovery classification — deterministic absence claimed for a state where a complete frame may already be durable — past its own exit criteria, because the enclosing test asserted every neighboring field except the one that was wrong (contract review 2026-07-24-A). Wave A's frame and recovery model is the same class of contract and a larger surface. Its review must therefore try to refute each durability claim against the physical state the storage stack can actually be in, not merely confirm that the model is self-consistent, and every frozen outcome must be individually asserted rather than covered by a catch-all arm. A reviewer that only reproduces the author's reasoning has not reviewed it.
- **NamespaceTxn:** `engine.rs`, `transaction.rs`, `snapshot.rs`, `staging.rs`; repo creation, bounded staging artifacts and inline/staged object-membership adoption, atomic ref/authority CAS, independent shard/repository sequences, typed evidence, linearizable status, receipts/idempotency/terminal retention.
- **StorageReviewer:** read-only durability/concurrency/security review.
Exit: P2 ≥75k, acknowledged crash recovery, namespace isolation, exact same-ref winner, multi-ref atomicity, no per-object fsync.
### Phase 2 — protocol, validation, and admission
Parallel wave A:
- **ProtocolV2:** canonical normal/fork envelopes, mirror/admin evidence, private-read/missing/status/deterministic-snapshot/token/request-pin/event codecs, projection-staging codecs, dual sequence fields, and client-side prepared body.
- **StreamingPack:** bounded Pack v1 streaming codec and fuzz/property coverage.
- **IdentitySession:** anchored whole-graph verifier, native/foreign fork-boundary and authority-transition ordering rules, release semantics, and authority-cache facts.
- **AdmissionAuth:** v2 ingestion/read authentication, checked retry and nonce-horizon bounds, pre-append deadline/terminal-resolution retention, replay guard, same-ID in-flight coalescing, snapshot request-pin admission, hard limits, and fairness/backpressure.
- **ProjectionCore:** one full/release/metadata native/foreign root/edge/boundary algorithm, inline/staged mirror snapshot/projected/cursor-only application, staged manifest validation, replay-event dependency retention, and golden projection digests used by ingest/read/mirror/backup/compaction.
- **ReceiveSpool:** dedicated bounded temporary I/O pool, quotas/backpressure, cleanup/scavenging, and no-filesystem-on-Tokio instrumentation.
Wave B after ProtocolV2 golden vectors:
- **InstanceHarness:** owns `tools/levcs-loadgen`, `bench/workloads`, result schema, deployed/recovery scripts, and invalidation fixtures. It proves byte-identical v2 requests against the decoder and intentionally rejects batch, dedupe, tmpfs, non-ACK, and missing-metadata results. A runnable archived dry-run bundle is a prerequisite for Phase 3.
- **InstanceMigrator:** implements typed legacy evidence, sibling/same-device atomic installation, production-recovery/digest gates, and golden/representative legacy fixtures before runtime cutover.
No agent edits instance composition or storage internals. The lead integrates shared errors/types after each slice passes targeted tests.
Exit: every adversarial matrix case rejects without creating a `ValidatedTransaction`; parse/hash/authority/graph visit instrumentation proves once-per-session behavior; ProjectionCore and migration fixtures pass; the deployed harness can produce and reject a complete dry-run bundle.
### Phase 3 — instance integration
Lead wires `StoreEngine`, pools, config, projection core, federation identity, and router composition only after the migrator and harness prerequisites pass.
Parallel wave:
- **IngestService:** init/push and projection-staging finalization pipelines through final speculative sequencing, typed status, and store submission.
- **SnapshotReads:** signed snapshot/info/refs/object/pack/transaction/missing endpoints, deterministic signed-generation/ephemeral-token separation, bounded base/request-pin acquisition/release/expiry, typed status/cursor expiry, and current-policy private-read authorization.
- **ClientCLI:** v2 client, normal/fork push kinds, signed private reads, snapshot-token restart, token-bound missing negotiation, projection staging, coalesced receipt/pending/resolving/terminal-expiry handling, Set/Delete, and receive verification.
- **InstanceMetrics:** readiness, stage/spool/status metrics, queue/fence/publication/storage telemetry.
Exit: P3 ≥66k plus its batch-1 floor, golden/representative migration passes, consumer feed/resnapshot tests pass, all current instance behavior is implemented on v2, and no v2 mutation path bypasses the store. This artifact is not deployed into a configured federation yet.
### Phase 4 — federation, compaction, backup, and operations
Parallel wave:
- **MirrorFeed:** complete deterministic signed snapshot evidence, inline/staged atomic snapshot adoption, projected/cursor-only event evidence, cursor expiry, mode transitions, lag, and full/release/metadata projection tests.
- **CompactionBackup:** safe-point base+tail compaction, atomic event-floor/object-dependency retention, snapshot/backup/staging-adoption pin interference and expired-session cleanup, active-journal checkpoint export, and absent-destination exact no-new-event restore.
- **DeployOps:** config/service/proxy/docs, spool/replay/staging limits, and reviewed root-only fault scripts.
- **NetworkMigration:** bounded staged source snapshot, final delta/fence, atomic membership adoption, and destination durable receipt.
Exit: exact projection digests, backup restore, compaction interference, federation trust/lag/catch-up, and deterministic/random/power-loss gates pass. Only now perform the maintenance cutover: stop mutations; migrate; upgrade and verify every writer and configured peer; start v2; return explicit upgrade-required responses to v1; remove v1 POST and direct instance loose-store/ref paths. Legacy rollback is allowed only before writes reopen; after a v2 ACK, remain v2 or validated-migrate those transactions.
### Phase 5 — aggressive-envelope optimization and release proof
Optimization follows measured profiles only. Expected candidates are allocation removal, batch sizing, index/cache layout, validator parallelism, authority/replay cache layout, fd reuse, proxy logging, and shard/device affinity. No optimization may change v2 semantics or skip validation.
Parallel evidence agents may collect independent CPU, storage, network/TLS, and allocator profiles. One reviewer audits benchmark integrity and another audits durability. The lead selects changes and reruns the complete gate; agents do not optimize from isolated microbenchmarks.
Exit: P4/P5 pass and the result evaluator emits independent `storage_primitive`, `in_process_protocol`, `deployed_30k`, `deployed_60k`, `recovery`, `overload`, `compaction`, `federation`, and `release` verdicts. Only `release=pass` permits the 60k claim.
## 13. Risks and stop conditions
- **Custom journal correctness:** stop performance work if crash invariants are not mechanically testable or recovery requires guessing. Fix format/recovery first.
- **Index memory at sustained rates:** result bundles must report bytes/object and checkpoint lookup fan-out. If the index cannot survive the P4 corpus within the reference RAM envelope, redesign before further tuning.
- **Authority/protocol ambiguity:** do not encode metadata mode or authority transitions until their signed logical contract is fixed in tests.
- **Proxy/network ceiling:** 60k × canonical bytes may exceed 1 GbE. The release profile must have measured network headroom; loopback does not support the deployed claim.
- **Batch gaming:** batch >64, reused objects, unreachable commits, disabled signatures/policy/fsync, tmpfs, or submitted rather than acknowledged counts invalidate the headline result.
- **Compaction starvation:** a store that reaches 60k only with compaction disabled does not pass.
- **Replay memory:** at the configured skew and rate, replay state must remain within its declared bound; silently extending TTL is not acceptable.
- **One hot branch:** do not market many-ref throughput as one-ref throughput. Publish both.
- **Format instability:** physical format remains internal until recovery, migration, compaction, and P5 pass. Future workflow code must bind only to logical snapshots/events.
## 14. Definition of done
The rewrite is complete only when:
1. Every online/import instance mutation uses one validated, durable transaction service; verified byte-exact offline restore is the sole exception and creates no transaction or event.
2. Successful init/push/mirror/migration receipts survive abrupt power loss with complete object/ref/authority state.
3. Repository identity substitution, stale authority, incomplete closure, type mismatch, policy bypass, and partial multi-ref publication are impossible under the test matrix.
4. Full, release, and metadata projections are explicit, enforced identically across ingest/read/mirror/compaction/backup, and verified end to end.
5. Existing object IDs/bytes, signatures, repo IDs, releases, merge records, force rules, and federation identity are preserved.
6. Offline instance migration and checkpoint backup/restore are proved on representative legacy data.
7. Memory, queues, replay state, receipt/resolution state, snapshot base/request pins, projection-staging sessions/artifacts, file descriptors, compaction debt, and overload behavior are bounded and observable.
8. P2, P3, deployed 30k, deployed 60k, recovery, overload, compaction, and federation gates independently pass.
9. Future instance software has one stable logical snapshot/transaction-event contract and no reason to inspect or mutate storage internals.
10. Documentation states only performance and durability claims directly supported by archived reproducible evidence.