Levi Neuwirth
ff9bd0fd06
Implements the schema-minor MUST (binary_format.tex, Schema Versioning) that no writer had ever honoured: a writer raises the chunk minor when it emits a discriminant appended after the minor it otherwise declares, so an unknown-discriminant decode failure is attributable to version skew rather than corruption. introduced_minor() lands on the five vocabularies with post-baseline variants - OperationKind, OperationKindTag, OperationPayload, ReanchorReason, and PreconditionFailureReason - each exhaustive with no wildcard arm, so a future variant cannot compile without being assigned an epoch. The tag epochs live inside operation_kind_tag_vocabulary! rather than beside it, because a sibling match is the parallel list that macro exists to prevent. The sentinel is Option<u16>, not 0, since 0 is a real baseline minor for V1-V3 and conflating them would make the max read correctly only by accident. An envelope's required minor is the max over every discriminant it actually emits; a block's is the max over its envelopes; major and minor derive independently. Baselines are not normalised - V0 keeps minor 1. The manifest seam keeps epiphany-bundle opaque: no ops or layout-ir dependency, and the aggregate version is supplied by the producer rather than derived, with CommitContext carrying the previous one so unchanged barrier content preserves it. The version rides the superblock slot that already exists; Manifest gains no field, which would have been schema-major and would have defeated the rung. bundle.rs's superblock check stays major-only - tightening it to full-version equality is a conformance regression, and s11 locks that. textproj carries the manifest SchemaVersion and never derives it, so COMPANION_VERSION moves 0.9.0 to 0.10.0 with the corpus regenerated. Not because of op-block stamping, which remains projection-invisible. A new normative requirement records the carry-never-derive rule in the companion itself; its rationale names layout-ir, which textproj genuinely lacks, rather than the operation vocabulary, which it has. Gate [7f] adds an independent oracle over decodable in-tree barrier fixtures, requiring exact equality rather than >=. Equality is load-bearing: >= catches under-stamping but not stale over-stamping after the sole maximum contributor is removed. Undecodable blobs are reported not-checkable, never as a pass. Also repairs binary_format.tex's stale claim that OperationKind and OperationKindTag append at 30 with a history stopping at 29, while 30-33 are taken and the normative tables already carry them. Gate: 1399 tests, clippy 0, fmt clean, conformance 8/8 with [7f] at four fixtures checked and one not-checkable, 102 decode vectors byte-identical (verified, not assumed), 13 text-projection vectors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjsEnYhm1gPpf6ii2iFxFV |
||
|---|---|---|
| .. | ||
| examples | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| DECISIONS.md | ||
| README.md | ||
README.md
epiphany-bundle
The Epiphany .musc file format, implementing the normative requirements of
Chapter 8 (File Format) of the core specification (spec/core_spec.pdf).
This is Agent D's crate per spec/QUICKSTART.md. It depends on
epiphany-determinism (Agent A) and on nothing else — not on epiphany-core
(Agent B) or epiphany-ops (Agent C):
bundles handle bytes, ops handles semantics. A canonical-base snapshot from the bundle's perspective is opaque bytes plus a frontier DVV; only
epiphany-opsinterprets it. — QUICKSTART
A bundle is a single file: a fixed 64-byte header at offset 0, two 256-byte superblock slots, then a body of immutable, content-addressed chunks. The superblocks are the only mutable on-disk objects. A commit appends new chunks, writes a new manifest chunk, then flips the active superblock by writing the inactive slot and durably flushing it — that flush is the commit point. Because commits only ever append and touch the inactive slot, a crash can never corrupt the active state.
What's here
| Area | Items | Spec |
|---|---|---|
| Prelude | FixedHeader (64 B, CRC-32C), Superblock/CommitState (256 B, CRC-32C), select_active |
Ch. 8 §"The Bundle Layout", §"Superblock Selection" |
| Atomic commit | Bundle::create/open/commit, the 7-step protocol, cold-open path |
Ch. 8 §"The Atomic Write Protocol", §"Streaming Reads" |
| Content addressing | chunk_content_hash/chunk_id, ChunkRef, ChunkKind, CompressionAlgorithm, domain separation |
Ch. 8 §"Content Hashing", §"Chunks" |
| Manifest | Manifest (canonical_base ≠ acceleration_snapshots), SnapshotRef, BlobRef, ProfileDeclaration, ExtensionDeclaration |
Ch. 8 §"The Manifest" |
| Retention | RetentionPolicy (first-class), ProfileConstraints |
Ch. 8 §"Garbage Collection and Retention" |
| Op blocks | pack_operation_blocks (1 MiB soft target), encode_block/decode_block |
Ch. 8 §"Operation Envelope Blocks" |
| Storage | BlockStore, MemStore, FileStore (real fsync), FaultStore (crash sim) |
Ch. 8 §"Durable Writes" |
| Gates | fuzz::run_crash_recovery_fuzz, fuzz::exhaustive_crash_check, fuzz::run_manifest_selection_harness |
QUICKSTART acceptance |
The crash-recovery contract (the acceptance gate)
Kill the process between any two syscalls in the commit protocol; reopen; the bundle must be valid in 100% of runs, and must recover to the previous generation when the crash precedes the durable flush. This is the most important single test in the entire prototype. — QUICKSTART, Agent D
Killing a real process between syscalls cannot be made deterministic, so the
fuzzer drives the commit against a FaultStore that distinguishes live
(page-cache) bytes from durable (survives-a-crash) bytes and can crash after
any chosen syscall — optionally tearing the in-flight superblock write, the
case the slot CRC must catch. After every simulated crash the bundle is reopened
from the durable image and must:
- open successfully (never corrupt);
- be at the previous generation or the new one, never anything else;
- if the commit returned
Ok, be at the new generation; and if the crash was clean (the in-flight flush persisted nothing) and the commit did not complete, be at the previous generation — the exact "recover to the previous generation when the crash precedes the durable flush" property. (A torn final flush may at a full prefix legitimately persist the whole superblock — the genuine post-commit case — so the torn branch admits either generation.) - report no integrity anomaly;
- have every canonical chunk present and hash-intact.
Two drivers exercise this: a randomized 10,000-iteration sweep, and an exhaustive per-commit sweep that tests every syscall boundary crossed with every tear point (clean, and torn at prefixes around the 252-byte CRC offset and the 256-byte slot size). The second leaves no step of the protocol untested.
The companion manifest_selection gate asserts the Chapter 8 superblock-
selection rule across every corruption scenario the QUICKSTART enumerates: slot A
corrupt + B valid (and vice versa), both valid at generation+1, both valid at the
same generation (equivalent, and divergent), a generation gap > 1, a
non-committed slot, a manifest-hash mismatch, and neither valid.
Building and testing
cargo test -p epiphany-bundle # unit + the two gates
cargo clippy -p epiphany-bundle --all-targets -- -D warnings
cargo run --release --example fuzz_crash -- 1000000 # extended crash soak
Hand-off criteria (QUICKSTART, Agent D)
cargo testclean.- Crash-recovery fuzzer passes 10,000 iterations
(
crash_recovery_fuzz_ten_thousand_iterations, two seeds; extended soak via the example binary; exhaustive per-syscall sweep inexhaustive_sweep_across_base_states_and_commit_shapes). - Manifest-selection harness handles every corruption scenario
(
every_selection_scenario_holds). - Real-filesystem
fsyncround-trip (file_store_real_fsync_round_trip).
Scope boundaries (per QUICKSTART "Don't do these")
v0 writes only uncompressed chunks (compression on the write path is deferred),
but reading zstd-compressed chunks and blobs is supported, per the spec's
§Compression MUST (the manifest is mandatory-uncompressed regardless, and a
compressed manifest is rejected). It carries the text-projection root but does
not implement the s-expression projection content, and it preserves extension
declarations and chunks but does not evaluate edit barriers — barrier operands
(OperationKindTag, ObjectKind, EditBarrier) are owned by Agents C and E.
Operation envelopes, snapshots, and causal frontiers are opaque bytes here.
See DECISIONS.md for the prototype byte-layout choices that anticipate the
deferred Binary Format companion, and the batched Pass 11 candidates.