Levi Neuwirth
2868f8c4b3
0.1.0 left kind, action, policy, constraints and barrier derived-but-unwritten and admitted it. All are now written. The grammar has no undefined nonterminal (machine-checked) and 31 operation-kind productions in exact discriminant order, cross-checked against envdecode.rs. The one real decision was how embedded Chapter-5 values appear. An operation payload carries an Event, a Pitch, a Region, a TimeSignature, and there were three ways to write them: Forty hand-written productions would restate the entire Chapter-5 data model in a second normative document -- two normative listings of one struct, which is the exact drift P13-I1 was opened to close. Opaque canonical-value byte strings would be lossless and zero-drift, but a pitch would be unreadable without binary tooling, failing the core spec's own "format inspection and debugging" use case. One mechanical rule (req:textproj:value-projection, ratified): a struct is (<type-name> <field>...) with fields positional in the ratified declaration order; a newtype is transparent, exactly as in the binary form; a tagged union is (<variant> <field>...); an option is () or (some v); a sequence keeps the binary form's order. A rule cannot drift from what it reads. Two leaf decisions follow from canonicality rather than taste. A rational is (ratio n d), lowest terms, sign on the numerator. A CanonicalF64 is the byte string of its eight canonical IEEE-754 bytes and never a decimal: decimal float text is not canonically unique -- shortest-round-trip and 17-significant-digit forms both round-trip, and -0.0 has two spellings -- so a decimal tempo would break req:textproj:canonical-text at the first tempo mark. Operation-kind names follow the Operation Catalog's sections (create-region, create-staff), not OperationKindTag's (InsertRegion, InsertStaff), which renamed three pairs for reasons internal to the tag space. The projection follows the semantics. One deferral, stated as such rather than left to inference: affected_object_kinds and edit_barriers have ratified structured shapes AND canonical byte encodings, and the bundle stores them opaquely. At 0.3.0 the projection does the same, on the principle that it interprets nothing the bundle does not. A later revision may project them structurally; their canonical bytes are unchanged by that, so it changes the text and not the document. Also corrected: the extension line now writes its fields in the ratified declaration order, which had preserved_chunk_roots before affected_object_kinds and edit_barriers. Gate: clippy 0, 31 targets / 1031 passed / 0 failed, conformance 8/8, zero golden churn; core_spec, binary_format and text_projection all build with no undefined references. Grammar closure machine-checked: no undefined nonterminal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| examples | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| DECISIONS.md | ||
| README.md | ||
README.md
epiphany-bundle
The Epiphany .musc file format, implementing the normative requirements of
Chapter 8 (File Format) of the core specification (spec/core_spec.pdf).
This is Agent D's crate per spec/QUICKSTART.md. It depends on
epiphany-determinism (Agent A) and on nothing else — not on epiphany-core
(Agent B) or epiphany-ops (Agent C):
bundles handle bytes, ops handles semantics. A canonical-base snapshot from the bundle's perspective is opaque bytes plus a frontier DVV; only
epiphany-opsinterprets it. — QUICKSTART
A bundle is a single file: a fixed 64-byte header at offset 0, two 256-byte superblock slots, then a body of immutable, content-addressed chunks. The superblocks are the only mutable on-disk objects. A commit appends new chunks, writes a new manifest chunk, then flips the active superblock by writing the inactive slot and durably flushing it — that flush is the commit point. Because commits only ever append and touch the inactive slot, a crash can never corrupt the active state.
What's here
| Area | Items | Spec |
|---|---|---|
| Prelude | FixedHeader (64 B, CRC-32C), Superblock/CommitState (256 B, CRC-32C), select_active |
Ch. 8 §"The Bundle Layout", §"Superblock Selection" |
| Atomic commit | Bundle::create/open/commit, the 7-step protocol, cold-open path |
Ch. 8 §"The Atomic Write Protocol", §"Streaming Reads" |
| Content addressing | chunk_content_hash/chunk_id, ChunkRef, ChunkKind, CompressionAlgorithm, domain separation |
Ch. 8 §"Content Hashing", §"Chunks" |
| Manifest | Manifest (canonical_base ≠ acceleration_snapshots), SnapshotRef, BlobRef, ProfileDeclaration, ExtensionDeclaration |
Ch. 8 §"The Manifest" |
| Retention | RetentionPolicy (first-class), ProfileConstraints |
Ch. 8 §"Garbage Collection and Retention" |
| Op blocks | pack_operation_blocks (1 MiB soft target), encode_block/decode_block |
Ch. 8 §"Operation Envelope Blocks" |
| Storage | BlockStore, MemStore, FileStore (real fsync), FaultStore (crash sim) |
Ch. 8 §"Durable Writes" |
| Gates | fuzz::run_crash_recovery_fuzz, fuzz::exhaustive_crash_check, fuzz::run_manifest_selection_harness |
QUICKSTART acceptance |
The crash-recovery contract (the acceptance gate)
Kill the process between any two syscalls in the commit protocol; reopen; the bundle must be valid in 100% of runs, and must recover to the previous generation when the crash precedes the durable flush. This is the most important single test in the entire prototype. — QUICKSTART, Agent D
Killing a real process between syscalls cannot be made deterministic, so the
fuzzer drives the commit against a FaultStore that distinguishes live
(page-cache) bytes from durable (survives-a-crash) bytes and can crash after
any chosen syscall — optionally tearing the in-flight superblock write, the
case the slot CRC must catch. After every simulated crash the bundle is reopened
from the durable image and must:
- open successfully (never corrupt);
- be at the previous generation or the new one, never anything else;
- if the commit returned
Ok, be at the new generation; and if the crash was clean (the in-flight flush persisted nothing) and the commit did not complete, be at the previous generation — the exact "recover to the previous generation when the crash precedes the durable flush" property. (A torn final flush may at a full prefix legitimately persist the whole superblock — the genuine post-commit case — so the torn branch admits either generation.) - report no integrity anomaly;
- have every canonical chunk present and hash-intact.
Two drivers exercise this: a randomized 10,000-iteration sweep, and an exhaustive per-commit sweep that tests every syscall boundary crossed with every tear point (clean, and torn at prefixes around the 252-byte CRC offset and the 256-byte slot size). The second leaves no step of the protocol untested.
The companion manifest_selection gate asserts the Chapter 8 superblock-
selection rule across every corruption scenario the QUICKSTART enumerates: slot A
corrupt + B valid (and vice versa), both valid at generation+1, both valid at the
same generation (equivalent, and divergent), a generation gap > 1, a
non-committed slot, a manifest-hash mismatch, and neither valid.
Building and testing
cargo test -p epiphany-bundle # unit + the two gates
cargo clippy -p epiphany-bundle --all-targets -- -D warnings
cargo run --release --example fuzz_crash -- 1000000 # extended crash soak
Hand-off criteria (QUICKSTART, Agent D)
cargo testclean.- Crash-recovery fuzzer passes 10,000 iterations
(
crash_recovery_fuzz_ten_thousand_iterations, two seeds; extended soak via the example binary; exhaustive per-syscall sweep inexhaustive_sweep_across_base_states_and_commit_shapes). - Manifest-selection harness handles every corruption scenario
(
every_selection_scenario_holds). - Real-filesystem
fsyncround-trip (file_store_real_fsync_round_trip).
Scope boundaries (per QUICKSTART "Don't do these")
v0 writes only uncompressed chunks (compression on the write path is deferred),
but reading zstd-compressed chunks and blobs is supported, per the spec's
§Compression MUST (the manifest is mandatory-uncompressed regardless, and a
compressed manifest is rejected). It carries the text-projection root but does
not implement the s-expression projection content, and it preserves extension
declarations and chunks but does not evaluate edit barriers — barrier operands
(OperationKindTag, ObjectKind, EditBarrier) are owned by Agents C and E.
Operation envelopes, snapshots, and causal frontiers are opaque bytes here.
See DECISIONS.md for the prototype byte-layout choices that anticipate the
deferred Binary Format companion, and the batched Pass 11 candidates.