Levi Neuwirth
7de9e479c4
A wire-decode fuzzer over Bundle::open, Manifest::decode, OperationIndex::decode, decode_block and envelope_offsets. The existing crash-recovery fuzzer corrupts an image the way a CRASH does -- torn writes at syscall boundaries. This one corrupts it the way an attacker or a bit-rotted disk does: arbitrary bytes, anywhere. It found a real defect. CompressionAlgorithm::None read its parameter byte and DISCARDED it, while encode writes zero. So [0, 0xFF] and [0, 0] both decoded to None, and the first re-encoded to the second: a lenient, non-injective codec, inherited by every structure embedding a ChunkRef. Whether that was visible depended entirely on the embedder: Manifest::decode has a whole-value re-encode guard, and it is TOTAL -- proved by exhaustive single-byte perturbation, every one rejected. It caught this. OperationIndex::decode has no guard; it validates per-site. It accepted both byte strings, while its own doc promised to "reject (never normalizing) any non-canonical form". That promise was false. That is the same two-layer lesson P2 recorded one commit ago, from the other side: a re-encode guard is complete only where the encoder normalizes, and its completeness can MASK a lenient sub-codec rather than fix it. Fixed at the source, not papered over at the index. An exhaustive sweep -- every byte, every value, plus an 8-byte extreme-integer window -- finds no remaining non-injective site. The fix contradicted ratified spec text, which said the byte was "present but zero, and ignored on read". Escalated rather than fixed unilaterally. The user ratified strict decode: core spec's clause is superseded, Binary Format gains req:binfmt:compression-none-parameter and moves 0.7.0 -> 0.8.0. No wire layout changed, and no conforming writer emits a non-zero byte, so this rejects only corrupt or adversarial input -- no existing file changes meaning. Coverage was the harness's problem again. The fuzzer's first run reached the operation index's accept path ZERO times -- random bytes never decode as an index -- so every assertion under it was vacuous. It found the bug only once the index corpus was built from real OperationIndex::build output. The smoke tests now assert on a WireFuzzCoverage so that cannot silently regress. 1.5M inputs across five seeds, ~1s each, clean after the fix. Three regressions, each mutation-verified by restoring the leniency: the codec itself, the index that exposed it, and the manifest guard's totality -- which is the asymmetry that hid it. Gate: fmt clean, clippy 0, 30 targets / 1012 passed / 0 failed, docs 0 under -D warnings, conformance 8/8, zero golden churn, both spec documents rebuild with no undefined references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| examples | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| DECISIONS.md | ||
| README.md | ||
README.md
epiphany-bundle
The Epiphany .musc file format, implementing the normative requirements of
Chapter 8 (File Format) of the core specification (spec/core_spec.pdf).
This is Agent D's crate per spec/QUICKSTART.md. It depends on
epiphany-determinism (Agent A) and on nothing else — not on epiphany-core
(Agent B) or epiphany-ops (Agent C):
bundles handle bytes, ops handles semantics. A canonical-base snapshot from the bundle's perspective is opaque bytes plus a frontier DVV; only
epiphany-opsinterprets it. — QUICKSTART
A bundle is a single file: a fixed 64-byte header at offset 0, two 256-byte superblock slots, then a body of immutable, content-addressed chunks. The superblocks are the only mutable on-disk objects. A commit appends new chunks, writes a new manifest chunk, then flips the active superblock by writing the inactive slot and durably flushing it — that flush is the commit point. Because commits only ever append and touch the inactive slot, a crash can never corrupt the active state.
What's here
| Area | Items | Spec |
|---|---|---|
| Prelude | FixedHeader (64 B, CRC-32C), Superblock/CommitState (256 B, CRC-32C), select_active |
Ch. 8 §"The Bundle Layout", §"Superblock Selection" |
| Atomic commit | Bundle::create/open/commit, the 7-step protocol, cold-open path |
Ch. 8 §"The Atomic Write Protocol", §"Streaming Reads" |
| Content addressing | chunk_content_hash/chunk_id, ChunkRef, ChunkKind, CompressionAlgorithm, domain separation |
Ch. 8 §"Content Hashing", §"Chunks" |
| Manifest | Manifest (canonical_base ≠ acceleration_snapshots), SnapshotRef, BlobRef, ProfileDeclaration, ExtensionDeclaration |
Ch. 8 §"The Manifest" |
| Retention | RetentionPolicy (first-class), ProfileConstraints |
Ch. 8 §"Garbage Collection and Retention" |
| Op blocks | pack_operation_blocks (1 MiB soft target), encode_block/decode_block |
Ch. 8 §"Operation Envelope Blocks" |
| Storage | BlockStore, MemStore, FileStore (real fsync), FaultStore (crash sim) |
Ch. 8 §"Durable Writes" |
| Gates | fuzz::run_crash_recovery_fuzz, fuzz::exhaustive_crash_check, fuzz::run_manifest_selection_harness |
QUICKSTART acceptance |
The crash-recovery contract (the acceptance gate)
Kill the process between any two syscalls in the commit protocol; reopen; the bundle must be valid in 100% of runs, and must recover to the previous generation when the crash precedes the durable flush. This is the most important single test in the entire prototype. — QUICKSTART, Agent D
Killing a real process between syscalls cannot be made deterministic, so the
fuzzer drives the commit against a FaultStore that distinguishes live
(page-cache) bytes from durable (survives-a-crash) bytes and can crash after
any chosen syscall — optionally tearing the in-flight superblock write, the
case the slot CRC must catch. After every simulated crash the bundle is reopened
from the durable image and must:
- open successfully (never corrupt);
- be at the previous generation or the new one, never anything else;
- if the commit returned
Ok, be at the new generation; and if the crash was clean (the in-flight flush persisted nothing) and the commit did not complete, be at the previous generation — the exact "recover to the previous generation when the crash precedes the durable flush" property. (A torn final flush may at a full prefix legitimately persist the whole superblock — the genuine post-commit case — so the torn branch admits either generation.) - report no integrity anomaly;
- have every canonical chunk present and hash-intact.
Two drivers exercise this: a randomized 10,000-iteration sweep, and an exhaustive per-commit sweep that tests every syscall boundary crossed with every tear point (clean, and torn at prefixes around the 252-byte CRC offset and the 256-byte slot size). The second leaves no step of the protocol untested.
The companion manifest_selection gate asserts the Chapter 8 superblock-
selection rule across every corruption scenario the QUICKSTART enumerates: slot A
corrupt + B valid (and vice versa), both valid at generation+1, both valid at the
same generation (equivalent, and divergent), a generation gap > 1, a
non-committed slot, a manifest-hash mismatch, and neither valid.
Building and testing
cargo test -p epiphany-bundle # unit + the two gates
cargo clippy -p epiphany-bundle --all-targets -- -D warnings
cargo run --release --example fuzz_crash -- 1000000 # extended crash soak
Hand-off criteria (QUICKSTART, Agent D)
cargo testclean.- Crash-recovery fuzzer passes 10,000 iterations
(
crash_recovery_fuzz_ten_thousand_iterations, two seeds; extended soak via the example binary; exhaustive per-syscall sweep inexhaustive_sweep_across_base_states_and_commit_shapes). - Manifest-selection harness handles every corruption scenario
(
every_selection_scenario_holds). - Real-filesystem
fsyncround-trip (file_store_real_fsync_round_trip).
Scope boundaries (per QUICKSTART "Don't do these")
v0 writes only uncompressed chunks (compression on the write path is deferred),
but reading zstd-compressed chunks and blobs is supported, per the spec's
§Compression MUST (the manifest is mandatory-uncompressed regardless, and a
compressed manifest is rejected). It carries the text-projection root but does
not implement the s-expression projection content, and it preserves extension
declarations and chunks but does not evaluate edit barriers — barrier operands
(OperationKindTag, ObjectKind, EditBarrier) are owned by Agents C and E.
Operation envelopes, snapshots, and causal frontiers are opaque bytes here.
See DECISIONS.md for the prototype byte-layout choices that anticipate the
deferred Binary Format companion, and the batched Pass 11 candidates.