Levi Neuwirth
a41596d329
spec/vectors/decode_vectors.txt -- 37 committed byte strings across five surfaces, each with its normative accept/reject verdict. The reference implementation's fuzzers prove its own decoders self-consistent, which says nothing about whether a foreign decoder agrees with the format. This is what one is checked against. Gated in the conformance suite as [7d], and drift-locked: the committed file must equal vectors::render(), so a wire-format change lands in the diff. It found a real defect on its first run. OperationKindTag::TransposeInterval encoded to [30] and its own decoder REJECTED it -- Push 4a added the variant to discriminant() and never to decode_canonical. OperationKindTag is what edit barriers persist, so a barrier prohibiting TransposeInterval could be written and never read back. Silent data loss on reopen. Four things should have caught it. None did, and two made it worse: The round-trip test enumerated DISCRIMINANTS -- (0u8..30).map(decode_canonical) -- starting from bytes the decoder already knew, so it structurally could not notice a variant the decoder was missing. It now enumerates VARIANTS from one all_tags() list, with a completeness check in both directions. The distinctness test's hand-written variant list omitted it too. Same list now. operation_kind_tag_decode_rejects_malformed_bytes asserted that tag 30 is REJECTED, and layout-ir's decode_rejects_unknown_discriminants asserted the same at the barrier surface. Both were locking the bug in place and made it look deliberate. Both now name 31, and a new barrier test round-trips a barrier prohibiting every tag -- the persistence surface where this actually bites. The P2 decode fuzzer fed valid corpus bytes to the tag decoder and tallied the failure as a REJECTION, like any garbage input. It never asserted that an unmutated corpus entry decodes. Both fuzzers now do, as a pre-pass. The harness had the same disease as the code. `check` collapsed "rejected" with "accepted but does not re-encode", so a decoder that silently normalizes non-canonical bytes PASSED the reject vectors it was written to catch. Verified: removing the whole-state guard, and restoring the lenient compression codec, both left the corpus green. `check` now returns Ok(injective) for accept and Err for reject and never conflates them -- silently normalizing non-canonical bytes IS accepting them. With that fixed, all four defect mutations fail the corpus, each naming its class. The corpus pins one vector per class this repo has shipped a bug in: non-canonical-map-order (a guard catches it; no per-site check exists), non-canonical-vec-order (only a per-site check catches it; a guard is blind), lenient-sub-codec (a guard masked it in the manifest; the index had none), plus trailing-bytes, truncated, unknown-discriminant, count-exceeds-remaining. A test fails if one goes missing. Gate: fmt clean, clippy 0, 30 targets / 1024 passed / 0 failed, docs 0 under -D warnings, conformance 8/8 (now including [7d]), zero golden churn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| examples | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| DECISIONS.md | ||
| README.md | ||
README.md
epiphany-bundle
The Epiphany .musc file format, implementing the normative requirements of
Chapter 8 (File Format) of the core specification (spec/core_spec.pdf).
This is Agent D's crate per spec/QUICKSTART.md. It depends on
epiphany-determinism (Agent A) and on nothing else — not on epiphany-core
(Agent B) or epiphany-ops (Agent C):
bundles handle bytes, ops handles semantics. A canonical-base snapshot from the bundle's perspective is opaque bytes plus a frontier DVV; only
epiphany-opsinterprets it. — QUICKSTART
A bundle is a single file: a fixed 64-byte header at offset 0, two 256-byte superblock slots, then a body of immutable, content-addressed chunks. The superblocks are the only mutable on-disk objects. A commit appends new chunks, writes a new manifest chunk, then flips the active superblock by writing the inactive slot and durably flushing it — that flush is the commit point. Because commits only ever append and touch the inactive slot, a crash can never corrupt the active state.
What's here
| Area | Items | Spec |
|---|---|---|
| Prelude | FixedHeader (64 B, CRC-32C), Superblock/CommitState (256 B, CRC-32C), select_active |
Ch. 8 §"The Bundle Layout", §"Superblock Selection" |
| Atomic commit | Bundle::create/open/commit, the 7-step protocol, cold-open path |
Ch. 8 §"The Atomic Write Protocol", §"Streaming Reads" |
| Content addressing | chunk_content_hash/chunk_id, ChunkRef, ChunkKind, CompressionAlgorithm, domain separation |
Ch. 8 §"Content Hashing", §"Chunks" |
| Manifest | Manifest (canonical_base ≠ acceleration_snapshots), SnapshotRef, BlobRef, ProfileDeclaration, ExtensionDeclaration |
Ch. 8 §"The Manifest" |
| Retention | RetentionPolicy (first-class), ProfileConstraints |
Ch. 8 §"Garbage Collection and Retention" |
| Op blocks | pack_operation_blocks (1 MiB soft target), encode_block/decode_block |
Ch. 8 §"Operation Envelope Blocks" |
| Storage | BlockStore, MemStore, FileStore (real fsync), FaultStore (crash sim) |
Ch. 8 §"Durable Writes" |
| Gates | fuzz::run_crash_recovery_fuzz, fuzz::exhaustive_crash_check, fuzz::run_manifest_selection_harness |
QUICKSTART acceptance |
The crash-recovery contract (the acceptance gate)
Kill the process between any two syscalls in the commit protocol; reopen; the bundle must be valid in 100% of runs, and must recover to the previous generation when the crash precedes the durable flush. This is the most important single test in the entire prototype. — QUICKSTART, Agent D
Killing a real process between syscalls cannot be made deterministic, so the
fuzzer drives the commit against a FaultStore that distinguishes live
(page-cache) bytes from durable (survives-a-crash) bytes and can crash after
any chosen syscall — optionally tearing the in-flight superblock write, the
case the slot CRC must catch. After every simulated crash the bundle is reopened
from the durable image and must:
- open successfully (never corrupt);
- be at the previous generation or the new one, never anything else;
- if the commit returned
Ok, be at the new generation; and if the crash was clean (the in-flight flush persisted nothing) and the commit did not complete, be at the previous generation — the exact "recover to the previous generation when the crash precedes the durable flush" property. (A torn final flush may at a full prefix legitimately persist the whole superblock — the genuine post-commit case — so the torn branch admits either generation.) - report no integrity anomaly;
- have every canonical chunk present and hash-intact.
Two drivers exercise this: a randomized 10,000-iteration sweep, and an exhaustive per-commit sweep that tests every syscall boundary crossed with every tear point (clean, and torn at prefixes around the 252-byte CRC offset and the 256-byte slot size). The second leaves no step of the protocol untested.
The companion manifest_selection gate asserts the Chapter 8 superblock-
selection rule across every corruption scenario the QUICKSTART enumerates: slot A
corrupt + B valid (and vice versa), both valid at generation+1, both valid at the
same generation (equivalent, and divergent), a generation gap > 1, a
non-committed slot, a manifest-hash mismatch, and neither valid.
Building and testing
cargo test -p epiphany-bundle # unit + the two gates
cargo clippy -p epiphany-bundle --all-targets -- -D warnings
cargo run --release --example fuzz_crash -- 1000000 # extended crash soak
Hand-off criteria (QUICKSTART, Agent D)
cargo testclean.- Crash-recovery fuzzer passes 10,000 iterations
(
crash_recovery_fuzz_ten_thousand_iterations, two seeds; extended soak via the example binary; exhaustive per-syscall sweep inexhaustive_sweep_across_base_states_and_commit_shapes). - Manifest-selection harness handles every corruption scenario
(
every_selection_scenario_holds). - Real-filesystem
fsyncround-trip (file_store_real_fsync_round_trip).
Scope boundaries (per QUICKSTART "Don't do these")
v0 writes only uncompressed chunks (compression on the write path is deferred),
but reading zstd-compressed chunks and blobs is supported, per the spec's
§Compression MUST (the manifest is mandatory-uncompressed regardless, and a
compressed manifest is rejected). It carries the text-projection root but does
not implement the s-expression projection content, and it preserves extension
declarations and chunks but does not evaluate edit barriers — barrier operands
(OperationKindTag, ObjectKind, EditBarrier) are owned by Agents C and E.
Operation envelopes, snapshots, and causal frontiers are opaque bytes here.
See DECISIONS.md for the prototype byte-layout choices that anticipate the
deferred Binary Format companion, and the batched Pass 11 candidates.