diff --git a/crates/epiphany-bundle/DECISIONS.md b/crates/epiphany-bundle/DECISIONS.md index c9e88c7..353638f 100644 --- a/crates/epiphany-bundle/DECISIONS.md +++ b/crates/epiphany-bundle/DECISIONS.md @@ -428,3 +428,54 @@ implies it, the canonical-base-stays-major-0 rule is now enforced per ROLE (`mis_stamped_canonical_base`, consulted at open and commit → read-only + `UnsupportedCanonicalChunkMajor`, regression-locked). The `SnapshotId` in the harness remains a hash-truncation stand-in (companion open question). + +## Push 5 / P3 — the bundle wire, and a lenient sub-codec (2026-07-09) + +A wire-decode fuzzer (`fuzz::run_wire_decode_fuzz`) over `Bundle::open`, +`Manifest::decode`, `OperationIndex::decode`, `decode_block`, and +`envelope_offsets`. The existing crash-recovery fuzzer corrupts the image the way +a *crash* does — torn writes at syscall boundaries. This one corrupts it the way +an attacker or a bit-rotted disk does: arbitrary bytes, anywhere. + +**One real defect: `CompressionAlgorithm::None` ignored its parameter byte.** +`decode` read it and discarded it; `encode` writes `0`. So `[0, 0xFF]` and +`[0, 0]` both decoded to `None`, and the first re-encoded to the second — a +lenient, **non-injective** codec, inherited by every structure embedding a +`ChunkRef`. + +Its visibility depended entirely on whether the embedder had a whole-value +re-encode guard: + +- `Manifest::decode` **has** one, and it is *total* — verified by exhaustive + single-byte perturbation, every one rejected. It caught this. (`encode_body` + sorts and deduplicates every vector, and `manifest_id` is derived from the + body, which is why the guard is complete here where `MaterializedState`'s is + not — see `epiphany-ops/DECISIONS.md` §"Push 5 / P2".) +- `OperationIndex::decode` has **no** guard; it validates per-site instead. It + accepted both byte strings while its own doc promised to *"reject (never + normalizing) any non-canonical form"*. That promise was false. + +Fixed at the source rather than papered over at the index: a non-zero `None` +parameter is now rejected. Exhaustive sweep (every byte × every value, plus an +8-byte extreme-integer window) finds no remaining non-injective site. + +**This contradicted ratified spec text**, which said the byte was *"present but +zero, and ignored on read"*. Escalated rather than fixed unilaterally; the user +ratified strict decode and the spec amendment. Core spec's clause is superseded; +Binary Format gains `req:binfmt:compression-none-parameter` and moves 0.7.0 → +0.8.0. No wire layout changed and no conforming writer emits a non-zero byte, so +this rejects only corrupt or adversarial input and no existing file changes +meaning. + +**Coverage is the harness's job, again.** The fuzzer's first run reached the +operation index's accept path **zero times** — random bytes never decode as an +index — so every assertion under it was vacuous. It found the bug only after the +index corpus was built from real `OperationIndex::build` output. The smoke tests +now assert on a `WireFuzzCoverage` so that can never silently regress. 1.5M +inputs across five seeds, ~1s each, clean after the fix. + +Three regressions, each mutation-verified by restoring the leniency: +`compression_none_rejects_a_non_zero_parameter_byte` (the codec), +`a_lenient_compression_byte_is_rejected_rather_than_normalized` (the index, the +surface that exposed it), and `every_single_byte_perturbation_of_a_manifest_is_rejected` +(the guard's totality, and the asymmetry that hid the bug). diff --git a/crates/epiphany-bundle/src/chunk.rs b/crates/epiphany-bundle/src/chunk.rs index 1c78f48..c9729ff 100644 --- a/crates/epiphany-bundle/src/chunk.rs +++ b/crates/epiphany-bundle/src/chunk.rs @@ -136,12 +136,27 @@ impl CompressionAlgorithm { }; } + /// Decodes strictly: `None` carries no parameter, so its parameter byte + /// **must** be zero. + /// + /// Accepting a non-zero one made this codec lenient, and therefore + /// non-injective: `[0, 0xFF]` and `[0, 0]` both decoded to `None`, which + /// re-encodes to `[0, 0]`. Any structure embedding a [`ChunkRef`] without a + /// whole-value re-encode guard inherited that — `OperationIndex::decode` + /// did, while promising in its own doc to reject non-canonical bytes rather + /// than normalize them. A conforming writer never emits a non-zero + /// parameter here, so this rejects only corrupt or adversarial input. #[inline] pub(crate) fn decode(r: &mut Reader) -> Result { let tag = r.get_u8()?; let param = r.get_u8()?; Ok(match tag { - 0 => CompressionAlgorithm::None, + 0 if param == 0 => CompressionAlgorithm::None, + 0 => { + return Err(DecodeError::Malformed( + "CompressionAlgorithm::None carries a non-zero parameter byte", + )) + } 1 => CompressionAlgorithm::Zstd { level: param }, 2 => CompressionAlgorithm::Reserved(param), other => { @@ -296,6 +311,45 @@ impl Ord for ChunkRef { mod tests { use super::*; + /// `CompressionAlgorithm::None` carries no parameter, so its parameter byte + /// is normative zero. Accepting a non-zero one made the codec lenient: two + /// distinct byte strings decoded to one value, and re-encoding produced a + /// third. Found by the P3 wire fuzzer via `OperationIndex`, which embeds a + /// `ChunkRef` and has no whole-value re-encode guard to hide it. + #[test] + fn compression_none_rejects_a_non_zero_parameter_byte() { + let mut w = Writer::new(); + CompressionAlgorithm::None.encode(&mut w); + let canonical = w.into_bytes(); + assert_eq!(canonical, vec![0, 0]); + assert_eq!( + CompressionAlgorithm::decode(&mut Reader::new(&canonical)).unwrap(), + CompressionAlgorithm::None + ); + + for param in [1u8, 0x7F, 0xFF] { + let lenient = vec![0, param]; + assert!( + CompressionAlgorithm::decode(&mut Reader::new(&lenient)).is_err(), + "None with parameter {param:#04x} must be rejected, never normalized to zero" + ); + } + + // The parameter is meaningful for the other two, so it round-trips. + for algo in [ + CompressionAlgorithm::Zstd { level: 0xFF }, + CompressionAlgorithm::Reserved(0xFF), + ] { + let mut w = Writer::new(); + algo.encode(&mut w); + let bytes = w.into_bytes(); + assert_eq!( + CompressionAlgorithm::decode(&mut Reader::new(&bytes)).unwrap(), + algo + ); + } + } + #[test] fn chunk_kind_discriminants_round_trip() { for kind in [ diff --git a/crates/epiphany-bundle/src/fuzz.rs b/crates/epiphany-bundle/src/fuzz.rs index d8198b2..b65598a 100644 --- a/crates/epiphany-bundle/src/fuzz.rs +++ b/crates/epiphany-bundle/src/fuzz.rs @@ -36,13 +36,16 @@ //! across every corruption scenario the QUICKSTART enumerates. use crate::bundle::{Bundle, CommitContext, StagedChunk, BODY_START}; +use crate::chunk::{ChunkKind, ChunkRef, CompressionAlgorithm}; use crate::error::IntegrityAnomaly; use crate::header::FixedHeader; use crate::ids::{DocumentId, FileUuid, ReductionAlgorithmVersion, SchemaVersion, WallClockTime}; use crate::manifest::Manifest; +use crate::opindex::OperationIndex; use crate::store::{CrashPoint, FaultStore, MemStore, Tear}; use crate::superblock::{CommitState, ProfileId, Slot, Superblock, SUPERBLOCK_LEN}; use crate::{block, manifest_chunk_hash}; +use epiphany_determinism::{ChunkId, ContentHash}; /// A tiny deterministic generator (SplitMix64), matching `epiphany-determinism`'s /// fuzz harness: reproducible across platforms, no `rand` dependency, so a @@ -175,6 +178,275 @@ fn build_base(rng: &mut SplitMix64, commits: u64) -> (Vec, u64) { (bundle.into_store().into_bytes(), gen) } +// --------------------------------------------------------------------------- +// Wire-decode fuzzing (P3 of the decode-hardening track). +// +// The crash-recovery fuzzer above corrupts the image the way a *crash* does: +// torn writes at syscall boundaries. This one corrupts it the way an *attacker* +// or a bit-rotted disk does — arbitrary bytes, anywhere — and drives every +// decode surface the bundle exposes: +// +// Bundle::open · Manifest::decode · OperationIndex::decode +// block::decode_block · block::envelope_offsets +// +// Two properties. A mutated image must never panic a decoder (a bundle is +// attacker-controlled input the moment it is emailed), and an *accepted* byte +// string must re-encode to itself where the type has a canonical encoding. +// +// P2 (`epiphany-ops`) established that a re-encode guard is complete only for +// fields the encoder NORMALIZES. `Manifest::encode_body` sorts and deduplicates +// every vector, so its guard genuinely is complete; `OperationIndex::decode` +// has no guard and instead checks its two `Vec`s for strict ascent per-site, +// which is the other correct answer. Both are asserted here. +// --------------------------------------------------------------------------- + +/// What a decode-fuzz run actually reached. A harness that never gets a decoder +/// to say `Ok` proves only the absence of a panic. +#[derive(Default, Debug, PartialEq, Eq)] +pub struct WireFuzzCoverage { + pub opens_ok: u64, + pub opens_rejected: u64, + pub manifests_ok: u64, + pub manifests_rejected: u64, + pub blocks_ok: u64, + pub blocks_rejected: u64, + pub indices_ok: u64, + pub indices_rejected: u64, +} + +fn wire_random_bytes(rng: &mut SplitMix64, n: usize) -> Vec { + (0..n).map(|_| rng.next_u64() as u8).collect() +} + +fn wire_substitute(rng: &mut SplitMix64, bytes: &mut [u8], k: usize) { + if bytes.is_empty() { + return; + } + for _ in 0..k { + let i = (rng.next_u64() as usize) % bytes.len(); + bytes[i] = rng.next_u64() as u8; + } +} + +/// Overwrites a random 4- or 8-byte window with an extreme integer: the +/// count/length/offset attack. A bundle's manifest carries file offsets and +/// lengths, so this is the mutation that matters most here. +fn wire_corrupt_int(rng: &mut SplitMix64, bytes: &mut [u8]) { + if bytes.len() < 8 { + return; + } + let i = (rng.next_u64() as usize) % (bytes.len() - 7); + match rng.next_u64() % 4 { + 0 => bytes[i..i + 4].copy_from_slice(&u32::MAX.to_le_bytes()), + 1 => bytes[i..i + 8].copy_from_slice(&u64::MAX.to_le_bytes()), + 2 => { + bytes[i..i + 8].copy_from_slice(&(rng.next_u64() | 0x8000_0000_0000_0000).to_le_bytes()) + } + _ => bytes[i..i + 4].copy_from_slice(&(rng.next_u64() as u32).to_le_bytes()), + } +} + +fn mutate_image(rng: &mut SplitMix64, base: &[u8]) -> Vec { + let mut b = base.to_vec(); + match rng.next_u64() % 6 { + 0 => return b, // unmutated: a live check that the corpus opens + 1 => { + let k = 1 + (rng.next_u64() % 6) as usize; + wire_substitute(rng, &mut b, k); + } + 2 => { + let t = (rng.next_u64() as usize) % (b.len() + 1); + b.truncate(t); + } + 3 => { + let n = 1 + (rng.next_u64() % 32) as usize; + let tail = wire_random_bytes(rng, n); + b.extend_from_slice(&tail); + } + 4 => wire_corrupt_int(rng, &mut b), + _ => { + // Corrupt the superblock region specifically: the two 256-byte slots + // whose CRCs gate `open`. + let n = b.len().min(512); + if n > 0 { + let i = (rng.next_u64() as usize) % n; + b[i] = rng.next_u64() as u8; + } + } + } + b +} + +/// Runs `iters` adversarial wire-decode iterations from `seed` over the bundle's +/// decode surfaces. A panic, or an accepted byte string that does not re-encode +/// to itself, fails the run; `seed` reproduces it exactly. +pub fn run_wire_decode_fuzz(iters: u64, seed: u64) -> WireFuzzCoverage { + let mut rng = SplitMix64::new(seed); + let mut cov = WireFuzzCoverage::default(); + + // A small pool of valid, populated images and valid manifest payloads. + let mut images: Vec> = Vec::new(); + let mut manifests: Vec> = Vec::new(); + for commits in 0..4u64 { + let (image, _) = build_base(&mut rng, commits); + let bundle = Bundle::open(MemStore::from_bytes(image.clone())).expect("valid image opens"); + manifests.push(bundle.manifest().encode()); + images.push(image); + } + let valid_blocks: Vec> = { + let payloads: Vec> = (0..3).map(|i| vec![i as u8 + 1; 8 + i * 5]).collect(); + block::pack_operation_blocks(&payloads) + }; + + // Valid operation-index payloads. Random bytes never decode, so an index + // corpus of noise leaves the decoder's accept path — and therefore every + // assertion below it — unreached (measured: `indices_ok` was 0). + let valid_indices: Vec> = { + let block_ref = |hash_byte: u8, offset: u64, len: u64| ChunkRef { + id: ChunkId(ContentHash([hash_byte; 32])), + kind: ChunkKind::OperationEnvelopeBlock, + schema_version: SchemaVersion::V0, + offset, + compressed_length: len, + uncompressed_length: len, + compression: CompressionAlgorithm::None, + hash: ContentHash([hash_byte; 32]), + }; + [ + OperationIndex::build(&[]).expect("empty index"), + OperationIndex::build(&[(block_ref(0x11, 576, 64), vec![([2; 16], 8)])]) + .expect("one block"), + OperationIndex::build(&[ + (block_ref(0x22, 1000, 64), vec![([3; 16], 8), ([1; 16], 40)]), + (block_ref(0x11, 576, 32), vec![([2; 16], 8)]), + ]) + .expect("two blocks"), + ] + .iter() + .map(|i| i.encode()) + .collect() + }; + + for _ in 0..iters { + // 1. Whole-image open. Must never panic; an Ok manifest must re-encode. + let pick = (rng.next_u64() as usize) % images.len(); + let image = mutate_image(&mut rng, &images[pick]); + match Bundle::open(MemStore::from_bytes(image)) { + Ok(bundle) => { + cov.opens_ok += 1; + let encoded = bundle.manifest().encode(); + assert_eq!( + Manifest::decode(&encoded).as_ref(), + Ok(bundle.manifest()), + "an opened bundle's manifest does not round-trip" + ); + // Reading every chunk the manifest names must be total. + for r in bundle.manifest().canonical_chunk_refs() { + let _ = bundle.read_chunk(&r); + } + } + Err(_) => cov.opens_rejected += 1, + } + + // 2. Manifest payload decode: strict-canonical, guard-backed. + let mut m = manifests[(rng.next_u64() as usize) % manifests.len()].clone(); + match rng.next_u64() % 4 { + 0 => {} + 1 => wire_substitute(&mut rng, &mut m, 1), + 2 => wire_corrupt_int(&mut rng, &mut m), + _ => { + let t = (rng.next_u64() as usize) % (m.len() + 1); + m.truncate(t); + } + } + match Manifest::decode(&m) { + Ok(manifest) => { + cov.manifests_ok += 1; + assert_eq!( + manifest.encode(), + m, + "the manifest decoder accepted a non-canonical byte string" + ); + } + Err(_) => cov.manifests_rejected += 1, + } + + // 3. Block payload framing. + let mut b = valid_blocks[(rng.next_u64() as usize) % valid_blocks.len()].clone(); + match rng.next_u64() % 4 { + 0 => {} + 1 => wire_substitute(&mut rng, &mut b, 1), + 2 => wire_corrupt_int(&mut rng, &mut b), + _ => { + let t = (rng.next_u64() as usize) % (b.len() + 1); + b.truncate(t); + } + } + match block::decode_block(&b) { + Ok(envelopes) => { + cov.blocks_ok += 1; + // `envelope_offsets` shares the code path: the two must agree, + // and each recorded offset must actually address its envelope. + let offsets = block::envelope_offsets(&b).expect("same validation"); + assert_eq!(offsets.len(), envelopes.len()); + for ((offset, slice), env) in offsets.iter().zip(envelopes.iter()) { + assert_eq!( + *slice, + &env[..], + "envelope_offsets disagrees with decode_block" + ); + let at = *offset as usize; + assert_eq!( + &b[at..at + env.len()], + &env[..], + "offset does not address the envelope" + ); + } + } + Err(_) => cov.blocks_rejected += 1, + } + + // 4. Operation index: no re-encode guard; per-site strict-ascent checks. + let mut idx = valid_indices[(rng.next_u64() as usize) % valid_indices.len()].clone(); + match rng.next_u64() % 5 { + 0 => {} + 1 => wire_substitute(&mut rng, &mut idx, 1), + 2 => wire_corrupt_int(&mut rng, &mut idx), + 3 => { + let t = (rng.next_u64() as usize) % (idx.len() + 1); + idx.truncate(t); + } + _ => { + let n = (rng.next_u64() % 96) as usize; + idx = wire_random_bytes(&mut rng, n); + } + } + match OperationIndex::decode(&idx) { + Ok(index) => { + cov.indices_ok += 1; + // No re-encode guard here; the decoder's per-site checks are the + // contract, so assert them directly — and assert injectivity, + // which the `Vec`-order preservation makes non-trivial. + assert_eq!( + index.encode(), + idx, + "the operation-index decoder accepted a non-canonical byte string" + ); + assert!( + index.blocks().windows(2).all(|w| w[0] < w[1]), + "the index decoder accepted unsorted blocks" + ); + assert!( + index.entries().windows(2).all(|w| w[0].id < w[1].id), + "the index decoder accepted unsorted entries" + ); + } + Err(_) => cov.indices_rejected += 1, + } + } + cov +} + /// The crash-recovery fuzzer: `iters` randomized scenarios from `seed`. Each /// iteration builds a base bundle at a random generation, then commits a random /// set of operation blocks while crashing at a random syscall with a random @@ -538,4 +810,34 @@ mod tests { fn manifest_selection_harness_passes() { run_manifest_selection_harness(); } + + /// Two deterministic smoke seeds over every bundle decode surface. + /// + /// The coverage assertions are load-bearing. A wire fuzzer that never gets a + /// decoder to say `Ok` proves only the absence of a panic — and this one + /// initially reached the operation index's accept path *zero* times, because + /// random bytes never decode as an index. It found the lenient + /// `CompressionAlgorithm::None` parameter byte only once its index corpus + /// was real. + #[test] + fn wire_decode_fuzz_smoke_seed_a() { + let cov = run_wire_decode_fuzz(20_000, 0x0DEC_0DE0_F022_1234); + assert!(cov.opens_ok > 1_000, "{cov:?}"); + assert!(cov.opens_rejected > 1_000, "{cov:?}"); + assert!(cov.manifests_ok > 1_000, "{cov:?}"); + assert!(cov.manifests_rejected > 1_000, "{cov:?}"); + assert!(cov.blocks_ok > 1_000, "{cov:?}"); + assert!(cov.blocks_rejected > 1_000, "{cov:?}"); + assert!(cov.indices_ok > 1_000, "{cov:?}"); + assert!(cov.indices_rejected > 1_000, "{cov:?}"); + } + + #[test] + fn wire_decode_fuzz_smoke_seed_b() { + let cov = run_wire_decode_fuzz(20_000, 0xF0FA_11BA_C0DE_5EED); + assert!(cov.opens_ok > 1_000, "{cov:?}"); + assert!(cov.manifests_ok > 1_000, "{cov:?}"); + assert!(cov.blocks_ok > 1_000, "{cov:?}"); + assert!(cov.indices_ok > 1_000, "{cov:?}"); + } } diff --git a/crates/epiphany-bundle/src/manifest.rs b/crates/epiphany-bundle/src/manifest.rs index a42d015..ba4cff0 100644 --- a/crates/epiphany-bundle/src/manifest.rs +++ b/crates/epiphany-bundle/src/manifest.rs @@ -709,6 +709,57 @@ mod tests { use crate::chunk::{chunk_id, ChunkKind}; use crate::ids::SnapshotId; + /// The manifest's whole-value re-encode guard is **total**: every single-byte + /// perturbation is rejected. `manifest_id` is derived from the body, so a + /// body edit fails the id check and an id edit fails the derivation; and + /// `encode_body` sorts and deduplicates every vector, so an out-of-order + /// encoding cannot round-trip either. + /// + /// This is what makes the guard complete *here* and not in + /// `MaterializedState` (epiphany-ops), whose encoder writes its `Vec` fields + /// verbatim, nor in `OperationIndex`, which has no guard at all. The lenient + /// `CompressionAlgorithm::None` parameter byte was invisible through the + /// manifest for exactly this reason, and visible through the index. + #[test] + fn every_single_byte_perturbation_of_a_manifest_is_rejected() { + use crate::chunk::{ChunkRef, CompressionAlgorithm}; + use crate::ids::SchemaVersion; + + let mut m = Manifest::empty(DocumentId([5; 16])); + m.operation_roots.push(ChunkRef { + id: ChunkId(ContentHash([0x11; 32])), + kind: ChunkKind::OperationEnvelopeBlock, + schema_version: SchemaVersion::V0, + offset: 576, + compressed_length: 64, + uncompressed_length: 64, + compression: CompressionAlgorithm::None, + hash: ContentHash([0x11; 32]), + }); + let bytes = m.encode(); + // `encode` re-derives `manifest_id` from the body, so the decoded + // manifest carries the derived id while `m` still holds the empty one. + // Compare the canonical bytes, which is what the guard compares. + assert_eq!(Manifest::decode(&bytes).unwrap().encode(), bytes); + + for i in 0..bytes.len() { + for delta in [1u8, 0x7F, 0xFF] { + let mut b = bytes.clone(); + b[i] ^= delta; + if b == bytes { + continue; + } + match Manifest::decode(&b) { + Err(_) => {} + Ok(decoded) => panic!( + "byte {i} ^ {delta:#04x} was accepted; re-encode matches input: {}", + decoded.encode() == b + ), + } + } + } + } + #[test] fn operation_block_summaries_round_trip_and_are_selectable() { let mut m = Manifest::empty(DocumentId([5; 16])); diff --git a/crates/epiphany-bundle/src/opindex.rs b/crates/epiphany-bundle/src/opindex.rs index d10a1f4..dc862f6 100644 --- a/crates/epiphany-bundle/src/opindex.rs +++ b/crates/epiphany-bundle/src/opindex.rs @@ -295,6 +295,36 @@ mod tests { .expect("valid build inputs") } + /// The index has **no** whole-value re-encode guard — it validates per-site + /// instead — so a lenient sub-codec is not caught for it, only for the + /// manifest. `CompressionAlgorithm::None` used to ignore its parameter byte, + /// which made this decoder accept two distinct byte strings for one value + /// while its own doc promised to "reject, never normalize". Found by the P3 + /// wire fuzzer; fixed in the sub-codec, and pinned here at the surface that + /// exposed it. + #[test] + fn a_lenient_compression_byte_is_rejected_rather_than_normalized() { + let index = OperationIndex::build(&[(block_ref(0x11, 576, 64), vec![([2; 16], 8)])]) + .expect("valid build inputs"); + let bytes = index.encode(); + assert_eq!(OperationIndex::decode(&bytes).unwrap(), index); + + // blocks count (4) + ChunkRef: id 32, kind 1, schema 4, offset 8, + // compressed_length 8, uncompressed_length 8, compression tag 1. + const PARAM_AT: usize = 4 + 32 + 1 + 4 + 8 + 8 + 8 + 1; + assert_eq!(bytes[PARAM_AT - 1], 0, "the compression tag is None"); + assert_eq!(bytes[PARAM_AT], 0, "its parameter byte is canonically zero"); + + let mut lenient = bytes.clone(); + lenient[PARAM_AT] = 0xFF; + assert_ne!(lenient, bytes); + assert!( + OperationIndex::decode(&lenient).is_err(), + "a non-zero None parameter must be rejected; accepting it made \ + decode non-injective (two byte strings, one value)" + ); + } + #[test] fn payload_encoding_is_golden() { // PROVISIONAL golden lock (DECISIONS.md "Operation index"): the byte diff --git a/spec/binary_format.pdf b/spec/binary_format.pdf index 320c7d5..00d3065 100644 Binary files a/spec/binary_format.pdf and b/spec/binary_format.pdf differ diff --git a/spec/binary_format.tex b/spec/binary_format.tex index 7968ad6..476da5b 100644 --- a/spec/binary_format.tex +++ b/spec/binary_format.tex @@ -235,7 +235,7 @@ {\Large\scshape\color{epiphanyslate}Binary Format}\\[6pt] {\large\itshape\color{epiphanyslate}A companion to the Core Specification}\\[14pt] {\color{epiphanygold}\rule{3in}{0.8pt}}\\[24pt] - {\normalsize\color{epiphanyink}Version 0.7.0 --- Transpose algebra (OperationKind/Tag 30; the major-0 minor append, and the strictly-increasing sequence)}\\[4pt] + {\normalsize\color{epiphanyink}Version 0.8.0 --- Strict CompressionAlgorithm decode (the non-zero None parameter is rejected, not ignored)}\\[4pt] {\small\color{epiphanyslate}Normative for the byte layouts it defines} \vfill \end{titlepage} @@ -1858,6 +1858,30 @@ Zstd$\{$level$\}$ $= (\tablenums{1}, \mathit{level})$; Reserved$(v)$ $= (\tablenums{2}, v)$. \texttt{None} is \emph{not} a bare tag --- its zero parameter byte is always present. +\begin{requirement} + \label{req:binfmt:compression-none-parameter} + A decoder \MUST{} reject a \texttt{CompressionAlgorithm} whose discriminant + is \tablenums{0} (\texttt{None}) and whose parameter byte is non-zero. It + \MUSTNOT{} ignore the byte, and \MUSTNOT{} normalize it to zero. +\end{requirement} + +\begin{rationale} + Ignoring it made the codec \emph{lenient}, and therefore non-injective: + $(\tablenums{0}, \tablenums{255})$ and $(\tablenums{0}, \tablenums{0})$ + decoded to one value, which re-encodes to the latter. Every structure + embedding a \texttt{ChunkRef} inherited that. A whole-value + re-encode-and-compare guard hides it --- the manifest has one --- but the + operation index does not, and so accepted two distinct payloads for one index + while its own contract promised to reject non-canonical bytes rather than + normalize them. Content-addressing rests on distinct bytes meaning distinct + values. + + No conforming writer emits a non-zero parameter for \texttt{None}, so this + rejects only corrupt or adversarial input; no existing file changes meaning. + Found by the wire-decode fuzzer (Push~5, P3); supersedes the core + specification's earlier ``present but zero, and ignored on read''. +\end{rationale} + \textbf{Read rules.} Writers in this format version emit \texttt{None} only; \emph{reading} zstd at any level is a conformance \MUST{}. A zstd payload is decompressed into a buffer sized \emph{exactly} by the declared @@ -3207,6 +3231,14 @@ layouts of Section~\ref{sec:values:representative}. changed. Semantics: Operation Catalog \sectionsc{TransposeInterval}, 0.8.0; algebra: core Chapter~2, \sectionsc{Transposition and the Interval Type}. \\ + \today & Chunk references & 0.8.0 --- Strict \texttt{CompressionAlgorithm} + decode (Push~5, P3): \texttt{req:binfmt:compression-none-parameter} \MUST{} + reject a non-zero parameter byte on the \texttt{None} discriminant. Ignoring + it made the codec non-injective --- two byte strings, one value --- which + every structure embedding a \texttt{ChunkRef} inherited; the operation index, + having no whole-value re-encode guard, accepted both. Found by the wire-decode + fuzzer. Supersedes the core specification's ``ignored on read''; no wire + layout changed and no conforming writer is affected. \\ \bottomrule \end{longtable} diff --git a/spec/core_spec.pdf b/spec/core_spec.pdf index 53a9394..61f06c4 100644 Binary files a/spec/core_spec.pdf and b/spec/core_spec.pdf differ diff --git a/spec/core_spec.tex b/spec/core_spec.tex index b6658e9..de7d6aa 100644 --- a/spec/core_spec.tex +++ b/spec/core_spec.tex @@ -10668,8 +10668,10 @@ pub struct ChunkId(pub ContentHash); \texttt{CompressionAlgorithm} encodes as a fixed \emph{two} bytes: a declaration-order discriminant byte followed by a single parameter - byte that is always present. \texttt{None} $= 0$ (the parameter byte - is present but zero, and ignored on read); \texttt{Zstd\{level\}} + byte that is always present. \texttt{None} $= 0$, and its parameter + byte \MUST{} be zero --- a reader \MUST{} reject a non-zero one rather + than ignore it (Section~\ref{sec:format:binary}, and the Binary Format + companion's \sectionsc{Chunk References}). \texttt{Zstd\{level\}} $= 1$ with the \texttt{level} in the parameter byte; \texttt{Reserved(u8)} $= 2$ with the reserved value in the parameter byte. The parameter byte is written even for \texttt{None}, so the