Push 5 / P3: fuzz the bundle wire, and find a lenient codec hiding behind a guard
A wire-decode fuzzer over Bundle::open, Manifest::decode, OperationIndex::decode, decode_block and envelope_offsets. The existing crash-recovery fuzzer corrupts an image the way a CRASH does -- torn writes at syscall boundaries. This one corrupts it the way an attacker or a bit-rotted disk does: arbitrary bytes, anywhere. It found a real defect. CompressionAlgorithm::None read its parameter byte and DISCARDED it, while encode writes zero. So [0, 0xFF] and [0, 0] both decoded to None, and the first re-encoded to the second: a lenient, non-injective codec, inherited by every structure embedding a ChunkRef. Whether that was visible depended entirely on the embedder: Manifest::decode has a whole-value re-encode guard, and it is TOTAL -- proved by exhaustive single-byte perturbation, every one rejected. It caught this. OperationIndex::decode has no guard; it validates per-site. It accepted both byte strings, while its own doc promised to "reject (never normalizing) any non-canonical form". That promise was false. That is the same two-layer lesson P2 recorded one commit ago, from the other side: a re-encode guard is complete only where the encoder normalizes, and its completeness can MASK a lenient sub-codec rather than fix it. Fixed at the source, not papered over at the index. An exhaustive sweep -- every byte, every value, plus an 8-byte extreme-integer window -- finds no remaining non-injective site. The fix contradicted ratified spec text, which said the byte was "present but zero, and ignored on read". Escalated rather than fixed unilaterally. The user ratified strict decode: core spec's clause is superseded, Binary Format gains req:binfmt:compression-none-parameter and moves 0.7.0 -> 0.8.0. No wire layout changed, and no conforming writer emits a non-zero byte, so this rejects only corrupt or adversarial input -- no existing file changes meaning. Coverage was the harness's problem again. The fuzzer's first run reached the operation index's accept path ZERO times -- random bytes never decode as an index -- so every assertion under it was vacuous. It found the bug only once the index corpus was built from real OperationIndex::build output. The smoke tests now assert on a WireFuzzCoverage so that cannot silently regress. 1.5M inputs across five seeds, ~1s each, clean after the fix. Three regressions, each mutation-verified by restoring the leniency: the codec itself, the index that exposed it, and the manifest guard's totality -- which is the asymmetry that hid it. Gate: fmt clean, clippy 0, 30 targets / 1012 passed / 0 failed, docs 0 under -D warnings, conformance 8/8, zero golden churn, both spec documents rebuild with no undefined references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
9596d1e824
commit
7de9e479c4
|
|
@ -428,3 +428,54 @@ implies it, the canonical-base-stays-major-0 rule is now enforced per ROLE
|
|||
(`mis_stamped_canonical_base`, consulted at open and commit → read-only +
|
||||
`UnsupportedCanonicalChunkMajor`, regression-locked). The `SnapshotId` in
|
||||
the harness remains a hash-truncation stand-in (companion open question).
|
||||
|
||||
## Push 5 / P3 — the bundle wire, and a lenient sub-codec (2026-07-09)
|
||||
|
||||
A wire-decode fuzzer (`fuzz::run_wire_decode_fuzz`) over `Bundle::open`,
|
||||
`Manifest::decode`, `OperationIndex::decode`, `decode_block`, and
|
||||
`envelope_offsets`. The existing crash-recovery fuzzer corrupts the image the way
|
||||
a *crash* does — torn writes at syscall boundaries. This one corrupts it the way
|
||||
an attacker or a bit-rotted disk does: arbitrary bytes, anywhere.
|
||||
|
||||
**One real defect: `CompressionAlgorithm::None` ignored its parameter byte.**
|
||||
`decode` read it and discarded it; `encode` writes `0`. So `[0, 0xFF]` and
|
||||
`[0, 0]` both decoded to `None`, and the first re-encoded to the second — a
|
||||
lenient, **non-injective** codec, inherited by every structure embedding a
|
||||
`ChunkRef`.
|
||||
|
||||
Its visibility depended entirely on whether the embedder had a whole-value
|
||||
re-encode guard:
|
||||
|
||||
- `Manifest::decode` **has** one, and it is *total* — verified by exhaustive
|
||||
single-byte perturbation, every one rejected. It caught this. (`encode_body`
|
||||
sorts and deduplicates every vector, and `manifest_id` is derived from the
|
||||
body, which is why the guard is complete here where `MaterializedState`'s is
|
||||
not — see `epiphany-ops/DECISIONS.md` §"Push 5 / P2".)
|
||||
- `OperationIndex::decode` has **no** guard; it validates per-site instead. It
|
||||
accepted both byte strings while its own doc promised to *"reject (never
|
||||
normalizing) any non-canonical form"*. That promise was false.
|
||||
|
||||
Fixed at the source rather than papered over at the index: a non-zero `None`
|
||||
parameter is now rejected. Exhaustive sweep (every byte × every value, plus an
|
||||
8-byte extreme-integer window) finds no remaining non-injective site.
|
||||
|
||||
**This contradicted ratified spec text**, which said the byte was *"present but
|
||||
zero, and ignored on read"*. Escalated rather than fixed unilaterally; the user
|
||||
ratified strict decode and the spec amendment. Core spec's clause is superseded;
|
||||
Binary Format gains `req:binfmt:compression-none-parameter` and moves 0.7.0 →
|
||||
0.8.0. No wire layout changed and no conforming writer emits a non-zero byte, so
|
||||
this rejects only corrupt or adversarial input and no existing file changes
|
||||
meaning.
|
||||
|
||||
**Coverage is the harness's job, again.** The fuzzer's first run reached the
|
||||
operation index's accept path **zero times** — random bytes never decode as an
|
||||
index — so every assertion under it was vacuous. It found the bug only after the
|
||||
index corpus was built from real `OperationIndex::build` output. The smoke tests
|
||||
now assert on a `WireFuzzCoverage` so that can never silently regress. 1.5M
|
||||
inputs across five seeds, ~1s each, clean after the fix.
|
||||
|
||||
Three regressions, each mutation-verified by restoring the leniency:
|
||||
`compression_none_rejects_a_non_zero_parameter_byte` (the codec),
|
||||
`a_lenient_compression_byte_is_rejected_rather_than_normalized` (the index, the
|
||||
surface that exposed it), and `every_single_byte_perturbation_of_a_manifest_is_rejected`
|
||||
(the guard's totality, and the asymmetry that hid the bug).
|
||||
|
|
|
|||
|
|
@ -136,12 +136,27 @@ impl CompressionAlgorithm {
|
|||
};
|
||||
}
|
||||
|
||||
/// Decodes strictly: `None` carries no parameter, so its parameter byte
|
||||
/// **must** be zero.
|
||||
///
|
||||
/// Accepting a non-zero one made this codec lenient, and therefore
|
||||
/// non-injective: `[0, 0xFF]` and `[0, 0]` both decoded to `None`, which
|
||||
/// re-encodes to `[0, 0]`. Any structure embedding a [`ChunkRef`] without a
|
||||
/// whole-value re-encode guard inherited that — `OperationIndex::decode`
|
||||
/// did, while promising in its own doc to reject non-canonical bytes rather
|
||||
/// than normalize them. A conforming writer never emits a non-zero
|
||||
/// parameter here, so this rejects only corrupt or adversarial input.
|
||||
#[inline]
|
||||
pub(crate) fn decode(r: &mut Reader) -> Result<Self, DecodeError> {
|
||||
let tag = r.get_u8()?;
|
||||
let param = r.get_u8()?;
|
||||
Ok(match tag {
|
||||
0 => CompressionAlgorithm::None,
|
||||
0 if param == 0 => CompressionAlgorithm::None,
|
||||
0 => {
|
||||
return Err(DecodeError::Malformed(
|
||||
"CompressionAlgorithm::None carries a non-zero parameter byte",
|
||||
))
|
||||
}
|
||||
1 => CompressionAlgorithm::Zstd { level: param },
|
||||
2 => CompressionAlgorithm::Reserved(param),
|
||||
other => {
|
||||
|
|
@ -296,6 +311,45 @@ impl Ord for ChunkRef {
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// `CompressionAlgorithm::None` carries no parameter, so its parameter byte
|
||||
/// is normative zero. Accepting a non-zero one made the codec lenient: two
|
||||
/// distinct byte strings decoded to one value, and re-encoding produced a
|
||||
/// third. Found by the P3 wire fuzzer via `OperationIndex`, which embeds a
|
||||
/// `ChunkRef` and has no whole-value re-encode guard to hide it.
|
||||
#[test]
|
||||
fn compression_none_rejects_a_non_zero_parameter_byte() {
|
||||
let mut w = Writer::new();
|
||||
CompressionAlgorithm::None.encode(&mut w);
|
||||
let canonical = w.into_bytes();
|
||||
assert_eq!(canonical, vec![0, 0]);
|
||||
assert_eq!(
|
||||
CompressionAlgorithm::decode(&mut Reader::new(&canonical)).unwrap(),
|
||||
CompressionAlgorithm::None
|
||||
);
|
||||
|
||||
for param in [1u8, 0x7F, 0xFF] {
|
||||
let lenient = vec![0, param];
|
||||
assert!(
|
||||
CompressionAlgorithm::decode(&mut Reader::new(&lenient)).is_err(),
|
||||
"None with parameter {param:#04x} must be rejected, never normalized to zero"
|
||||
);
|
||||
}
|
||||
|
||||
// The parameter is meaningful for the other two, so it round-trips.
|
||||
for algo in [
|
||||
CompressionAlgorithm::Zstd { level: 0xFF },
|
||||
CompressionAlgorithm::Reserved(0xFF),
|
||||
] {
|
||||
let mut w = Writer::new();
|
||||
algo.encode(&mut w);
|
||||
let bytes = w.into_bytes();
|
||||
assert_eq!(
|
||||
CompressionAlgorithm::decode(&mut Reader::new(&bytes)).unwrap(),
|
||||
algo
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunk_kind_discriminants_round_trip() {
|
||||
for kind in [
|
||||
|
|
|
|||
|
|
@ -36,13 +36,16 @@
|
|||
//! across every corruption scenario the QUICKSTART enumerates.
|
||||
|
||||
use crate::bundle::{Bundle, CommitContext, StagedChunk, BODY_START};
|
||||
use crate::chunk::{ChunkKind, ChunkRef, CompressionAlgorithm};
|
||||
use crate::error::IntegrityAnomaly;
|
||||
use crate::header::FixedHeader;
|
||||
use crate::ids::{DocumentId, FileUuid, ReductionAlgorithmVersion, SchemaVersion, WallClockTime};
|
||||
use crate::manifest::Manifest;
|
||||
use crate::opindex::OperationIndex;
|
||||
use crate::store::{CrashPoint, FaultStore, MemStore, Tear};
|
||||
use crate::superblock::{CommitState, ProfileId, Slot, Superblock, SUPERBLOCK_LEN};
|
||||
use crate::{block, manifest_chunk_hash};
|
||||
use epiphany_determinism::{ChunkId, ContentHash};
|
||||
|
||||
/// A tiny deterministic generator (SplitMix64), matching `epiphany-determinism`'s
|
||||
/// fuzz harness: reproducible across platforms, no `rand` dependency, so a
|
||||
|
|
@ -175,6 +178,275 @@ fn build_base(rng: &mut SplitMix64, commits: u64) -> (Vec<u8>, u64) {
|
|||
(bundle.into_store().into_bytes(), gen)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Wire-decode fuzzing (P3 of the decode-hardening track).
|
||||
//
|
||||
// The crash-recovery fuzzer above corrupts the image the way a *crash* does:
|
||||
// torn writes at syscall boundaries. This one corrupts it the way an *attacker*
|
||||
// or a bit-rotted disk does — arbitrary bytes, anywhere — and drives every
|
||||
// decode surface the bundle exposes:
|
||||
//
|
||||
// Bundle::open · Manifest::decode · OperationIndex::decode
|
||||
// block::decode_block · block::envelope_offsets
|
||||
//
|
||||
// Two properties. A mutated image must never panic a decoder (a bundle is
|
||||
// attacker-controlled input the moment it is emailed), and an *accepted* byte
|
||||
// string must re-encode to itself where the type has a canonical encoding.
|
||||
//
|
||||
// P2 (`epiphany-ops`) established that a re-encode guard is complete only for
|
||||
// fields the encoder NORMALIZES. `Manifest::encode_body` sorts and deduplicates
|
||||
// every vector, so its guard genuinely is complete; `OperationIndex::decode`
|
||||
// has no guard and instead checks its two `Vec`s for strict ascent per-site,
|
||||
// which is the other correct answer. Both are asserted here.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// What a decode-fuzz run actually reached. A harness that never gets a decoder
|
||||
/// to say `Ok` proves only the absence of a panic.
|
||||
#[derive(Default, Debug, PartialEq, Eq)]
|
||||
pub struct WireFuzzCoverage {
|
||||
pub opens_ok: u64,
|
||||
pub opens_rejected: u64,
|
||||
pub manifests_ok: u64,
|
||||
pub manifests_rejected: u64,
|
||||
pub blocks_ok: u64,
|
||||
pub blocks_rejected: u64,
|
||||
pub indices_ok: u64,
|
||||
pub indices_rejected: u64,
|
||||
}
|
||||
|
||||
fn wire_random_bytes(rng: &mut SplitMix64, n: usize) -> Vec<u8> {
|
||||
(0..n).map(|_| rng.next_u64() as u8).collect()
|
||||
}
|
||||
|
||||
fn wire_substitute(rng: &mut SplitMix64, bytes: &mut [u8], k: usize) {
|
||||
if bytes.is_empty() {
|
||||
return;
|
||||
}
|
||||
for _ in 0..k {
|
||||
let i = (rng.next_u64() as usize) % bytes.len();
|
||||
bytes[i] = rng.next_u64() as u8;
|
||||
}
|
||||
}
|
||||
|
||||
/// Overwrites a random 4- or 8-byte window with an extreme integer: the
|
||||
/// count/length/offset attack. A bundle's manifest carries file offsets and
|
||||
/// lengths, so this is the mutation that matters most here.
|
||||
fn wire_corrupt_int(rng: &mut SplitMix64, bytes: &mut [u8]) {
|
||||
if bytes.len() < 8 {
|
||||
return;
|
||||
}
|
||||
let i = (rng.next_u64() as usize) % (bytes.len() - 7);
|
||||
match rng.next_u64() % 4 {
|
||||
0 => bytes[i..i + 4].copy_from_slice(&u32::MAX.to_le_bytes()),
|
||||
1 => bytes[i..i + 8].copy_from_slice(&u64::MAX.to_le_bytes()),
|
||||
2 => {
|
||||
bytes[i..i + 8].copy_from_slice(&(rng.next_u64() | 0x8000_0000_0000_0000).to_le_bytes())
|
||||
}
|
||||
_ => bytes[i..i + 4].copy_from_slice(&(rng.next_u64() as u32).to_le_bytes()),
|
||||
}
|
||||
}
|
||||
|
||||
fn mutate_image(rng: &mut SplitMix64, base: &[u8]) -> Vec<u8> {
|
||||
let mut b = base.to_vec();
|
||||
match rng.next_u64() % 6 {
|
||||
0 => return b, // unmutated: a live check that the corpus opens
|
||||
1 => {
|
||||
let k = 1 + (rng.next_u64() % 6) as usize;
|
||||
wire_substitute(rng, &mut b, k);
|
||||
}
|
||||
2 => {
|
||||
let t = (rng.next_u64() as usize) % (b.len() + 1);
|
||||
b.truncate(t);
|
||||
}
|
||||
3 => {
|
||||
let n = 1 + (rng.next_u64() % 32) as usize;
|
||||
let tail = wire_random_bytes(rng, n);
|
||||
b.extend_from_slice(&tail);
|
||||
}
|
||||
4 => wire_corrupt_int(rng, &mut b),
|
||||
_ => {
|
||||
// Corrupt the superblock region specifically: the two 256-byte slots
|
||||
// whose CRCs gate `open`.
|
||||
let n = b.len().min(512);
|
||||
if n > 0 {
|
||||
let i = (rng.next_u64() as usize) % n;
|
||||
b[i] = rng.next_u64() as u8;
|
||||
}
|
||||
}
|
||||
}
|
||||
b
|
||||
}
|
||||
|
||||
/// Runs `iters` adversarial wire-decode iterations from `seed` over the bundle's
|
||||
/// decode surfaces. A panic, or an accepted byte string that does not re-encode
|
||||
/// to itself, fails the run; `seed` reproduces it exactly.
|
||||
pub fn run_wire_decode_fuzz(iters: u64, seed: u64) -> WireFuzzCoverage {
|
||||
let mut rng = SplitMix64::new(seed);
|
||||
let mut cov = WireFuzzCoverage::default();
|
||||
|
||||
// A small pool of valid, populated images and valid manifest payloads.
|
||||
let mut images: Vec<Vec<u8>> = Vec::new();
|
||||
let mut manifests: Vec<Vec<u8>> = Vec::new();
|
||||
for commits in 0..4u64 {
|
||||
let (image, _) = build_base(&mut rng, commits);
|
||||
let bundle = Bundle::open(MemStore::from_bytes(image.clone())).expect("valid image opens");
|
||||
manifests.push(bundle.manifest().encode());
|
||||
images.push(image);
|
||||
}
|
||||
let valid_blocks: Vec<Vec<u8>> = {
|
||||
let payloads: Vec<Vec<u8>> = (0..3).map(|i| vec![i as u8 + 1; 8 + i * 5]).collect();
|
||||
block::pack_operation_blocks(&payloads)
|
||||
};
|
||||
|
||||
// Valid operation-index payloads. Random bytes never decode, so an index
|
||||
// corpus of noise leaves the decoder's accept path — and therefore every
|
||||
// assertion below it — unreached (measured: `indices_ok` was 0).
|
||||
let valid_indices: Vec<Vec<u8>> = {
|
||||
let block_ref = |hash_byte: u8, offset: u64, len: u64| ChunkRef {
|
||||
id: ChunkId(ContentHash([hash_byte; 32])),
|
||||
kind: ChunkKind::OperationEnvelopeBlock,
|
||||
schema_version: SchemaVersion::V0,
|
||||
offset,
|
||||
compressed_length: len,
|
||||
uncompressed_length: len,
|
||||
compression: CompressionAlgorithm::None,
|
||||
hash: ContentHash([hash_byte; 32]),
|
||||
};
|
||||
[
|
||||
OperationIndex::build(&[]).expect("empty index"),
|
||||
OperationIndex::build(&[(block_ref(0x11, 576, 64), vec![([2; 16], 8)])])
|
||||
.expect("one block"),
|
||||
OperationIndex::build(&[
|
||||
(block_ref(0x22, 1000, 64), vec![([3; 16], 8), ([1; 16], 40)]),
|
||||
(block_ref(0x11, 576, 32), vec![([2; 16], 8)]),
|
||||
])
|
||||
.expect("two blocks"),
|
||||
]
|
||||
.iter()
|
||||
.map(|i| i.encode())
|
||||
.collect()
|
||||
};
|
||||
|
||||
for _ in 0..iters {
|
||||
// 1. Whole-image open. Must never panic; an Ok manifest must re-encode.
|
||||
let pick = (rng.next_u64() as usize) % images.len();
|
||||
let image = mutate_image(&mut rng, &images[pick]);
|
||||
match Bundle::open(MemStore::from_bytes(image)) {
|
||||
Ok(bundle) => {
|
||||
cov.opens_ok += 1;
|
||||
let encoded = bundle.manifest().encode();
|
||||
assert_eq!(
|
||||
Manifest::decode(&encoded).as_ref(),
|
||||
Ok(bundle.manifest()),
|
||||
"an opened bundle's manifest does not round-trip"
|
||||
);
|
||||
// Reading every chunk the manifest names must be total.
|
||||
for r in bundle.manifest().canonical_chunk_refs() {
|
||||
let _ = bundle.read_chunk(&r);
|
||||
}
|
||||
}
|
||||
Err(_) => cov.opens_rejected += 1,
|
||||
}
|
||||
|
||||
// 2. Manifest payload decode: strict-canonical, guard-backed.
|
||||
let mut m = manifests[(rng.next_u64() as usize) % manifests.len()].clone();
|
||||
match rng.next_u64() % 4 {
|
||||
0 => {}
|
||||
1 => wire_substitute(&mut rng, &mut m, 1),
|
||||
2 => wire_corrupt_int(&mut rng, &mut m),
|
||||
_ => {
|
||||
let t = (rng.next_u64() as usize) % (m.len() + 1);
|
||||
m.truncate(t);
|
||||
}
|
||||
}
|
||||
match Manifest::decode(&m) {
|
||||
Ok(manifest) => {
|
||||
cov.manifests_ok += 1;
|
||||
assert_eq!(
|
||||
manifest.encode(),
|
||||
m,
|
||||
"the manifest decoder accepted a non-canonical byte string"
|
||||
);
|
||||
}
|
||||
Err(_) => cov.manifests_rejected += 1,
|
||||
}
|
||||
|
||||
// 3. Block payload framing.
|
||||
let mut b = valid_blocks[(rng.next_u64() as usize) % valid_blocks.len()].clone();
|
||||
match rng.next_u64() % 4 {
|
||||
0 => {}
|
||||
1 => wire_substitute(&mut rng, &mut b, 1),
|
||||
2 => wire_corrupt_int(&mut rng, &mut b),
|
||||
_ => {
|
||||
let t = (rng.next_u64() as usize) % (b.len() + 1);
|
||||
b.truncate(t);
|
||||
}
|
||||
}
|
||||
match block::decode_block(&b) {
|
||||
Ok(envelopes) => {
|
||||
cov.blocks_ok += 1;
|
||||
// `envelope_offsets` shares the code path: the two must agree,
|
||||
// and each recorded offset must actually address its envelope.
|
||||
let offsets = block::envelope_offsets(&b).expect("same validation");
|
||||
assert_eq!(offsets.len(), envelopes.len());
|
||||
for ((offset, slice), env) in offsets.iter().zip(envelopes.iter()) {
|
||||
assert_eq!(
|
||||
*slice,
|
||||
&env[..],
|
||||
"envelope_offsets disagrees with decode_block"
|
||||
);
|
||||
let at = *offset as usize;
|
||||
assert_eq!(
|
||||
&b[at..at + env.len()],
|
||||
&env[..],
|
||||
"offset does not address the envelope"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(_) => cov.blocks_rejected += 1,
|
||||
}
|
||||
|
||||
// 4. Operation index: no re-encode guard; per-site strict-ascent checks.
|
||||
let mut idx = valid_indices[(rng.next_u64() as usize) % valid_indices.len()].clone();
|
||||
match rng.next_u64() % 5 {
|
||||
0 => {}
|
||||
1 => wire_substitute(&mut rng, &mut idx, 1),
|
||||
2 => wire_corrupt_int(&mut rng, &mut idx),
|
||||
3 => {
|
||||
let t = (rng.next_u64() as usize) % (idx.len() + 1);
|
||||
idx.truncate(t);
|
||||
}
|
||||
_ => {
|
||||
let n = (rng.next_u64() % 96) as usize;
|
||||
idx = wire_random_bytes(&mut rng, n);
|
||||
}
|
||||
}
|
||||
match OperationIndex::decode(&idx) {
|
||||
Ok(index) => {
|
||||
cov.indices_ok += 1;
|
||||
// No re-encode guard here; the decoder's per-site checks are the
|
||||
// contract, so assert them directly — and assert injectivity,
|
||||
// which the `Vec`-order preservation makes non-trivial.
|
||||
assert_eq!(
|
||||
index.encode(),
|
||||
idx,
|
||||
"the operation-index decoder accepted a non-canonical byte string"
|
||||
);
|
||||
assert!(
|
||||
index.blocks().windows(2).all(|w| w[0] < w[1]),
|
||||
"the index decoder accepted unsorted blocks"
|
||||
);
|
||||
assert!(
|
||||
index.entries().windows(2).all(|w| w[0].id < w[1].id),
|
||||
"the index decoder accepted unsorted entries"
|
||||
);
|
||||
}
|
||||
Err(_) => cov.indices_rejected += 1,
|
||||
}
|
||||
}
|
||||
cov
|
||||
}
|
||||
|
||||
/// The crash-recovery fuzzer: `iters` randomized scenarios from `seed`. Each
|
||||
/// iteration builds a base bundle at a random generation, then commits a random
|
||||
/// set of operation blocks while crashing at a random syscall with a random
|
||||
|
|
@ -538,4 +810,34 @@ mod tests {
|
|||
fn manifest_selection_harness_passes() {
|
||||
run_manifest_selection_harness();
|
||||
}
|
||||
|
||||
/// Two deterministic smoke seeds over every bundle decode surface.
|
||||
///
|
||||
/// The coverage assertions are load-bearing. A wire fuzzer that never gets a
|
||||
/// decoder to say `Ok` proves only the absence of a panic — and this one
|
||||
/// initially reached the operation index's accept path *zero* times, because
|
||||
/// random bytes never decode as an index. It found the lenient
|
||||
/// `CompressionAlgorithm::None` parameter byte only once its index corpus
|
||||
/// was real.
|
||||
#[test]
|
||||
fn wire_decode_fuzz_smoke_seed_a() {
|
||||
let cov = run_wire_decode_fuzz(20_000, 0x0DEC_0DE0_F022_1234);
|
||||
assert!(cov.opens_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.opens_rejected > 1_000, "{cov:?}");
|
||||
assert!(cov.manifests_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.manifests_rejected > 1_000, "{cov:?}");
|
||||
assert!(cov.blocks_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.blocks_rejected > 1_000, "{cov:?}");
|
||||
assert!(cov.indices_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.indices_rejected > 1_000, "{cov:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wire_decode_fuzz_smoke_seed_b() {
|
||||
let cov = run_wire_decode_fuzz(20_000, 0xF0FA_11BA_C0DE_5EED);
|
||||
assert!(cov.opens_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.manifests_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.blocks_ok > 1_000, "{cov:?}");
|
||||
assert!(cov.indices_ok > 1_000, "{cov:?}");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -709,6 +709,57 @@ mod tests {
|
|||
use crate::chunk::{chunk_id, ChunkKind};
|
||||
use crate::ids::SnapshotId;
|
||||
|
||||
/// The manifest's whole-value re-encode guard is **total**: every single-byte
|
||||
/// perturbation is rejected. `manifest_id` is derived from the body, so a
|
||||
/// body edit fails the id check and an id edit fails the derivation; and
|
||||
/// `encode_body` sorts and deduplicates every vector, so an out-of-order
|
||||
/// encoding cannot round-trip either.
|
||||
///
|
||||
/// This is what makes the guard complete *here* and not in
|
||||
/// `MaterializedState` (epiphany-ops), whose encoder writes its `Vec` fields
|
||||
/// verbatim, nor in `OperationIndex`, which has no guard at all. The lenient
|
||||
/// `CompressionAlgorithm::None` parameter byte was invisible through the
|
||||
/// manifest for exactly this reason, and visible through the index.
|
||||
#[test]
|
||||
fn every_single_byte_perturbation_of_a_manifest_is_rejected() {
|
||||
use crate::chunk::{ChunkRef, CompressionAlgorithm};
|
||||
use crate::ids::SchemaVersion;
|
||||
|
||||
let mut m = Manifest::empty(DocumentId([5; 16]));
|
||||
m.operation_roots.push(ChunkRef {
|
||||
id: ChunkId(ContentHash([0x11; 32])),
|
||||
kind: ChunkKind::OperationEnvelopeBlock,
|
||||
schema_version: SchemaVersion::V0,
|
||||
offset: 576,
|
||||
compressed_length: 64,
|
||||
uncompressed_length: 64,
|
||||
compression: CompressionAlgorithm::None,
|
||||
hash: ContentHash([0x11; 32]),
|
||||
});
|
||||
let bytes = m.encode();
|
||||
// `encode` re-derives `manifest_id` from the body, so the decoded
|
||||
// manifest carries the derived id while `m` still holds the empty one.
|
||||
// Compare the canonical bytes, which is what the guard compares.
|
||||
assert_eq!(Manifest::decode(&bytes).unwrap().encode(), bytes);
|
||||
|
||||
for i in 0..bytes.len() {
|
||||
for delta in [1u8, 0x7F, 0xFF] {
|
||||
let mut b = bytes.clone();
|
||||
b[i] ^= delta;
|
||||
if b == bytes {
|
||||
continue;
|
||||
}
|
||||
match Manifest::decode(&b) {
|
||||
Err(_) => {}
|
||||
Ok(decoded) => panic!(
|
||||
"byte {i} ^ {delta:#04x} was accepted; re-encode matches input: {}",
|
||||
decoded.encode() == b
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn operation_block_summaries_round_trip_and_are_selectable() {
|
||||
let mut m = Manifest::empty(DocumentId([5; 16]));
|
||||
|
|
|
|||
|
|
@ -295,6 +295,36 @@ mod tests {
|
|||
.expect("valid build inputs")
|
||||
}
|
||||
|
||||
/// The index has **no** whole-value re-encode guard — it validates per-site
|
||||
/// instead — so a lenient sub-codec is not caught for it, only for the
|
||||
/// manifest. `CompressionAlgorithm::None` used to ignore its parameter byte,
|
||||
/// which made this decoder accept two distinct byte strings for one value
|
||||
/// while its own doc promised to "reject, never normalize". Found by the P3
|
||||
/// wire fuzzer; fixed in the sub-codec, and pinned here at the surface that
|
||||
/// exposed it.
|
||||
#[test]
|
||||
fn a_lenient_compression_byte_is_rejected_rather_than_normalized() {
|
||||
let index = OperationIndex::build(&[(block_ref(0x11, 576, 64), vec![([2; 16], 8)])])
|
||||
.expect("valid build inputs");
|
||||
let bytes = index.encode();
|
||||
assert_eq!(OperationIndex::decode(&bytes).unwrap(), index);
|
||||
|
||||
// blocks count (4) + ChunkRef: id 32, kind 1, schema 4, offset 8,
|
||||
// compressed_length 8, uncompressed_length 8, compression tag 1.
|
||||
const PARAM_AT: usize = 4 + 32 + 1 + 4 + 8 + 8 + 8 + 1;
|
||||
assert_eq!(bytes[PARAM_AT - 1], 0, "the compression tag is None");
|
||||
assert_eq!(bytes[PARAM_AT], 0, "its parameter byte is canonically zero");
|
||||
|
||||
let mut lenient = bytes.clone();
|
||||
lenient[PARAM_AT] = 0xFF;
|
||||
assert_ne!(lenient, bytes);
|
||||
assert!(
|
||||
OperationIndex::decode(&lenient).is_err(),
|
||||
"a non-zero None parameter must be rejected; accepting it made \
|
||||
decode non-injective (two byte strings, one value)"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_encoding_is_golden() {
|
||||
// PROVISIONAL golden lock (DECISIONS.md "Operation index"): the byte
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -235,7 +235,7 @@
|
|||
{\Large\scshape\color{epiphanyslate}Binary Format}\\[6pt]
|
||||
{\large\itshape\color{epiphanyslate}A companion to the Core Specification}\\[14pt]
|
||||
{\color{epiphanygold}\rule{3in}{0.8pt}}\\[24pt]
|
||||
{\normalsize\color{epiphanyink}Version 0.7.0 --- Transpose algebra (OperationKind/Tag 30; the major-0 minor append, and the strictly-increasing sequence)}\\[4pt]
|
||||
{\normalsize\color{epiphanyink}Version 0.8.0 --- Strict CompressionAlgorithm decode (the non-zero None parameter is rejected, not ignored)}\\[4pt]
|
||||
{\small\color{epiphanyslate}Normative for the byte layouts it defines}
|
||||
\vfill
|
||||
\end{titlepage}
|
||||
|
|
@ -1858,6 +1858,30 @@ Zstd$\{$level$\}$ $= (\tablenums{1}, \mathit{level})$; Reserved$(v)$ $=
|
|||
(\tablenums{2}, v)$. \texttt{None} is \emph{not} a bare tag --- its zero
|
||||
parameter byte is always present.
|
||||
|
||||
\begin{requirement}
|
||||
\label{req:binfmt:compression-none-parameter}
|
||||
A decoder \MUST{} reject a \texttt{CompressionAlgorithm} whose discriminant
|
||||
is \tablenums{0} (\texttt{None}) and whose parameter byte is non-zero. It
|
||||
\MUSTNOT{} ignore the byte, and \MUSTNOT{} normalize it to zero.
|
||||
\end{requirement}
|
||||
|
||||
\begin{rationale}
|
||||
Ignoring it made the codec \emph{lenient}, and therefore non-injective:
|
||||
$(\tablenums{0}, \tablenums{255})$ and $(\tablenums{0}, \tablenums{0})$
|
||||
decoded to one value, which re-encodes to the latter. Every structure
|
||||
embedding a \texttt{ChunkRef} inherited that. A whole-value
|
||||
re-encode-and-compare guard hides it --- the manifest has one --- but the
|
||||
operation index does not, and so accepted two distinct payloads for one index
|
||||
while its own contract promised to reject non-canonical bytes rather than
|
||||
normalize them. Content-addressing rests on distinct bytes meaning distinct
|
||||
values.
|
||||
|
||||
No conforming writer emits a non-zero parameter for \texttt{None}, so this
|
||||
rejects only corrupt or adversarial input; no existing file changes meaning.
|
||||
Found by the wire-decode fuzzer (Push~5, P3); supersedes the core
|
||||
specification's earlier ``present but zero, and ignored on read''.
|
||||
\end{rationale}
|
||||
|
||||
\textbf{Read rules.} Writers in this format version emit \texttt{None} only;
|
||||
\emph{reading} zstd at any level is a conformance \MUST{}. A zstd payload is
|
||||
decompressed into a buffer sized \emph{exactly} by the declared
|
||||
|
|
@ -3207,6 +3231,14 @@ layouts of Section~\ref{sec:values:representative}.
|
|||
changed. Semantics: Operation Catalog \sectionsc{TransposeInterval},
|
||||
0.8.0; algebra: core Chapter~2, \sectionsc{Transposition and the
|
||||
Interval Type}. \\
|
||||
\today & Chunk references & 0.8.0 --- Strict \texttt{CompressionAlgorithm}
|
||||
decode (Push~5, P3): \texttt{req:binfmt:compression-none-parameter} \MUST{}
|
||||
reject a non-zero parameter byte on the \texttt{None} discriminant. Ignoring
|
||||
it made the codec non-injective --- two byte strings, one value --- which
|
||||
every structure embedding a \texttt{ChunkRef} inherited; the operation index,
|
||||
having no whole-value re-encode guard, accepted both. Found by the wire-decode
|
||||
fuzzer. Supersedes the core specification's ``ignored on read''; no wire
|
||||
layout changed and no conforming writer is affected. \\
|
||||
\bottomrule
|
||||
\end{longtable}
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -10668,8 +10668,10 @@ pub struct ChunkId(pub ContentHash);
|
|||
|
||||
\texttt{CompressionAlgorithm} encodes as a fixed \emph{two} bytes: a
|
||||
declaration-order discriminant byte followed by a single parameter
|
||||
byte that is always present. \texttt{None} $= 0$ (the parameter byte
|
||||
is present but zero, and ignored on read); \texttt{Zstd\{level\}}
|
||||
byte that is always present. \texttt{None} $= 0$, and its parameter
|
||||
byte \MUST{} be zero --- a reader \MUST{} reject a non-zero one rather
|
||||
than ignore it (Section~\ref{sec:format:binary}, and the Binary Format
|
||||
companion's \sectionsc{Chunk References}). \texttt{Zstd\{level\}}
|
||||
$= 1$ with the \texttt{level} in the parameter byte;
|
||||
\texttt{Reserved(u8)} $= 2$ with the reserved value in the parameter
|
||||
byte. The parameter byte is written even for \texttt{None}, so the
|
||||
|
|
|
|||
Loading…
Reference in New Issue