A wire-decode fuzzer over Bundle::open, Manifest::decode, OperationIndex::decode,
decode_block and envelope_offsets. The existing crash-recovery fuzzer corrupts an
image the way a CRASH does -- torn writes at syscall boundaries. This one
corrupts it the way an attacker or a bit-rotted disk does: arbitrary bytes,
anywhere.
It found a real defect. CompressionAlgorithm::None read its parameter byte and
DISCARDED it, while encode writes zero. So [0, 0xFF] and [0, 0] both decoded to
None, and the first re-encoded to the second: a lenient, non-injective codec,
inherited by every structure embedding a ChunkRef.
Whether that was visible depended entirely on the embedder:
Manifest::decode has a whole-value re-encode guard, and it is TOTAL -- proved
by exhaustive single-byte perturbation, every one rejected. It caught this.
OperationIndex::decode has no guard; it validates per-site. It accepted both
byte strings, while its own doc promised to "reject (never normalizing) any
non-canonical form". That promise was false.
That is the same two-layer lesson P2 recorded one commit ago, from the other
side: a re-encode guard is complete only where the encoder normalizes, and its
completeness can MASK a lenient sub-codec rather than fix it. Fixed at the
source, not papered over at the index. An exhaustive sweep -- every byte, every
value, plus an 8-byte extreme-integer window -- finds no remaining non-injective
site.
The fix contradicted ratified spec text, which said the byte was "present but
zero, and ignored on read". Escalated rather than fixed unilaterally. The user
ratified strict decode: core spec's clause is superseded, Binary Format gains
req:binfmt:compression-none-parameter and moves 0.7.0 -> 0.8.0. No wire layout
changed, and no conforming writer emits a non-zero byte, so this rejects only
corrupt or adversarial input -- no existing file changes meaning.
Coverage was the harness's problem again. The fuzzer's first run reached the
operation index's accept path ZERO times -- random bytes never decode as an
index -- so every assertion under it was vacuous. It found the bug only once the
index corpus was built from real OperationIndex::build output. The smoke tests
now assert on a WireFuzzCoverage so that cannot silently regress. 1.5M inputs
across five seeds, ~1s each, clean after the fix.
Three regressions, each mutation-verified by restoring the leniency: the codec
itself, the index that exposed it, and the manifest guard's totality -- which is
the asymmetry that hid it.
Gate: fmt clean, clippy 0, 30 targets / 1012 passed / 0 failed, docs 0 under
-D warnings, conformance 8/8, zero golden churn, both spec documents rebuild
with no undefined references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit reopened P12-K2, whose Pass-12 pin promised the repair would be "a
payload schema-major landing with the Chapter 4 tuning catalog". Both halves
were wrong, and the operation was more broken than the pin admitted.
Measured through EditorSession, not inferred. On a C4: +12 yields alteration
12 (six double-sharps, not C5); +128 clamps to 127 and still reports Applied;
targets [p, p] transposes twice; a non-Cmn position is silently untouched.
transpose(1000) then transpose(-1000) lands on -128, so the operation is not
invertible. Nothing downstream is at fault -- prepass::accidental_ids renders
alteration 12 faithfully. The defect is entirely in what Transpose means.
The false coupling is why this looked big. Pitch has orthogonal scale_position
and acoustic fields. Transposition adds an interval to a scale position;
tuning decides what frequency a scale position sounds at. Adding a fifth to C4
needs no tuning catalog. The same weld had spread: PitchSpaceMismatch was
"Reserved: requires the Chapter 4 tuning catalog" (it reads a discriminant),
and TranspositionInterval was "ADVISORY until the Chapter 4 tuning catalog
pins interval algebra". Push 4 splits: 4a is the algebra and needs no catalog;
4b is the catalog, which has its own blockers (cmn-24 is in the pitch-space
table but cannot exist while Cmn.alteration is i8 semitones).
Ratified by the user:
- New kind, freeze the old. An operation is history; a corrected reduction
rule would rewrite every score that used one. Transpose (disc 9) keeps its
exact semantics, now written as normative replay semantics rather than as
apologies. TransposeInterval takes disc 30. This is cheap: appending a kind
at >= 30 is a schema MINOR, and the payload's constituents are all major-0
layouts, so it stamps major 0. No major 3, no migration.
- Diatonic + chromatic interval, reusing TranspositionInterval -- which
already existed in graph.rs at major 2 for Instrument.transposition, already
codec'd, byte-for-byte the required pair. Minting an Interval beside it
would have been a second normative listing of one type, the drift P13-I1
just closed. Declared once now, in Chapter 2; Chapter 5 references it.
- Atomic refusal. Non-Cmn, AbsoluteHz, or an out-of-range result refuses the
whole operation. Never saturate, never partially apply. Tombstoned and
SYSTEM_DERIVED targets are still skipped: a deleted pitch is not an
untransposable pitch, it is one the operation has nothing to say about.
targets becomes CanonicalSet<PitchId> at the type level, not a Vec plus a
dedup() someone can forget (PitchId's Ord is its canonical byte order). This
was never a convergence bug -- every replica replaying [p, p] double-
transposes identically -- but a canonicalization one. It is free today because
no operation-payload decoder exists yet; once one lands in Push 5, dedup
normalization would change the meaning of stored operations. Push 4a blocks
Push 5, and that is why.
Spec: req:pitch:transposition (algebra + the three refusals), and four
req:opcat:transpose-* requirements. Operation Catalog 0.7.0 -> 0.8.0; Binary
Format 0.6.0 -> 0.7.0 (disc 30, and a seq-strictly-increasing notation whose
decoder must reject a duplicate rather than normalize it away).
This commit is the design gate: the spec now declares MUSTs the code does not
yet satisfy -- editor-core still authors Transpose, and TransposeInterval does
not exist. The implementation follows in this push.
Also recorded: the two existing transpose tests are false locks. Gutting
graph_transpose_pitch leaves both green -- they call base-free reduce(), where
graph is None and the function never runs, and assert only OperationEffect.
Only editor-core's undo_and_redo_a_transpose, three crates away, catches it.
Gate: clippy 0, 30 targets / 964 passed / 0 failed, docs 0 under -D warnings,
conformance 8/8, all three spec documents build with no undefined references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CreateRepeatStructure / DeleteRepeatStructure enter the ratified
operation set — the dedicated pair for RepeatStructure (repeats live in
the cross-cutting registry but are deliberately NOT CrossCuttingValue
wire kinds; the reconciliation note follows Ch6's CrossCuttingStructure
listing).
operation_catalog (0.6.0 -> 0.7.0):
- New K0 §"Repeat Structures": the full six-part schema for the pair.
Set-union mint with the ALL-anchors-live precondition (start/end, the
kind's jump targets, each volta's span — the mint must leave the graph
satisfying reference-resolution invariants; dead anchor =>
TargetMissing no-op); live-id re-create reads AlreadyApplied without
value comparison (cross-cutting discipline; RecreateContentMismatch
scope unchanged); delete-wins tombstone; create-undo tombstones the
mint, delete-undo does not restore (P11-C8); volta well-formedness
stays advisory.
- K1 gains the "Added in the schema-major-2 revision" entry (net-new
primitives, never drafted as slots).
binary_format (0.5.0 -> 0.6.0):
- OperationKind wire table appends 28 (lp(RepeatStructure)) / 29 (bare
identifier); OperationKindTag 28/29, name-verbatim projection;
requirement bounds move past 29.
- Honest per-op stamping ratified: the CREATE is born at v2 (kind/voltas
are unconditional fields — no lower-major payload layout exists, so
every block carrying one stamps major 2 under minimal stamping); the
DELETE's bare-id payload is a major-0 layout, so minimal stamping
gives its blocks major 0 — the kind discriminant itself being a
schema-minor vocabulary append (mechanism claim only; the stamp always
follows minimal stamping, as discriminant 24's always-2 CreateStaff
shows).
- §Schema Major 2 restructured: RepeatStructure moves from the
snapshot-only bullet to the canonical operation layer (eight embedding
payloads now); the CanonicalValue seam grows to twenty-three
(RepeatStructure joins — CreateRepeatStructure embeds it per
req:catalog:value-encoding).
core_spec:
- Ch6 OperationKind + Ch8 OperationKindTag listings gain the variants;
the CrossCuttingStructure/wire-vocabulary reconciliation note; the
re-anchoring rule table gains the "Repeat structure / Anchor" row
(re-anchor to the nearest surviving anchor across EVERY
event-referencing anchor site; cascade-delete only when none
survives — the spanner discipline); Ch8's schema-versioning paragraph
now enumerates eight embedding payloads with the born-at-v2/major-0
split; revision-history row.
Review-hardened: five-dimension workflow review, 22 findings, 9
confirmed + 1 recovered from a failed verifier (an API-error casualty,
assessed by hand) — all fixed (the load-bearing ones: Ch8's stale
seven-payload enumeration; the mis-attributed "Phase-3 precedent" gloss
that would have taught a false stamping rule; the pair filed under
"Snapshot-only"; the CanonicalValue seam omission). All three documents
compile clean, zero undefined references.
Code tranche follows this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEs4aYiu8MXjdYdMxw8PTd
The first phase of the first-ever binary-format schema-major bump (v0 -> v1),
the machinery-first minimal major. Spec-only: ratifies the contract that
Phases B-F build against; no code changes.
core_spec.tex: defines the two referenced-but-undefined types that were the
P12-I7 / P12-K7 gaps -- CanvasLayoutDefaults (with core geometry primitives
CanvasSize/CanvasMargins in staff spaces via CanonicalF64, A4/8mm default,
since core has no geometry types and must not depend on layout-ir) and
PitchRange (advisory pitch compass used by Instrument.range and
IndeterminacyHints) -- and adds Region.permits_spanning_slurs (default false).
Records schema major 1 as the first data-model expansion major and tightens
the minor-version rule (a field add, even Option, is major; minor = append
discriminants to the companion's append-safe vocabularies only).
binary_format.tex -> 0.3.0: the full "Schema Major 1" section --
- Where the changed fields reach: Canvas.layout_defaults and
Instrument.range are snapshot-only (no CreateCanvas/CreateInstrument op),
but Region.permits_spanning_slurs also reaches the CANONICAL CreateRegion
operation payload (CreateRegion embeds the full Region). The canonical-base
MaterializedState embeds none of these and stays major 0, byte-identical.
- Cross-major reader rules: discard-and-regenerate non-canonical chunks;
parse-or-read-only for canonical ones, so a major-0 reader opens a bundle
carrying v1 CreateRegion ops read-only.
- Accept-set gate [min,max] (rejects majors outside the set); per-payload-
type major assignment; the changed v1 value layouts (the wire form ratifies
the reduced reference-code layout, not the fuller data model); the total
default-filling v0->v1 migration table (including the CreateRegion payload).
- Length-prefix unification NARROWED to the resolved-layout (its own
non-canonical LayoutCache): the barrier/extension blobs stay regime (b) u64
because they ride the canonical manifest, which stays major 0.
Two review passes hardened this checkpoint. The first caught that Region is a
canonical operation payload (not cache-only, as the architecture analysis had
assumed) -- user chose to embrace it and build the canonical op-payload
migration. It also surfaced the barrier-blobs-in-manifest constraint that
narrows the unification. The second refined the minor-version delegation, the
accept-set outside-[min,max] semantics, and stale "no defined type" text in
the reference-suite / quality-metric companions and the engrave DECISIONS.
P12-I7 moved to IN PROGRESS (spec type defined here; code graph home lands in
Phase C). Both companions and the engrave DECISIONS reworded accordingly. All
four affected PDFs rebuild clean (0 undefined references).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEs4aYiu8MXjdYdMxw8PTd
The chosen Phase-3 attack, run as two parallel waves. 829 workspace
tests pass; clippy -D warnings, fmt, and rustdoc clean; all three
spec documents build with zero undefined references.
Casting-off (epiphany-engrave/src/casting.rs, wired into the
Engraver):
- Greedy first-fit system breaking per region at measure-start
barline columns; a measure that would overflow the content width
starts a new system. Hard SystemBreakAt/PageBreakAt always
honoured; soft breaks honoured unless pathological (skipped with
the spec's warning + IrOverride-recorded decision).
- Vertical stacking from real content extents with the inter-system
gap read from the vertical band model; page overflow starts the
next page. World frame: pages stacked vertically, coordinates
baked into glyphs/strokes, so the SVG renderer, hit-testing, and
the GUI viewport are unchanged.
- Real ResolvedPage/ResolvedSystem trees (1-based page numbers,
content bounding boxes, staves from staff-line segments, measures
from barline columns); every chosen break appends an
EngravingDecision with MUSCLOID EngravedBreak provenance,
UserOverride-attributed via the new ConstrainedLayoutIR
break_origins sidecar; staff lines split per system with
synthesized continuation provenance.
- Break-constraint evaluation flips: satisfied iff the layout breaks
at the slot. The two single-system tests invert deliberately
(a hard break is now honoured; a user break is honoured and
attributed instead of warned). Geometric constraints evaluate in
the pre-casting spaced frame (documented).
- Page geometry is engraver-side PageGeometry (A4 portrait at an
8 mm staff: page 105 x 148.5 staff spaces, margins 7.5, content
90 x 133.5; arithmetic documented) — Canvas.layout_defaults has no
graph home and is a schema-major addition (P12-I7).
ENGRAVER_VERSION = 2. Goldens regenerated: ten_measure_single_staff
engraves as 2 systems (viewBox 84x20.6, was 103x11);
valid_score_rich as 3 systems; stub goldens byte-identical.
K1 schema-fill (Operation Catalog 0.4.0 -> 0.5.0, ratified first;
wire discriminants strictly appended):
- CreateStaff (24 / tag InsertStaff 24): set-union mint of a global
Staff; CreateStaffInstance now preconditions that its referenced
staff is live.
- SetTimeSignature (25): value-carrying meter-change LWW keyed by
(region, resolved position); the carried TimeSignature mints
set-union; StructuralFieldCollision on meter_sequence.
- SetTempoSegment (26): LWW keyed by (scope, resolved start) over
the score or region tempo map; a write that would malform the map
refuses with the appended PreconditionFailureReason 11
(TempoMapMalformed).
- SetStaffLayout (27): LWW advisory over the staff instance's three
inline layout fields.
- Create score/canvas remain deliberately unavailable slots: the
root and canvas are inline singletons with no addressable object
model (P12-K8), not force-designed.
Value-restoring undo (P11-C8 narrowed; catalog §UndoTransaction
rewritten and per-primitive undo notes updated):
- Canonical-order write chains (base-seeded) across all eleven LWW
families. StrictInverse restores each written key to its
chain-predecessor value iff the transaction's write is still the
key's last writer, else refuses the whole undo with a
TransactionConflict naming the superseder; BestEffort restores the
still-last keys. Clean compensations are Applied; only minted-
object tombstone repairs ride AppliedWithRepair (no new repair
vocabulary). Mixed mint+overwrite transactions compose; strand
guards refuse tombstoning mints still referenced by live
non-members.
- Undo-of-undo pinned and tested: restorations are chain writes, so
undoing the undo's transaction restores the undone value, and a
second undo of the same transaction conflicts (absence-restores
repeat idempotently — documented asymmetry, P12-K11).
- Permutation invariance pinned across five delivery orders; the
convergence generators gain the new ops and a tx-then-undo flow.
- Still deferred in normative text: delete resurrection (needs a
system-derived tag outside the ratified closed set), Transpose
inversion (P12-K2), Cascade dependent closure.
Trackers: Binary Format companion 0.1.0 -> 0.2.0 (appended wire/tag
tables, PreconditionFailureReason 11, payload layouts, history row —
a schema-minor evolution under its own rules); nine new Pass-12 rows
(C5, K8-K11, I7-I10); core-spec OperationKind listing gains the four
kinds; revision-history rows in core spec and companion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEs4aYiu8MXjdYdMxw8PTd
The audit's fourth push: the biggest outstanding Phase-2 item plus the
performance gate. 793 workspace tests pass; clippy -D warnings, fmt,
and rustdoc (deny-warnings) clean; all three spec documents build with
zero undefined references.
Binary Format companion (spec/binary_format.tex, v0.1.0 — Agent J's
deliverable, 43 pages):
- Twelve chapters transcribed from the golden-locked implementation:
encoding conventions (the three prefix/endianness regimes, a
normative no-varint rule, reject-never-normalize decode discipline),
identifiers imported from the core spec's Canonical Byte-Layout
Reference, primitive value encodings, the whole-Score positional
codec ratified as the schema-major-0 wire form, operation wire
forms (envelope field order with the normative id-leads property,
the OperationPayload 0..=3 and OperationKind 0..=23 tables,
effects/conflict/anomaly/MaterializedState vocabulary), the bundle
physical layout (64-byte header, 256-byte superblock, chunk
preimages and framing, ChunkRef, manifest body order), the
operation-index payload, and the extension-blob/edit-barrier byte
forms.
- Ratifies P12-D1 (req:binfmt:opindex), P12-E1 (req:binfmt:ext-blobs),
P12-E2 (req:binfmt:condition-depth, MAX_CONDITION_DEPTH = 64
normative), and P12-E3 (req:binfmt:object-kind-open) — batch rows
struck through; discharges the provisional-codec notes in core
(P11-4), ops, and bundle (P11-D2/D4/D5) DECISIONS with ratification
cross-references.
- Pins the frozen-layout schema-evolution keystone: within schema
major 0 every positional struct layout is frozen; a field-set change
is a schema-major change with migration — formally grounding the
data-model-expansion staging decision. Open questions kept honest
in-document: SnapshotId derivation, index-refresh threshold, u64/u32
prefix unification at the next major.
- Not yet delivered from J's charter: the cross-implementation decoder
test and the wire-format fuzzer (follow-up harnesses).
F1 benches (crates/epiphany-testkit/benches/, per the F0 decision):
- criterion 0.5.1 (workspace dev-dependency; MSRV 1.77 respected with
documented transitive pins: clap 4.5.53, half 2.4.1).
- reduction bench at 1K/10K/50K envelopes with the Chapter-10 budget
(>10,000 envelopes/second cold) written in the bench as a Pass/Xfail
gate; bundle benches for the typical-edit commit (<=50 ms; measured
~14.7 ms on real disk after catching that tmpfs neuters fsync) and
the open/bootstrap read (<=200 ms; measured ~60 us).
- CI: quick budget gates in the conformance job, full gates nightly.
Subquadratic canonical_reduction_order (the F-surfaces/K-fixes
handshake, closing K's 10K-envelope acceptance gate):
- The bench documented the failure (50K at ~1.7K env/s, a 29 s cold
reduction; two O(n^2) loops); the fix replaces pair enumeration with
threshold/frontier readiness per replica plus explicit-dot dependent
lists and a stamp-tuple binary heap — O((n + sum(context)) log n),
never materializing covered pairs.
- Byte-identical order: same edge relation, same ready predicate, same
total order; the old implementation is retained as a test-only
oracle with element-for-element order-equality property tests over
fuzz sets, adversarial sets, and directed shapes (2,000-envelope
full-coverage chains, dot cycles, duplicate-id stamp ties),
mutation-tested for sensitivity.
- Measured: 1K 155K->674K env/s, 10K 12.5K->257K, 50K 1.7K->87K; all
three scale points now pass and the 50K row is promoted from Xfail.
Also: fixed nine rustdoc private/unresolved intra-doc links that had
accumulated across the pushes (the CI deny-doc-warnings job would have
failed on them).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEs4aYiu8MXjdYdMxw8PTd
Levi Neuwirth