epiphany/crates/epiphany-ops
Levi Neuwirth 3e91a8302a Push 4: Binary Format companion, F1 benches, subquadratic reduction order
The audit's fourth push: the biggest outstanding Phase-2 item plus the
performance gate. 793 workspace tests pass; clippy -D warnings, fmt,
and rustdoc (deny-warnings) clean; all three spec documents build with
zero undefined references.

Binary Format companion (spec/binary_format.tex, v0.1.0 — Agent J's
deliverable, 43 pages):
- Twelve chapters transcribed from the golden-locked implementation:
  encoding conventions (the three prefix/endianness regimes, a
  normative no-varint rule, reject-never-normalize decode discipline),
  identifiers imported from the core spec's Canonical Byte-Layout
  Reference, primitive value encodings, the whole-Score positional
  codec ratified as the schema-major-0 wire form, operation wire
  forms (envelope field order with the normative id-leads property,
  the OperationPayload 0..=3 and OperationKind 0..=23 tables,
  effects/conflict/anomaly/MaterializedState vocabulary), the bundle
  physical layout (64-byte header, 256-byte superblock, chunk
  preimages and framing, ChunkRef, manifest body order), the
  operation-index payload, and the extension-blob/edit-barrier byte
  forms.
- Ratifies P12-D1 (req:binfmt:opindex), P12-E1 (req:binfmt:ext-blobs),
  P12-E2 (req:binfmt:condition-depth, MAX_CONDITION_DEPTH = 64
  normative), and P12-E3 (req:binfmt:object-kind-open) — batch rows
  struck through; discharges the provisional-codec notes in core
  (P11-4), ops, and bundle (P11-D2/D4/D5) DECISIONS with ratification
  cross-references.
- Pins the frozen-layout schema-evolution keystone: within schema
  major 0 every positional struct layout is frozen; a field-set change
  is a schema-major change with migration — formally grounding the
  data-model-expansion staging decision. Open questions kept honest
  in-document: SnapshotId derivation, index-refresh threshold, u64/u32
  prefix unification at the next major.
- Not yet delivered from J's charter: the cross-implementation decoder
  test and the wire-format fuzzer (follow-up harnesses).

F1 benches (crates/epiphany-testkit/benches/, per the F0 decision):
- criterion 0.5.1 (workspace dev-dependency; MSRV 1.77 respected with
  documented transitive pins: clap 4.5.53, half 2.4.1).
- reduction bench at 1K/10K/50K envelopes with the Chapter-10 budget
  (>10,000 envelopes/second cold) written in the bench as a Pass/Xfail
  gate; bundle benches for the typical-edit commit (<=50 ms; measured
  ~14.7 ms on real disk after catching that tmpfs neuters fsync) and
  the open/bootstrap read (<=200 ms; measured ~60 us).
- CI: quick budget gates in the conformance job, full gates nightly.

Subquadratic canonical_reduction_order (the F-surfaces/K-fixes
handshake, closing K's 10K-envelope acceptance gate):
- The bench documented the failure (50K at ~1.7K env/s, a 29 s cold
  reduction; two O(n^2) loops); the fix replaces pair enumeration with
  threshold/frontier readiness per replica plus explicit-dot dependent
  lists and a stamp-tuple binary heap — O((n + sum(context)) log n),
  never materializing covered pairs.
- Byte-identical order: same edge relation, same ready predicate, same
  total order; the old implementation is retained as a test-only
  oracle with element-for-element order-equality property tests over
  fuzz sets, adversarial sets, and directed shapes (2,000-envelope
  full-coverage chains, dot cycles, duplicate-id stamp ties),
  mutation-tested for sensitivity.
- Measured: 1K 155K->674K env/s, 10K 12.5K->257K, 50K 1.7K->87K; all
  three scale points now pass and the 50K row is promoted from Xfail.

Also: fixed nine rustdoc private/unresolved intra-doc links that had
accumulated across the pushes (the CI deny-doc-warnings job would have
failed on them).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEs4aYiu8MXjdYdMxw8PTd
2026-07-02 19:02:07 -04:00
..
examples A B C D F 2026-06-19 12:42:31 -04:00
src Push 4: Binary Format companion, F1 benches, subquadratic reduction order 2026-07-02 19:02:07 -04:00
tests Pushes 1+3: fix the MUST-level violations, wire the types-only machinery 2026-07-02 17:10:50 -04:00
Cargo.toml A B C D F 2026-06-19 12:42:31 -04:00
DECISIONS.md Push 4: Binary Format companion, F1 benches, subquadratic reduction order 2026-07-02 19:02:07 -04:00
README.md Land M1 + M2 (Agent C): framework edge fixes and real-Score graph integration 2026-06-21 16:37:51 -04:00

README.md

epiphany-ops

The Epiphany concurrent semantics: the operations through which the score graph becomes a live model, and the deterministic reduction by which a set of operations becomes a materialized score state. Implements the normative requirements of Chapter 6 (Semantic Operations and Concurrent Reduction) of the core specification (spec/core_spec.pdf). This is Agent C's crate per spec/QUICKSTART.md, building on Agent A's epiphany-determinism and Agent B's epiphany-core.

A score's state is defined by the set of operations committed to it. Any materialized graph is a deterministic reduction of that set; caches, snapshots, and partial reductions are acceleration structures, never the source of truth. — Chapter 6, Design Principles

The thesis in one paragraph

The replicated operation set is a grow-only CRDT: replicas accumulate envelopes and converge on the same set. The materialized graph is not a CRDT — it is the deterministic reduction of that set in a single canonical order (causal-first, then the HLC tuple). Any permutation of the same envelopes reduces to byte-identical materialized state. That property is the determinism heart of the architecture, and the reduction fuzzer is its tripwire.

What's here

Area Items Spec
Stamps HybridLogicalClock, OperationStamp, the reduction & monotonicity tuples Ch. 6 §"Operation Identity and Stamps"
Causal context CausalContext (dotted version vector), covers, the missing-predecessor signal Ch. 6 §6.2
Payloads OperationKind, the discriminator-only OperationKindTag, OperationPayload, the §6.10 representative ops Ch. 6 §"Operation Envelopes", §6.10
Envelopes OperationEnvelope, EnvelopeHash (MUSCENVH), well_formed (incl. stamp.id == id) Ch. 6 §6.4
Slots OperationSlot::{Single, Equivocated}, the order-independent (Pass-10) transitions Ch. 6 §6.5
Anomalies AnomalousReplicaSegment, IntegrityAnomaly/Kind, the HLC-monotonicity detector Ch. 6 §6.6; Ch. 5 §"System-Derived Counter Collisions"
Effects OperationEffect, NoOpReason, the typed PreconditionFailureReason, RepairRecord/RepairKind Ch. 6 §6.3.2, §6.5
Conflicts ConflictRecord, ConflictKind, content-derived ConflictId (derive_conflict_id), the registry, resolution Ch. 6 §6.4
Transactions / undo TransactionDescriptor with the causal-prior-descriptor rule, UndoTransactionPayload / UndoPolicy Ch. 6 §6.6, §6.8
Operation set OperationSet: accept pipeline (well-formedness → slot → causal), grow-only Ch. 6 §"Envelope Acceptance"
Reduction canonical_reduction_order (single function), MaterializedState, the reduction driver Ch. 6 §6.3

The determinism this crate enforces

  1. A single reduction-order function. canonical_reduction_order performs deterministic causal topological ordering, using the intrinsic stamp tuple (physical, logical, replica, counter) only among ready operations.
  2. Order-independent equivocation. A duplicate OperationId with different canonical bytes transitions its slot to Equivocated regardless of which envelope arrived first (Pass 10). Equivocated slots contribute nothing to reduction; dependents are held pending.
  3. Content-derived facts. ConflictId and IntegrityAnomalyId are derived from content, so two replicas reducing the same set agree on every conflict and anomaly id — the conflict registry and anomaly register are deterministic materialized facts, not local bookkeeping.
  4. Byte-identical materialized state. MaterializedState::canonical_bytes serializes the effect log, conflict registry, anomaly register, object existence, spellings, and LWW fields in their normative orders.
  5. Real graph materialization. OperationSet::reduce_onto(&base_score) returns GraphMaterialization { state, score }. The graph is mutated in the same canonical order and compares by canonical event identity, independent of arena storage order.

Hand-off gates

Run the gate harnesses (QUICKSTART, Agent C):

cargo test -p epiphany-ops
cargo run --release -p epiphany-ops --example fuzz_reduction          # 10k iters, seed 0
cargo run --release -p epiphany-ops --example fuzz_reduction 100000 7 # soak, seed 7
  • Reduction determinism — every randomized envelope set reduces to byte-identical materialized state under any acceptance order (v0 acceptance criteria 1 and 5).
  • Equivocation order-independence — every duplicate-id-with-different-bytes scenario equivocates regardless of arrival order (v0 acceptance criterion 3).

The integration tests (tests/concurrent_reduction.rs) exercise these plus transaction atomicity, descriptor precedence, anomaly exclusion, and forward undo through the public API.

Scope and decisions

Chapter 6 specifies the framework and a representative selection of operations; the full ~6080-primitive catalog is an explicit open question (§6.11) deferred to the Operation Catalog companion. This crate implements the framework in full and the representative operations, which is sufficient to exercise every reduction discipline. The representative operations can also reduce onto an epiphany_core::Score: insert/delete, voice promotion, supported cross-cutting structures, system breaks, migration checks, transaction rollback, and undo mutate the real graph while preserving Agent B's invariants. reduce() remains the base-free CRDT/bookkeeping API; reduce_onto() is the graph-aware editing path. See DECISIONS.md for remaining payload boundaries.

Per QUICKSTART "Don't do these": undo is the spec's forward compensating operation, never inverse-based; unsafe is forbidden; everything is sync.