200 lines
11 KiB
Markdown
200 lines
11 KiB
Markdown
# epiphany-testkit
|
|
|
|
Agent F's crate per [`spec/QUICKSTART.md`](../../spec/QUICKSTART.md): the
|
|
cross-cutting conformance testkit. It is the architecture's tripwire — the suite
|
|
that proves the other crates work end to end and that runs in CI (see
|
|
[`.github/workflows/ci.yml`](../../.github/workflows/ci.yml)).
|
|
|
|
It provides:
|
|
|
|
- **Deterministic property-test generators** for the public types of A
|
|
(`epiphany-determinism`), B (`epiphany-core`), C (`epiphany-ops`), D
|
|
(`epiphany-bundle`), and E (`epiphany-layout-ir`). Agent B's score-graph
|
|
generators/shrinkers are re-exported as `generators::graph`.
|
|
- **The canonical round-trip harness** (`roundtrip`) — v0 acceptance criterion 4
|
|
(typed values + bundle container; the bookkeeping `MaterializedState` round-trip
|
|
is retained as `assert_reduction_serialization_stable`).
|
|
- **The CRDT convergence harness** (`convergence`) — criteria 1 and 5. Criterion
|
|
1 proper is **real-Score** convergence through `reduce_onto`
|
|
(`run_graph_convergence`); the byte-canonical bookkeeping-projection
|
|
convergence (`assert_convergence`) backs criterion 5.
|
|
- **The equivocation harness** (`equivocation`) — criterion 3.
|
|
- **The crash-recovery harness** (`bundle_harness`) — Agent D's gate, criterion 2.
|
|
- **The manifest-selection harness** (`bundle_harness`).
|
|
- **The layout round-trip harness** (`layout_stub`) — criterion 6.
|
|
- **The audit regression guards** (`negative`) — one guard per defect the Agent C
|
|
framework audit surfaced (the M1 fixes), so a regression trips this suite
|
|
directly.
|
|
|
|
## All harnesses are real
|
|
|
|
The QUICKSTART charters Agent F to *"build against A and stubs for the others."*
|
|
All five implementation crates — A, B, C, D, and now E (`epiphany-layout-ir`) —
|
|
have shipped, so every harness drives the **real** crate.
|
|
|
|
| Harness | Backend | Status |
|
|
|---------|---------|--------|
|
|
| `roundtrip` (criterion 4) | A + B + C + D, real | **real** |
|
|
| `bundle_harness` (criterion 2, manifest selection) | D, real | **real** |
|
|
| `convergence` (criteria 1, 5) | C (`epiphany-ops`), real | **real** |
|
|
| `equivocation` (criterion 3) | C (`epiphany-ops`), real | **real** |
|
|
| `layout_stub` (criterion 6) | E (`epiphany-layout-ir`), real | **real** |
|
|
|
|
For criteria 1, 3, and 5 the testkit drives the real
|
|
`epiphany_ops::OperationSet` / `canonical_reduction_order` / reduce and also
|
|
re-exports Agent C's own authoritative gates
|
|
(`convergence::ops_reduction_determinism_fuzz`,
|
|
`equivocation::ops_equivocation_fuzz`). The `layout_stub` module — once a
|
|
faithful in-tree stub of Chapters 7 & 9 — now re-exports the real
|
|
`epiphany-layout-ir` IR types and stub solver behind the same `round_trip`
|
|
signature; the provenance-preservation contract is implemented and tested inside
|
|
that crate. (The "stub" in the module name now refers to the spec-sanctioned
|
|
*stub constraint solver*, not to a stubbed crate.)
|
|
|
|
## Criterion 1: real-Score vs. reducer-bookkeeping convergence
|
|
|
|
Criterion 1 proper (`convergence::run_graph_convergence`, the acceptance
|
|
`criterion_1_convergence` test) is **real-Score** convergence: a real ~50-bar,
|
|
two-voice base `epiphany_core::Score` is edited by two replicas through
|
|
`OperationSet::reduce_onto`, and the entire materialized graph — arena, voices,
|
|
tombstones, cross-cutting, *and* the bookkeeping state — must be **identical**
|
|
under every delivery order, pass `check_invariants`, and genuinely grow both
|
|
edited voices (non-vacuity). The session targets the base's actual voice ids
|
|
(`generators::graph_edit_session`), so it exercises the integration point, not a
|
|
synthetic id space.
|
|
|
|
The earlier, narrower gate is retained and honestly renamed
|
|
(`reducer_bookkeeping_convergence`): it converges the byte-canonical
|
|
**bookkeeping projection** (`OperationSet::reduce` →
|
|
`MaterializedState::canonical_bytes`) — the Chapter 6 §6.3 ledger (effects,
|
|
conflicts, anomalies, tombstones, spellings, pending), not the full musical
|
|
graph. It still backs criterion 5 and proves causal-first ordering
|
|
(`convergence::assert_causal_order_respected`,
|
|
`run_authoritative_reduction_gate`). The bookkeeping two-staff scenario remains
|
|
*instantiated* — a real ~50-bar (`TWO_STAFF_BARS`) session whose staves are
|
|
asserted populated by `generators::assert_two_staff_populated`, not just modeled.
|
|
|
|
## Criterion 4: what is and isn't tested
|
|
|
|
Criterion 4 has three tiers — two asserted now, one pending item 5:
|
|
|
|
- **Real decode round-trips** (these catch decoder / canonicalization defects):
|
|
the generic `CanonicalEncode`/`CanonicalDecode` property swept across every
|
|
typed identifier, both `RationalTime` arms, and every `TypedObjectId`
|
|
discriminant; the bundle `Manifest` (`encode → decode → encode` fixpoint, with
|
|
a *rich* generator exercising snapshots, blobs, extensions, varied profiles,
|
|
retention, and the optional roots); the `FixedHeader`; and the `Superblock`
|
|
slot encoding. Crucially, the **decoders are shown to validate**: corrupting a
|
|
manifest or header makes `decode` *reject* it
|
|
(`assert_manifest_decode_rejects_corruption`,
|
|
`assert_header_decode_rejects_corruption`).
|
|
|
|
- **A reducer-bookkeeping serialization tier** (`reducer_bookkeeping_serialization`,
|
|
via `assert_reduction_serialization_stable`): a real `OperationSet` is reduced
|
|
to its `MaterializedState::canonical_bytes()` — the canonical *bookkeeping*
|
|
state, **not** the whole musical `Score` — which is stored as a `Snapshot`
|
|
chunk referenced by the manifest's `canonical_base`, survives the bundle's
|
|
content-addressed store (hash-verified on reopen), decodes through
|
|
`MaterializedState::decode_canonical`, compares structurally with the original
|
|
reduction, and re-serializes byte-identically. The decoder validates nested
|
|
tags, lengths, primitive values, canonical form, and trailing bytes. Musical
|
|
sensitivity is proven two ways:
|
|
`assert_content_mutation_changes_serialization` (a cloned operation set with
|
|
**identical** ids/stamps/causal contexts but one changed payload reduces to
|
|
*different* bytes — the rebuttal to an id-only serializer) and
|
|
`assert_distinct_scores_serialize_differently`. The materialized real `Score`
|
|
itself is shown reproducible today (`full_score_materialization_is_reproducible`,
|
|
structural equality across delivery orders) — the determinism precondition a
|
|
byte codec depends on.
|
|
|
|
- **The full-`Score` byte round-trip** (`criterion_4_full_score_byte_roundtrip`,
|
|
via `assert_score_serialization_stable`): item 5's whole-score codec
|
|
(`epiphany_core::Score::canonical_bytes` / `decode_canonical`) has landed, so a
|
|
real ~50-bar `Score` — materialized through Agent C's `reduce_onto` — now
|
|
`encode → decode → re-encode`s byte-identically through a real bundle snapshot
|
|
(hash-verified on reopen), with the decoded `Score` structurally equal to the
|
|
original. This is the whole musical graph (arena, voices, regions,
|
|
cross-cutting, tombstones), not the bookkeeping projection.
|
|
|
|
## Decisions (per QUICKSTART "Make each one once and document it")
|
|
|
|
1. **No platform entropy in the harness.** Appendix D §"Randomness" forbids
|
|
platform entropy in canonical state; the testkit holds itself to the stronger
|
|
rule that *no* platform entropy enters the harness at all. Everything draws
|
|
from `rng::Rng` (a wrapper over Agent A's vendored SplitMix64, with unbiased
|
|
bounded draws and an overflow-safe full-range `range`), so every failure
|
|
reproduces from its seed.
|
|
2. **Drive the real crate once it ships; stub only what hasn't landed.** Earlier
|
|
in development `epiphany-ops` (C) and `epiphany-layout-ir` (E) were in-flight
|
|
and their harnesses ran against faithful in-tree stubs; now that both have
|
|
shipped, every harness drives the real crate and re-exports its gates.
|
|
|
|
## Flagged for a future spec pass (Pass 11 candidates)
|
|
|
|
Per the QUICKSTART, implementation-discovered gaps are batched, not improvised:
|
|
|
|
- **Whole-graph (`epiphany_core::Score`) wire format — landed (item 5).** A
|
|
direct canonical byte codec for the core `Score` now exists
|
|
(`epiphany_core::Score::canonical_bytes` / `decode_canonical`), and
|
|
`criterion_4_full_score_byte_roundtrip` exercises it on a real `reduce_onto`
|
|
materialization through a bundle snapshot. The prototype byte form predates the
|
|
Binary Format companion specification and is to be reconciled with it (see
|
|
`epiphany-core/DECISIONS.md`, P11-4).
|
|
- **Layout harness re-pointed.** `epiphany-layout-ir` has landed, so `layout_stub`
|
|
now drives the real IR types behind the same `round_trip` signature (done). IR
|
|
coordinates are f32 staff spaces, quantized only when serializing canonical
|
|
`ResolvedLayoutIR` (Appendix D); see that crate's `DECISIONS.md` for the
|
|
remaining layout-specific Pass 11 candidates (the `OperationKindTag` variant set
|
|
and the layout-object id derivation).
|
|
|
|
## Performance benches (Chapter 10 budgets, worklist F1)
|
|
|
|
`benches/` holds the criterion benches for the spec's measurable Chapter 10
|
|
budgets (see `DECISIONS.md` F0 for why they live in this crate, F1 for every
|
|
call made). Criterion measures; the **budget gate** (`src/budget.rs`) asserts:
|
|
each bench's `main()` ends by re-timing every budget row and exiting nonzero if
|
|
a `Pass`-marked row misses its threshold. Known-pending rows are marked
|
|
`Xfail(reason)` *in the bench source* next to the numeric budget — a miss is
|
|
reported and tolerated, and a pass prints a loud promotion notice so stale
|
|
markings cannot linger. This is the "F surfaces, K fixes" handshake, and its
|
|
inaugural round has completed: the bench documented the reducer's O(n²)
|
|
`canonical_reduction_order` failure at scale, and Agent K's subquadratic
|
|
rewrite (see `epiphany-ops/DECISIONS.md`) flipped the xfail row to `Pass`.
|
|
|
|
| row | budget (spec Chapter 10) | expectation |
|
|
|-----|--------------------------|-------------|
|
|
| `reduction/1000` | > 10,000 envelopes/s, cold | Pass (~674K env/s measured) |
|
|
| `reduction/10000` | > 10,000 envelopes/s, cold | Pass (~257K env/s measured) |
|
|
| `reduction/50000` | > 10,000 envelopes/s, cold | Pass (~87K env/s measured; promoted from Xfail by Agent K's reducer fix — was ~1.7K env/s) |
|
|
| `bundle/typical_edit_commit` | ≤ 50 ms (append + manifest + superblock flip, fsync'd) | Pass (~15 ms) |
|
|
| `bundle/open_bootstrap_read` | ≤ 200 ms (manifest + bootstrap chunks) | Pass (moderate-corpus stand-in) |
|
|
|
|
```sh
|
|
# Full run (includes the 50K cold-reduction point, ~0.6 s per iteration):
|
|
cargo bench -p epiphany-testkit
|
|
|
|
# The reduced CI shape: smaller sampling, 50K point skipped (PR CI runs this):
|
|
EPIPHANY_BENCH_QUICK=1 cargo bench -p epiphany-testkit
|
|
```
|
|
|
|
The gate is a calibrated median over a few iterations, deliberately not the
|
|
spec's p99-over-1000-iterations conformance methodology (that is the reference
|
|
suite's job; the deviation is documented in `src/budget.rs`).
|
|
|
|
## Running
|
|
|
|
```sh
|
|
# Unit + acceptance tests (a meaningful slice, under the cargo test timeout):
|
|
cargo test -p epiphany-testkit
|
|
|
|
# The full conformance suite at scale, outside the test timeout (includes Agent
|
|
# A's 1,000,000-iteration determinism gate and Agent C's reduction/equivocation
|
|
# gates):
|
|
cargo run --release -p epiphany-testkit --example conformance_suite # scale 1
|
|
cargo run --release -p epiphany-testkit --example conformance_suite 10 # soak
|
|
cargo run --release -p epiphany-testkit --example conformance_suite 0 # smoke
|
|
```
|
|
|
|
The six v0 acceptance criteria are asserted in `tests/acceptance.rs`, one test
|
|
per architecture layer.
|