10 KiB

Raw Blame History

Text Projection — the document layer: scope and plan

Status: scoping complete, two rulings needed before dispatch. Prepared against master @ fdb4a57. Every claim was checked against the code; where I ran a probe I say so.

This is the last phase. Chapter-5 values (cf81074) and Chapter-6 operations (fdb4a57) already project and parse strictly. What remains is the document around them: the header, identity, profile, extension, canonical-base and blob lines — and then the two round-trip equations over a whole bundle.

1. Decisions needed

Ruling A — no blob can be canonical today. What should the projector do?

req:textproj:canonical-blobs and core_spec §"Canonical and Non-Canonical Manifest Roots" agree exactly: a blob is canonical iff it is referenced by a canonical operation or by canonical reduced state. A blob referenced only by acceleration structures is non-canonical and MUSTNOT be projected. Canonicality is by reachability, not by membership in blob_roots.

I grepped epiphany-core and epiphany-ops for BlobId: no hit. No operation payload and no Chapter-5 value carries a blob reference. There is no mechanism by which a blob can be reached from canonical state, so today the correct projection of every real bundle emits zero (blob ...) lines, and the production is future-proofing.

The dangerous wrong answer is projecting manifest.blob_roots wholesale — it looks obviously right, is one line of code, and violates the requirement by emitting non-canonical blobs.

Recommendation. Implement the reachability predicate as a real function that today provably returns the empty set, and prove it rather than asserting it: a test that fails the moment any core or ops type gains a BlobId field, so the hook is implemented when it is first needed rather than silently skipped. Same discipline as computing a bound instead of spelling it. The (blob ...) projection and parse must still be written and tested against a synthetic document, because the parser must accept texts that a future writer emits.

Ruling B — the header version is the companion version, and that couples every byte

The companion says the header names "the version of this companion the text conforms to." So a projection written today starts (text-projection (0 6 0)).

Two consequences worth accepting deliberately:

The companion's own worked example currently reads (text-projection (0 3 0)) — stale. My conformance lock covers the envelope line of that example, not the header above it.
Every companion version bump changes the first line of every projection, so req:textproj:canonical-text is version-relative and the text vector corpus must be regenerated on each bump.

Recommendation. Keep the coupling — a text that does not say which rules it follows cannot be strictly parsed against them — fix the stale example, and make the corpus regeneration a documented, one-command step so the cost is mechanical rather than a surprise. If instead you want the header to track a slower-moving format version independent of editorial revisions, that is a spec change and should be decided now, not after a corpus exists.

2. What I verified

A new crate is unavoidable, and the orphan rule does not bite. epiphany-bundle depends only on epiphany-determinism — not on epiphany-core. So TextValue (a core trait) cannot be implemented for Manifest, BlobRef, ProfileDeclaration and friends from any third crate.

That would have been a real problem, except: no document-line production contains a value position. Every one bottoms out in leaves — bytes, integer, bool, string, option — plus the closed vocabularies profile-id, chunk-kind, schema. So the document layer needs free functions, not trait impls, exactly as the grammar-directed operation layer did. No orphan problem, no new dependency edge, and epiphany-bundle stays independent of the music model as designed.

epiphany-textproj depends on core (for Sexp, render, read_sexp), ops (for project_envelope/parse_envelope and canonical_reduction_order), and bundle.

Parse cannot produce a Manifest. The text carries payloads inline — the canonical base's root chunk payload, each extension's preserved chunk payloads, each blob's payload — whereas a Manifest holds ChunkRefs with offsets and lengths that the text deliberately erases. The parse output needs its own type (§4), and the bundle writer synthesizes the physical layer.

The write path already does the synthesis. Bundle::create + commit assigns offsets, content-addresses and de-duplicates chunks, and derives every ChunkId/ContentHash. That is precisely what req:textproj:derive-or-carry assumes, so serialization stages payloads and lets commit do the rest.

Envelope ordering is available. epiphany_ops::canonical_reduction_order is public and exported; Bundle::read_operation_block yields raw envelope bytes for decode_envelope.

Accelerators are dropped, by design. operation_index_root, acceleration_snapshots, text_projection_root, integrity_root and operation_block_summaries are non-canonical and are not projected. A bundle that goes text → binary comes back without them. That is correct and should be stated in a doc comment, because it looks like data loss and is not.

3. The round trip, stated precisely

req:textproj:roundtrip gives two equations. Both hold; neither says bytes survive.

  semantics(parse(project(B))) = semantics(B)     -- weaker
  project(serialize(parse(T))) = T                -- byte-checkable, the corpus test

Going bundle → text → bundle is deliberately lossy in the physical layer and in three specific ways an implementer will otherwise read as bugs:

Duplicate blobs collapse. req:textproj:derived-ordering de-duplicates blob lines and extension chunk roots by projected form. Two byte-identical blobs stored twice are one blob; that is the erasure working.
Layout is regenerated. Offsets, compression, block splitting, chunk count.
Accelerators vanish, as above.

All three preserve semantics(...), and none of them affects the second equation, which quantifies over texts, not bundles. Write the corpus against the second equation.

4. Work breakdown

New crate crates/epiphany-textproj. Three waves; file boundaries are one-per-agent.

Wave 1 — scaffold and types (blocks everything; small)

Cargo.toml, lib.rs, and the parse-output type. Roughly:

pub struct TextDocument {
    pub document_id: DocumentId,
    pub lineage_id: Option<LineageId>,
    pub profiles: Vec<ProfileDeclaration>,
    pub extensions: Vec<TextExtension>,      // chunk payloads INLINE
    pub canonical_base: Option<TextCanonicalBase>,  // root payload INLINE
    pub blobs: Vec<TextBlob>,                // payloads INLINE
    pub envelopes: Vec<OperationEnvelope>,   // canonical operation order
}

The Text* types exist because the corresponding bundle types carry ChunkRefs the text erases. Do not try to reuse SnapshotRef/ExtensionDeclaration verbatim.

Wave 2 — three parallel agents

project.rs — Bundle<S> → TextDocument → text. Reads chunk and blob payloads through the bundle, decodes envelopes, sorts by canonical_reduction_order, and applies derived-ordering's sort-and-dedup by projected form to blob lines and extension chunk roots.
parse.rs — text → TextDocument, strictly. Line-oriented: each line is one complete s-expression (req:textproj:envelope-per-line), and the line order is itself constrained by the projection production — header, document, lineage?, profile*, extension*, canonical-base?, blob*, envelope*. A out-of-order or repeated section is a rejection, not a reordering.
serialize.rs — TextDocument → Bundle<S>. Stages the base root chunk, extension chunks, blobs and one operation-envelope block; builds the manifest; commits. Block splitting is a free physical choice — one block is fine and simplest.

Wave 3 — conformance

Text vectors in the shape of spec/vectors/decode_vectors.txt, drift-locked and gated as conformance step [7e], plus the two round-trip equations over the generated-score corpus. This is also where the stale (0 3 0) header in the companion's example gets fixed and locked.

5. Traps

(blob ...) is unreachable from real data. Its tests need a synthetic document. See Ruling A.
Section order is normative. The projection production is a sequence, not a set. A parser that accepts lines in any order and sorts them is normalizing.
derived-ordering sorts by projected form, not by any binary key — order and de-duplicate on the rendered UTF-8 bytes of each line, which is the whole point (a chunk's offset must not decide the text's order).
The extension line is easy to get subtly wrong. Six fields in ratified declaration order — id, version, required, chunks, affected-kinds, barriers — and affected_object_kinds/edit_barriers are opaque byte strings, never structured. The Vec<u8>-is-not-a-sequence trap that bit the ops layer lives here too.
ProfileId::Custom carries a 16-byte registry id and is the one profile that is not a bare symbol.
Do not project manifest.blob_roots. See Ruling A; it is the plausible wrong answer.
Two WallClockDuration types exist — one in epiphany-core, one in epiphany-bundle. The retention policy uses the bundle's.

6. Verification contract

The standing gate, plus:

Mutation-verify every check, anchor asserted before substitution, and report per-check with output.
Exercise every outbound normalization — the contract rule added after five such normalizations shipped untested in the ops layer. derived-ordering's sort-and-dedup is exactly this shape: build a document with duplicate blobs and out-of-order chunk roots in memory and prove the projection collapses and orders them.
Assert the corpus's own reach. A round-trip suite that never exercises an extension, a canonical base, or a multi-envelope document proves less than its green tick suggests. State the counts.

10 KiB Raw Blame History