Text Projection: the document layer, and the equation closes

`epiphany-textproj` projects a bundle to its canonical text and reads it back:
the header, identity, profile, extension, canonical-base and blob productions,
plus the pipeline that joins them to the operation layer already in place.

Free functions, not `TextValue` impls. `epiphany-bundle` does not depend on
`epiphany-core`, so the trait and the manifest types are both foreign to this
crate and the orphan rule forbids the impls. That looked like it would force a new
dependency edge until the productions were read properly: no document line
contains a `value` position, so none is needed. The bundle stays independent of
the music model, as designed.

Blobs are handled on both sides and neither is the obvious one. The emit side
filters `blob_roots` through a real reachability predicate that today provably
returns empty -- never the field wholesale, which is the plausible wrong answer.
The parse side rejects any blob line outright, while still parsing the production
in isolation so both halves are ready together. A trip-wire source-scans
epiphany-core and epiphany-ops for `BlobId` and names both obligations when it
fires; it asserts it actually scanned files, so it cannot pass vacuously.

Two rejections were added on the agents' own reading of the spec rather than the
brief, and both are right. Envelopes must already be in canonical reduction order
(text_projection.tex:458) -- without that, two texts differing only in envelope
order would denote one document. And a hand-built `TextDocument` carrying blobs is
refused rather than silently dropped, since serializing it would produce a bundle
whose next projection loses those bytes.

Section order is enforced as a sequence, not a set: a repeated or out-of-order
section is a rejection, never something the parser sorts back into place.
`derived-ordering` sorts and de-duplicates blob lines and extension chunk roots by
*rendered* form, proved against fixtures that are unsorted in memory rather than
already canonical -- the blind spot that let five outbound normalizations ship
untested in the operation layer.

Serialization stages payloads and lets `commit` assign every offset, hash and
chunk id, which is exactly what `req:textproj:derive-or-carry` assumes; nothing is
hashed by hand. The emitted bundle deliberately carries no accelerators.

And the three directions compose: `project(serialize(parse(T))) == T` holds
end-to-end over the companion's own worked example. Eight checks re-mutated
independently of the agents' reports, each killed by its own named test; all eight
document productions diffed against the grammar.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Levi Neuwirth 2026-07-21 15:05:28 -04:00
parent 50ad97a31b
commit 244bd6bd96
10 changed files with 3438 additions and 0 deletions

10
Cargo.lock generated
View File

@ -1286,6 +1286,16 @@ dependencies = [
"epiphany-ops",
]
[[package]]
name = "epiphany-textproj"
version = "0.0.0"
dependencies = [
"epiphany-bundle",
"epiphany-core",
"epiphany-determinism",
"epiphany-ops",
]
[[package]]
name = "equivalent"
version = "1.0.2"

View File

@ -5,6 +5,7 @@ members = [
"crates/epiphany-core",
"crates/epiphany-bundle",
"crates/epiphany-ops",
"crates/epiphany-textproj",
"crates/epiphany-layout-ir",
"crates/epiphany-engrave",
"crates/epiphany-render-svg",
@ -44,6 +45,9 @@ epiphany-bundle = { path = "crates/epiphany-bundle" }
# testkit drives its convergence, reduction-determinism, and equivocation gates,
# so its path is declared here alongside the other intra-workspace crates.
epiphany-ops = { path = "crates/epiphany-ops" }
# The Text Projection companion is the document-level bridge between canonical
# bundles and their normative s-expression form.
epiphany-textproj = { path = "crates/epiphany-textproj" }
# Agent E's epiphany-layout-ir is the layout IR + constraint-solver interface
# (Chapters 7 & 9). Agent F's testkit drives its layout round-trip gate
# (v0 acceptance criterion 6), so its path is declared here alongside the other

View File

@ -0,0 +1,14 @@
[package]
name = "epiphany-textproj"
version = "0.0.0"
edition.workspace = true
rust-version.workspace = true
authors.workspace = true
repository.workspace = true
description = "The Epiphany Text Projection companion: the canonical s-expression document representation and its strict bundle round trip."
[dependencies]
epiphany-determinism.workspace = true
epiphany-core.workspace = true
epiphany-bundle.workspace = true
epiphany-ops.workspace = true

View File

@ -0,0 +1,117 @@
#![forbid(unsafe_code)]
//! The Epiphany Text Projection companion document layer.
//!
//! This crate bridges canonical bundle documents and the normative line-oriented
//! s-expression representation defined by `spec/text_projection.tex`.
pub mod parse;
pub mod project;
pub mod serialize;
use epiphany_bundle::{
ChunkKind, DocumentId, ExtensionId, FrontierBytes, LineageId, ProfileDeclaration, ProfileId,
ReductionAlgorithmVersion, SchemaVersion, SemVer, SnapshotId,
};
use epiphany_ops::OperationEnvelope;
/// The one Text Projection companion version implemented by this crate.
///
/// A parser must reject every other version rather than migrating or
/// normalizing it on read.
pub const COMPANION_VERSION: (u32, u32, u32) = (0, 7, 0);
/// A parsed canonical Text Projection document.
///
/// Under `req:textproj:derive-or-carry`, this representation deliberately erases
/// physical offsets, compressed and uncompressed storage lengths, and
/// compression choices because serialization is free to choose a new physical
/// layout. It also omits the derivable `ChunkId`, `ContentHash`, and `BlobId`;
/// those identities are recomputed from kind, schema, and inline payload.
/// Finally, it drops the non-canonical `operation_index_root`,
/// `acceleration_snapshots`, `text_projection_root`, `integrity_root`, and
/// `operation_block_summaries` accelerators. None contributes to canonical
/// document semantics, so a bundle serialized from this form correctly rebuilds
/// or omits them rather than carrying stale physical metadata.
#[derive(Debug, PartialEq)]
pub struct TextDocument {
/// Logical identity of the projected document.
pub document_id: DocumentId,
/// Optional shared-ancestor identity used for document genealogy.
pub lineage_id: Option<LineageId>,
/// Profile declarations in canonical manifest order.
pub profiles: Vec<ProfileDeclaration>,
/// Extension declarations with every preserved chunk payload inline.
pub extensions: Vec<TextExtension>,
/// Optional canonical base with its snapshot root payload inline.
pub canonical_base: Option<TextCanonicalBase>,
/// Canonically reachable blobs with their payloads inline.
pub blobs: Vec<TextBlob>,
/// Operation envelopes in canonical reduction order.
pub envelopes: Vec<OperationEnvelope>,
}
/// An extension declaration in its text-document form.
///
/// Unlike the bundle's `ExtensionDeclaration`, this type carries preserved
/// chunks as semantic kind/schema/payload triples, not physical `ChunkRef`s.
#[derive(Debug, PartialEq)]
pub struct TextExtension {
/// Opaque identity of the extension.
pub extension_id: ExtensionId,
/// Semantic version of the extension declaration.
pub version: SemVer,
/// Whether an implementation unaware of the extension must refuse editing.
pub required: bool,
/// Preserved extension chunks, inline and ordered by projected form.
pub chunks: Vec<TextChunk>,
/// Canonical opaque encoding of affected object kinds.
pub affected_object_kinds: Vec<u8>,
/// Canonical opaque encoding of the extension's edit barriers.
pub edit_barriers: Vec<u8>,
}
/// One preserved extension chunk with all physical reference data erased.
#[derive(Debug, PartialEq)]
pub struct TextChunk {
/// Semantic role of the chunk.
pub kind: ChunkKind,
/// Schema version governing the payload bytes.
pub schema_version: SchemaVersion,
/// Uncompressed chunk payload carried inline.
pub payload: Vec<u8>,
}
/// A canonical base snapshot in its text-document form.
///
/// The snapshot identity is carried because it is opaque, while the root chunk
/// identity and content hash are derived from its schema and inline payload.
#[derive(Debug, PartialEq)]
pub struct TextCanonicalBase {
/// Opaque snapshot identity, carried verbatim.
pub snapshot_id: SnapshotId,
/// Opaque causal frontier materialized by the snapshot.
pub covers_causal_frontier: FrontierBytes,
/// Reduction algorithm version used to produce the snapshot.
pub reduction_algorithm_version: ReductionAlgorithmVersion,
/// Profile under which the snapshot was produced.
pub profile_id: ProfileId,
/// Schema version of the snapshot root chunk.
pub root_schema_version: SchemaVersion,
/// Uncompressed snapshot root payload carried inline.
pub root_payload: Vec<u8>,
}
/// A canonical blob in its text-document form.
///
/// The payload is inline; its bundle `BlobId`, content hash, offset, lengths,
/// and compression metadata are deliberately absent and are derived or chosen
/// when serialized.
#[derive(Debug, PartialEq)]
pub struct TextBlob {
/// RFC 6838 media type.
pub media_type: String,
/// Optional declared maximum uncompressed size.
pub declared_max_uncompressed_length: Option<u64>,
/// Uncompressed blob payload carried inline.
pub payload: Vec<u8>,
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,561 @@
//! Serialization of parsed Text Projection documents into canonical bundles.
//!
//! [`serialize_document`] stages the payloads a [`TextDocument`] carries inline
//! — the canonical base's root chunk, every extension's preserved chunks, and
//! one operation-envelope block — and lets [`Bundle::create`] /
//! [`Bundle::commit`] do everything `req:textproj:derive-or-carry` says a
//! serializer must not: assign offsets, content-address every chunk, and
//! de-duplicate identical content. **Nothing here computes a `ChunkId`, a
//! `ContentHash`, or an offset** — every physical field written into the
//! manifest comes straight from the [`ChunkRef`](epiphany_bundle::ChunkRef)s
//! `commit` hands back.
//!
//! # Block splitting is a free physical choice
//!
//! `req:textproj:roundtrip` only requires the *bundle's* physical layout to
//! round-trip in the second, byte-checkable equation (`project(serialize(parse(T)))
//! == T`, quantified over texts, not bundles); how a serializer packs envelopes
//! into blocks is unconstrained. This module always emits **exactly one**
//! operation-envelope block, containing every envelope the document carries (in
//! the order the document carries them) — the simplest possible choice, and
//! sufficient because block boundaries are storage artifacts, never semantic
//! structure (Chapter 8: *"the set of envelopes is the union of all envelopes
//! across all referenced blocks"*).
//!
//! # No accelerator is (re)written
//!
//! A [`TextDocument`] carries no `operation_index_root`, `acceleration_snapshots`,
//! `text_projection_root`, `integrity_root`, or `operation_block_summaries` —
//! they are non-canonical and the companion text does not carry them
//! (`req:textproj:derive-or-carry`). The manifest this module builds leaves every
//! one of those fields at its empty/`None` default. A bundle serialized from a
//! `TextDocument` therefore comes back from this module *without* any
//! accelerator a previous generation might have had. That reads as data loss; it
//! is not — none of those fields contributes to canonical document semantics,
//! and a consumer that wants them back rebuilds them the same way any bundle
//! writer does (e.g. `epiphany_bundle::fuzz` and the testkit's operation-index
//! harness both scan-and-rebuild rather than trust a carried-forward index).
//!
//! # No blob is staged
//!
//! [`TextDocument::blobs`] is never staged into `manifest.blob_roots` here. Today
//! that vector is always empty: no operation payload or Chapter-5 value in
//! `epiphany-core`/`epiphany-ops` carries a `BlobId`, so no blob is reachable from
//! canonical state (`req:textproj:canonical-blobs`), and this companion's own
//! parser rejects every `(blob ...)` line rather than populate the vector (see
//! `parse`). [`serialize_document`] still checks: a `TextDocument` assembled
//! directly against the type (bypassing `parse`) with a populated `blobs` vector
//! is refused with [`SerializeError::NonEmptyBlobs`] rather than silently
//! dropped — wiring a blob into the manifest here would require a canonical
//! `declared_max_uncompressed_length`/compression policy this companion version
//! does not specify, and silently discarding the caller's bytes is worse.
use std::fmt;
use epiphany_bundle::{
encode_block, BlockStore, Bundle, BundleError, ChunkKind, CommitContext, ExtensionDeclaration,
FileUuid, Manifest, SchemaVersion, SnapshotRef, StagedChunk,
};
use epiphany_determinism::CanonicalEncode;
use epiphany_ops::OperationEnvelope;
use crate::TextDocument;
/// An error [`serialize_document`] cannot recover from.
#[derive(Debug)]
pub enum SerializeError {
/// The document's `blobs` vector is non-empty. See the module documentation:
/// no blob is canonical today, so a document built by this companion's
/// `parse` never carries one, and staging one here would either wire a
/// non-canonical root into the manifest or silently drop the caller's
/// payload bytes. Neither is acceptable, so serialization refuses instead.
NonEmptyBlobs,
/// The bundle itself rejected the manifest or a staged chunk — e.g. the
/// document declares no profile this implementation understands, or a
/// staged root fails `Bundle::commit`'s structural validation.
Bundle(BundleError),
}
impl fmt::Display for SerializeError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
SerializeError::NonEmptyBlobs => f.write_str(
"the document carries a populated blobs vector, but no blob can be canonical today",
),
SerializeError::Bundle(error) => {
write!(f, "the bundle rejected the serialized document: {error}")
}
}
}
}
impl std::error::Error for SerializeError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
SerializeError::Bundle(error) => Some(error),
SerializeError::NonEmptyBlobs => None,
}
}
}
impl From<BundleError> for SerializeError {
fn from(error: BundleError) -> Self {
SerializeError::Bundle(error)
}
}
/// Serializes a [`TextDocument`] into a freshly created bundle over `store`.
///
/// Two-phase, because [`Bundle::create`] requires a manifest with no canonical
/// roots or blobs (there is nothing to reference before any chunk is written):
/// this creates an empty generation-0 bundle carrying only the document's
/// identity and profile declarations, then stages every payload the document
/// carries inline and commits once, building the real manifest from the
/// [`ChunkRef`](epiphany_bundle::ChunkRef)s that commit assigns.
///
/// Returns [`SerializeError::NonEmptyBlobs`] if `document.blobs` is non-empty
/// (see the module documentation), or [`SerializeError::Bundle`] if the bundle
/// itself refuses the manifest or a staged root.
pub fn serialize_document<S: BlockStore>(
document: &TextDocument,
store: S,
file_uuid: FileUuid,
) -> Result<Bundle<S>, SerializeError> {
if !document.blobs.is_empty() {
return Err(SerializeError::NonEmptyBlobs);
}
let mut bundle = Bundle::create(store, file_uuid, empty_manifest(document))?;
let mut staged = Vec::new();
if let Some(base) = &document.canonical_base {
// The canonical base's root chunk: kind and schema version are carried
// verbatim from the document (they are not derivable from anything),
// and its payload is staged exactly as the document carries it. `commit`
// computes the chunk's id, hash, and offset.
staged.push(StagedChunk {
kind: ChunkKind::Snapshot,
schema_version: base.root_schema_version,
payload: base.root_payload.clone(),
});
}
for extension in &document.extensions {
for chunk in &extension.chunks {
staged.push(StagedChunk {
kind: chunk.kind,
schema_version: chunk.schema_version,
payload: chunk.payload.clone(),
});
}
}
staged.push(stage_operation_envelope_block(document));
bundle.commit(&staged, |ctx| build_manifest(document, ctx))?;
Ok(bundle)
}
/// The manifest [`Bundle::create`] is given: the document's identity and
/// declared profiles, and nothing else. `create` itself rejects a manifest
/// carrying canonical roots or blobs, so every root is added by the subsequent
/// commit (see [`build_manifest`]). If `document.profiles` is empty or
/// otherwise unemittable, `Bundle::create` reports that itself — this function
/// does not invent a fallback profile the document did not declare.
fn empty_manifest(document: &TextDocument) -> Manifest {
let mut manifest = Manifest::empty(document.document_id);
manifest.lineage_id = document.lineage_id;
manifest.profile_declarations = document.profiles.clone();
manifest
}
/// Encodes every envelope the document carries, in the order it carries them,
/// into a single operation-envelope block payload (see the module
/// documentation on why one block is the right choice here). The block's
/// schema version is the max over its envelopes' `schema_major` — never a fixed
/// baseline — so a block that carries a higher-major payload (e.g. a v1
/// `CreateRegion` or a v2 cross-cutting value) is never mis-stamped major 0,
/// mirroring `StagedChunk::operation_block_versioned`'s own contract.
fn stage_operation_envelope_block(document: &TextDocument) -> StagedChunk {
let payloads: Vec<Vec<u8>> = document
.envelopes
.iter()
.map(CanonicalEncode::to_canonical_bytes)
.collect();
let major = document
.envelopes
.iter()
.map(OperationEnvelope::schema_major)
.max()
.unwrap_or(0);
StagedChunk::operation_block_versioned(encode_block(&payloads), SchemaVersion::for_major(major))
}
/// Builds the committed manifest from the previous (empty) manifest and the
/// [`ChunkRef`](epiphany_bundle::ChunkRef)s [`Bundle::commit`] assigned to the
/// chunks [`serialize_document`] staged, in the same order it staged them: the
/// canonical base's root (if any), then each extension's preserved chunks in
/// turn, then the single operation-envelope block. Every physical field
/// (`ChunkId`, `ContentHash`, offset) comes from `ctx.new_chunks` — nothing here
/// recomputes one (`req:textproj:derive-or-carry`). `operation_index_root`,
/// `acceleration_snapshots`, `text_projection_root`, `integrity_root`,
/// `operation_block_summaries`, and `blob_roots` are left untouched at the
/// previous manifest's empty defaults; see the module documentation for why
/// that is correct rather than lossy.
fn build_manifest(document: &TextDocument, ctx: &CommitContext) -> Manifest {
let mut manifest = ctx.previous_manifest.clone();
let mut cursor = 0usize;
if let Some(base) = &document.canonical_base {
let root = ctx.new_chunks[cursor];
cursor += 1;
manifest.canonical_base = Some(SnapshotRef {
snapshot_id: base.snapshot_id,
covers_causal_frontier: base.covers_causal_frontier.clone(),
reduction_algorithm_version: base.reduction_algorithm_version,
profile_id: base.profile_id,
root,
hash: root.hash,
});
}
let mut extension_declarations = Vec::with_capacity(document.extensions.len());
for extension in &document.extensions {
let mut preserved_chunk_roots = Vec::with_capacity(extension.chunks.len());
for _ in &extension.chunks {
preserved_chunk_roots.push(ctx.new_chunks[cursor]);
cursor += 1;
}
extension_declarations.push(ExtensionDeclaration {
extension_id: extension.extension_id,
version: extension.version,
required: extension.required,
preserved_chunk_roots,
affected_object_kinds: extension.affected_object_kinds.clone(),
edit_barriers: extension.edit_barriers.clone(),
});
}
manifest.extension_declarations = extension_declarations;
manifest.operation_roots = vec![ctx.new_chunks[cursor]];
cursor += 1;
debug_assert_eq!(
cursor,
ctx.new_chunks.len(),
"every chunk staged by serialize_document must be wired into the manifest exactly once"
);
manifest
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{TextCanonicalBase, TextChunk, TextExtension};
use epiphany_bundle::{
DocumentId, ExtensionId, FrontierBytes, LineageId, MemStore, ProfileConstraints,
ProfileDeclaration, ProfileId, ReductionAlgorithmVersion, SemVer, SnapshotId,
};
use epiphany_determinism::fuzz::SplitMix64;
use epiphany_ops::{decode_envelope, fuzz::gen_envelope_set};
/// A real, varied envelope set (every operation kind the generator reaches),
/// so the schema-major derivation and the envelope round trip both have
/// something to bite on.
fn envelopes(seed: u64, n: usize) -> Vec<OperationEnvelope> {
let mut rng = SplitMix64::new(seed);
gen_envelope_set(&mut rng, n)
}
/// A profile declaration distinguishable from `ProfileDeclaration::full()`'s
/// default version, so a test that forgets to carry `document.profiles`
/// through cannot pass by accident (`full()` would still be a `Full`
/// profile, just at a different version).
fn base_profile() -> ProfileDeclaration {
ProfileDeclaration {
profile_id: ProfileId::Full,
version: SemVer::new(0, 2, 0),
constraints: ProfileConstraints::DEFAULT_FULL,
}
}
fn minimal_document(seed: u64) -> TextDocument {
TextDocument {
document_id: DocumentId([seed as u8; 16]),
lineage_id: None,
profiles: vec![base_profile()],
extensions: Vec::new(),
canonical_base: None,
blobs: Vec::new(),
envelopes: envelopes(seed, 5),
}
}
fn document_with_extension(seed: u64) -> TextDocument {
let mut document = minimal_document(seed);
document.extensions.push(TextExtension {
extension_id: ExtensionId([7; 16]),
version: SemVer::new(1, 0, 0),
required: false,
chunks: vec![
TextChunk {
kind: ChunkKind::ExtensionData,
schema_version: SchemaVersion::V0,
payload: b"chunk-a".to_vec(),
},
TextChunk {
kind: ChunkKind::ExtensionData,
schema_version: SchemaVersion::V0,
payload: b"chunk-b".to_vec(),
},
],
affected_object_kinds: vec![0xAA],
edit_barriers: vec![0xBB, 0xCC],
});
document
}
fn document_with_canonical_base(seed: u64) -> TextDocument {
let mut document = minimal_document(seed);
document.canonical_base = Some(TextCanonicalBase {
snapshot_id: SnapshotId([9; 16]),
covers_causal_frontier: FrontierBytes(vec![1, 2, 3]),
reduction_algorithm_version: ReductionAlgorithmVersion(1),
profile_id: ProfileId::Full,
root_schema_version: SchemaVersion::V0,
root_payload: b"snapshot-root".to_vec(),
});
document
}
/// Carries an extension, a canonical base, and many envelopes at once — the
/// document every round-trip law needs to be checked against together.
fn rich_document(seed: u64) -> TextDocument {
let mut document = document_with_extension(seed);
document.canonical_base = document_with_canonical_base(seed).canonical_base;
document.envelopes = envelopes(seed, 60);
document
}
fn serialize_and_reopen(document: &TextDocument) -> Bundle<MemStore> {
let bundle = serialize_document(document, MemStore::new(), FileUuid([1; 16]))
.expect("a well-formed document serializes");
let image = bundle.into_store().into_bytes();
Bundle::open(MemStore::from_bytes(image)).expect("the serialized bundle reopens")
}
#[test]
fn document_identity_round_trips() {
let mut document = minimal_document(1);
document.lineage_id = Some(LineageId([2; 16]));
let reopened = serialize_and_reopen(&document);
assert_eq!(reopened.manifest().document_id, document.document_id);
assert_eq!(reopened.manifest().lineage_id, document.lineage_id);
}
#[test]
fn profile_declarations_round_trip() {
let document = minimal_document(2);
let reopened = serialize_and_reopen(&document);
assert_eq!(reopened.manifest().profile_declarations, document.profiles);
}
#[test]
fn missing_profile_declaration_surfaces_the_bundles_own_rejection() {
// serialize_document must not invent a fallback profile the document
// did not declare: an empty `profiles` propagates Bundle::create's own
// "no declared profile" rejection.
let mut document = minimal_document(3);
document.profiles.clear();
let result = serialize_document(&document, MemStore::new(), FileUuid([1; 16]));
assert!(matches!(result, Err(SerializeError::Bundle(_))));
}
#[test]
fn canonical_base_round_trips_snapshot_id_and_payload() {
let document = document_with_canonical_base(4);
let reopened = serialize_and_reopen(&document);
let expected = document
.canonical_base
.as_ref()
.expect("fixture carries a canonical base");
let base = reopened
.manifest()
.canonical_base
.as_ref()
.expect("canonical base present after reopen");
assert_eq!(base.snapshot_id, expected.snapshot_id);
assert_eq!(base.covers_causal_frontier, expected.covers_causal_frontier);
assert_eq!(
base.reduction_algorithm_version,
expected.reduction_algorithm_version
);
assert_eq!(base.profile_id, expected.profile_id);
assert_eq!(base.root.kind, ChunkKind::Snapshot);
assert_eq!(base.root.schema_version, expected.root_schema_version);
let payload = reopened
.read_chunk(&base.root)
.expect("root chunk reads and verifies");
assert_eq!(payload, expected.root_payload);
}
#[test]
fn extension_round_trips_fields_and_chunk_payloads() {
let document = document_with_extension(5);
let reopened = serialize_and_reopen(&document);
let expected = &document.extensions[0];
let declaration = reopened
.manifest()
.extension_declarations
.iter()
.find(|d| d.extension_id == expected.extension_id)
.expect("extension declaration present after reopen");
assert_eq!(declaration.version, expected.version);
assert_eq!(declaration.required, expected.required);
assert_eq!(
declaration.affected_object_kinds,
expected.affected_object_kinds
);
assert_eq!(declaration.edit_barriers, expected.edit_barriers);
assert_eq!(
declaration.preserved_chunk_roots.len(),
expected.chunks.len()
);
// The manifest's own canonical encoding sorts preserved_chunk_roots by
// ChunkRef (kind, hash, offset), not by the document's chunk order, so
// compare payload sets rather than assume position survives.
let mut payloads: Vec<Vec<u8>> = declaration
.preserved_chunk_roots
.iter()
.map(|root| reopened.read_chunk(root).expect("preserved chunk reads"))
.collect();
let mut expected_payloads: Vec<Vec<u8>> =
expected.chunks.iter().map(|c| c.payload.clone()).collect();
payloads.sort();
expected_payloads.sort();
assert_eq!(
payloads, expected_payloads,
"preserved chunk payloads survive, order aside"
);
}
#[test]
fn envelopes_round_trip_through_a_single_operation_block() {
let document = minimal_document(6);
assert!(document.envelopes.len() > 1, "fixture reach check");
let reopened = serialize_and_reopen(&document);
assert_eq!(
reopened.manifest().operation_roots.len(),
1,
"one operation-envelope block, by design"
);
let root = reopened.manifest().operation_roots[0];
let payloads = reopened
.read_operation_block(&root)
.expect("operation block reads");
let recovered: Vec<OperationEnvelope> = payloads
.iter()
.map(|bytes| decode_envelope(bytes).expect("canonical envelope decodes"))
.collect();
assert_eq!(recovered, document.envelopes);
}
#[test]
fn operation_block_schema_major_is_derived_not_hardcoded() {
let document = rich_document(7);
let expected_major = document
.envelopes
.iter()
.map(OperationEnvelope::schema_major)
.max()
.unwrap_or(0);
assert!(
expected_major > 0,
"fixture must include a schema-major-bearing operation to exercise the derivation"
);
let reopened = serialize_and_reopen(&document);
let root = reopened.manifest().operation_roots[0];
assert_eq!(
root.schema_version,
SchemaVersion::for_major(expected_major)
);
}
#[test]
fn a_document_with_no_envelopes_still_stages_one_empty_operation_block() {
let mut document = minimal_document(8);
document.envelopes.clear();
let reopened = serialize_and_reopen(&document);
assert_eq!(reopened.manifest().operation_roots.len(), 1);
let root = reopened.manifest().operation_roots[0];
assert!(reopened
.read_operation_block(&root)
.expect("empty operation block still reads")
.is_empty());
}
#[test]
fn nonempty_blobs_are_rejected_rather_than_dropped() {
let mut document = minimal_document(9);
document.blobs.push(crate::TextBlob {
media_type: "audio/wav".to_string(),
declared_max_uncompressed_length: None,
payload: b"nope".to_vec(),
});
let result = serialize_document(&document, MemStore::new(), FileUuid([1; 16]));
assert!(matches!(result, Err(SerializeError::NonEmptyBlobs)));
}
#[test]
fn a_rich_document_round_trips_every_root_at_once() {
let document = rich_document(10);
let reopened = serialize_and_reopen(&document);
assert_eq!(reopened.manifest().document_id, document.document_id);
assert!(reopened.manifest().canonical_base.is_some());
assert_eq!(reopened.manifest().extension_declarations.len(), 1);
assert_eq!(reopened.manifest().operation_roots.len(), 1);
reopened
.verify_canonical_chunks()
.expect("every canonical chunk this module wrote is intact");
assert!(!reopened.is_read_only());
assert!(reopened.anomalies().is_empty());
}
/// `req:textproj` verification discipline item 3: a round-trip suite that
/// never exercises an extension, a canonical base, or a multi-envelope
/// document proves far less than its green tick suggests. Count what this
/// suite actually covers.
#[test]
fn the_suite_exercises_an_extension_a_canonical_base_and_multiple_envelopes() {
let documents = vec![
minimal_document(20),
document_with_extension(21),
document_with_canonical_base(22),
rich_document(23),
];
let with_extension = documents
.iter()
.filter(|d| !d.extensions.is_empty())
.count();
let with_canonical_base = documents
.iter()
.filter(|d| d.canonical_base.is_some())
.count();
let with_multiple_envelopes = documents.iter().filter(|d| d.envelopes.len() > 1).count();
assert!(
with_extension >= 1,
"reach: no document carries an extension"
);
assert!(
with_canonical_base >= 1,
"reach: no document carries a canonical base"
);
assert!(
with_multiple_envelopes >= 1,
"reach: no document carries more than one envelope"
);
for document in &documents {
let reopened = serialize_and_reopen(document);
assert_eq!(reopened.manifest().document_id, document.document_id);
}
}
}

View File

@ -0,0 +1,59 @@
use epiphany_textproj::COMPANION_VERSION;
fn title_page_version(source: &str) -> Result<(u32, u32, u32), String> {
let title_page = source
.split_once("\\begin{titlepage}")
.and_then(|(_, after_start)| after_start.split_once("\\end{titlepage}"))
.map(|(title_page, _)| title_page)
.ok_or_else(|| "companion must contain one complete titlepage environment".to_owned())?;
let candidates: Vec<_> = title_page
.lines()
.filter_map(|line| line.split_once("Version ").map(|(_, suffix)| suffix))
.collect();
if candidates.len() != 1 {
return Err(format!(
"title page must contain exactly one `Version <major>.<minor>.<patch>` anchor; found {}",
candidates.len()
));
}
let token: String = candidates[0]
.chars()
.take_while(|character| character.is_ascii_digit() || *character == '.')
.collect();
let components: Vec<_> = token.split('.').collect();
if components.len() != 3 || components.iter().any(|component| component.is_empty()) {
return Err(format!(
"title-page version must have exactly three numeric components; found `{token}`"
));
}
let parse_component = |component: &str| {
component
.parse::<u32>()
.map_err(|_| format!("invalid numeric component `{component}` in title-page version"))
};
Ok((
parse_component(components[0])?,
parse_component(components[1])?,
parse_component(components[2])?,
))
}
#[test]
fn companion_version_matches_title_page() {
let companion_path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../spec/text_projection.tex"
);
let companion = std::fs::read_to_string(companion_path)
.unwrap_or_else(|error| panic!("failed to read {companion_path}: {error}"));
let title_version = title_page_version(&companion)
.unwrap_or_else(|error| panic!("failed to parse {companion_path}: {error}"));
assert_eq!(
title_version, COMPANION_VERSION,
"title-page companion version differs from epiphany_textproj::COMPANION_VERSION"
);
}

View File

@ -0,0 +1,155 @@
# Contract: the Text Projection document layer
Repo root `/home/jeans/Repos/active/epiphany`. Read this in full before writing a
line. The plan is `spec/PLAN_TEXTPROJ_DOCUMENT.md`; both its rulings are granted
and this contract states them as law.
The two layers below you are done and are your model: `epiphany-core`'s
`textvalue*.rs` (Chapter-5 values) and `epiphany-ops`'s `textproj_*.rs`
(Chapter-6 operations). Read the latter — the document layer is the same shape.
## Grammar-directed, and free functions
`req:textproj:operation-vocabulary` established that the grammar's productions
govern, not the mechanical value rule. The document lines are the same: find your
production in `spec/text_projection.tex`'s Grammar chapter and implement exactly
what it says.
**Do not implement `TextValue` for bundle types.** `epiphany-bundle` does not
depend on `epiphany-core`, so the trait is foreign and so is the type — the
orphan rule forbids it from `epiphany-textproj`. You do not need it: no
document-line production contains a `value` position; every one bottoms out in
`bytes`, `integer`, `bool`, `string`, `option`, or a closed vocabulary. Write
**free functions** — `fn project_profile(&ProfileDeclaration) -> Sexp`,
`fn parse_profile(&Sexp) -> Result<ProfileDeclaration, TextError>`. Use
`epiphany_core::textvalue::{Sexp, read_sexp, TextError}` for the machinery.
Do **not** add `epiphany-core` to `epiphany-bundle`'s dependencies. The bundle is
a container format and its independence from the music model is deliberate.
## Blobs: emit none, and reject them on parse
A blob is canonical **iff** referenced by a canonical operation or by canonical
reduced state (`req:textproj:canonical-blobs`; `core_spec` §"Canonical and
Non-Canonical Manifest Roots"). Nothing in `epiphany-core` or `epiphany-ops`
references a `BlobId` — verified by source scan — so **no blob is canonical
today** and the correct projection of every real bundle emits zero `(blob ...)`
lines.
Three obligations, and they are not the obvious ones:
1. **Emit side.** Implement the reachability predicate as a real function that
today provably returns the empty set. **Never project
`manifest.blob_roots`** — that is the plausible wrong answer: one line, looks
right, emits non-canonical blobs in violation of the requirement. Even
`manifest.rs`'s own doc comment says "canonical `blob_roots`" — the subset, not
the field.
2. **Parse side: reject.** Document-level validation **MUST reject any
`(blob ...)` line** as unreferenced, with a test asserting the rejection.
A blob-bearing text at this companion version is necessarily non-canonical.
Accepting one would stage a blob into the bundle that the next projection
silently drops, which loses data *and* falsifies
`project(serialize(parse(T))) == T` for that text. Forward compatibility is
owned by header-version gating, not by leniency here.
Line-level `project`/`parse` of the `(blob ...)` production must still be
written and unit-tested in **both** directions against synthetic data, so that
when the predicate becomes non-empty both sides are ready at once.
3. **A trip-wire.** A test that source-scans `crates/epiphany-core/src` and
`crates/epiphany-ops/src` for the token `BlobId` and fails if it appears — same
family as the existing drift locks. Its failure message must name the
reachability predicate and tell the finder that both the emit side and the
parse-side rejection now need their real implementation.
## The header carries exactly one version
The header names the version of the companion the text conforms to. The parser
**accepts exactly one**: the companion version this crate implements. Anything
else is a rejection at line one. Multi-version acceptance and migrate-on-read for
text are future spec decisions — do not improvise them.
Define the version once, as a constant, and lock it against the companion with a
test that reads `spec/text_projection.tex` and asserts the title version matches.
A constant that silently disagrees with the document it claims conformance to is
the whole failure mode this project keeps paying for.
## Section order is normative
`projection ::= header document lineage? profile* extension* canonical-base?
blob* envelope*` is a **sequence**, not a set. A repeated or out-of-order section
is a rejection. A parser that accepts lines in any order and sorts them is
normalizing, which `req:textproj:strict-parse` forbids.
## What must not appear (`req:textproj:derive-or-carry`)
* **Physical attributes**`offset`, `compressed_length`, `compression`,
`uncompressed_length`. A serializer chooses them freely.
* **Derivable identities**`ChunkId`, `ContentHash`, `BlobId`. Re-derived from
content.
* The **one** identity carried verbatim is `SnapshotId`, which has nothing to
derive from.
* **Non-canonical accelerators are not projected**: `operation_index_root`,
`acceleration_snapshots`, `text_projection_root`, `integrity_root`,
`operation_block_summaries`. A bundle that round-trips through text comes back
without them. That is correct; say so in a doc comment, because it reads as data
loss and is not.
## `derived-ordering` sorts by projected form
`req:textproj:derived-ordering` applies to exactly two sequences: the `(blob ...)`
lines and an extension's preserved chunk roots. Order **and de-duplicate** them by
the **UTF-8 bytes of their rendered form** — not by any binary key, because the
binary key reads the offset, and a chunk's file position must not decide the
text's order. Every other sequence keeps the binary order.
## Traps
* **The extension line has six fields** in ratified declaration order: id,
version, required, chunks, affected-kinds, barriers. `affected_object_kinds`
and `edit_barriers` are **opaque byte strings**, never structured — the
`Vec<u8>`-is-not-a-sequence trap that bit the operation layer lives here too.
Each preserved chunk root projects as kind + schema_version + **uncompressed
payload**, never as a `ChunkRef`.
* **`ProfileId::Custom`** carries a 16-byte registry id and is the one profile
that is not a bare symbol.
* **Two `WallClockDuration` types exist**, one in `epiphany-core` and one in
`epiphany-bundle`. The retention policy uses the bundle's.
* **`canonical-base` carries the root chunk's uncompressed payload inline**, as a
single byte string, plus the `SnapshotId` verbatim.
* **Envelopes are emitted in `canonical_reduction_order`**
(`epiphany_ops::canonical_reduction_order`, public and exported).
## Style
Match the surrounding code. Doc comments on every public item and every
non-obvious decision — especially *why* a parse rejects rather than normalizes.
`rustfmt` clean, no `clippy` warnings, no `#[allow(...)]`, no `unwrap`/`expect` on
a parse path. Touch only the file you are told to create; do not run
`cargo fmt --all`.
## Verification
The standing gate: `cargo fmt --all --check`; `cargo clippy --workspace
--all-targets` → 0; full workspace tests; `RUSTDOCFLAGS="-D warnings" cargo doc
--workspace --no-deps` → 0; `cargo run -q -p epiphany-testkit --example
conformance_suite`; zero golden churn.
Plus the three this project learned the hard way:
1. **Mutation-verify every check you write.** Assert the anchor is present, delete
or invert the check, confirm a *named* test of yours fails, restore. Report per
check, with the actual command output. A survivor is a finding — say so.
2. **Exercise every outbound normalization.** For every value you normalize on the
way out, construct a non-normalized input and prove the normalization happens.
`derived-ordering` is exactly this shape: build a document with duplicate blobs
and out-of-order chunk roots *in memory* and prove the projection collapses and
orders them. Five such normalizations shipped untested in the operation layer
because every fixture was already sorted.
3. **Assert your suite's own reach.** A round-trip suite that never exercises an
extension, a canonical base, or a multi-envelope document proves far less than
its green tick suggests. Count what you covered and assert the counts.
Report the actual commands and their actual output. A previous agent reported
"verification passes" when errors did point into its own file.

View File

@ -0,0 +1,217 @@
# Text Projection — the document layer: scope and plan
Status: **scoping complete, two rulings needed before dispatch.**
Prepared against `master` @ `fdb4a57`. Every claim was checked against the code;
where I ran a probe I say so.
This is the last phase. Chapter-5 values (`cf81074`) and Chapter-6 operations
(`fdb4a57`) already project and parse strictly. What remains is the document
around them: the header, identity, profile, extension, canonical-base and blob
lines — and then the two round-trip equations over a whole bundle.
---
## 1. Decisions needed
### Ruling A — no blob can be canonical today. What should the projector do?
`req:textproj:canonical-blobs` and `core_spec` §"Canonical and Non-Canonical
Manifest Roots" agree exactly: a blob is canonical **iff** it is *referenced by a
canonical operation or by canonical reduced state*. A blob referenced only by
acceleration structures is non-canonical and `MUSTNOT` be projected. Canonicality
is by **reachability**, not by membership in `blob_roots`.
I grepped `epiphany-core` and `epiphany-ops` for `BlobId`: **no hit.** No
operation payload and no Chapter-5 value carries a blob reference. There is no
mechanism by which a blob can be reached from canonical state, so **today the
correct projection of every real bundle emits zero `(blob ...)` lines**, and the
production is future-proofing.
The dangerous wrong answer is projecting `manifest.blob_roots` wholesale — it
looks obviously right, is one line of code, and violates the requirement by
emitting non-canonical blobs.
**Recommendation.** Implement the reachability predicate as a real function that
today provably returns the empty set, and *prove* it rather than asserting it: a
test that fails the moment any core or ops type gains a `BlobId` field, so the
hook is implemented when it is first needed rather than silently skipped. Same
discipline as computing a bound instead of spelling it. The `(blob ...)`
projection and parse must still be written and tested against a synthetic
document, because the parser must accept texts that a future writer emits.
### Ruling B — the header version is the companion version, and that couples every byte
The companion says the header names *"the version of **this companion** the text
conforms to."* So a projection written today starts `(text-projection (0 6 0))`.
Two consequences worth accepting deliberately:
* The companion's own worked example currently reads `(text-projection (0 3 0))`
**stale**. My conformance lock covers the envelope line of that example, not
the header above it.
* Every companion version bump changes the first line of every projection, so
`req:textproj:canonical-text` is *version-relative* and the text vector corpus
must be regenerated on each bump.
**Recommendation.** Keep the coupling — a text that does not say which rules it
follows cannot be strictly parsed against them — fix the stale example, and make
the corpus regeneration a documented, one-command step so the cost is mechanical
rather than a surprise. If instead you want the header to track a slower-moving
*format* version independent of editorial revisions, that is a spec change and
should be decided now, not after a corpus exists.
---
## 2. What I verified
**A new crate is unavoidable, and the orphan rule does not bite.**
`epiphany-bundle` depends only on `epiphany-determinism`**not** on
`epiphany-core`. So `TextValue` (a core trait) cannot be implemented for
`Manifest`, `BlobRef`, `ProfileDeclaration` and friends from any third crate.
That would have been a real problem, except: **no document-line production
contains a `value` position.** Every one bottoms out in leaves — `bytes`,
`integer`, `bool`, `string`, `option` — plus the closed vocabularies
`profile-id`, `chunk-kind`, `schema`. So the document layer needs **free
functions**, not trait impls, exactly as the grammar-directed operation layer
did. No orphan problem, no new dependency edge, and `epiphany-bundle` stays
independent of the music model as designed.
`epiphany-textproj` depends on core (for `Sexp`, `render`, `read_sexp`), ops (for
`project_envelope`/`parse_envelope` and `canonical_reduction_order`), and bundle.
**Parse cannot produce a `Manifest`.** The text carries **payloads inline** — the
canonical base's root chunk payload, each extension's preserved chunk payloads,
each blob's payload — whereas a `Manifest` holds `ChunkRef`s with offsets and
lengths that the text deliberately erases. The parse output needs its own type
(§4), and the bundle writer synthesizes the physical layer.
**The write path already does the synthesis.** `Bundle::create` + `commit`
assigns offsets, content-addresses and de-duplicates chunks, and derives every
`ChunkId`/`ContentHash`. That is precisely what `req:textproj:derive-or-carry`
assumes, so serialization stages payloads and lets `commit` do the rest.
**Envelope ordering is available.** `epiphany_ops::canonical_reduction_order` is
public and exported; `Bundle::read_operation_block` yields raw envelope bytes for
`decode_envelope`.
**Accelerators are dropped, by design.** `operation_index_root`,
`acceleration_snapshots`, `text_projection_root`, `integrity_root` and
`operation_block_summaries` are non-canonical and are not projected. A bundle
that goes text → binary comes back without them. That is correct and should be
stated in a doc comment, because it looks like data loss and is not.
---
## 3. The round trip, stated precisely
`req:textproj:roundtrip` gives two equations. **Both hold; neither says bytes
survive.**
```
semantics(parse(project(B))) = semantics(B) -- weaker
project(serialize(parse(T))) = T -- byte-checkable, the corpus test
```
Going bundle → text → bundle is **deliberately lossy in the physical layer** and
in three specific ways an implementer will otherwise read as bugs:
1. **Duplicate blobs collapse.** `req:textproj:derived-ordering` de-duplicates
blob lines and extension chunk roots by *projected form*. Two byte-identical
blobs stored twice are one blob; that is the erasure working.
2. **Layout is regenerated.** Offsets, compression, block splitting, chunk count.
3. **Accelerators vanish**, as above.
All three preserve `semantics(...)`, and none of them affects the second
equation, which quantifies over *texts*, not bundles. Write the corpus against
the second equation.
---
## 4. Work breakdown
New crate `crates/epiphany-textproj`. Three waves; file boundaries are
one-per-agent.
### Wave 1 — scaffold and types (blocks everything; small)
`Cargo.toml`, `lib.rs`, and the parse-output type. Roughly:
```rust
pub struct TextDocument {
pub document_id: DocumentId,
pub lineage_id: Option<LineageId>,
pub profiles: Vec<ProfileDeclaration>,
pub extensions: Vec<TextExtension>, // chunk payloads INLINE
pub canonical_base: Option<TextCanonicalBase>, // root payload INLINE
pub blobs: Vec<TextBlob>, // payloads INLINE
pub envelopes: Vec<OperationEnvelope>, // canonical operation order
}
```
The `Text*` types exist because the corresponding bundle types carry `ChunkRef`s
the text erases. Do not try to reuse `SnapshotRef`/`ExtensionDeclaration`
verbatim.
### Wave 2 — three parallel agents
* **`project.rs`** — `Bundle<S>``TextDocument` → text. Reads chunk and blob
payloads through the bundle, decodes envelopes, sorts by
`canonical_reduction_order`, and applies `derived-ordering`'s sort-and-dedup by
projected form to blob lines and extension chunk roots.
* **`parse.rs`** — text → `TextDocument`, strictly. Line-oriented: each line is
one complete s-expression (`req:textproj:envelope-per-line`), and the line
*order* is itself constrained by the `projection` production — header, document,
lineage?, profile*, extension*, canonical-base?, blob*, envelope*. A
out-of-order or repeated section is a rejection, not a reordering.
* **`serialize.rs`** — `TextDocument``Bundle<S>`. Stages the base root chunk,
extension chunks, blobs and one operation-envelope block; builds the manifest;
commits. Block splitting is a free physical choice — one block is fine and
simplest.
### Wave 3 — conformance
Text vectors in the shape of `spec/vectors/decode_vectors.txt`, drift-locked and
gated as conformance step **`[7e]`**, plus the two round-trip equations over the
generated-score corpus. This is also where the stale `(0 3 0)` header in the
companion's example gets fixed and locked.
---
## 5. Traps
* **`(blob ...)` is unreachable from real data.** Its tests need a synthetic
document. See Ruling A.
* **Section order is normative.** The `projection` production is a sequence, not
a set. A parser that accepts lines in any order and sorts them is normalizing.
* **`derived-ordering` sorts by *projected form*, not by any binary key** — order
and de-duplicate on the rendered UTF-8 bytes of each line, which is the whole
point (a chunk's offset must not decide the text's order).
* **The extension line is easy to get subtly wrong.** Six fields in ratified
declaration order — id, version, required, chunks, affected-kinds, barriers —
and `affected_object_kinds`/`edit_barriers` are opaque **byte strings**, never
structured. The `Vec<u8>`-is-not-a-sequence trap that bit the ops layer lives
here too.
* **`ProfileId::Custom` carries a 16-byte registry id** and is the one profile
that is not a bare symbol.
* **Do not project `manifest.blob_roots`.** See Ruling A; it is the plausible
wrong answer.
* **Two `WallClockDuration` types exist** — one in `epiphany-core`, one in
`epiphany-bundle`. The retention policy uses the bundle's.
---
## 6. Verification contract
The standing gate, plus:
* **Mutation-verify every check**, anchor asserted before substitution, and report
per-check with output.
* **Exercise every outbound normalization** — the contract rule added after five
such normalizations shipped untested in the ops layer. `derived-ordering`'s
sort-and-dedup is exactly this shape: build a document with duplicate blobs and
out-of-order chunk roots *in memory* and prove the projection collapses and
orders them.
* **Assert the corpus's own reach.** A round-trip suite that never exercises an
extension, a canonical base, or a multi-envelope document proves less than its
green tick suggests. State the counts.