diff --git a/crates/epiphany-bundle/DECISIONS.md b/crates/epiphany-bundle/DECISIONS.md index 1baac81..94f1c8d 100644 --- a/crates/epiphany-bundle/DECISIONS.md +++ b/crates/epiphany-bundle/DECISIONS.md @@ -549,3 +549,50 @@ shapes are normative; `kind`, `action`, `policy`, `constraints`, `barrier` are derived from the Operation Catalog and the wire table rather than spelled out. That is the difference between a design gate and a finished companion, and it is written into the companion rather than left for a reader to discover. + +### Text Projection 0.2.0 — the gate reopened: canonical-manifest coverage + +A review found the 0.1.0 companion **lossy for documents that are valid today**. +Its claim to preserve the manifest's canonical roots was false in three ways, and +all three had one cause I had not named. + +- **A canonical blob had no representation.** `blob_roots` referenced by canonical + operations or reduced state are canonical roots (`core_spec` §"Canonical and + Non-Canonical Roots"), and the document structure had no blob line. An embedded + image, font, or recording would vanish from a projection *silently* — the + operations referencing it would still be there, pointing at a blob id the text + no longer contained. +- **An `ExtensionDeclaration` lost its semantic version and its + `affected_object_kinds`**, and left its `preserved_chunk_roots` undefined. +- **`ProfileId::Custom(ProfileRegistryId)` was unrepresentable**: the grammar + required a symbol where sixteen registry bytes are carried. + +**The cause: `ChunkRef` and `BlobRef` are physical references.** Offset, +compressed length, compression — exactly what the projection may not preserve. +And they carry derivable identities — `ChunkId`, `ContentHash`, `BlobId` — which +it may not duplicate. Having no rule for that, I dropped the references entirely +and took their contents with them. + +`req:textproj:derive-or-carry` states the rule, and it is the same rule +`req:textproj:reduced-state-derived` already applied one level up: **carry exactly +what the document does not determine, and nothing it does.** Physical attributes +never appear. Derivable identities never appear. Content and semantic attributes +always appear. The sole non-derivable identity in schema major 0 is `SnapshotId`, +which the Binary Format companion pins as opaque and forbids readers to derive — +an exception for a stated reason, not an oversight. + +Consequently: `req:textproj:canonical-blobs`, `req:textproj:profile-id`, +`req:textproj:extension-declaration`, and `req:textproj:base-snapshot-inline` +extended to say what the inlined payload *is* (the canonical byte form of the +reduced state) and that the root `ChunkRef` and the `SnapshotRef.hash` are +**re-derived** from `hash(Snapshot, schema, payload)`, never read from the text. + +**The gap started upstream.** `core_spec`'s own list of what the projection +preserves omitted canonical blobs while classifying `blob_roots` as canonical +roots — an inconsistency inside one document. Corrected there too, along with +withdrawing the permission to reference a base snapshot "externally", which the +inline ratification had already made untenable. + +The 0.1.0 ratifications — reduced state derived, base inlined, hex, one envelope +per line, strict parsing — stand unchanged. Implementation stays deferred: the +companion is a gate, and a gate that is lossy is not one. diff --git a/spec/core_spec.pdf b/spec/core_spec.pdf index a3d79e0..9ba1f67 100644 Binary files a/spec/core_spec.pdf and b/spec/core_spec.pdf differ diff --git a/spec/core_spec.tex b/spec/core_spec.tex index e70dea8..011b1e8 100644 --- a/spec/core_spec.tex +++ b/spec/core_spec.tex @@ -11314,13 +11314,22 @@ This section states \emph{what} is projected; the companion states \emph{how}. \item All operation envelopes (with their identities, stamps, causal contexts, transaction groupings, and payloads). \item All profile declarations. - \item All extension declarations, including their preserved - chunk roots, with extension payloads encoded as base64 or - another canonical text form. - \item The active canonical base snapshot's reference, - frontier, and reduction algorithm version (the snapshot - payload itself \MAY{} be encoded compactly or referenced - externally). + \item All extension declarations \emph{in full} --- identity, semantic + version, required flag, affected object kinds, edit barriers, and + preserved chunk roots --- with opaque payloads in a canonical text + encoding. + \item Every \textbf{canonical blob}: a blob referenced by a canonical + operation or by canonical reduced state is a canonical root of this + document (Section~\ref{sec:format:manifest}), so the projection carries + its media type and its uncompressed payload. Blobs referenced only by + acceleration structures are non-canonical and are not projected. + \item The active canonical base snapshot's identity, frontier, and + reduction-algorithm version, \emph{with its payload inline}. The core + specification once permitted the payload to be ``referenced externally''; + that is withdrawn, because a compacted document's base is derivable from + nothing else and an external reference would make the projection lossy + (Text Projection companion, + \texttt{req:textproj:base-snapshot-inline}). \item All canonical reduced state --- preserved by \emph{determining} it, never by carrying a second literal copy. Reduced state is a deterministic function of the operation set and the canonical base, so a projection diff --git a/spec/text_projection.pdf b/spec/text_projection.pdf index 0a3fec8..0a4fbc0 100644 Binary files a/spec/text_projection.pdf and b/spec/text_projection.pdf differ diff --git a/spec/text_projection.tex b/spec/text_projection.tex index 7198e3e..09e0bc5 100644 --- a/spec/text_projection.tex +++ b/spec/text_projection.tex @@ -59,7 +59,7 @@ linkcolor=epiphanyteal, citecolor=epiphanyteal, urlcolor=epiphanygold, - pdftitle={Epiphany --- Operation Catalog}, + pdftitle={Epiphany --- Text Projection}, pdfauthor={The Epiphany Project}, pdfsubject={Operation Catalog companion for the Epiphany music notation platform}, pdfkeywords={music notation, operations, CRDT, reduction, serialization}, @@ -105,7 +105,7 @@ \fancyhf{} \renewcommand{\headrulewidth}{0pt} \renewcommand{\footrulewidth}{0pt} -\fancyhead[L]{\small\scshape\color{epiphanyslate}Epiphany --- Operation Catalog} +\fancyhead[L]{\small\scshape\color{epiphanyslate}Epiphany --- Text Projection} \fancyhead[R]{\small\itshape\color{epiphanyslate}\leftmark} \fancyfoot[C]{\small\color{epiphanyslate}\thepage} \renewcommand{\headrule}{ @@ -226,7 +226,7 @@ {\Large\scshape\color{epiphanyslate}Text Projection}\\[6pt] {\large\itshape\color{epiphanyslate}A companion to the Core Specification}\\[14pt] {\color{epiphanygold}\rule{3in}{0.8pt}}\\[24pt] - {\normalsize\color{epiphanyink}Version 0.1.0 --- The canonical s-expression form}\\[4pt] + {\normalsize\color{epiphanyink}Version 0.2.0 --- Canonical-manifest coverage (blobs, extensions, custom profiles, the derived-reference rule)}\\[4pt] {\small\color{epiphanyslate}Normative for the text form it defines} \vfill \end{titlepage} @@ -383,7 +383,63 @@ other escape exists. \chapter{What Is Projected} \label{ch:content} +\section{Derive, or Carry --- Never Both} +\label{sec:content:derive-or-carry} + +The manifest's references are \emph{physical}. A \texttt{ChunkRef} carries an +offset, a compressed length, and a compression algorithm; a \texttt{BlobRef} +carries the same. Those are exactly the things the projection \MUSTNOT{} +preserve. But a reference also carries an \emph{identity} --- a chunk id, a +content hash, a blob id --- and every one of those is a function of the content. + +One rule resolves both, and it is the rule +Requirement~\ref{req:textproj:reduced-state-derived} already applies to reduced +state. + +\begin{requirement} + \label{req:textproj:derive-or-carry} + A projection \MUST{} carry exactly what the document does not determine, and + \MUSTNOT{} carry anything it does. + + \begin{itemize} + \item \textbf{Physical attributes} --- a chunk's or blob's \texttt{offset}, + \texttt{compressed\_length}, and \texttt{compression}, and a chunk's + \texttt{uncompressed\_length} --- \MUSTNOT{} appear. A serializer chooses + them freely. + \item \textbf{Derivable identities} --- \texttt{ChunkId}, + \texttt{ContentHash}, \texttt{BlobId} --- \MUSTNOT{} appear. They are + re-derived from the content by the derivations the Binary Format companion + pins (\sectionsc{Content Hashing}, \sectionsc{Domain-Separated Preimages}). + \item \textbf{Non-derivable identities} \MUST{} appear. In schema major~0 + there is exactly one: \texttt{SnapshotId}, which the Binary Format + companion declares opaque, with readers forbidden from deriving it + (\texttt{req:binfmt:snapshot-id-opaque}). + \item \textbf{Content and semantic attributes} \MUST{} appear: a chunk's + \texttt{kind} and \texttt{schema\_version} and its uncompressed payload; a + blob's media type, declared maximum uncompressed length if any, and its + payload. + \end{itemize} + + A parser \MUST{} reject a projection carrying a value this requirement forbids. +\end{requirement} + +\begin{rationale} + Carrying a derivable identity would reproduce, at the level of a chunk, the + defect Requirement~\ref{req:textproj:reduced-state-derived} rules out at the + level of the document: two sources of truth for one fact, with nothing to stop + them disagreeing. Carrying a physical attribute would make two encodings of one + document project to two texts, breaking + Requirement~\ref{req:textproj:canonical-text}. + + \texttt{SnapshotId} is the sole exception, and it is an exception for a stated + reason rather than an oversight: v0 has no snapshot producer, so the identity + has nothing to derive \emph{from}, and the Binary Format companion accordingly + pins it as sixteen opaque bytes that a reader \MUSTNOT{} attempt to verify. A + projection must carry what it cannot recompute. +\end{rationale} + \section{Document Structure} +\label{sec:content:structure} A projection is, in order: @@ -395,6 +451,7 @@ A projection is, in order: \item zero or more \texttt{(profile ...)} lines, in canonical order; \item zero or more \texttt{(extension ...)} lines, in canonical order; \item at most one \texttt{(canonical-base ...)} line; + \item zero or more \texttt{(blob ...)} lines, in canonical order; \item zero or more \texttt{(envelope ...)} lines, in canonical operation order (core specification Appendix~D). \end{enumerate} @@ -402,6 +459,79 @@ A projection is, in order: Every sequence is written in the normative order its binary counterpart uses. The projection introduces no ordering of its own. +\section{Canonical Blobs} +\label{sec:content:blobs} + +\begin{requirement} + \label{req:textproj:canonical-blobs} + A blob referenced by a canonical operation or by canonical reduced state is + itself canonical (core specification Chapter~8, + \sectionsc{Canonical and Non-Canonical Roots}). Every such blob \MUST{} be + projected as a \texttt{(blob ...)} line carrying its media type, its declared + maximum uncompressed length if it declares one, and its uncompressed payload. + + Its \texttt{BlobId}, content hash, offset, lengths, and compression are + re-derived (Requirement~\ref{req:textproj:derive-or-carry}). + + A blob referenced only by acceleration structures is non-canonical and + \MUSTNOT{} be projected. +\end{requirement} + +\begin{rationale} + An embedded image, font, or audio recording that a canonical operation + references is part of the document. Omitting it would make the projection lossy + for exactly the documents most in need of archival --- and lossy \emph{silently}, + since the operations that reference the blob would still be there, pointing at a + blob id the text no longer contains. This requirement was absent from version + 0.1.0 of this companion, and from the core specification's own list of what the + projection preserves; both are corrected. +\end{rationale} + +\section{Profile Declarations} +\label{sec:content:profiles} + +A \texttt{(profile ...)} line carries the profile's identity, its semantic +version, and its constraints. + +\begin{requirement} + \label{req:textproj:profile-id} + A profile identity is a symbol for each closed-vocabulary profile + (\texttt{full}, \texttt{read-only}, \texttt{lite}), and + \texttt{(custom \#x)} for \texttt{ProfileId::Custom}, whose + registry id is sixteen bytes. +\end{requirement} + +\begin{rationale} + Version 0.1.0 required a symbol for every profile, which made a custom profile + unrepresentable and its claim to preserve ``all profile declarations'' false. +\end{rationale} + +\section{Extension Declarations} +\label{sec:content:extensions} + +\begin{requirement} + \label{req:textproj:extension-declaration} + An \texttt{(extension ...)} line \MUST{} carry every field of the declaration: + its identity, its semantic version, whether it is required, its affected object + kinds, its edit barriers, and its preserved chunk roots. The affected object + kinds and the edit barriers are opaque to the bundle and are projected as byte + strings, verbatim. + + Each preserved chunk root is projected as its \texttt{kind}, its + \texttt{schema\_version}, and its uncompressed payload + (Requirement~\ref{req:textproj:derive-or-carry}), never as a + \texttt{ChunkRef}: a \texttt{ChunkRef} is a physical reference, and the + projection has no file to point into. +\end{requirement} + +\begin{rationale} + Version 0.1.0 carried only identity, the required flag, barriers, and a list of + chunks, dropping the semantic version and the affected object kinds outright and + leaving the chunks' representation undefined. An extension's declaration governs + how the canonical document is \emph{interpreted}; a projection that loses part + of it does not determine the document. +\end{rationale} + \section{Reduced State} \begin{requirement} @@ -430,9 +560,27 @@ projection introduces no ordering of its own. \begin{requirement} \label{req:textproj:base-snapshot-inline} If the manifest declares a canonical base, the projection \MUST{} carry a - \texttt{(canonical-base ...)} line bearing the snapshot's identity, its causal - frontier, its reduction-algorithm version, its profile, \emph{and its payload - inline} as a single byte-string atom. + \texttt{(canonical-base ...)} line bearing: + + \begin{itemize} + \item the \texttt{SnapshotId}, verbatim --- the one identity that is carried + rather than derived (Requirement~\ref{req:textproj:derive-or-carry}); + \item the causal frontier it materializes, as opaque bytes; + \item its reduction-algorithm version; + \item the profile under which it was produced; + \item its root chunk's \texttt{schema\_version}; and + \item \emph{the root chunk's uncompressed payload, inline}, as a single byte + string. That payload is the canonical byte form of the reduced state the + snapshot materializes. + \end{itemize} + + The root chunk's kind is \texttt{Snapshot} by role and is not written. The + \texttt{SnapshotRef}'s \texttt{root} and \texttt{hash} are \textbf{re-derived}: + the chunk's content hash is + $\textrm{hash}(\texttt{Snapshot}, \textit{schema}, \textit{payload})$ under the + Binary Format companion's chunk-hash preimage, and it is both the root's + \texttt{ChunkId} and the \texttt{SnapshotRef}'s \texttt{hash}. A parser + \MUST{} perform that derivation rather than read it from the text. \end{requirement} \begin{rationale} @@ -449,15 +597,12 @@ projection introduces no ordering of its own. which point the whole line changes anyway. A later revision \MAY{} project the snapshot structurally; doing so does not break the round trip, because the document it denotes is unchanged. Ratified at 0.1.0. + + Schema major~0 has no snapshot producer --- pruning and canonical-base creation + are deferred (Binary Format, \sectionsc{SnapshotId}) --- so this requirement + binds whoever writes the first one, and cannot be exercised before then. \end{rationale} -\section{Extensions} - -An \texttt{(extension ...)} line carries the extension's identity, its -required-or-optional flag, its edit barriers, and its preserved chunk roots. An -extension payload is a byte string -(Requirement~\ref{req:textproj:hex}); the projection never interprets it. - % =========================================================================== \chapter{Requirements} \label{ch:requirements} @@ -535,7 +680,7 @@ extension payload is a byte string \begin{lstlisting} projection ::= header document lineage? profile* extension* - canonical-base? envelope* + canonical-base? blob* envelope* header ::= "(text-projection " version ")" LF version ::= integer "." integer "." integer @@ -543,14 +688,24 @@ version ::= integer "." integer "." integer document ::= "(document " bytes ")" LF lineage ::= "(lineage " bytes ")" LF -profile ::= "(profile " symbol " " version " " constraints ")" LF -extension ::= "(extension " bytes " " ("required"|"optional") - " (barriers " barrier* ") (chunks " bytes* "))" LF +profile ::= "(profile " profile-id " " version " " constraints ")" LF +profile-id ::= "full" | "read-only" | "lite" | "(custom " bytes ")" +constraints ::= "(constraints " integer " " retention ")" +retention ::= "(retention " integer " " (integer|"()") " " bool ")" + +extension ::= "(extension " bytes " " version " " bool + " (kinds " bytes ") (barriers " bytes ")" + " (chunks " chunk* "))" LF +chunk ::= "(chunk " symbol " " schema " " bytes ")" ; kind, schema, payload +schema ::= "(schema " integer " " integer ")" ; major, minor canonical-base ::= "(canonical-base " bytes " (frontier " bytes ")" - " (reduction " integer ") (profile " symbol ")" - " (payload " bytes "))" LF + " (reduction " integer ") (profile " profile-id ")" + " " schema " (payload " bytes "))" LF + +blob ::= "(blob " string " " (integer|"()") " " bytes ")" LF + ; media type, declared-max-uncompressed-length, payload envelope ::= "(envelope " bytes " (author " bytes ")" " (stamp " integer " " integer " " bytes ")" @@ -568,10 +723,17 @@ dots ::= "(" bytes* ")" ; operation ids bytes ::= "#x" hexdigit* ; even count, lowercase integer ::= "-"? digit+ ; no leading zeros, no "-0" +bool ::= "true" | "false" symbol ::= [a-z] [a-z0-9-]* string ::= '"' schar* '"' \end{lstlisting} +Observe what does \emph{not} appear: no offset, no compressed length, no +compression algorithm, no chunk id, no content hash, no blob id. Every one is +either physical or derivable, and +Requirement~\ref{req:textproj:derive-or-carry} forbids both. The lone opaque +identity the grammar carries is the \texttt{SnapshotId}. + \textbf{What this grammar is, and is not.} The atom productions (\texttt{bytes}, \texttt{integer}, \texttt{symbol}, \texttt{string}) and the line shapes above are \textbf{normative}. The productions left unexpanded --- @@ -597,12 +759,14 @@ vectors that pin real bytes are a deliverable of the implementation, not of this gate; elisions are marked \texttt{\dots}. A document of one operation --- a transposition of two pitches up a perfect -fifth --- projects to four lines: +fifth, over a compacted base and one embedded image --- projects to six lines: \begin{lstlisting} -(text-projection 0.1.0) +(text-projection 0.2.0) (document #x05050505050505050505050505050505) -(canonical-base #x1f8b... (frontier #x00) (reduction 1) (profile full) (payload #x0000...)) +(profile full (0 1 0) (constraints 67108864 (retention 1 () true))) +(canonical-base #x1f8b... (frontier #x00) (reduction 1) (profile full) (schema 0 1) (payload #x0000...)) +(blob "image/png" () #x89504e47...) (envelope #x00000000000000070000000000000001 (author #x00000000000000000000000011223344) (stamp 42 7 #x00000000000000070000000000000001) (causal ((#x0000000000000001 3)) (#x00000000000000020000000000000009)) (transaction #x00000000000000070000000000000005) (primitive (transpose-interval (targets #x0000000000000007000000000000000100000000000000070000000000000002) (interval 4 7)))) \end{lstlisting} @@ -637,6 +801,34 @@ absorb it, exactly as the binary decoder does. accepting it --- and conformance requires \emph{both} directions (\texttt{req:textproj:conformance}). No implementation yet; this document is the design gate. \\ + \today & Chapters 3, 5 & 0.2.0 --- Canonical-manifest coverage. 0.1.0 was + \emph{lossy for documents that are valid today}, and its claim to preserve the + manifest's canonical roots was false in three ways: a canonical blob (an + embedded image, font, or recording referenced by a canonical operation) had no + representation at all; an \texttt{ExtensionDeclaration} lost its semantic + version and its affected object kinds, and left its preserved chunk roots + undefined; and \texttt{ProfileId::Custom} was unrepresentable, a symbol being + required where a sixteen-byte registry id is carried. + + The three share one cause, now stated as + \texttt{req:textproj:derive-or-carry}: a \texttt{ChunkRef} and a + \texttt{BlobRef} are \emph{physical} references --- offset, compressed length, + compression --- which the projection may not preserve, and they also carry + \emph{derivable} identities, which it may not duplicate. Carry the content and + the semantic attributes; re-derive the rest. The sole non-derivable identity in + schema major~0 is \texttt{SnapshotId}, which the Binary Format companion pins + as opaque and forbids readers to derive. + + Consequently: \texttt{req:textproj:canonical-blobs} (a canonical blob is + projected; a non-canonical one is not), \texttt{req:textproj:profile-id} + (\texttt{(custom \#x...)}), \texttt{req:textproj:extension-declaration} + (every field, chunks as kind + schema + payload), and + \texttt{req:textproj:base-snapshot-inline} extended to say what the inlined + payload \emph{is} and how the root chunk id and the snapshot hash are + re-derived rather than read. The core specification's own list of what the + projection preserves omitted canonical blobs; it is corrected there too. The + 0.1.0 ratifications --- reduced state derived, base inlined, hex, one envelope + per line, strict parsing --- stand unchanged. \\ \bottomrule \end{longtable}