neuwirth
/
epiphany
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
epiphany/spec/text_projection.tex

1019 lines
47 KiB
TeX
Raw Blame History

% !TEX program = xelatex
%
% Epiphany --- Text Projection (companion specification)
% Companion to the Core Specification. Compile with XeLaTeX.
%
% This document is versioned independently of the Core Specification
% (independent semver; see the Versioning note in the front matter). Its preamble
% is intentionally a self-contained copy of the core specification's preamble so
% the two documents build independently; factoring a shared preamble file is a
% later cleanup, not a v0.1 deliverable.
\documentclass[11pt,letterpaper]{report}
% ---------------------------------------------------------------------------
% Packages
% ---------------------------------------------------------------------------
\usepackage{fontspec}
\usepackage{geometry}
\geometry{
letterpaper,
top=1.05in,
bottom=1.05in,
left=1.15in,
right=1.15in,
headheight=15pt
}
\usepackage[english]{babel}
\usepackage{microtype}
\usepackage{parskip}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{titlesec}
\usepackage{fancyhdr}
\usepackage{booktabs}
\usepackage{array}
\usepackage{longtable}
\usepackage{listings}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tcolorbox}
\tcbuselibrary{breakable, skins}
% ---------------------------------------------------------------------------
% Color palette (shared with the core specification)
% ---------------------------------------------------------------------------
\definecolor{epiphanyteal}{HTML}{1A4044}
\definecolor{epiphanygold}{HTML}{8E6E2E}
\definecolor{epiphanyink}{HTML}{1F1B16}
\definecolor{epiphanyslate}{HTML}{6B6660}
\definecolor{epiphanycream}{HTML}{F8F4ED}
\definecolor{epiphanymist}{HTML}{ECE8E0}
\definecolor{epiphanycode}{HTML}{2A2520}
\definecolor{epiphanycrimson}{HTML}{7A2424}
\hypersetup{
colorlinks=true,
linkcolor=epiphanyteal,
citecolor=epiphanyteal,
urlcolor=epiphanygold,
pdftitle={Epiphany --- Text Projection},
pdfauthor={The Epiphany Project},
pdfsubject={Operation Catalog companion for the Epiphany music notation platform},
pdfkeywords={music notation, operations, CRDT, reduction, serialization},
bookmarksnumbered=true,
bookmarksopen=true
}
% ---------------------------------------------------------------------------
% Typography (shared with the core specification)
% ---------------------------------------------------------------------------
\setmainfont{TeX Gyre Pagella}[Numbers={OldStyle, Proportional}, Ligatures={TeX, Common}]
\setsansfont{TeX Gyre Heros}[Scale=0.94, Ligatures={TeX, Common}]
\setmonofont{TeX Gyre Cursor}[Scale=0.88, Ligatures={TeX}]
\newfontfamily\titlefont{TeX Gyre Pagella}[Numbers={OldStyle}, Ligatures={TeX, Common}]
\newcommand{\tablenums}[1]{{\addfontfeatures{Numbers={Lining,Tabular}}#1}}
\newcommand{\sectionsc}[1]{{\addfontfeatures{Letters=SmallCaps}#1}}
% ---------------------------------------------------------------------------
% Section styling (shared with the core specification)
% ---------------------------------------------------------------------------
\titleformat{\chapter}[display]
{\normalfont\filright}
{\raggedright\color{epiphanygold}\fontsize{14pt}{16pt}\selectfont
\scshape Chapter\ \thechapter}
{16pt}
{\raggedright\color{epiphanyteal}\fontsize{32pt}{36pt}\selectfont\bfseries}
[\vspace{4pt}{\color{epiphanygold}\rule{2in}{0.6pt}}]
\titlespacing*{\chapter}{0pt}{-20pt}{30pt}
\titleformat{\section}
{\normalfont\Large\bfseries\color{epiphanyteal}}
{\color{epiphanygold}\thesection}{1em}{}
\titleformat{\subsection}
{\normalfont\large\bfseries\color{epiphanyteal}}
{\color{epiphanygold}\thesubsection}{1em}{}
\titleformat{\subsubsection}
{\normalfont\normalsize\bfseries\color{epiphanyink}}
{\thesubsubsection}{1em}{}
% ---------------------------------------------------------------------------
% Headers and footers (shared with the core specification)
% ---------------------------------------------------------------------------
\pagestyle{fancy}
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\fancyhead[L]{\small\scshape\color{epiphanyslate}Epiphany --- Text Projection}
\fancyhead[R]{\small\itshape\color{epiphanyslate}\leftmark}
\fancyfoot[C]{\small\color{epiphanyslate}\thepage}
\renewcommand{\headrule}{
\color{epiphanygold!50}\hrule width\headwidth height 0.4pt
\vspace{1pt}
\color{epiphanygold!30}\hrule width\headwidth height 0.2pt
}
% ---------------------------------------------------------------------------
% Code listing style (shared with the core specification)
% ---------------------------------------------------------------------------
\lstdefinelanguage{Rust}{
keywords={fn,let,mut,pub,struct,enum,impl,trait,for,in,if,else,match,return,
use,mod,crate,self,Self,as,where,move,async,await,const,static,
ref,type,unsafe,extern,dyn,box,break,continue,loop,while},
keywordstyle=\color{epiphanyteal}\bfseries,
ndkeywords={i8,i16,i32,i64,i128,u8,u16,u32,u64,u128,f32,f64,bool,char,str,
String,Vec,Option,Result,Box,Rc,Arc,HashMap,BTreeMap,
NonZeroU16,NonZeroU32,NonZeroU64,Duration,Timestamp},
ndkeywordstyle=\color{epiphanygold}\bfseries,
sensitive=true,
comment=[l]{//},
morecomment=[s]{/*}{*/},
commentstyle=\color{epiphanyslate}\itshape,
stringstyle=\color{epiphanycrimson},
morestring=[b]",
morestring=[b]'
}
\lstset{
basicstyle=\ttfamily\small\color{epiphanycode},
backgroundcolor=\color{epiphanycream},
frame=leftline,
rulecolor=\color{epiphanygold!60},
framesep=8pt,
framerule=1.5pt,
xleftmargin=10pt,
xrightmargin=4pt,
breaklines=true,
showstringspaces=false,
numberstyle=\tiny\color{epiphanyslate},
numbersep=10pt,
captionpos=b,
aboveskip=10pt,
belowskip=10pt,
language=Rust
}
% ---------------------------------------------------------------------------
% Custom environments (shared with the core specification)
% ---------------------------------------------------------------------------
\newtcolorbox{openquestion}[1][]{
enhanced, breakable,
colback=epiphanymist, colframe=epiphanycrimson,
fonttitle=\bfseries\color{white}, title={\scshape\hspace{2pt}Open Question},
coltitle=white, colbacktitle=epiphanycrimson,
arc=1pt, boxrule=0pt, leftrule=2pt,
left=10pt, right=10pt, top=8pt, bottom=8pt,
attach boxed title to top left={xshift=0pt, yshift=0pt},
boxed title style={arc=0pt, sharp corners, boxrule=0pt, left=6pt, right=8pt, top=2pt, bottom=2pt},
#1
}
\newtcolorbox{rationale}[1][]{
enhanced, breakable,
colback=epiphanymist, colframe=epiphanyteal,
fonttitle=\bfseries\color{white}, title={\scshape\hspace{2pt}Rationale},
coltitle=white, colbacktitle=epiphanyteal,
arc=1pt, boxrule=0pt, leftrule=2pt,
left=10pt, right=10pt, top=8pt, bottom=8pt,
attach boxed title to top left={xshift=0pt, yshift=0pt},
boxed title style={arc=0pt, sharp corners, boxrule=0pt, left=6pt, right=8pt, top=2pt, bottom=2pt},
#1
}
\newtcolorbox{requirement}[1][]{
enhanced, breakable,
colback=white, colframe=epiphanygold,
fonttitle=\bfseries\color{white}, title={\scshape\hspace{2pt}Requirement},
coltitle=white, colbacktitle=epiphanygold,
arc=1pt, boxrule=0pt, leftrule=2pt,
left=10pt, right=10pt, top=8pt, bottom=8pt,
attach boxed title to top left={xshift=0pt, yshift=0pt},
boxed title style={arc=0pt, sharp corners, boxrule=0pt, left=6pt, right=8pt, top=2pt, bottom=2pt},
#1
}
\newtcolorbox{nongoal}[1][]{
enhanced, breakable,
colback=epiphanymist, colframe=epiphanyslate,
fonttitle=\bfseries\color{white}, title={\scshape\hspace{2pt}Non-Goal},
coltitle=white, colbacktitle=epiphanyslate,
arc=1pt, boxrule=0pt, leftrule=2pt,
left=10pt, right=10pt, top=8pt, bottom=8pt,
attach boxed title to top left={xshift=0pt, yshift=0pt},
boxed title style={arc=0pt, sharp corners, boxrule=0pt, left=6pt, right=8pt, top=2pt, bottom=2pt},
#1
}
\newcommand{\MUST}{\textbf{MUST}}
\newcommand{\MUSTNOT}{\textbf{MUST}\nobreak\ \textbf{NOT}}
\newcommand{\SHOULD}{\textbf{SHOULD}}
\newcommand{\SHOULDNOT}{\textbf{SHOULD}\nobreak\ \textbf{NOT}}
\newcommand{\MAY}{\textbf{MAY}}
\setlist[itemize]{topsep=2pt, itemsep=3pt, parsep=0pt}
\setlist[enumerate]{topsep=2pt, itemsep=3pt, parsep=0pt}
\setlist[description]{topsep=2pt, itemsep=5pt, parsep=0pt}
\AtBeginDocument{\color{epiphanyink}}
% ---------------------------------------------------------------------------
% Document
% ---------------------------------------------------------------------------
\begin{document}
\begin{titlepage}
\thispagestyle{empty}
\centering
\vspace*{2.2in}
{\color{epiphanygold}\rule{3in}{0.8pt}}\\[18pt]
{\titlefont\fontsize{34pt}{38pt}\selectfont\color{epiphanyteal}\bfseries Epiphany}\\[10pt]
{\Large\scshape\color{epiphanyslate}Text Projection}\\[6pt]
{\large\itshape\color{epiphanyslate}A companion to the Core Specification}\\[14pt]
{\color{epiphanygold}\rule{3in}{0.8pt}}\\[24pt]
{\normalsize\color{epiphanyink}Version 0.3.0 --- Every production expanded; the value-projection rule}\\[4pt]
{\small\color{epiphanyslate}Normative for the text form it defines}
\vfill
\end{titlepage}
\tableofcontents
% ===========================================================================
\chapter{About This Companion}
\label{ch:about}
The \emph{Text Projection} is a companion to the Epiphany Core Specification. It
fulfils the delegation the core specification makes in Chapter~8,
\sectionsc{Text Projection} (\texttt{sec:format:textproj}), which declares that
``the format admits a deterministic projection to a canonical s-expression text
form'' and that ``the text projection is normative'' --- while leaving the form
itself unwritten. The Binary Format companion likewise excludes it: ``the
canonical s-expression form --- that is the \emph{Text Projection} companion's''.
This document supplies that form.
\section{What This Document Covers}
\begin{itemize}
\item The canonical text syntax: atoms, byte strings, text, and the layout that
makes the projection deterministic.
\item What is projected, and what is deliberately not.
\item The projection and parse requirements, including the bidirectional
round-trip the core specification demands.
\end{itemize}
It does \emph{not} cover the binary encoding of anything --- that is the Binary
Format companion's --- nor the semantics of any operation, which is the Operation
Catalog's. The projection is a \emph{re-presentation} of the canonical document,
never a second definition of it. Wherever the two could disagree, the binary form
is normative and the projection is wrong.
\section{The Subject of the Projection}
The projection's subject is the \textbf{canonical document}, not the file. Per the
core specification's Chapter~8 \sectionsc{Text Projection}, the projection
\MUSTNOT{} be required to preserve chunk offsets, compression choices, cache
chunks (operation indexes, layout caches, integrity indexes), garbage bytes from
prior commits, or superblock generation numbers, slot assignments, and CRCs.
Consequently two bundles that differ only in physical layout project to the same
text, and a text re-serializes to \emph{a} bundle rather than to \emph{the}
bundle it came from. That is the intent, not a limitation: the physical file is
an encoding of the document, and the projection is of the document.
A bundle \MAY{} cache its own projection in a \texttt{TextProjection} chunk named
by \texttt{Manifest.text\_projection\_root}. That chunk is a \textbf{non-canonical
accelerator} (core specification Chapter~8, \sectionsc{Schema Versioning}): a
reader need not understand it, and a writer preserves it verbatim or discards it.
A cached projection that disagrees with the operations it claims to project is
\emph{stale}, not authoritative.
% ===========================================================================
\chapter{The Canonical Text Form}
\label{ch:form}
\section{Encoding and Character Set}
\begin{requirement}
\label{req:textproj:charset}
A text projection \MUST{} be UTF-8, with no byte-order mark. Every text field
it carries \MUST{} be in Unicode NFC, matching the canonical-text rule of the
core specification's Appendix~D, \sectionsc{Text and Unicode}. A parser
\MUST{} reject non-NFC text rather than normalize it.
\end{requirement}
\section{Atoms}
An \emph{atom} is a symbol, an integer, a byte string, or a text string.
\textbf{Symbols} are lowercase ASCII words, possibly hyphenated:
\texttt{envelope}, \texttt{insert-event}, \texttt{strict-inverse}. They name
constructors and enumeration cases. A symbol is never quoted.
\textbf{Integers} are written in base ten, with a leading \texttt{-} for negative
values, no leading zeros, and no leading \texttt{+}. Zero is \texttt{0}, never
\texttt{-0}.
\textbf{Byte strings} carry every identifier, hash, and opaque payload.
\begin{requirement}
\label{req:textproj:hex}
A byte string \MUST{} be written as \texttt{\#x} followed by an even number of
\textbf{lowercase} hexadecimal digits, one pair per byte, in the order the
bytes appear in the canonical binary form. The empty byte string is
\texttt{\#x}. A parser \MUST{} reject uppercase digits, an odd digit count, and
any separator within the digits.
\end{requirement}
\begin{rationale}
One rule for every byte string. Hexadecimal has no alphabet variant and no
padding to canonicalize, it is greppable, and a corrupted character is locally
obvious. Identifiers are 16 or 32 bytes, so its expansion costs nothing where
it is read; only an inlined snapshot pays. The core specification permits
``base64 \emph{or another canonical text form}''; a second encoding would buy
a quarter of the bytes of the one body nobody reads, at the price of pinning an
alphabet, a padding rule, and a line-wrapping rule, and of choosing which
encoding applies where. Ratified at 0.1.0.
\end{rationale}
\textbf{Text strings} are double-quoted. Inside a string, \texttt{\textbackslash{}"}
denotes a quotation mark, \texttt{\textbackslash{}\textbackslash{}} a backslash,
\texttt{\textbackslash{}n} a line feed, and \texttt{\textbackslash{}t} a tab; no
other escape exists.
\begin{requirement}
\label{req:textproj:string-escapes}
A text string \MUST{} escape exactly the characters that require it: the
quotation mark, the backslash, U+000A, and U+0009. Every other character
\MUST{} appear literally. A parser \MUST{} reject an escape sequence outside
this set, and \MUST{} reject a literal character that the writer was required
to escape.
\end{requirement}
\begin{rationale}
``Escape exactly'' rather than ``escape at least'': the text is canonical, so
two spellings of one string cannot both be valid. This is the same injectivity
the binary form rests on (Binary Format,
\texttt{req:binfmt:decode-vectors}), stated for text.
\end{rationale}
\section{Layout}
\begin{requirement}
\label{req:textproj:envelope-per-line}
A projection is a sequence of lines separated by a single U+000A, with a final
U+000A and no other trailing whitespace. Each line is one complete
s-expression. Tokens within a line are separated by exactly one space; there is
no other whitespace, and no indentation.
Each operation envelope \MUST{} occupy exactly one line.
\end{requirement}
\begin{rationale}
The core specification's stated use case is that ``merge conflicts surface at
the operation-envelope level, which is the meaningful level for collaborative
editing''. One envelope per line makes a line-based three-way merge conflict
\emph{exactly} an envelope conflict --- never a conflict inside an envelope,
which could otherwise produce a syntactically valid operation that neither side
wrote. It also disposes of indentation: there is no whitespace to canonicalize,
so the ``identical semantics project to identical text'' requirement below has
nothing to hide in.
The lines are long. Readability is a \emph{tooling} concern, and a
pretty-printer is free to reformat for display; what it must not do is write
the reformatted text back and call it a projection. Ratified at 0.1.0.
\end{rationale}
% ===========================================================================
\chapter{What Is Projected}
\label{ch:content}
\section{Derive, or Carry --- Never Both}
\label{sec:content:derive-or-carry}
The manifest's references are \emph{physical}. A \texttt{ChunkRef} carries an
offset, a compressed length, and a compression algorithm; a \texttt{BlobRef}
carries the same. Those are exactly the things the projection \MUSTNOT{}
preserve. But a reference also carries an \emph{identity} --- a chunk id, a
content hash, a blob id --- and every one of those is a function of the content.
One rule resolves both, and it is the rule
Requirement~\ref{req:textproj:reduced-state-derived} already applies to reduced
state.
\begin{requirement}
\label{req:textproj:derive-or-carry}
A projection \MUST{} carry exactly what the document does not determine, and
\MUSTNOT{} carry anything it does.
\begin{itemize}
\item \textbf{Physical attributes} --- a chunk's or blob's \texttt{offset},
\texttt{compressed\_length}, and \texttt{compression}, and a chunk's
\texttt{uncompressed\_length} --- \MUSTNOT{} appear. A serializer chooses
them freely.
\item \textbf{Derivable identities} --- \texttt{ChunkId},
\texttt{ContentHash}, \texttt{BlobId} --- \MUSTNOT{} appear. They are
re-derived from the content by the derivations the Binary Format companion
pins (\sectionsc{Content Hashing}, \sectionsc{Domain-Separated Preimages}).
\item \textbf{Non-derivable identities} \MUST{} appear. In schema major~0
there is exactly one: \texttt{SnapshotId}, which the Binary Format
companion declares opaque, with readers forbidden from deriving it
(\texttt{req:binfmt:snapshot-id-opaque}).
\item \textbf{Content and semantic attributes} \MUST{} appear: a chunk's
\texttt{kind} and \texttt{schema\_version} and its uncompressed payload; a
blob's media type, declared maximum uncompressed length if any, and its
payload.
\end{itemize}
A parser \MUST{} reject a projection carrying a value this requirement forbids.
\end{requirement}
\begin{rationale}
Carrying a derivable identity would reproduce, at the level of a chunk, the
defect Requirement~\ref{req:textproj:reduced-state-derived} rules out at the
level of the document: two sources of truth for one fact, with nothing to stop
them disagreeing. Carrying a physical attribute would make two encodings of one
document project to two texts, breaking
Requirement~\ref{req:textproj:canonical-text}.
\texttt{SnapshotId} is the sole exception, and it is an exception for a stated
reason rather than an oversight: v0 has no snapshot producer, so the identity
has nothing to derive \emph{from}, and the Binary Format companion accordingly
pins it as sixteen opaque bytes that a reader \MUSTNOT{} attempt to verify. A
projection must carry what it cannot recompute.
\end{rationale}
\section{Document Structure}
\label{sec:content:structure}
A projection is, in order:
\begin{enumerate}
\item a \texttt{(text-projection <version>)} header line, naming the version of
\emph{this companion} the text conforms to;
\item a \texttt{(document \#x<document-id>)} line, and a
\texttt{(lineage \#x<lineage-id>)} line if the manifest declares one;
\item zero or more \texttt{(profile ...)} lines, in canonical order;
\item zero or more \texttt{(extension ...)} lines, in canonical order;
\item at most one \texttt{(canonical-base ...)} line;
\item zero or more \texttt{(blob ...)} lines, in canonical order;
\item zero or more \texttt{(envelope ...)} lines, in canonical operation order
(core specification Appendix~D).
\end{enumerate}
Every sequence is written in the normative order its binary counterpart uses. The
projection introduces no ordering of its own.
\section{Canonical Blobs}
\label{sec:content:blobs}
\begin{requirement}
\label{req:textproj:canonical-blobs}
A blob referenced by a canonical operation or by canonical reduced state is
itself canonical (core specification Chapter~8,
\sectionsc{Canonical and Non-Canonical Roots}). Every such blob \MUST{} be
projected as a \texttt{(blob ...)} line carrying its media type, its declared
maximum uncompressed length if it declares one, and its uncompressed payload.
Its \texttt{BlobId}, content hash, offset, lengths, and compression are
re-derived (Requirement~\ref{req:textproj:derive-or-carry}).
A blob referenced only by acceleration structures is non-canonical and
\MUSTNOT{} be projected.
\end{requirement}
\begin{rationale}
An embedded image, font, or audio recording that a canonical operation
references is part of the document. Omitting it would make the projection lossy
for exactly the documents most in need of archival --- and lossy \emph{silently},
since the operations that reference the blob would still be there, pointing at a
blob id the text no longer contains. This requirement was absent from version
0.1.0 of this companion, and from the core specification's own list of what the
projection preserves; both are corrected.
\end{rationale}
\section{Profile Declarations}
\label{sec:content:profiles}
A \texttt{(profile ...)} line carries the profile's identity, its semantic
version, and its constraints.
\begin{requirement}
\label{req:textproj:profile-id}
A profile identity is a symbol for each closed-vocabulary profile
(\texttt{full}, \texttt{read-only}, \texttt{lite}), and
\texttt{(custom \#x<registry-id>)} for \texttt{ProfileId::Custom}, whose
registry id is sixteen bytes.
\end{requirement}
\begin{rationale}
Version 0.1.0 required a symbol for every profile, which made a custom profile
unrepresentable and its claim to preserve ``all profile declarations'' false.
\end{rationale}
\section{Extension Declarations}
\label{sec:content:extensions}
\begin{requirement}
\label{req:textproj:extension-declaration}
An \texttt{(extension ...)} line \MUST{} carry every field of the declaration:
its identity, its semantic version, whether it is required, its affected object
kinds, its edit barriers, and its preserved chunk roots. The affected object
kinds and the edit barriers are opaque to the bundle and are projected as byte
strings, verbatim.
Fields appear in the ratified declaration order (core specification Chapter~8,
\sectionsc{Extension Declarations}).
Each preserved chunk root is projected as its \texttt{kind}, its
\texttt{schema\_version}, and its uncompressed payload
(Requirement~\ref{req:textproj:derive-or-carry}), never as a
\texttt{ChunkRef}: a \texttt{ChunkRef} is a physical reference, and the
projection has no file to point into.
\end{requirement}
\begin{rationale}
\texttt{affected\_object\_kinds} and \texttt{edit\_barriers} have ratified
structured shapes (\texttt{ObjectKind}, \texttt{EditBarrier}) \emph{and}
canonical byte encodings, and the bundle stores them opaquely: it preserves
them across reads and writes without interpreting them. At 0.3.0 the projection
does the same, carrying their canonical bytes, on the principle that the
projection interprets nothing the bundle does not.
This is a deferral, not a conclusion. A later revision \MAY{} project them
structurally under
Requirement~\ref{req:textproj:value-projection}; because their canonical bytes
are unchanged by that, doing so does not change the document, only its text ---
and that is a version-gated change to this companion, not to the format.
\end{rationale}
\begin{rationale}
Version 0.1.0 carried only identity, the required flag, barriers, and a list of
chunks, dropping the semantic version and the affected object kinds outright and
leaving the chunks' representation undefined. An extension's declaration governs
how the canonical document is \emph{interpreted}; a projection that loses part
of it does not determine the document.
\end{rationale}
\section{Projecting Canonical Values}
\label{sec:content:values}
An operation payload embeds canonical values from the core specification's
Chapter~5 --- an \texttt{Event}, a \texttt{Pitch}, a \texttt{Region}, a
\texttt{TimeSignature}. This document does \emph{not} restate their shapes. It
states one rule for turning any of them into text, and the shapes stay where they
are ratified.
\begin{requirement}
\label{req:textproj:value-projection}
A canonical value is projected thus:
\begin{enumerate}
\item A \textbf{struct} becomes \texttt{(<type-name> <field>\ldots)}, where
the type name is the core specification's name in lower-case
hyphenated form and the fields appear \emph{positionally}, in the order
that specification's ratified listing declares them. Field names are not
written.
\item A \textbf{newtype} --- a struct of exactly one unnamed field ---
is projected as that field alone, with no wrapper. This mirrors the binary
form, in which a newtype delegates to its field and adds no bytes.
\item A \textbf{tagged union} becomes
\texttt{(<variant-name> <field>\ldots)}. A variant with no fields is the
bare symbol \texttt{<variant-name>}.
\item An \textbf{option} is \texttt{()} when absent and
\texttt{(some <value>)} when present.
\item A \textbf{sequence}, \textbf{set}, or \textbf{map} is a parenthesised
list of its elements, \emph{in the order the binary form writes them}; a
map entry is \texttt{(<key> <value>)}. The projection introduces no
ordering of its own, and a set that the binary form writes strictly
increasing is written strictly increasing here
(Requirement~\ref{req:textproj:strict-parse}).
\item \textbf{Leaves.} An identifier or hash is a byte string. An integer is
an integer. A boolean is \texttt{true} or \texttt{false}. Canonical text is
a quoted string. A rational is \texttt{(ratio <numerator> <denominator>)},
in lowest terms with a positive denominator and the sign on the numerator;
zero is \texttt{(ratio 0 1)}. A \texttt{CanonicalF64} is a byte string of
its eight canonical little-endian IEEE~754 bytes.
\end{enumerate}
\end{requirement}
\begin{rationale}
\textbf{One rule, not forty productions.} Spelling out a production per value
type would restate the entire Chapter~5 data model in a second normative
document, and two normative listings of one struct is precisely the drift this
project has already been bitten by. A rule cannot drift from the listing it
reads.
\textbf{Positional fields.} The declaration order is already normative --- the
binary form depends on it --- so field names would be redundant, would double
the length of every line, and would create a second thing to keep in step with
a rename. The constructor name carries the context a reader needs.
\textbf{Newtypes are transparent} for the same reason they are transparent in
the binary form: a \texttt{MusicalPosition} \emph{is} a rational, and wrapping
it would put a distinction in the text that the document does not make.
\textbf{Floats are bytes, never decimal.} A decimal rendering of an
\texttt{f64} is not canonically unique --- shortest-round-trip and
seventeen-significant-digit forms both round-trip, and $-0.0$ has two
spellings --- so a decimal float would break
Requirement~\ref{req:textproj:canonical-text} at the first tempo mark. The core
specification already forbids computed floats in canonical state and stores the
eight bytes; the projection carries those eight bytes. This costs readability
in exactly one place, and buys canonicality everywhere.
\end{rationale}
\section{Reduced State}
\begin{requirement}
\label{req:textproj:reduced-state-derived}
The projection \MUST{} preserve canonical reduced state \emph{by preserving the
operations that determine it}. It \MUSTNOT{} carry a second, literal copy of
the reduced state.
\end{requirement}
\begin{rationale}
Reduced state is a deterministic function of the operation set and the
canonical base (core specification Chapter~6, \sectionsc{Design Principles}).
A text that carried both would have two sources of truth for one fact, and
nothing could stop them disagreeing --- a projection with an internally
contradictory document is worse than no projection. This reading is what the
core specification's own round-trip clause already implies: text that ``parses
to identical canonical document semantics'' must re-serialize to bundles with
identical semantics, and reduced state is a function of semantics.
Read the core specification's ``all canonical reduced state'' as
\emph{determines}, not \emph{contains}. Ratified at 0.1.0.
\end{rationale}
\section{The Canonical Base Snapshot}
\begin{requirement}
\label{req:textproj:base-snapshot-inline}
If the manifest declares a canonical base, the projection \MUST{} carry a
\texttt{(canonical-base ...)} line bearing:
\begin{itemize}
\item the \texttt{SnapshotId}, verbatim --- the one identity that is carried
rather than derived (Requirement~\ref{req:textproj:derive-or-carry});
\item the causal frontier it materializes, as opaque bytes;
\item its reduction-algorithm version;
\item the profile under which it was produced;
\item its root chunk's \texttt{schema\_version}; and
\item \emph{the root chunk's uncompressed payload, inline}, as a single byte
string. That payload is the canonical byte form of the reduced state the
snapshot materializes.
\end{itemize}
The root chunk's kind is \texttt{Snapshot} by role and is not written. The
\texttt{SnapshotRef}'s \texttt{root} and \texttt{hash} are \textbf{re-derived}:
the chunk's content hash is
$\textrm{hash}(\texttt{Snapshot}, \textit{schema}, \textit{payload})$ under the
Binary Format companion's chunk-hash preimage, and it is both the root's
\texttt{ChunkId} and the \texttt{SnapshotRef}'s \texttt{hash}. A parser
\MUST{} perform that derivation rather than read it from the text.
\end{requirement}
\begin{rationale}
A canonical base exists precisely so that the operations before its frontier
need not be retained. Where they have been pruned, the snapshot is \emph{not}
derivable from anything else in the document, and a projection that carried
only a reference would be \textbf{lossy} --- the text would no longer determine
the document, which is the one thing it is for. The core specification permits
the payload to be ``encoded compactly or referenced externally''; inline and
compact is the choice that keeps the text self-contained.
The snapshot diffs as one opaque atom. That is acceptable: merges happen among
operations, and a base snapshot changes only when the document is compacted, at
which point the whole line changes anyway. A later revision \MAY{} project the
snapshot structurally; doing so does not break the round trip, because the
document it denotes is unchanged. Ratified at 0.1.0.
Schema major~0 has no snapshot producer --- pruning and canonical-base creation
are deferred (Binary Format, \sectionsc{SnapshotId}) --- so this requirement
binds whoever writes the first one, and cannot be exercised before then.
\end{rationale}
% ===========================================================================
\chapter{Requirements}
\label{ch:requirements}
\section{Canonicality}
\begin{requirement}
\label{req:textproj:canonical-text}
Two bundles whose canonical document semantics are identical \MUST{} project to
\textbf{byte-identical} text. A projector \MUSTNOT{} have any freedom the
document does not determine: no optional whitespace, no alternative spelling of
an atom, no ordering choice.
\end{requirement}
\section{Round Trip}
\begin{requirement}
\label{req:textproj:roundtrip}
Parsing a projection and re-serializing it to binary \MUST{} yield a bundle
whose canonical document semantics are identical to the original's. The
bundle's physical layout, chunking, and compression \MAY{} differ.
Equivalently, and more usefully to an implementer: for every bundle $B$,
\[
\textrm{semantics}(\textrm{parse}(\textrm{project}(B))) =
\textrm{semantics}(B),
\]
and for every valid projection $T$,
\[
\textrm{project}(\textrm{serialize}(\textrm{parse}(T))) = T .
\]
The second equation is the text's own injectivity: it is \emph{stronger} than
the first, and it is the one a conformance test can check with byte equality.
\end{requirement}
\section{Strict Parsing}
\begin{requirement}
\label{req:textproj:strict-parse}
A parser \MUST{} reject any text that is not the canonical projection of the
document it denotes. It \MUSTNOT{} normalize: not whitespace, not letter case
in a byte string, not an escape sequence, not an out-of-order sequence, not a
duplicate in a set-typed field.
Accepting non-canonical text and normalizing it \emph{is} accepting it, and
does not satisfy this requirement.
\end{requirement}
\begin{rationale}
This is the same discipline the binary decoders carry, and it exists for the
same reason: a lenient parser makes two texts denote one document, and the
projection's contract is that a text \emph{determines} its document. The Binary
Format companion learned this concretely --- a whole-value re-encode guard
catches the fields a decoder normalizes and is blind to order-preserving
sequences, and a guard on an outer value can \emph{mask} a lenient inner codec
rather than fix it (Binary Format,
\sectionsc{The Decode Vector Corpus}). A text parser inherits both hazards, and
the cheapest total defence is the same one: re-project the parsed document and
compare, \emph{and} check per-site the orders that re-projection would restore.
\end{rationale}
\section{Conformance}
\begin{requirement}
\label{req:textproj:conformance}
An implementation claiming Text Projection conformance \MUST{} implement both
directions. A projector alone does not conform: the round trip
(Requirement~\ref{req:textproj:roundtrip}) is the requirement, and half of it
is not checkable.
\end{requirement}
% ===========================================================================
\chapter{Grammar}
\label{ch:grammar}
\begin{lstlisting}
projection ::= header document lineage? profile* extension*
canonical-base? blob* envelope*
header ::= "(text-projection " version ")" LF
version ::= "(" integer " " integer " " integer ")"
document ::= "(document " bytes ")" LF
lineage ::= "(lineage " bytes ")" LF
profile ::= "(profile " profile-id " " version " " constraints ")" LF
profile-id ::= "full" | "read-only" | "lite" | "(custom " bytes ")"
constraints ::= "(constraints " integer " " retention ")"
retention ::= "(retention " integer " " option " " bool ")"
extension ::= "(extension " bytes " " version " " bool
" (" chunk* ") " bytes " " bytes ")" LF
; id, version, required, chunks, affected-kinds, barriers
; (the ratified declaration order)
chunk ::= "(chunk " chunk-kind " " schema " " bytes ")"
chunk-kind ::= "operation-envelope-block" | "operation-index" | "snapshot"
| "blob" | "extension-data" | "text-projection"
| "layout-cache" | "integrity-index" | "manifest"
schema ::= "(schema " integer " " integer ")"
canonical-base
::= "(canonical-base " bytes " " bytes " " integer
" " profile-id " " schema " " bytes ")" LF
; snapshot-id, frontier, reduction version, profile,
; root schema, root payload
blob ::= "(blob " string " " option " " bytes ")" LF
; media type, declared max uncompressed length, payload
envelope ::= "(envelope " bytes " " bytes " " stamp " " causal
" " option " " payload ")" LF
; id, author, stamp, causal context, transaction, payload
stamp ::= "(stamp " integer " " integer " " bytes ")"
causal ::= "(causal (" replica-seen* ") (" bytes* "))"
replica-seen ::= "(" bytes " " integer ")"
payload ::= "(primitive " kind ")"
| "(resolve-conflict " bytes " " action ")"
| "(undo " bytes " " policy ")"
| "(resolve-equivocation " bytes " " bytes ")"
action ::= "accept-loser" | "keep-winner" | "dismiss"
| "(override " bytes ")" | "(reanchor " bytes ")"
| "(registered " bytes ")"
policy ::= "strict-inverse" | "best-effort" | "cascade"
; --- Operation kinds. Fields are the Operation Catalog's payload schema,
; --- positionally, in declaration order. Embedded Chapter-5 values follow
; --- req:textproj:value-projection and are written <value> below.
kind ::= "(insert-event " bytes " " value ")"
| "(delete-event " bytes " " tuplet-comp ")"
| "(respell-pitch " bytes " " value ")"
| "(create-cross-cutting " cross-cutting ")"
| "(change-region-time-model " bytes " " value
" (" bytes* ") " remapping ")"
| "(set-user-system-break " bytes " " value " " bool ")"
| "(declare-transaction " bytes " " string " " option ")"
| "(registered " bytes " " bytes ")"
| "(modify-event " value ")"
| "(transpose (" bytes* ") " integer ")"
| "(insert-identified-pitch " bytes " " value ")"
| "(delete-identified-pitch " bytes ")"
| "(modify-identified-pitch " bytes " " value ")"
| "(delete-cross-cutting " bytes ")"
| "(modify-cross-cutting " cross-cutting ")"
| "(create-region " value ")"
| "(delete-region " bytes ")"
| "(create-staff-instance " bytes " " value ")"
| "(delete-staff-instance " bytes ")"
| "(create-voice " bytes " " value ")"
| "(delete-voice " bytes ")"
| "(set-metadata " value ")"
| "(set-metric-grid " bytes " " option ")"
| "(set-user-page-break " bytes " " value " " bool ")"
| "(create-staff " value ")"
| "(set-time-signature " bytes " " value " " option ")"
| "(set-tempo-segment " option " " value " " option ")"
| "(set-staff-layout " bytes " " option " " option " " bool ")"
| "(create-repeat-structure " value ")"
| "(delete-repeat-structure " bytes ")"
| "(transpose-interval (" bytes* ") (interval " integer
" " integer "))"
tuplet-comp ::= "not-in-tuplet" | "(replace-with-rest " value ")"
| "(rewrite-tuplets (" bytes* "))"
| "(cascade-delete-tuplets (" bytes* "))"
cross-cutting
::= "(tie " value ")" | "(slur " value ")"
| "(beam " value ")" | "(spanner " value ")"
remapping ::= "preserve-time" | "(reassign (" reassign-entry* "))"
reassign-entry
::= "(" bytes " " ratio ")" ; event id, musical position
; --- Values and leaves.
value ::= "(" symbol " " value* ")" ; req:textproj:value-projection
| symbol | bytes | integer | bool | string | ratio | option
option ::= "()" | "(some " value ")"
ratio ::= "(ratio " integer " " integer ")"
bytes ::= "#x" hexdigit* ; even count, lowercase
integer ::= "-"? digit+ ; no leading zeros, no "-0"
bool ::= "true" | "false"
symbol ::= [a-z] [a-z0-9-]*
string ::= '"' schar* '"'
digit ::= [0-9]
hexdigit ::= [0-9a-f]
schar ::= unescaped | '\"' | '\\' | '\n' | '\t'
unescaped ::= [^"\ LF TAB] ; any other character
\end{lstlisting}
Observe what does \emph{not} appear: no offset, no compressed length, no
compression algorithm, no chunk id, no content hash, no blob id. Every one is
either physical or derivable, and
Requirement~\ref{req:textproj:derive-or-carry} forbids both. The lone opaque
identity the grammar carries is the \texttt{SnapshotId}.
\textbf{Every production is expanded.} Version 0.1.0 left \texttt{kind},
\texttt{action}, \texttt{policy}, \texttt{constraints}, and \texttt{barrier}
derived-but-unwritten, and said so; that gap is closed. Barriers and affected
object kinds are byte strings, because the bundle holds them opaquely and the
projection interprets nothing the bundle does not.
The one thing still \emph{read} rather than restated is a Chapter-5 value's field
list, and that is deliberate: Requirement~\ref{req:textproj:value-projection} is a
rule applied to the core specification's ratified listings, not a copy of them.
A rule cannot drift from what it reads. An implementation that projects a value's
fields in an order other than the declaration order disagrees with the core
specification, not with this document.
Note the operation-kind names are the \emph{Operation Catalog's} section names
(\texttt{create-region}, \texttt{create-staff}), not the \texttt{OperationKindTag}
names (\texttt{InsertRegion}, \texttt{InsertStaff}). The tag space renamed three
pairs for reasons of its own (Binary Format,
\sectionsc{\texttt{OperationKindTag}}); the projection follows the semantics, not
the tag.
% ===========================================================================
\chapter{A Worked Example}
\label{ch:example}
\emph{Non-normative.} The byte strings below are illustrative. The conformance
vectors that pin real bytes are a deliverable of the implementation, not of this
gate; elisions are marked \texttt{\dots}.
A document of one operation --- a transposition of two pitches up a perfect
fifth, over a compacted base and one embedded image --- projects to six lines:
\begin{lstlisting}
(text-projection (0 3 0))
(document #x05050505050505050505050505050505)
(profile full (0 1 0) (constraints 67108864 (retention 1 () true)))
(canonical-base #x1f8b... #x00 1 full (schema 0 1) #x0000...)
(blob "image/png" () #x89504e47...)
(envelope #x00000000000000070000000000000001 #x00000000000000000000000011223344 (stamp 42 7 #x00000000000000070000000000000001) (causal ((#x0000000000000001 3)) (#x00000000000000020000000000000009)) (some #x00000000000000070000000000000005) (primitive (transpose-interval (#x00000000000000070000000000000001 #x00000000000000070000000000000002) (interval 4 7))))
\end{lstlisting}
The envelope's targets are a \emph{set}: strictly increasing, no duplicates
(Operation Catalog, \texttt{req:opcat:transpose-interval-targets}; Binary Format
$\mathrm{seq}^{\Uparrow}$). A parser \MUST{} reject a duplicate rather than
absorb it, exactly as the binary decoder does.
% ===========================================================================
\chapter{Revision History}
\label{ch:history}
\begin{longtable}{p{2cm} p{2.5cm} p{9cm}}
\toprule
\textbf{Date} & \textbf{Section} & \textbf{Change} \\
\midrule
\endhead
\today & All & 0.1.0 --- Initial companion. Supplies the canonical
s-expression form the core specification's Chapter~8
\sectionsc{Text Projection} declares normative and leaves unwritten, and which
the Binary Format companion excludes as ``the \emph{Text Projection}
companion's''. Ratified: reduced state is preserved by \emph{determining} it,
never by a second literal copy (\texttt{req:textproj:reduced-state-derived});
a canonical base snapshot is inlined as one opaque byte string, because a
pruned document's base is derivable from nothing and a reference-only
projection would be lossy (\texttt{req:textproj:base-snapshot-inline});
lowercase hex is the single byte-string encoding
(\texttt{req:textproj:hex}); one envelope per line, so a line-based merge
conflict is exactly an envelope conflict
(\texttt{req:textproj:envelope-per-line}). Parsing is strict-canonical
(\texttt{req:textproj:strict-parse}) --- normalizing non-canonical text is
accepting it --- and conformance requires \emph{both} directions
(\texttt{req:textproj:conformance}). No implementation yet; this document is
the design gate. \\
\today & Chapters 3, 5 & 0.2.0 --- Canonical-manifest coverage. 0.1.0 was
\emph{lossy for documents that are valid today}, and its claim to preserve the
manifest's canonical roots was false in three ways: a canonical blob (an
embedded image, font, or recording referenced by a canonical operation) had no
representation at all; an \texttt{ExtensionDeclaration} lost its semantic
version and its affected object kinds, and left its preserved chunk roots
undefined; and \texttt{ProfileId::Custom} was unrepresentable, a symbol being
required where a sixteen-byte registry id is carried.
The three share one cause, now stated as
\texttt{req:textproj:derive-or-carry}: a \texttt{ChunkRef} and a
\texttt{BlobRef} are \emph{physical} references --- offset, compressed length,
compression --- which the projection may not preserve, and they also carry
\emph{derivable} identities, which it may not duplicate. Carry the content and
the semantic attributes; re-derive the rest. The sole non-derivable identity in
schema major~0 is \texttt{SnapshotId}, which the Binary Format companion pins
as opaque and forbids readers to derive.
Consequently: \texttt{req:textproj:canonical-blobs} (a canonical blob is
projected; a non-canonical one is not), \texttt{req:textproj:profile-id}
(\texttt{(custom \#x...)}), \texttt{req:textproj:extension-declaration}
(every field, chunks as kind + schema + payload), and
\texttt{req:textproj:base-snapshot-inline} extended to say what the inlined
payload \emph{is} and how the root chunk id and the snapshot hash are
re-derived rather than read. The core specification's own list of what the
projection preserves omitted canonical blobs; it is corrected there too. The
0.1.0 ratifications --- reduced state derived, base inlined, hex, one envelope
per line, strict parsing --- stand unchanged. \\
\today & Chapters 3, 5 & 0.3.0 --- Every production expanded. 0.1.0 left
\texttt{kind}, \texttt{action}, \texttt{policy}, \texttt{constraints} and
\texttt{barrier} derived-but-unwritten and admitted it; all are now written.
Barriers and affected object kinds are byte strings, because the bundle holds
them opaquely and the projection interprets nothing the bundle does not.
Operation payloads embed Chapter-5 canonical values, and those are projected by
\emph{one rule} rather than by forty productions
(\texttt{req:textproj:value-projection}): a struct is
\texttt{(<type-name> <field>\ldots)} with fields positional in the ratified
declaration order; a newtype is transparent, as it is in the binary form; a
tagged union is \texttt{(<variant> <field>\ldots)}; an option is \texttt{()}
or \texttt{(some v)}; a sequence keeps the binary form's order. Restating the
Chapter-5 model here would have put two normative listings on one struct, which
is the drift P13-I1 was opened to close --- and a rule cannot drift from what it
reads.
Two leaf decisions follow from canonicality rather than taste. A rational is
\texttt{(ratio n d)} in lowest terms with the sign on the numerator. A
\texttt{CanonicalF64} is the byte string of its eight canonical IEEE~754 bytes,
never a decimal: decimal float text is not canonically unique, so a decimal
tempo would break \texttt{req:textproj:canonical-text} at the first tempo mark.
Operation-kind names are the Operation Catalog's section names, not the
\texttt{OperationKindTag} names, which renamed three pairs for reasons of the
tag space. Still no implementation. \\
\bottomrule
\end{longtable}
\end{document}
Reference in New Issue View Git Blame Copy Permalink