Found by starting the implementation: the grammar could not derive an ordinary
pitched note.
`value` had no alternative for a sequence at all, though value-projection clause 5
required one. Adding it exposed why it was missing. A sequence whose first element
is a fieldless variant is shape-identical to a struct, and `()` is both the empty
sequence and the absent option. Both collisions are reachable from the first
pitched note in any score: `PitchedEvent` carries `articulations` and `ornaments`,
sequences over the zero-field `ArticulationMark` and `OrnamentMark`, beside an
optional `DynamicMark`. One `insert-event` line holds both.
The collision is irreducible without new syntax, and new syntax buys nothing --
`req:textproj:strict-parse` already obliges a parser to reject a duplicate in a
set-typed field, which it cannot do without knowing the field is set-typed. The
parser consults the schema either way. So `req:textproj:schema-directed` states
what the ratified rules already required, `value` collapses to `"(" value* ")"` or
a leaf -- all shape can honestly say -- and the requirement assigns meaning by the
expected type. The binary form is schema-directed for the same reason and pays the
same price: its bytes do not say what they are either.
Three consequences stated: a struct with no fields is the bare symbol, as a
fieldless variant is; a byte string is not a sequence, so an opaque extension
payload and a `SoundConfiguration` project as byte strings, never as lists of
integers; and the grammar's repetitions now carry a notation rule -- adjacent
elements separated by exactly one space -- without which `"(transpose (" bytes* ")"`
spelled two targets as one undelimited run of hex.
Checker gains a seventh test, mutation-verified by restoring the 0.4.0 `value`
production and by stripping the schema-directed citations. It asserts the
symbol-headed struct alternative is *absent*: a grammar claiming to tell a struct
from a sequence by shape would be lying.
Gate green -- clippy 0, 1038 tests, doc 0, conformance 8/8, no golden churn, three
spec documents build clean with no undefined references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 0.3.0 audit raised two normative issues and one process issue. All three
land here, plus a fourth defect that reviewing the fix turned up.
The grammar contradicted its own escape requirement. `req:textproj:string-escapes`
obliges a writer to escape the backslash and a parser to reject a bare one, while
`unescaped` admitted it. Escapes are now four two-character sequences and
`unescaped` excludes U+0022, U+005C, U+000A, U+0009 by codepoint.
"Keep the binary order" was not available for every sequence. It holds only where
the binary order reads data the projection preserves, and two sequences fail that
test: `blob_roots` sorts by the full `BlobRef` encoding (offset, compressed
length, compression), and an extension's preserved chunk roots sort by
`ChunkRef`'s order, keyed on kind, then content hash, then *offset*. Under the
blanket rule, relocating a chunk -- which changes no semantics -- would change the
text, and two entries indistinguishable after erasure would produce duplicate
lines. `req:textproj:derived-ordering` orders and de-duplicates exactly those two
by projected form, and states that every other sequence keeps the binary order:
profile and extension declarations sort on semantic `(id, version)` keys, and
envelopes on canonical operation order.
The "machine-checked" grammar was checked by a throwaway script -- true of one run
and of nothing durable, the same evidence gap P2-P4 kept exposing.
`text_projection_grammar.rs` is the committed form: no nonterminal undefined or
unreachable, the escape rule admits exactly its four sequences, and the operation
and chunk productions are *derived* from `OperationKindTag::PAYLOAD_FREE` and
`ChunkKind` through an exhaustive match, so a kind added to the enum and not to
the grammar fails to compile. Every locator finds its production by name; writing
it exposed four bugs in itself, three of them column-anchored checks that a reflow
would have silently switched off.
Reviewing the escape fix found it reintroduced the audited defect in disguise: a
quoted terminal `"\\"` reads as two backslashes, so every escape became three
characters long. Both characters are now codepoints. Relatedly the mono font's
`Ligatures={TeX}` rendered U+0022 as a right curly quote and `--` as an en dash,
so the grammar misprinted the very delimiters it defines; `core_spec.tex` already
omitted it. Both are asserted against.
All six tests mutation-verified: the anchor asserted present before substitution,
then the named test observed to fail. Gate green -- clippy 0, 1037 tests, doc 0,
conformance 8/8, no golden churn, three spec documents build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.1.0 left kind, action, policy, constraints and barrier derived-but-unwritten
and admitted it. All are now written. The grammar has no undefined nonterminal
(machine-checked) and 31 operation-kind productions in exact discriminant order,
cross-checked against envdecode.rs.
The one real decision was how embedded Chapter-5 values appear. An operation
payload carries an Event, a Pitch, a Region, a TimeSignature, and there were
three ways to write them:
Forty hand-written productions would restate the entire Chapter-5 data model in
a second normative document -- two normative listings of one struct, which is
the exact drift P13-I1 was opened to close.
Opaque canonical-value byte strings would be lossless and zero-drift, but a
pitch would be unreadable without binary tooling, failing the core spec's own
"format inspection and debugging" use case.
One mechanical rule (req:textproj:value-projection, ratified): a struct is
(<type-name> <field>...) with fields positional in the ratified declaration
order; a newtype is transparent, exactly as in the binary form; a tagged union
is (<variant> <field>...); an option is () or (some v); a sequence keeps the
binary form's order. A rule cannot drift from what it reads.
Two leaf decisions follow from canonicality rather than taste. A rational is
(ratio n d), lowest terms, sign on the numerator. A CanonicalF64 is the byte
string of its eight canonical IEEE-754 bytes and never a decimal: decimal float
text is not canonically unique -- shortest-round-trip and 17-significant-digit
forms both round-trip, and -0.0 has two spellings -- so a decimal tempo would
break req:textproj:canonical-text at the first tempo mark.
Operation-kind names follow the Operation Catalog's sections (create-region,
create-staff), not OperationKindTag's (InsertRegion, InsertStaff), which renamed
three pairs for reasons internal to the tag space. The projection follows the
semantics.
One deferral, stated as such rather than left to inference:
affected_object_kinds and edit_barriers have ratified structured shapes AND
canonical byte encodings, and the bundle stores them opaquely. At 0.3.0 the
projection does the same, on the principle that it interprets nothing the bundle
does not. A later revision may project them structurally; their canonical bytes
are unchanged by that, so it changes the text and not the document.
Also corrected: the extension line now writes its fields in the ratified
declaration order, which had preserved_chunk_roots before affected_object_kinds
and edit_barriers.
Gate: clippy 0, 31 targets / 1031 passed / 0 failed, conformance 8/8, zero golden
churn; core_spec, binary_format and text_projection all build with no undefined
references. Grammar closure machine-checked: no undefined nonterminal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A review found the 0.1.0 companion lossy for documents that are valid today. Its
claim to preserve the manifest's canonical roots was false three ways:
A canonical blob had no representation at all. blob_roots referenced by
canonical operations or reduced state ARE canonical roots, and the document
structure had no blob line. An embedded image, font or recording would vanish
from a projection silently -- the operations referencing it still there,
pointing at a blob id the text no longer contained.
An ExtensionDeclaration lost its semantic version and its affected_object_kinds
outright, and left its preserved_chunk_roots undefined.
ProfileId::Custom(ProfileRegistryId) was unrepresentable: a symbol was required
where sixteen registry bytes are carried.
All three share one cause I had not named. A ChunkRef and a BlobRef are PHYSICAL
references -- offset, compressed length, compression -- which the projection may
not preserve; and they carry DERIVABLE identities -- ChunkId, ContentHash, BlobId
-- which it may not duplicate. Having no rule for either, I dropped the
references and took their contents with them.
req:textproj:derive-or-carry states it, and it is the same rule
req:textproj:reduced-state-derived already applied one level up: carry exactly
what the document does not determine, and nothing it does. Physical attributes
never appear; derivable identities never appear; content and semantic attributes
always do. The one non-derivable identity in schema major 0 is SnapshotId, which
the Binary Format companion pins as opaque and forbids readers to derive -- an
exception for a stated reason rather than an oversight. The grammar now visibly
contains no offset, no length, no compression, no chunk id, no hash, no blob id.
So: req:textproj:canonical-blobs (canonical blobs projected, non-canonical ones
not), req:textproj:profile-id ((custom #x...)), req:textproj:extension-declaration
(every field; chunks as kind + schema + payload, never as a ChunkRef -- the
projection has no file to point into), and req:textproj:base-snapshot-inline
extended to say what the inlined payload IS and that the root ChunkRef and the
SnapshotRef hash are re-derived from hash(Snapshot, schema, payload), never read.
The gap started upstream. core_spec's own list of what the projection preserves
omitted canonical blobs while classifying blob_roots as canonical roots -- an
inconsistency inside one document. Corrected there, along with withdrawing the
permission to reference a base snapshot "externally", which the inline
ratification had already made untenable.
Also: the generated PDF metadata and page header still said Operation Catalog,
inherited from the copied preamble.
The four 0.1.0 ratifications stand unchanged. Implementation stays deferred: a
gate that is lossy is not a gate.
Gate: clippy 0, 31 targets / 1031 passed / 0 failed, conformance 8/8, zero golden
churn; core_spec, binary_format, operation_catalog and text_projection all build
with no undefined references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
core_spec Chapter 8 declares the text projection normative and leaves the form
unwritten; binary_format excludes it as "the Text Projection companion's". That
companion did not exist. spec/text_projection.tex v0.1.0 supplies it. No
implementation -- this is the gate.
It was blocked on P5. The projection's normative requirement is bidirectionality
WITH THE BINARY FORM, which needs bytes -> OperationEnvelope, and that decoder
did not exist until 3baf8d0.
Four ratified calls:
Reduced state is preserved by DETERMINING it, never by a second literal copy. It
is a deterministic function of the operation set and the canonical base; a text
carrying both would hold two sources of truth for one fact and nothing could stop
them disagreeing. core_spec's "all canonical reduced state" now carries that
reading inline rather than being silently reinterpreted.
A canonical base snapshot is INLINED as one opaque byte string. This is the call
with teeth. A base exists precisely so prior operations need not be retained, and
where they are pruned the base is derivable from nothing else -- so a
reference-only projection of a compacted document would be LOSSY, and the text
would not determine the document it claims to project. core_spec permits "encoded
compactly or referenced externally"; inline is the choice that keeps archival
honest, and core_spec now says why.
Lowercase hex everywhere: one rule, no alphabet or padding to canonicalize,
greppable. Base64 would buy a quarter of the bytes of the one body nobody reads,
at the price of a second encoding and a rule for which applies where.
One envelope per line: the stated use case is that merge conflicts surface at the
envelope level, and one line per envelope makes a three-way merge conflict
exactly an envelope conflict -- never a conflict inside one, yielding an operation
neither side wrote. It also removes all indentation, so canonicality has nothing
to hide in. Readability is a pretty-printer's job; what a pretty-printer must not
do is write its output back and call it a projection.
Strict parsing is stated in the terms P2-P5 taught: normalizing non-canonical
text IS accepting it. The rationale names both hazards this repo hit in binary --
a re-encode guard is blind to order-preserving sequences, and a guard on an outer
value can mask a lenient inner codec -- and prescribes the same total defence.
Conformance requires both directions: a projector alone cannot be checked.
One gap is stated in the document rather than left to be discovered: the atom
productions and line shapes are normative, while kind/action/policy/constraints/
barrier are derived from the Operation Catalog and the wire table rather than
spelled out. That is the difference between a design gate and a finished
companion.
Gate: clippy 0, 31 targets / 1031 passed / 0 failed, conformance 8/8, zero golden
churn; core_spec, binary_format and the new companion all build with no undefined
references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Levi Neuwirth