epiphany/spec/PLAN_GMINOR_SCHEMA_MINOR.md

414 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Plan — G-minor: the chunk schema minor
**Filed as** P13-S14. **Ruled** 2026-07-28: its own rung, sequenced **after
G2a and before G2b** (`spec/PLAN_GENESIS_OPS.md` §4).
**Status:** policy ratified 2026-07-28 (§4); vocabulary audit landed (§5.1,
`spec/AUDIT_GMINOR_VOCABULARIES.md`, c63258d); **epoch ladder ratified
2026-07-28** (§4, minors 29). **Every gate in §5 is now discharged — the
implementation contract may be drafted.** Two things it must carry that are
not yet designed: the manifest derivation seam across bundle-opaque barrier
bytes (§4), and the `binary_format.tex:2373` correction (§5.1).
> **Revision 2026-07-28.** The first draft of this plan got three things wrong
> and recommended a policy that cannot work. Corrections are marked inline
> rather than deleted, because two of them are the kind of mistake that
> regenerates: an under-scoped vocabulary, and a corpus claim asserted from a
> `grep -c` over *lines* rather than occurrences.
---
## 1. The requirement, and the evidence it has never been met
`binary_format.tex:2330` (§"Schema Versioning", the Minor bullet):
> **Minor** = additive. v0 readers verify the major only; the minor is a
> *record*, not a gate — but it is a mandatory record: a writer **MUST** raise
> the chunk schema minor when it emits any discriminant appended after the
> minor it otherwise declares, so that a decode failure on an unknown appended
> discriminant is attributable to a version skew rather than corruption.
Three facts, each read from the tree:
1. `SchemaVersion::for_major` (`bundle/src/ids.rs:204`) maps a major to a
**fixed constant** and **accepts only a major**. There is no parameter a
minor could travel through. (`V0` is `{0, 1}`; `V1`/`V2`/`V3` are `{n, 0}`.
The baselines are already inconsistent — §4 preserves that rather than
normalising it.)
2. Both writer-side staging paths derive only the major
(`testkit/src/bundle_harness.rs:32`, `textproj/src/serialize.rs:189`).
3. So every post-baseline append ships with no additive record. The
requirement's own failure mode is what the gap produces: a reader meeting an
unknown discriminant cannot distinguish a stale vocabulary from damaged
bytes.
## 2. The minor is content-addressed
`chunk_content_hash` (`bundle/src/chunk.rs:177`) pushes
`schema.canonical_bytes()` into the preimage, and that is "major then minor,
little-endian" (`ids.rs:222`). `chunk_id` dispatches to the same function
(`chunk.rs:202`).
**So raising a chunk's minor changes its `ChunkId`**, which changes the
manifest body naming it, and therefore the `ManifestId`. This is real **address
churn** — a migration and deduplication cost — and it is the main reason the
rung is not a patch.
> **Correction 1 (was P2).** The first draft framed this as tension with
> `req:format:manifest-id`. It is not. That requirement
> (`core_spec.tex:11213`) promises that *"two conforming writers committing
> **the same manifest body** at the same generation of the same document derive
> identical `ManifestId`s"*. Once a `ChunkRef` changes, the bodies are not the
> same, so the promise does not apply. Two conforming writers both following
> one normative minor derivation still agree — and the *historical* writer was
> already violating the minor MUST. Churn, not a broken guarantee.
> **Correction 2 (was P1).** The first draft claimed op-block minor changes
> reach the text projection and therefore force another `COMPANION_VERSION`
> bump. **They do not.** Operation blocks are decoded into envelopes and the
> block's physical schema is discarded during projection
> (`textproj/src/project.rs:424`); schemas are carried only for preserved
> extension chunks and the canonical base (`:454`). The committed corpus holds
> **seven** decoded `(schema 0 1)` forms in its *accepted* vectors — not the
> six the draft claimed, an error from counting matching lines instead of
> occurrences — and every one belongs to an extension chunk or a canonical
> base. **No companion bump is required for op-block stamping.** A separate
> decision to move canonical-base or extension schemas would change this;
> the op-block sweep alone does not.
>
> **Still true, but no longer the whole answer (2026-07-28).** The manifest
> seam ruling (§4) makes the manifest's aggregate `SchemaVersion` a *carried*
> `TextDocument` attribute, so **G-minor does bump the companion** — 0.9.0 →
> 0.10.0, with corpus regeneration. Op-block stamping remains
> projection-invisible exactly as this correction says; the bump comes from the
> manifest, not from it.
**Sixty-six** occurrences of `SchemaVersion::{V0, new, for_major}` across
sixteen files (corrected from 62), concentrated in `epiphany-bundle`,
`epiphany-testkit`, and `epiphany-textproj`. Most are fixtures stamping `V0`
and are unaffected; the load-bearing ones are the two staging paths.
## 3. The vocabulary inventory — the part the first draft missed
> **Correction 3 (was P1), and the reason the recommended policy died.** The
> draft reduced the problem to a per-`OperationKind` minor. An envelope also
> emits the **independent outer `OperationPayload` discriminant**, and
> `ResolveEquivocation` — appended at 3 — contains no `OperationKind` at all
> (`ops/src/payload.rs:60`). So the ambiguity lives **inside one role and one
> block**, not merely between roles, and no per-kind method can discharge the
> MUST.
`binary_format.tex:2369` states that the *only* minor-additive mechanism in
schema major 0 is appending discriminants to open vocabularies, and enumerates
them:
| Vocabulary | Appends at | Note |
|---|---|---|
| `OperationKind` | ≥ 30 | 2427, 2829 also took this mechanism |
| `OperationKindTag` | ≥ 30 | independent space from `OperationKind` |
| `OperationPayload` | ≥ 4 | **outer**; `ResolveEquivocation` carries no kind |
| value-layer unions closed under `req:binfmt:frozen-layout` | — | appendable only via a ratified revision of that document; also minor-additive |
Two boundaries that keep the audit finite:
* **`ChunkKind` is closed.** It has no `Registered` variant and its
discriminant enters every chunk's hash preimage, so a new chunk kind is a
format-**major** event, not a minor one.
* **`Registered` escape variants are not schema changes at all** — the wire
form is already defined. `binary_format.tex` lists fifteen carriers
(`RepairKind`, `ReanchorReason`, `PreconditionFailureReason`,
`IntegrityAnomalyKind`, `TypedObjectId`, …). Extension through an escape is
out of scope; **appending a native variant to one of those same enums is
not**, and that distinction is the audit's sharpest edge.
**The rung must inventory every append-only discriminant reachable from each
affected chunk payload**, not just `OperationKind`. For the canonical base that
means the vocabularies `MaterializedState` actually emits — `OperationEffect`,
`NoOpReason`, `PreconditionFailureReason`, `ConflictKind`,
`IntegrityAnomalyKind`, `PendingReason`, `ObjectState`, `TypedObjectId`
which notably do **not** include `OperationKind`.
## 4. Policy — RATIFIED 2026-07-28
**A global additive epoch with content-minimal stamping.** The first draft
recommended "minor = highest discriminant emitted"; that is rejected. It cannot
represent multiple independent vocabularies in one block, and generalising it
to "highest from any vocabulary" is worse than useless — an old `OperationKind`
23 would numerically mask a newly appended `OperationPayload` 3.
The ratified rule:
1. Each additive format revision receives **one globally meaningful minor
epoch**.
2. Every appended variant is annotated with the epoch that introduced it.
3. An **envelope's** required minor is the maximum across its outer payload
variant, its primitive kind, and every nested additive variant **actually
emitted**.
4. A **block** takes the maximum required minor over its envelopes. The major
remains the independent maximum of `schema_major()`.
5. Old content keeps its major's baseline — `V0`'s existing `1`, the others'
`0`. Baselines are not normalised.
**Per-major counters are also rejected:** a mixed block can carry a new major-0
kind beside a major-2 payload, so independently numbered minor namespaces do
not compose after the block takes `max_major`.
**Maintenance risk, and how it is controlled.** The obvious objection to an
epoch table is the one this track keeps proving — hand-maintained parallel
lists go stale (four at Push 4a, six found during G2a). The control is to
**co-locate `introduced_minor` with each discriminant in an exhaustive macro or
match with no wildcard arm**, so a newly appended variant *cannot compile*
without being assigned an epoch. That assignment is an unavoidable schema
decision, not a fallible parallel list — the same reasoning that made
`operation_kind_tag_vocabulary!` safe.
### The epoch mapping — RATIFIED 2026-07-28
**These are schema-minor epochs, not companion semver numbers.** The two
numbering spaces are unrelated and must not be cross-read.
| Minor | Additive event | Variants introduced |
|---|---|---|
| 2 | M2c | `PreconditionFailureReason::ContainerNotEmpty` = 10 |
| 3 | Push 3 | `OperationPayload::ResolveEquivocation` = 3 |
| 4 | Phase-3 first tranche | `OperationKind`/`OperationKindTag` 2427; `PreconditionFailureReason::TempoMapMalformed` = 11 |
| 5 | Pass-12 G-pass | `ReanchorReason::SameCanvasNearer` = 6; `PreconditionFailureReason` 1213 |
| 6 | Schema-major-2 repeat revision | `OperationKind`/`OperationKindTag` 2829 |
| 7 | Push 4a | `OperationKind`/`OperationKindTag` 30; `PreconditionFailureReason` 1415 |
| 8 | Genesis G1 | `OperationKind`/`OperationKindTag` 31 |
| 9 | Genesis G2a | `OperationKind`/`OperationKindTag` 3233 |
| 10 | Genesis G2b | `OperationKind`/`OperationKindTag` 34 (`SetTuningContext`) |
| 11 | Genesis G3a | `OperationKind`/`OperationKindTag` 3538 (`CreateStaffGroup`, `CreatePartDefinition`, `CreateAnalysisLayer`, `CreateView`) |
| 12 | Genesis G3b | `OperationKind`/`OperationKindTag` 39 (`CreateMeasure`); `PreconditionFailureReason` 1618 (`MeasureMeterMismatch`, `MeasureOutOfOrder`, `MeasureOrderUnverifiable`) |
> **Epoch 10 ratified 2026-07-28**, with G2b as the event. This is the **first
> exercise of the ladder's own growth path**: G-minor's `introduced_minor()` is
> exhaustive with no wildcard arm, so kind/tag 34 *cannot compile* without an
> epoch — the control working exactly as designed. The ladder stays monotonic
> (G2b follows G2a) and prefix-closed. Epoch assignment remains a ratified
> schema decision, never an implementer's choice.
> **Epoch 11 ratified 2026-07-29**, with G3a as the event (introducing
> `6c5e69f`, per the same discipline as the G2a correction above). One epoch for
> the whole rung's four kinds, the G2a precedent (two kinds at epoch 9) rather
> than one epoch per kind. The ladder stays monotonic (G3a follows G2b) and
> prefix-closed.
> **Epoch 12 ratified 2026-07-30**, with G3b as the event — the final rung of
> the genesis ladder (`spec/PLAN_GENESIS_OPS.md`), closing it. One epoch for
> the rung's one kind and its three `PreconditionFailureReason` appends
> together, the Push 4a precedent (epoch 7 bundled kind 30 with reasons 1415)
> rather than a kind epoch and a reason epoch separately. The introducing
> commit is **`e64a4b7`** (2026-07-30), where kind/tag 39 enters
> `ops/src/payload.rs` and reasons 1618 enter `ops/src/effect.rs` — G3b landed
> across six commits, and the four that follow packet 1 introduce no
> discriminant, exactly the distinction the 2026-07-28 correction below draws
> between `7df5ca1` and `55eff00`. The rung *completes* at **`d58eee8`**
> (2026-07-30), the documentation half; the hash was filled in by the pre-push
> repair that follows it, because a commit cannot cite itself and G3b closed
> the ladder, leaving no later rung to do it.
> The ladder stays monotonic (G3b follows G3a) and prefix-closed.
**The ladder is complete against the audit** — every post-baseline variant in
`AUDIT_GMINOR_VOCABULARIES.md` appears exactly once: all fifteen kind/tag pairs
through G3a, plus G3b's kind/tag 39 and its three reason appends,
`OperationPayload` 3, `ReanchorReason` 6, and all nine
`PreconditionFailureReason` appends.
**And it is monotonic in real time**, which is what makes a minor prefix-closed
(a reader declaring minor *n* supports every epoch ≤ *n*). Verified against the
introducing commits rather than assumed: M2c `a207077` (2026-06-25) → Push 3
`92aaccf` (07-02) → Phase-3 `0316160` (07-02) → G-pass `e4edea6` (07-07) →
repeat pair `9b5339f` (07-07) → Push 4a `2740a6c` (07-09) → G1 `3b09595`
(07-24) → **G2a `7df5ca1`** (07-28) → **G2b `13c3d2f`** (07-29) → **G3a
`6c5e69f`** (07-29) → **G3b `e64a4b7`** (07-30, the packet-1 commit where
kind/tag 39 and reasons 1618 enter the vocabularies; the rung completes at
`d58eee8`, which introduces no discriminant). The two
events sharing 2026-07-07 are ordered correctly: the G-pass precedes the
repeat revision. The two events sharing 2026-07-29 are ordered **by ancestry,
not timestamp**: `13c3d2f` is an ancestor of `6c5e69f` (verified with
`git merge-base --is-ancestor 13c3d2f 6c5e69f`), so G2b precedes G3a. **Only
these three additive events — G2b, G3a, G3b — extend the chain past G2a**;
G-minor (`ff9bd0f`) built the epoch machinery itself and introduced no
additive variant, and P13-S17's restoration commit (`6170015`) repaired a
document's history rather than appending vocabulary, so neither belongs in
this chain.
> **Correction 2026-07-28.** G2a's introducing commit is **`7df5ca1`**, where
> kinds/tags 3233 enter `ops/src/payload.rs`. `55eff00` is the *subsequent
> review-fix* commit and introduces no discriminant. The distinction matters
> because these hashes are the evidence for monotonicity: citing a follow-up
> commit would make the ladder unverifiable at the one rung a future reader is
> most likely to re-derive.
**Existing major baselines are unchanged**: `V0` uses minor 1; `V1``V3` use
minor 0. **Baseline variants impose no additive override** — M2c's *operation
kinds* stay baseline (inside the golden-locked 0..=23) while M2c's
`PreconditionFailureReason` append requires epoch 2. Same tranche, two
vocabularies, two answers.
**Baseline boundaries are per vocabulary, even though the epoch space is
global.** The epoch answers *which revision introduced this variant*; the
boundary answers *was this variant present at that vocabulary's own lock*.
Only the second varies. M2c is the proof: it is baseline for `OperationKind`
(absorbed into the golden-locked 0..=23) and a genuine append for
`PreconditionFailureReason` (`ContainerNotEmpty` = 10, whose 0..=9 lock
predates it). So `introduced_minor` is assigned **per variant, per
vocabulary**.
### Consequent calls, all ruled 2026-07-28
* **Keep the MUST.** Its cost is lower than §2 first claimed.
* **No text-companion bump** for op-block stamping.
* **The canonical base does not move merely because its source operations are
newer.** It never emits the op-kind discriminant. It retains its current
minor while its payload uses only baseline vocabulary, and rises only when
the `MaterializedState` bytes themselves emit a later-added discriminant —
a newly introduced effect or reason variant, say. *(This replaces the first
draft's "is the base exempt?", which was the wrong binary question: the base
is neither exempt nor automatically dragged, it is content-minimally
stamped like everything else.)*
* ~~**`Manifest::SCHEMA` stays unchanged.**~~ **NARROWLY SUPERSEDED
2026-07-28 by the audit.** The original call was right about its own case and
wrong as an unconditional rule:
* **Still true** — a changed child `ChunkRef` is changed manifest *data*, not
a new manifest-layout discriminant. Its body, id, and chunk hash move
naturally, and **that alone never raises the manifest minor.**
* **Newly true** — **emitted barrier tags do.** The manifest reaches
`OperationKindTag` through `ExtensionDeclaration::edit_barriers`
`EditBarrier::prohibited_operation_kinds` (audit §5.1, Part 2). **A
manifest containing an edit barrier naming a tag in 2433 takes that tag's
epoch**; a manifest naming only baseline tags retains its baseline. This is
content-minimal stamping applied to the manifest like any other role — the
original call had simply not seen that the manifest emits an additive
vocabulary at all.
* **The manifest major remains 0.**
**The implementation contract must design the derivation seam across the
bundle-opaque barrier bytes**, and this is the hard part, not a detail.
`epiphany-bundle`'s entire dependency list is `epiphany-determinism` and
`zstd` — **not `epiphany-ops`, not `epiphany-layout-ir`, not
`epiphany-core`**. It holds `edit_barriers` as `Vec<u8>` by design
(`manifest.rs:283`: the barrier family "is owned by Agents C and E"), so the
layer that stamps the manifest is structurally incapable of reading the tags
that determine its minor. Three shapes are available — a new dependency edge,
a producer-supplied minor travelling beside the blob, or a
decode-and-inspect step at a higher layer that already depends on both —
and choosing among them is contract work.
**The genuinely hard case:** barrier bytes are *preserved verbatim* across
writes, including for extensions the writer does not understand. A repack can
carry a barrier blob it cannot decode, whose tags it cannot enumerate, and
whose epoch it cannot derive. **"Decode and inspect" silently does not cover
this** — it looks complete until a foreign extension is present.
### The seam — RULED 2026-07-28
**`epiphany-bundle` stays opaque. No `epiphany-ops` or `epiphany-layout-ir`
dependency is added.** The producer supplies the aggregate manifest
`SchemaVersion` explicitly; the bundle records it and never derives it.
1. **Producers supply the aggregate.** The write paths take the manifest's
`SchemaVersion` as an input instead of selecting a constant.
2. **`CommitContext` must expose the previous manifest version**, so unchanged
barrier content can preserve it. **This is new plumbing, not a read-through:**
`CommitContext::previous_manifest` is a `&Manifest`, and **`Manifest` has no
schema-version field at all** — the version lives in the *superblock*
(`superblock.rs:174`, bytes 64..68). It must therefore travel as its own
`CommitContext` field.
3. **It must NOT become a `Manifest` body field.** Adding one is a schema
**major** change regardless of type (`binary_format.tex:2360`), which would
defeat the entire rung. The wire slot already exists in the superblock; no
new one is needed.
4. **Changed barrier bytes require an aware producer.** One that cannot
establish the exact new epoch **must refuse**. Blindly retaining the previous
aggregate is specifically wrong: **removing the sole barrier that contributed
the maximum would leave the manifest over-stamped**, claiming a vocabulary
level its content no longer needs.
5. **Ordinary repacks preserving the complete barrier content preserve the
carried version exactly.** This is the common path and it stays cheap.
**Consequently `Manifest::SCHEMA` becomes a baseline constant, not the
universally emitted version** (`manifest.rs:591`, currently
`SchemaVersion::V0` = `{0, 1}`). **Three** sites silently select it for
hashing and must stop: `bundle.rs:220` (create), `bundle.rs:724` (commit), and
the re-exported harness helper `manifest_chunk_hash` (`bundle.rs:1298`) —
which is **public API**, so it is a signature change, not an internal edit. The
major check at `bundle.rs:301` stays as-is and stays correct: it compares
`.major` only, which is exactly the v0 rule that the minor is a record, not a
gate.
### The text-projection consequence — G-minor DOES need a companion bump
**This reverses Correction 2 in §2 for the manifest, though not for op
blocks.** The reasoning there still holds where it was aimed: op-block minors
are discarded during projection, so **op-block stamping remains
projection-invisible**. But the seam above makes the manifest's aggregate
`SchemaVersion` a **carried document attribute**, and the projection cannot
derive it — that is the whole point of keeping the bundle opaque, and the bytes
may come from a future or unknown extension.
So `TextDocument` gains the carried manifest `SchemaVersion`
(it currently has no such field), which is a document-surface change:
* **`COMPANION_VERSION` 0.9.0 → 0.10.0** (`textproj/src/lib.rs:34`).
* **Corpus regeneration**, on the same reasoning as G2a's kind-production bump:
holding the version while changing the surface would leave two incompatible
grammars both claiming `(0 9 0)`.
**Not** because of op-block stamping — because the manifest schema becomes a
carried attribute. Recording the distinction because §2's "no companion bump"
is otherwise still true and will read as contradictory.
* **No migration.** Existing bundles are accepted as-is; newly emitted or
repacked blocks are stamped correctly.
* **Scope is "all additive discriminants reachable in affected chunk
payloads"**, not "kinds 2433".
## 5. What must complete before a contract
1. ~~**The vocabulary audit** (§3).~~ **DONE** — see §5.1.
2. ~~Ratify the epoch mapping.~~ **DONE** — §4, minors 29, complete against
the audit and monotonic in real time.
3. Confirm no third staging path has appeared beside the two in §1. *(Contract
work — cheap, but a check rather than an assumption.)*
4. Decide whether `decode_vectors.txt` moves — it is value-level, so it should
not, but that is a check rather than an assumption. *(Contract work.)*
Items 3 and 4 are checks the contract performs, not gates on drafting it.
### 5.1 The audit landed — and it is the governing inventory
**`spec/AUDIT_GMINOR_VOCABULARIES.md` (c63258d) supersedes §3's table as the
authoritative reachability inventory.** §3 is retained as the reasoning that
led here; where the two differ, the audit governs. It walks each of the nine
`ChunkKind` roles' payload encoders and classifies every variant of every
vocabulary actually emitted.
What it changes about §3:
* **The canonical base reaches more than §3 lists** — add `RepairKind`,
`ReanchorReason`, and `SpellingNominal`. Not cosmetic: `ReanchorReason` is
one of only two vocabularies in the tree with a real native append.
* **The manifest reaches `OperationKindTag` with no operation envelope in
it**, through `ExtensionDeclaration::edit_barriers`
`EditBarrier::prohibited_operation_kinds` (`layout-ir/src/barrier.rs:392`).
§3's table is framed entirely around the operation layer and names neither
the role nor the path. A manifest declaring a barrier on `CreateInstrument`
(tag 31) needs the same stamping discipline as a block containing one, and
the manifest chunk's minor is a separate decision from any op block's.
* **`CompressionAlgorithm` is excluded** (audit §1.8): `ChunkRef` transport
metadata, not a discriminant emitted by the payload `SchemaVersion` governs,
and deliberately outside the chunk-identity preimage. Recorded so it is not
reopened.
* **`binary_format.tex:2373` is stale narrative** — "append at ≥30" with a
history stopping at 29, while 3033 are taken. The normative tables
(`:1443`, `:1526`) are current and authoritative. **The `.tex` correction is
owed by the G-minor implementation**, which must edit that paragraph
regardless, since it defines the mechanism the rung implements.
*Related: `spec/AUDIT_GMINOR_VOCABULARIES.md`,
`spec/PASS13_CANDIDATES.md` (P13-S14), `spec/PLAN_GENESIS_OPS.md` §4,
`spec/binary_format.tex` §"Schema Versioning" / §"What ``Additive'' Means
Here".*