From e6a058d32b9bb5e357c6e104b90ee6a8a46f8053 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Sun, 26 Jul 2026 19:53:11 -0400 Subject: [PATCH] doc: non-binding first pass on the multi-agent change fabric Deliberately a separate commit, outside the Wave A freeze (5ee9c6b). This document decides nothing and is not part of any frozen contract: it is a place to hold the thinking on whether LeVCS should become a change fabric for multi-agent work, pending its own review. Records where the proposal maps onto the existing plan, where it cuts against it -- levcsd must not own ref transactions, the ephemeral stratum invariant, delegation being identity-shaped rather than protocol-shaped, attestations not fitting TransactionEvidenceV1, and the canonical benchmark workload being the wrong shape for swarm traffic -- and where the proposal is simply wrong. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QNy4Ve7mogg4X1ezJTnFxG --- doc/swarm-fabric-roadmap-exploration.md | 339 ++++++++++++++++++++++++ 1 file changed, 339 insertions(+) create mode 100644 doc/swarm-fabric-roadmap-exploration.md diff --git a/doc/swarm-fabric-roadmap-exploration.md b/doc/swarm-fabric-roadmap-exploration.md new file mode 100644 index 0000000..ed2ecba --- /dev/null +++ b/doc/swarm-fabric-roadmap-exploration.md @@ -0,0 +1,339 @@ +# Swarm fabric and IDE integration — roadmap exploration + +**Status: non-binding first pass. This document decides nothing.** + +It supersedes no plan, freezes no contract, and gates no work. +`doc/instance-throughput-rewrite-plan.md` remains the only authoritative roadmap, and +`doc/phase1-storage-spine-scope.md` remains the only authoritative Phase 1 scope. Nothing +here changes either until it is deliberately folded into them. + +Written 2026-07-24, before the Wave A freeze review. If that review turns up something +structural in the frame or recovery model, it changes the sequencing conversation more +than anything in this document does, and this should be re-read afterward rather than +acted on before. + +## 0. What prompted this + +A proposal to make LeVCS the transactional substrate for a population of human and machine +contributors — thousands of speculative code states coexisting, being tested, synthesized, +audited, and selectively promoted without becoming thousands of branches — with an IDE as +the control room. Its flagship recommendation is "virtual swarm workspaces plus continuous +conflict radar" as the first non-Git-shaped feature. + +The question this document answers is narrow: **if we incorporate some of it, what changes +about the roadmap?** Not whether the product direction is right. + +## 1. What the existing plan already covers + +More than the proposal assumes. This matters for sequencing, because it means most of the +substrate work is already scheduled rather than additional. + +| Proposal asks for | Already in the plan | +|---|---| +| An event feed for IDE clients | §6.2 `GET /repos/{repo_id}/transactions?after={cursor}` — source-signed, hash-chained, fixed upper bound, typed `CursorExpired` | +| A stable contract for workflow tooling | §5.5, titled "Future instance software contract", naming review, CI, webhook, indexing, and workflow components as the consumers; `CommittedTransactionV1` frozen in Phase 0 for exactly this | +| Consumers that can fall behind and recover | §5.5 event low watermark, `CursorExpired` with the authenticated snapshot needed to rebuild, and Phase 3's exit requiring consumer feed/resnapshot tests | +| Parallel object ingestion instead of a per-repository mutex | The entire premise of this rewrite; §7's executor model and §5.2's group commit | +| Quarantine for untrusted agent objects | §4 identity invariant 7 — unreachable extras rejected, normal push never grants membership to pre-uploaded bytes | +| Bounded invisible staging that cannot publish | §8 `ProjectionStageSessionV1` — bounded, expiring, quota-accounted, sealing cannot publish membership, only `submit` may adopt | +| Not overloading the commit object | §5.5 keeps physical detail private and the commit compact; §13 "Format instability" says future workflow code binds only to logical snapshots and events | +| Repository identity independent of hosting | §4 identity invariant 2, and the federation model throughout §8 | + +Two of these deserve emphasis. + +**The signed feed is better than what the proposal asks for.** The proposal suggests SSE +"initially", then correctly says events should be notifications rather than trusted state. +The plan already has the trusted version: hash-chained, signed, page-bounded, with gap +detection via `previous_event_digest`. SSE belongs as a latency optimization *over* that +feed, never as a replacement for it. If SSE is ever added, the invariant to write down is +that a client must be able to reconstruct identical state from the signed feed alone, and +that no client action may depend on an SSE message it has not verified. + +**Projection staging is most of an ephemeral workspace already.** It is bounded, invisible, +non-membership-granting, expiring, and debt-accounted. What it is not is *general*: §8 +restricts it to initial mirror, resnapshot, oversized Full fork closure, and network +migration, and states plainly that normal push cannot reference it and it "is not an +ingestion loophole". Extending it to agent workspaces is a real scope amendment with real +security consequences, not a free reuse. + +## 2. Where the proposal genuinely cuts against the plan + +These are the items where a decision now is cheap and a decision later is expensive. + +### 2.1 `levcsd` must not own ref transactions + +The proposal lists ref transactions among the daemon's responsibilities. Definition of done +#1 is that *every* online or import mutation uses one validated, durable transaction +service, with verified byte-exact offline restore as the sole exception. A local daemon +writing refs directly would be a second mutation path — precisely what this rewrite exists +to eliminate. + +Two acceptable shapes: + +- `levcsd` **embeds** `StoreEngine` and *is* the transaction service for a local repository. + §5.1's public API is already the embedding seam; this costs nothing. +- `levcsd` is a **client** of an instance and holds no authority of its own. + +What is not acceptable is a daemon that sometimes writes through the store and sometimes +around it. Deciding this now is free; unwinding it after an IDE depends on the fast path +is not. + +### 2.2 The ephemeral stratum must stay outside the durable service + +The proposal's three-stratum split — canonical VCS, durable workflow, ephemeral +coordination — is right, and the third stratum is the one that will erode if it is not +written down as an invariant. + +Heartbeats, cursors, intent leases, and conflict forecasts must never become a +`ValidatedTransaction`. A thousand agents heartbeating through group commit would spend the +75k budget on coordination noise and make every resource bound in §4 meaningless. Worse, it +would do so gradually and defensibly, one convenient exception at a time. + +Proposed invariant, if this direction is taken: + +> Ephemeral coordination state is never durable, never fenced, never federated, and never +> occupies a shard sequence. It is TTL-governed, bounded, and reconstructible from durable +> state after a restart. A feature that cannot tolerate losing it on process death does not +> belong in this stratum. + +### 2.3 Delegation is identity-shaped, and identity freezes in Phase 2 + +§13's stop condition is explicit: do not encode authority transitions until their signed +logical contract is fixed in tests. Phase 2's **IdentitySession** builds the anchored +whole-graph verifier, the native/foreign fork-boundary rules, and the authority-transition +ordering rules. Adding scoped, expiring delegation certificates after that means reopening +the verifier. + +We have direct evidence of what that costs. Contract review 2026-07-24-A corrected an +unsound recovery classification that had shipped past its own exit criteria. Contract review +2026-07-24-B had to be amended twice in one day — once for `validation_flags`, then again +for `verification.*`, which had the identical defect one block over. Both were cheap only +because nothing depended on them yet. + +If delegation is wanted at all, its **contract** belongs in the Phase 2 freeze, even if no +code uses it until much later. Freezing an unused contract is nearly free. Retrofitting one +into a verifier with consumers is not. + +### 2.4 Attestations do not fit `TransactionEvidenceV1` + +`TransactionEvidenceV1` is per-transaction and describes the *source of a mutation* — +client, mirror, migration, projection admin, administrative. It is one-to-one with a frame +and is bound into the state digest. + +An attestation is a different shape entirely: + +| | Evidence | Attestation | +|---|---|---| +| Subject | The transaction it accompanies | A tree, commit, proposal, or integration node | +| Cardinality | Exactly one per transaction | Many per subject, unbounded over time | +| Producer | The mutating principal | Third parties, often much later | +| Timing | Before append, bound into the frame | Any time after the subject exists | +| Retention | Tied to the transaction | Independent; a benchmark result may outlive interest in it | + +Forcing attestations into the evidence union would corrupt a frozen contract to fit a +different lifetime. The proposal's signed artifact envelope is the right answer and matches +the plan's existing instinct to keep the commit object small — but it is a **new signed +object class**, so it is a new frozen contract rather than an extension of one. + +### 2.5 The canonical workload is the wrong benchmark for swarm scale + +`bench/workloads/small-commit.toml` is frozen and measures new, valid, durably acknowledged +commits per second, with an elaborate anti-gaming apparatus protecting the 60k claim. + +A swarm workload is a different animal: mostly overlay writes and merge previews, heavy +reads, comparatively few commits. Measuring it against `small-commit.toml` would either +understate it or, worse, invite loosening the canonical workload's rules to accommodate it. + +If swarm scale becomes a goal it needs a **second frozen workload** with its own gates and +its own `gate` enum value — additive, never a modification of the canonical one. §3 already +establishes the precedent that a storage primitive result can never be promoted to an +instance throughput claim; the same firewall applies here. + +## 3. The roadmap delta, by phase + +### Phases 0 and 1 — no change + +The storage spine is agnostic to all of this. It stores namespaced objects, refs, and +evidence behind one durability fence. Nothing in the proposal needs a different journal +format, a different recovery model, or a different index. + +Wave A is at its freeze review as this is written. Perturbing it now would be the worst +available timing, and there is no benefit to doing so: every item below lands in Phase 2 or +later. + +### Phase 2 — three contract freezes and one scope amendment + +Added to the Phase 2 lead freeze, alongside ProtocolV2 rather than after it: + +1. **Delegation certificate contract** — issuer, ephemeral subject key, repo, run, expiry, + allowed base refs, allowed operations, path or symbol scopes, commit ceiling, publication + rights, signature. Frozen with golden vectors and adversarial fixtures; no implementation + required in Phase 2 beyond what IdentitySession needs to not preclude it. +2. **Artifact envelope contract** — kind, schema version, subject IDs, dependency IDs, + issuer key, timestamp, payload, signature, with typed payloads for proposal, attestation, + review, integration plan, and supersession. The kernel stays small; the payload schemas + evolve independently. +3. **Attestation subject binding** — how an attestation names its subject (tree, commit, + proposal, integration node) and what makes it independently verifiable: environment + digest, command digest, input dependency closure, result, payload hash. + +Plus one amendment: **extend §8 projection staging's permitted uses** to cover ephemeral +agent workspaces. Today §8 enumerates mirror, resnapshot, oversized Full fork, and network +migration, and states that normal push cannot reference a session. Widening that list is a +security-relevant change and needs its own adversarial fixtures — the existing ones prove +that no partial or abandoned session affects membership, reads, dedupe, refs, receipts, or +events, and that property must survive the widening. + +Estimated cost: a few days of contract work. It buys the option without putting any +throughput gate at risk. + +### Phase 3 — two cheap additions + +- **Structured machine output** for status, log, diff, refs, authority, and merge + explanation. Prerequisite for any IDE client, and useful independently. +- **SSE transport over the existing signed feed**, with the invariant from §1: a client must + be able to reconstruct identical state from the signed feed alone. + +Phase 3's exit already requires consumer feed and resnapshot tests, so the hard part is +scheduled. + +### New phase after Phase 3 — the swarm substrate + +`levcsd`, overlay workspaces, lazy materialization, and conflict radar. Gated on P3 passing, +for the reason in §5 below. + +This is where the proposal's flagship lives, and notably it needs Phases 1, 2, and 3 and +almost nothing from Phases 4 and 5. That is a genuine reordering opportunity, discussed +below. + +### Phases 4 and 5 — mostly unchanged, with one new load + +Federation already replicates signed objects, so attestations and artifacts travel free if +they are objects. + +Compaction gains a real new requirement. §5.4 reasons about events, refs, and object +dependency retention. It does not reason about ten thousand expired agent overlays, which +would become a major reclamation consumer with a very different access pattern — +short-lived, bursty, and almost entirely garbage. §5.4's grace rules use committed insertion +sequence and time rather than file mtime, which is the right foundation, but the debt +accounting and low-watermark behavior would need to be re-derived for that load. + +## 4. Where the proposal is wrong or underweights something + +Recording these so they are not silently absorbed. + +**"levcsd owns ref transactions."** Addressed in §2.1. This is the one item in the proposal +that would break a definition-of-done clause if implemented literally. + +**"Add an event feed... using Server-Sent Events initially."** The plan already has a +stronger feed. SSE is a transport, not the contract. The proposal half-recognizes this two +sentences later; the correction is to make the signed feed primary by construction rather +than by convention. + +**The `levcs-core` index critique is accurate but out of scope for this plan.** The sorted +`Vec` with linear lookup and the double working-tree walk in `status` are real, but they are +*client-side*. The instance rewrite deliberately scoped itself to the instance. See §5.2. + +**"Parallelize object ingestion... the instance currently uses a per-repository serializing +mutex."** This is a description of the problem we are already 1.5 phases into fixing. Worth +noting only because it means the proposal's scalability section is largely satisfied by work +already scheduled. + +**The 255-parent commit limit** is correctly identified as a non-issue for the integration +forest, since the forest's whole point is that no node has a thousand parents. No change. + +**"Confidence should not initially be an LLM opinion."** Agreed and worth preserving as a +principle. The proposal's list of deterministic inputs — cascade success, handler used, +syntax validity, type checking, test deltas, same-symbol edits, interface changes — is +computable from artifacts the plan already produces or could produce cheaply. Keeping the +AI boundary at "proposes and explains, never decides or signs" matches the original +specification's instinct and should be written into whatever phase owns this. + +## 5. Sequencing risk + +### 5.1 The failure mode to avoid + +The discipline that makes the current plan good is that it refuses claims without archived +reproducible evidence, and it has an explicit stop condition for this exact failure: §13's +"Custom journal correctness — stop performance work if crash invariants are not mechanically +testable." + +The analogous risk here is starting the exciting layer before the boring one is proven and +finishing neither. A swarm fabric on an unproven storage engine gives you a demo that cannot +be trusted and an engine that never gets its gates run. + +The flagship needing nothing from Phases 4 and 5 makes reordering *tempting* specifically +because it looks free. It is not free: P4 and P5 are where power-loss, compaction +interference, and soak behavior get proven, and those are exactly the properties a swarm +workload stresses hardest. + +**Recommendation: take the Phase 2 contract freezes now, and gate everything else on P3.** +That is the smallest commitment that preserves the option. + +### 5.2 An unhomed workstream + +The client-side index and working-tree scanning work has **no home on the current roadmap**. +It is not in the instance rewrite, which is correctly scoped to the instance, and it is not +in any other document. + +It becomes load-bearing the moment an IDE does incremental analysis over live buffers: a +linear-lookup index and a double tree walk per status are fine for a CLI invocation and +untenable for a daemon serving an editor at keystroke latency. + +If the IDE direction is taken, this needs to become a real phase somewhere, with its own +scope. Flagging it here rather than assuming it will be absorbed. + +### 5.3 What the swarm workload does to the throughput claim + +Worth thinking through before any of this is committed to. + +The rewrite's headline is 60k durable commits/s. The proposal argues agents should +*not* produce commits — ephemeral overlays instead, with only worthwhile results promoted. +Taken seriously, that reduces commit pressure and increases pressure on: + +- Overlay writes and staging session churn (bounded by §8's quotas, which were sized for + mirror transfers, not for thousands of concurrent small sessions). +- Merge previews, which are CPU-bound tree-sitter work on the validation pool. +- Reads against many concurrent snapshots, each pinning a generation and holding segments + against reclamation. + +That last one is the interesting risk. §6.2's snapshot lease machinery has per-principal and +global ceilings on lease count, pinned bytes, duration, and compaction debt. A swarm holding +hundreds of concurrent workspace snapshots is a lease-pressure workload nobody has sized. +It is not obviously a problem, but it is unmeasured, and it is the kind of thing that shows +up as compaction starvation under soak rather than as an obvious failure. + +## 6. Open questions + +Not to be answered now. + +1. Does `levcsd` embed `StoreEngine` or client an instance? (§2.1 — both are acceptable; + the hybrid is not.) +2. Is delegation wanted at all, or is per-agent key issuance under an existing authority + acceptable for the first iteration? The contract freeze is cheap; the verifier work is + not. +3. Do ephemeral workspaces reuse projection staging, or get their own bounded mechanism? + Reuse is less code and more risk, since staging's security properties were proved for a + narrower set of callers. +4. Second benchmark workload: what does it actually measure, and what are its gates? "Swarm + throughput" is not yet a number anyone could pass or fail. +5. Where does the client-side index work live? (§5.2.) +6. What is the smallest demonstration that would be worth building — and is it the + proposal's ten-step one, or something smaller that proves the same thing? + +## 7. Summary + +- **Phases 0 and 1: no change.** Do not perturb Wave A. +- **Phase 2: three contract freezes** (delegation, artifact envelope, attestation subject + binding) **and one scope amendment** (§8 staging uses). This is the only part where + deferral is genuinely expensive, and it is a few days of work. +- **Phase 3: structured output and SSE-over-signed-feed.** Cheap, independently useful. +- **New phase after Phase 3** for `levcsd`, workspaces, and conflict radar, gated on P3. +- **Phases 4 and 5 stand**, with compaction gaining an ephemeral-debris workload it has not + been designed against. +- **One unhomed workstream** — client-side index and incremental scanning — needs a home if + the IDE direction is taken. +- **One invariant worth writing down early** — ephemeral coordination never enters the + durable transaction service. + +None of this is binding. Revisit after the Wave A freeze review.