21 KiB

Raw Blame History

Multi-language injections — framing (side quest, highlight family)

Intent. Teach the syntax engine that one buffer can hold more than one language. Today pmacs parses a buffer with exactly one grammar and runs exactly one highlights.scm over it ("first language wins"). This adds tree-sitter injection layers: after the root parse, run the grammar's injections.scm to find embedded regions (a markdown code fence, an HTML <script>, a rust macro! body), parse each with the injected language's grammar, and merge every layer's highlight spans. It is the load-bearing enabler the side-quest backlog names: it unlocks markdown fenced code, embedded languages, and is the honest gate on the per-cell notebook path.

The first consumer that ships with the engine is markdown fenced code blocks plus inline markdown, chosen because it needs zero new grammars: the markdown block grammar is already bundled and already ships an injection query, and every language a fence can name (rust, python, bash, js, ts, go, toml, …) already has a grammar from the grammar-gap PR (#118). The engine is grammar-agnostic; other injection sites (HTML embedding, JS template literals, comment-embedded langs) become follow-ups as their grammars land.

Ground truth (as of `main` @ `0ba01fe`, #119)

The syntax stack assumes one tree, one language, one query per buffer end to end. Every layer of that assumption has to grow.

ParseView / ParseTreeBundle (src/syntax.rs) — the per-buffer parse state holds a single language, source mirror, pending-InputEdit list, and current: Option<Arc<ParseTreeBundle>>. ParseTreeBundle holds one tree, one source, one language_name. run_parse (syntax.rs:106) sets one language and returns one tree.
Highlight query — SyntaxRegistry::highlights_query(name) (syntax.rs:655) lazily compiles and caches one highlights.scm per language into the main-thread Rc<SyntaxRegistry>. LanguageEntry (syntax.rs:318) carries highlights_query: &'static [&'static str] but no injection query.
Two style producers, both single-tree:
- Grid/TUI: SyntaxHighlightView (src/highlight.rs:289) is constructed with one Arc<Query> and caches spans for the one parse.current() bundle, invalidated by Arc::ptr_eq.
- Daemon→GPU wire: scoped_style_spans (src/semantic_render.rs:1640) reads the one bundle, the one highlights_query(language_name), runs compute_highlight_spans_in_range scoped to the viewport, maps captures→theme→StyleSpan. Perf-gated by StyleGate (semantic_render.rs:1612) keyed on the bundle Arc.
The wire path re-sorts spans by start — producer order is NOT preserved. The GPU applies StyleSpans through replace_style_spans (full, pmacs-gpu/src/main.rs:4112 — collects segments then sort_by_key(range.start)) and merge_style_spans (incremental, main.rs:4130 — clips, appends, re-sorts). So any "emit root-then-depth order and rely on it downstream" scheme is dead on arrival; overlaps must be resolved before the wire (Q#IJ6).
The two StyleSpan consumers also disagree on overlap: SemanticModel::effective_style_at (src/semantic_client.rs:334) folds every covering span via merge_styles; the GPU source_color_at (pmacs-gpu/src/main.rs:6718) returns the first covering span's fg and stops. Both are addressed by Q#IJ6.
Policy A — grammar-backed buffer ⇒ styled solely by tree-sitter; else solely by LSP tokens; never both on the wire (semantic_render.rs:1641).
Dispatch/settle (builtin/runtime/syntax.lua) — one language per buffer, pinned at first attach; the _dispatch wrapper seam is syntax.lua:26. The async tick installs the settled bundle.
Worker discipline — ParseRequest is fully owned (Send); parsing runs on a worker. The registry (Rc) and any Lua table are main-thread only — neither highlight-query resolution nor Lua-set config can happen on the worker (Q#IJ2, Q#IJ4).

Confirmed tree-sitter mechanics this design rests on (tree-sitter 0.26, verified against vendored sources + the tree-sitter injection docs):

Parser::set_included_ranges(&[Range]) restricts a parse to given byte ranges of the full source; resulting node offsets stay absolute into the full buffer — injected spans are already in buffer coordinates. Ranges must be sorted, non-overlapping, non-empty or the call returns IncludedRangesError.
tree_sitter_md::INJECTION_QUERY_BLOCK exists on the already-bundled crate, using both forms: dynamic (info_string (language) @injection.language) + (code_fence_content) @injection.content, and static ((inline) @injection.content (#set! injection.language "markdown_inline")) (also html/yaml/toml).
The injection contract excludes child ranges and intersects with the parent. Unless #set! injection.include-children is set, the injected ranges are the content node's extent minus its NAMED children's ranges, then intersected with the parent layer's own included ranges so a nested injection cannot reintroduce bytes its parent excluded. (Implementation note, round 1: excluding ALL children — not just named — shreds a markdown block inline node, whose children are anonymous text tokens, into unparseable fragments. Excluding only named children matches tree-sitter-md's own inline splitter, parser.rs:406-425, and is correct for every real injection site.) This makes markdown_inline a genuine multi-range case (Q#IJ5).
LanguageEntry.loader (fn() -> Language) and query-source &'static str consts are Send and touch no grammar C-object until called — a worker resolves injected languages lazily by indexing &'static BUILTIN_LANGUAGES, preserving the M4.2 lazy-load invariant.
Injection-query availability is per-crate and inconsistently named (rust INJECTIONS_QUERY; markdown INJECTION_QUERY_BLOCK; bash none) — the same pattern highlights_query already absorbs as a &'static [&'static str] slice.

Decisions

Q#IJ1 — Two node types: worker `RawLayer`, settled `Layer`; the layer set lives in `ParseTreeBundle`

RawLayer { language_name: String, tree: Tree, depth: u16 }          // worker out
Layer    { language_name: String, tree: Tree, depth: u16,           // settled
           highlight_query: Option<Arc<Query>> }

ParseTreeBundle grows to an ordered Vec<Layer> (root = layer 0, whole buffer; children depth-ascending). One Arc<ParseTreeBundle> is installed atomically per settle, so the StyleGate Arc::ptr_eq gate and the SyntaxHighlightView cache stay unchanged — a layered reparse mints a fresh Arc and flips both gates. bundle.language_name stays (root label). Per-layer included_ranges: Vec<Range> (Q#IJ5) are the worker's parse input, not retained on the settled Layer (styling reads the tree; offsets are absolute).

Q#IJ2 — Two-stage handoff: worker builds trees, settle resolves queries

Stage 1 — worker (run_parse_layered) builds the whole layer structure: root tree (incremental) + every child tree, via injection queries and set_included_ranges. It resolves injected languages by indexing &'static BUILTIN_LANGUAGES (loaders + a new injections_query: &'static [&'static str] field on LanguageEntry, mirroring highlights_query), loading/compiling only what a file injects. Touches no highlight query, no theme. Output Vec<RawLayer>.
Stage 2 — settle/install (main thread) resolves each raw layer's highlights_query(name) from the registry cache, builds Vec<Layer>, wraps one ParseTreeBundle, installs the Arc atomically.

Highlight-query compilation stays main-thread/cached/shared with the producers; tree parsing stays on the worker. The only dynamic worker input beyond the static table is the alias snapshot (Q#IJ4), carried in ParseRequest.

Rejected: (a) layer-build on the main thread in settle — moves child parsing onto the frame path; (b) a Send + Sync query store so the worker resolves everything — larger registry blast radius, deferred as an option if Stage 2 bottlenecks.

Limitation named: injection targets resolve only against BUILTIN_LANGUAGES; runtime/Lua-registered languages are not injectable in v1. Every headline case is bundled.

Q#IJ3 — Bounded recursion: depth cap, generous layer backstop, visited guard, child-only failure

max depth (default 3),
max total layer count — a runaway backstop set well above any real document (default 4096), decoupled from performance; the real perf bound is the Q#IJ10 settle-time guard. Hitting it is surfaced, not silent: run_parse sets a ParseTreeBundle::injection_capped flag, and syntax.lua's settle tick raises it once per buffer via pmacs.error (_injection_capped). A modest markdown doc's inline layers (one per paragraph/heading) sit far under the backstop. A real boundary test drives just over 4096 fences and asserts the flag + capped count + intact root.
a visited set on (language_name, ranges) so a same-language self-injection over a non-shrinking region can't reproduce itself.

Failure is isolated to the child: unknown/unresolvable language, cap hit, or child parse error drops that child layer only — never the root, never a sibling. The root always installs.

Q#IJ4 — Dynamic fence names: registry-held alias map, case-folded, snapshot to the worker

Raw @injection.language text (JS, ts, sh, py, Rust, c++, jsx, tsx) won't exact-match bundled names — the job tree-sitter's per-language injection-regex does. pmacs does it with a case-folded alias table: lowercase the text, look up an alias table before the bundled-name table (js→javascript, jsx→javascriptreact, ts→typescript, tsx→typescriptreact, py→python, rs→rust, sh/shell→bash, c++/cxx→cpp, yml→yaml, …). Unresolved → region skipped (no error, root intact).

Worker-safe extensibility: the map lives in the registry (static defaults + Lua-driven overrides via a Rust setter that pmacs.parse.injection_aliases writes through). Because the worker can't read Lua or the Rc registry, each _dispatch (the syntax.lua:26 seam) snapshots the merged map into an Arc<HashMap<String,String>> carried in ParseRequest. Acceptance mutates the alias set from Lua, then runs an asynchronous injected parse and asserts the new alias resolves — proving the bridge, not just the static resolver.

Q#IJ5 — Included ranges are `Vec<Range>` per layer: exclude children, intersect with parent

A layer's ranges are built from its @injection.content node(s):

#set! injection.include-children → [node.range];
otherwise (default; the markdown-inline case) → the node's extent minus its NAMED children's ranges (anonymous token children are the injected text itself and are kept — mechanic #3), then intersected with the parent layer's included ranges. Ranges come out ordered and non-overlapping; empty ranges dropped.

Fenced code (code_fence_content, no children) → one range; a multi-line container (a blockquote/list whose inline node carries a named block_continuation child) → several. Core, not deferred — markdown_inline needs it. Acceptance asserts content_node_ranges returns >1 range for a multi-line blockquote and that both sides parse and highlight (a one-line paragraph would give a single range and could not falsify multi-range support).

Q#IJ6 — The wire producer flattens layers into disjoint effective spans

Because the wire re-sorts by start (main.rs:4117/4130), producer order cannot carry overlap precedence. So overlaps are resolved in the producer, before the wire:

scoped_style_spans flattens all layers into disjoint effective spans — an ordered active-set event sweep: activate a span at its start boundary, expire it at its end, and at each interval fold the active set (deeper layer / later same-depth sibling / narrower capture wins, keyed by (layer_index, capture_order)). Cost is O(n·log n + Σ active) in the span count — linear in practice, since the active set is bounded by overlap depth, not the total span count. This bound matters because the file-style summary runs the flattener over the whole buffer, not just the viewport. Positional re-sorting downstream is then a no-op on precedence, and the disjoint output aligns with the model's existing style-tile disjointness invariant. (Per-byte effective style is unchanged from today for single-layer buffers; only the shape goes overlapping→disjoint, so existing StyleSpans shape assertions are updated to match.)
The GPU source_color_at fold fix is kept (fold all covering spans in order, matching effective_style_at) — defense-in-depth and a contract alignment, correct even for any residual same-start overlap.
The grid producer (SyntaxHighlightView) has no wire and no sort: it paints layer-by-layer in depth order (later overrides via the cell merge), which is already correct.

Acceptance test #9 drives the real full-frame message transform (spans_from_segments, extracted from replace_style_spans) then source_color_at, with an overlapping parent-red / child-green case — exercising the start-sort + fold, not a hand-rolled sort. (A live-State render pass needs a GPU device and is out of unit-test scope; the extracted transform is the code that matters here.)

Q#IJ7 — Both producers walk layers; Policy A unchanged at buffer scope

scoped_style_spans and SyntaxHighlightView iterate bundle.layers using each layer.highlight_query, reusing viewport-scoped compute_highlight_spans_in_range per layer. Policy A unchanged at the buffer level (grammar-backed ⇒ tree-sitter across all layers; LSP-only untouched; no new LspStyleView interaction). SyntaxHighlightView drops its single-query constructor param and reads per-layer Arc<Query> from the bundle, so it still holds only Send state — the largest single code change.

Q#IJ8 — Incrementality: root incremental, children cold each settle

Root keeps InputEdit-accumulation + prior_tree. Child layers rebuild cold each settle (an edit can add/remove/resize regions, so child identity isn't stable). Deferred: child-tree incrementality and range-scoped rebuild. Named cost: one cold inline layer per markdown paragraph/heading — made a measured acceptance guard by Q#IJ10, not a hope.

Q#IJ9 — `injection.combined` deferred

Each injection match is its own layer/parse (multi-range within a match is Q#IJ5, in scope). Combined injections (many matches → one shared parse; PHP-in-HTML, some comment schemes) are deferred. Markdown fenced/inline are not combined.

Q#IJ10 — First consumer: markdown fenced code and inline

Ships fenced-code and markdown_inline (zero new grammars; same crate; block grammar already injects inline via #set!). Inline exercises the static path and the multi-range path (Q#IJ5), completes markdown, and retires the M9.7 "block-only" floor. Its cost is real (a cold child parse per paragraph/heading), so a many-paragraph settle-time acceptance test guards it and asserts the final paragraph receives a layer/capture (not merely that parsing finished). While that stays green, child incrementality (Q#IJ8) is not required in v1. HTML, JS/TS template literals, and doc-comment code are follow-ups gated on their grammars.

Q#IJ11 — Perf gate & parse-in-flight stay correct for free

StyleGate.bundle is the root bundle Arc; a layered reparse installs a fresh Arc atomically (single dispatch), flipping the gate with no half-styled frame. grammar_style_parse_not_ready unchanged.

Bets

Absolute node offsets under set_included_ranges make layer spans buffer-coordinate-native (mechanic #1).
Static-table worker + two-stage settle + alias snapshot (Q#IJ2/IJ4) preserves lazy loading, keeps parsing off the main thread, and keeps query caching where it is.
Producer-side flattening (Q#IJ6) is the one definite overlap strategy given the wire re-sort; the GPU fold fix rides along.
Cold child reparse is fast enough at real paragraph/fence counts; the Q#IJ10 guard is the measured backstop, Q#IJ8 the escape hatch.

Deferred (named)

injection.combined (Q#IJ9).
Child-tree incrementality + range-scoped rebuild (Q#IJ8), gated open by the Q#IJ10 perf guard.
Injectable runtime/Lua-registered languages (Q#IJ2).
HTML/CSS/GraphQL/SQL grammars and their injection sites (Q#IJ10).
Notebook per-cell layering (needs JSON grammar + this engine).
Send + Sync query store so the worker resolves highlight queries directly (Q#IJ2 alternative b).

Acceptance (bite-verified where it guards a real gap)

injection_query_block_compiles — INJECTION_QUERY_BLOCK compiles against the md grammar.
layered_parse_builds_child_for_fenced_code — a ```rust fence yields ≥2 layers; child language == rust, roots at source_file.
child_layer_offsets_are_absolute — a fn inside the fence has byte offsets matching its position in the full markdown source.
dynamic_alias_resolves — ```py→python, ```JS→javascript (case-folded); ```nonsense → no child, no error, root intact.
lua_alias_override_resolves_on_async_parse — mutate injection_aliases in Lua, then an async injected parse resolves the new alias (Q#IJ4 bridge). Plus sync_parse_now_resolves_alias (round 1): the same must hold on the synchronous _parse_now path.
inline_layer_multi_range_excludes_block_continuation — a multi-line blockquote's inline node carries a named block_continuation; content_node_ranges returns >1 range and both sides parse + highlight (Q#IJ5). (A one-line paragraph gives a single range and can't falsify multi-range.)
recursion_bounds_terminate — rust macro self-injection terminates within the depth bound; injection_layer_cap_surfaces_and_preserves_root drives >4096 fences (bundle-flag level), and injection_cap_surfaced_once_and_rearms_via_lua (round 2) drives the observable Lua settle path: pmacs.error once, suppressed on an unchanged re-parse, and re-armed after dropping below the cap and exceeding it again. A failing child drops only itself (Q#IJ3).
wire_producer_emits_disjoint_child_spans_in_fence — scoped_style_spans over a ```rust fence emits disjoint StyleSpans covering a rust keyword inside the fence. Bite-verified vs the single-layer producer. flatten_same_depth_sibling_later_layer_wins (round 2): a later same-depth sibling layer wins the fold, matching grid paint order. full_buffer_summary_flatten_scales_on_large_grammar_file (round 1): the whole-buffer flatten stays ~linear under the event sweep (a quadratic flatten regresses it; the summary's per-line tally is a separate pre-existing loop).
source_color_folds_overlapping_child_over_parent — parent-red / child-green overlap driven through the real spans_from_segments (replace_style_spans body): the child (green) wins the fold. Bite-verified vs the first-span-wins path (Q#IJ6).
grid_paints_injected_child_keyword — SyntaxHighlightView paints a rust-keyword cell inside the fence.
incremental_edit_reflects_in_child_and_new_fence_adds_layer — editing inside a fence shows in child spans after reparse; a new fence adds a layer (Q#IJ8).
many_paragraph_settle_under_budget_with_tail_covered — a large all-inline markdown buffer settles within a comfortable budget and the final paragraph receives an inline layer (Q#IJ3 cap + Q#IJ10 guard).
non_injecting_buffer_single_layer — a plain .rs buffer still yields exactly one layer (regression guard).

Risks / interactions

Perf (Q#IJ8/IJ10) — cold child reparse per settle, one inline layer per paragraph; test 12 is the measured guard (with tail coverage), Q#IJ8 the release valve.
Wire overlap (Q#IJ6) — the wire re-sorts by start, so flattening in the producer is mandatory; the GPU fold fix rides along. Tests 8/9 pin both.
Single-layer wire shape — flattening turns overlapping spans disjoint for all grammar buffers; per-byte style is unchanged, but existing StyleSpans shape assertions are updated.
set_included_ranges contract — sorted, non-overlapping, non-empty; the Q#IJ5 exclusion+intersection yields this by construction, but guard empty inline nodes and non-UTF-8 info strings.
M9.7 prompt-result markdown buffers now get fenced + inline highlighting (a bonus, retiring the block-only floor); verify the _attach_highlight-for-markdown path doesn't crash.
Themes main quest — untouched. Highlight structure, not color; capture→style still flows through Theme::lookup. No protocol bump (StyleSpan wire shape unchanged).

21 KiB Raw Blame History