neuwirth
/
pmacs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(injections): multi-language injection layers 

Teach the syntax engine that one buffer can hold more than one
language. After the root parse, run the grammar's injections.scm, parse
each embedded region with the injected language, and merge every
layer's highlight spans. First consumer: markdown fenced code + inline
(zero new grammars — the block grammar already ships an injection query
and the injected langs already have grammars from #118).

Engine (src/syntax.rs):
- ParseTreeBundle now holds Vec<Layer> (root layer 0 + injected
  children, depth-ascending); installed atomically so the existing
  Arc::ptr_eq style gate and highlight cache keep working (Q#IJ1).
- run_parse builds layers on the worker: run injections.scm, resolve
  the injected language, compute Vec<Range> (exclude NAMED children,
  intersect the parent's ranges), set_included_ranges cold-parse,
  recurse — bounded by depth (3), a layer backstop (4096), and a
  (lang,ranges) visited guard; any child failure drops that child only
  (Q#IJ3/IJ5). LanguageEntry gains injections_query; markdown_inline is
  registered (retires the M9.7 block-only floor); markdown/rust carry
  injection queries.
- Injected languages resolve off the static BUILTIN_LANGUAGES table
  (Send loaders + query sources), preserving lazy loading. Dynamic
  fence names go through a case-folded alias map seeded with defaults
  and Lua-extensible via pmacs.parse.injection_aliases, snapshotted into
  ParseRequest at dispatch so the worker never touches the Rc registry
  or a Lua table (Q#IJ2/IJ4). Highlight queries are resolved at settle
  (resolve_layer_queries), keeping query compilation main-thread/cached.

Producers:
- SyntaxHighlightView (grid) iterates layers shallow-to-deep so a
  deeper layer's styling wins within its region (Q#IJ6/IJ7).
- scoped_style_spans (wire) flattens all layers into DISJOINT effective
  spans via a boundary sweep, since the GPU re-sorts spans by start
  (replace_style_spans / merge_style_spans) and would otherwise destroy
  producer order. The GPU source_color_at consumer is fixed to fold all
  covering spans (matching semantic_client's effective_style_at) rather
  than returning the first.

Named-children exclusion: content ranges exclude only NAMED children
(matching tree-sitter-md's own inline splitter) — excluding a block
inline node's anonymous text tokens would shred the paragraph into
unparseable fragments.

13 acceptance gates (framing docs/multi-language-injections-framing.md):
layer structure, absolute child offsets, alias resolution (static +
case-folded dynamic + unknown-skip + Lua-async override), multi-range
inline, recursion bounds, wire + grid + GPU producers, incremental edit
/ new fence, many-paragraph settle budget with tail coverage, and the
single-layer regression guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
This commit is contained in:
Levi Neuwirth 2026-07-15 10:57:41 +01:00
parent 3edfa47281
commit 4282b1c333
9 changed files with 1465 additions and 164 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
builtin/runtime/syntax.lua
View File

@ -44,6 +44,20 @@ function pmacs.parse._dispatch(buf, lang)
return job_id
end
-- Injection language aliases (framing Q#IJ4). The registry holds the
-- merged map (seeded with defaults on the Rust side), and each dispatch
-- snapshots it into the parse request so the worker can resolve dynamic
-- fence names (`py` → python, `ts` → typescript). Exposed as a
-- write-through proxy so users add fence-name aliases from init.lua:
-- pmacs.parse.injection_aliases.mylang = "rust"
-- Reads are not proxied (the canonical map lives Rust-side); this is a
-- write-only extension surface.
pmacs.parse.injection_aliases = setmetatable({}, {
__newindex = function(_, alias, lang)
pmacs.parse._register_injection_alias(alias, lang)
end,
})
-- Shebang → language detection ------------------------------------------
--
-- Extension detection (`language_for_path`) misses extensionless scripts

56
pmacs-gpu/src/main.rs
View File

@ -6732,12 +6732,24 @@ fn adornment_text_color(fg: CellColor) -> glyphon::Color {
}
fn source_color_at(byte: u64, spans: &[StyleSpan]) -> Option<glyphon::Color> {
for sp in spans {
if sp.range.start <= byte && byte < sp.range.end {
return cell_color_to_glyphon(sp.style.fg);
// Fold every covering span in order, matching the semantic-client
// `effective_style_at` contract (last covering span with a non-default
// fg wins) rather than returning the first. The daemon flattens
// injection layers into disjoint spans (framing Q#IJ6), so usually at
// most one covers a byte — but where spans do overlap (a styled
// markdown parent under an injected child), the topmost color must win,
// never the outermost. Spans arrive start-sorted, so a narrower nested
// span (later start) folds after and overrides its enclosing span.
let mut color = None;
for sp in spans
.iter()
.filter(|sp| sp.range.start <= byte && byte < sp.range.end)
{
if let Some(c) = cell_color_to_glyphon(sp.style.fg) {
color = Some(c);
}
}
None
color
}
/// Convert a `pmacs-protocol::cell::Color` to a `glyphon::Color`.
@ -8969,4 +8981,40 @@ mod tests {
"the peer cursor line must keep the CurrentLine constant"
);
}
#[test]
fn source_color_folds_overlapping_child_over_parent() {
// Framing acceptance #9 (GPU consumer): a styled markdown parent
// span [0,10) (red) with an injected child span [3,6) (green) on top.
// `replace_style_spans` sorts incoming spans by start, so we mirror
// that here; `source_color_at` must FOLD the covering spans (child
// green wins at a shared byte), not return the first (parent) span.
// Bite: the pre-fix first-covering-span code returns red at byte 4.
let red = style_with_fg(CellColor::Rgb(200, 0, 0));
let green = style_with_fg(CellColor::Rgb(0, 200, 0));
let mut spans = vec![
StyleSpan {
range: ByteRange { start: 0, end: 10 },
style: red,
},
StyleSpan {
range: ByteRange { start: 3, end: 6 },
style: green,
},
];
spans.sort_by_key(|s| s.range.start); // as replace_style_spans does
// Byte 4 is covered by both: the child (green) wins the fold.
assert_eq!(
source_color_at(4, &spans),
Some(glyphon::Color::rgb(0, 200, 0)),
"the injected child color wins over the parent at a shared byte"
);
// Byte 1 is parent-only: stays red.
assert_eq!(
source_color_at(1, &spans),
Some(glyphon::Color::rgb(200, 0, 0)),
"a parent-only byte keeps the parent color"
);
}
}

4
src/async_runtime.rs
View File

@ -2700,6 +2700,7 @@ mod tests {
language_name: "rust".to_owned(),
prior_tree: None,
edits: Vec::new(),
injection_aliases: Arc::new(std::collections::HashMap::new()),
};
let id = rt.dispatch_parse(req, None);
pump_until(&rt, || rt.is_complete(id));
@ -2713,7 +2714,7 @@ mod tests {
other => panic!("unexpected outcome: {other:?}"),
}
assert_eq!(bundle.language_name, "rust");
assert_eq!(bundle.tree.root_node().kind(), "source_file");
assert_eq!(bundle.root_tree().root_node().kind(), "source_file");
// take_parse_tree was already drained, so handoff is empty.
assert_eq!(rt.parse_handoff_len(), 0);
}
@ -2729,6 +2730,7 @@ mod tests {
language_name: "rust".to_owned(),
prior_tree: None,
edits: Vec::new(),
injection_aliases: Arc::new(std::collections::HashMap::new()),
};
let id = rt.dispatch_parse(req, None);
pump_until(&rt, || rt.is_complete(id));

257
src/highlight.rs
View File

@ -39,7 +39,7 @@ use crate::buffer::Buffer;
use crate::cell::{CellCoord, CellGrid, Color, Style, UnderlineStyle};
use crate::lsp::SharedLspManager;
use crate::overlay::merge_styles;
use crate::syntax::{HighlightSpan, ParseTreeBundle, ParseViewHandle, compute_highlight_spans};
use crate::syntax::{HighlightSpan, ParseTreeBundle, ParseViewHandle, compute_highlight_spans_for};
use crate::view::{View, Viewport};
// ---------------------------------------------------------------------------
@ -248,6 +248,19 @@ pub type ThemeHandle = Arc<Mutex<Theme>>;
// SyntaxHighlightView
// ---------------------------------------------------------------------------
/// Per-layer cached highlight spans + capture names (framing Q#IJ7).
/// One entry per bundle layer that has a highlight query; kept in the
/// bundle's depth order so the render paints shallow-to-deep.
struct LayerSpans {
/// Spans for this layer, sorted wider-first per
/// [`compute_highlight_spans_for`].
spans: Vec<HighlightSpan>,
/// Capture names indexed by `HighlightSpan::capture_index` for this
/// layer's own query. Layers may use different grammars, so this is
/// per-layer, not shared.
capture_names: Arc<[String]>,
}
/// Cached per-bundle highlight state. Keyed by the underlying
/// `Arc<ParseTreeBundle>`'s identity (compared via `Arc::ptr_eq`),
/// so a freshly-installed bundle invalidates the cache.
@ -255,25 +268,19 @@ struct HighlightCache {
/// The bundle the cache was built against. `None` until the
/// first render observes a settled bundle.
bundle: Option<Arc<ParseTreeBundle>>,
/// Compiled spans, sorted wider-first per
/// [`compute_highlight_spans`].
spans: Vec<HighlightSpan>,
/// Per-layer spans in depth-ascending order (framing Q#IJ6).
layers: Vec<LayerSpans>,
/// Per-row first-byte offsets into `bundle.source`. `Vec<u32>`
/// because pmacs files cap at 4 GiB.
line_offsets: Vec<u32>,
/// Capture names indexed by `HighlightSpan::capture_index`.
/// Populated alongside `spans` so the render path doesn't need
/// to keep a reference into the [`tree_sitter::Query`].
capture_names: Arc<[String]>,
}
impl HighlightCache {
fn empty() -> Self {
Self {
bundle: None,
spans: Vec::new(),
layers: Vec::new(),
line_offsets: Vec::new(),
capture_names: Arc::from(Vec::<String>::new().into_boxed_slice()),
}
}
}
@ -283,39 +290,41 @@ impl HighlightCache {
const TAB_WIDTH: u32 = 8;
/// View that renders syntax highlighting from a tree-sitter parse
/// tree. Composes over [`crate::text_view::TextView`] per the M2.9
/// view-composition contract --- it never writes glyphs, only merges
/// styles into cells the base view has already painted.
/// tree, including its injection layers. Composes over
/// [`crate::text_view::TextView`] per the M2.9 view-composition
/// contract --- it never writes glyphs, only merges styles into cells
/// the base view has already painted.
pub struct SyntaxHighlightView {
parse: ParseViewHandle,
query: Arc<tree_sitter::Query>,
theme: ThemeHandle,
cache: HighlightCache,
}
impl SyntaxHighlightView {
/// Construct a highlight view over `parse` using `query` and
/// `theme`. The initial cache is empty; the first render with
/// a settled bundle populates it.
/// Construct a highlight view over `parse` using `theme`. Per-layer
/// highlight queries come from each `Layer::highlight_query` in the
/// bundle (resolved at settle, framing Q#IJ2), so no query is passed
/// here. The initial cache is empty; the first render with a settled
/// bundle populates it.
#[must_use]
pub fn new(parse: ParseViewHandle, query: Arc<tree_sitter::Query>, theme: ThemeHandle) -> Self {
pub fn new(parse: ParseViewHandle, theme: ThemeHandle) -> Self {
Self {
parse,
query,
theme,
cache: HighlightCache::empty(),
}
}
/// Test helper: number of cached highlight spans.
/// Test helper: total cached highlight spans across all layers.
#[must_use]
pub fn cached_span_count(&self) -> usize {
self.cache.spans.len()
self.cache.layers.iter().map(|l| l.spans.len()).sum()
}
/// Refresh `self.cache` if the parse view's current bundle
/// differs from the cached one. No-op when the bundle pointer
/// is unchanged --- the steady-state cost between parses.
/// is unchanged --- the steady-state cost between parses. Rebuilds
/// spans for every layer that carries a highlight query.
fn refresh_cache_if_stale(&mut self) {
let Some(bundle) = self.parse.current() else {
return;
@ -328,26 +337,39 @@ impl SyntaxHighlightView {
if !stale {
return;
}
let spans = compute_highlight_spans(&self.query, &bundle);
let line_offsets = compute_line_offsets(bundle.source.as_ref());
let capture_names: Arc<[String]> = self
.query
.capture_names()
.iter()
.map(|s| (*s).to_owned())
.collect();
let source = bundle.source.as_ref();
let mut layers = Vec::new();
for layer in &bundle.layers {
let Some(query) = layer.highlight_query.as_ref() else {
continue;
};
let spans = compute_highlight_spans_for(query, &layer.tree, source, None);
if spans.is_empty() {
continue;
}
let capture_names: Arc<[String]> = query
.capture_names()
.iter()
.map(|s| (*s).to_owned())
.collect();
layers.push(LayerSpans {
spans,
capture_names,
});
}
let line_offsets = compute_line_offsets(source);
self.cache = HighlightCache {
bundle: Some(bundle),
spans,
layers,
line_offsets,
capture_names,
};
}
/// Look up the style for span `s`, consulting the active theme.
fn style_for(&self, theme: &Theme, s: HighlightSpan) -> Style {
/// Look up the style for span `s` within `layer`, consulting the
/// active theme.
fn style_for(theme: &Theme, layer: &LayerSpans, s: HighlightSpan) -> Style {
let idx = s.capture_index as usize;
let Some(name) = self.cache.capture_names.get(idx) else {
let Some(name) = layer.capture_names.get(idx) else {
return theme.default_style;
};
theme.lookup(name)
@ -364,7 +386,7 @@ impl View for SyntaxHighlightView {
let Some(bundle) = self.cache.bundle.clone() else {
return;
};
if self.cache.spans.is_empty() || self.cache.line_offsets.is_empty() {
if self.cache.layers.is_empty() || self.cache.line_offsets.is_empty() {
return;
}
let source: &[u8] = bundle.source.as_ref();
@ -376,61 +398,64 @@ impl View for SyntaxHighlightView {
let cell_origin = viewport.cell_origin;
let total_lines = self.cache.line_offsets.len() as u32;
for row_offset in 0..max_rows {
let line_idx = start_line + row_offset;
if line_idx >= total_lines {
break;
}
let line_start = self.cache.line_offsets[line_idx as usize];
let line_end = self
.cache
.line_offsets
.get(line_idx as usize + 1)
.copied()
.unwrap_or(source.len() as u32);
// Trim a single trailing newline if any --- the text
// view doesn't paint it as a glyph either.
let line_end_no_nl = if line_end > line_start
&& source.get(line_end as usize - 1).copied() == Some(b'\n')
{
line_end - 1
} else {
line_end
};
let line_bytes = &source[line_start as usize..line_end_no_nl as usize];
// Paint layers shallow-to-deep (framing Q#IJ6): a deeper layer
// merges on top, so an injected child's styling wins within its
// region. Within a layer, wider-first ordering (from
// `compute_highlight_spans_for`) lets narrower captures override.
for layer in &self.cache.layers {
for row_offset in 0..max_rows {
let line_idx = start_line + row_offset;
if line_idx >= total_lines {
break;
}
let line_start = self.cache.line_offsets[line_idx as usize];
let line_end = self
.cache
.line_offsets
.get(line_idx as usize + 1)
.copied()
.unwrap_or(source.len() as u32);
// Trim a single trailing newline if any --- the text
// view doesn't paint it as a glyph either.
let line_end_no_nl = if line_end > line_start
&& source.get(line_end as usize - 1).copied() == Some(b'\n')
{
line_end - 1
} else {
line_end
};
let line_bytes = &source[line_start as usize..line_end_no_nl as usize];
// Spans whose start lies on this line. Wider-first
// ordering means parents apply before children.
// Spans that *cross* lines apply on every line they
// touch --- this loop only filters by start, then
// intersects with the line range below.
for span in self
.cache
.spans
.iter()
.filter(|s| s.start_byte < line_end_no_nl && s.end_byte > line_start)
.copied()
{
let style = self.style_for(&theme, span);
if style == Style::default() {
// Nothing to merge --- skip the per-cell loop.
continue;
}
let s_start = span.start_byte.max(line_start);
let s_end = span.end_byte.min(line_end_no_nl);
let byte_col_start = (s_start - line_start) as usize;
let byte_col_end = (s_end - line_start) as usize;
let (start_col, end_col) =
byte_range_to_display_cols(line_bytes, byte_col_start, byte_col_end);
if end_col <= start_col {
continue;
}
let cell_row = cell_origin.row + row_offset;
let clamped_start = start_col.min(max_cols);
let clamped_end = end_col.min(max_cols);
for col in clamped_start..clamped_end {
let cell = cells.at(CellCoord::new(cell_row, cell_origin.col + col));
cell.style = merge_styles(cell.style, style);
// Spans whose start lies on this line. Spans that *cross*
// lines apply on every line they touch --- this loop filters
// by intersection, then clips to the line range below.
for span in layer
.spans
.iter()
.filter(|s| s.start_byte < line_end_no_nl && s.end_byte > line_start)
.copied()
{
let style = Self::style_for(&theme, layer, span);
if style == Style::default() {
// Nothing to merge --- skip the per-cell loop.
continue;
}
let s_start = span.start_byte.max(line_start);
let s_end = span.end_byte.min(line_end_no_nl);
let byte_col_start = (s_start - line_start) as usize;
let byte_col_end = (s_end - line_start) as usize;
let (start_col, end_col) =
byte_range_to_display_cols(line_bytes, byte_col_start, byte_col_end);
if end_col <= start_col {
continue;
}
let cell_row = cell_origin.row + row_offset;
let clamped_start = start_col.min(max_cols);
let clamped_end = end_col.min(max_cols);
for col in clamped_start..clamped_end {
let cell = cells.at(CellCoord::new(cell_row, cell_origin.col + col));
cell.style = merge_styles(cell.style, style);
}
}
}
}
@ -1228,4 +1253,56 @@ mod tests {
);
assert!(!c.style.bold);
}
#[test]
fn grid_paints_injected_child_keyword() {
// Framing acceptance #10: the grid `SyntaxHighlightView` paints a
// rust keyword cell INSIDE a markdown ```rust fence — styling only
// the injected rust layer can produce. Layers paint shallow-to-deep
// (framing Q#IJ6), so the rust keyword wins over the markdown root.
use crate::buffer::{Buffer, BufferId, EditOp};
use crate::cell::{Cell, CellSize};
use crate::syntax::{ParseView, SyntaxRegistry};
let reg = SyntaxRegistry::new();
let language = reg.language("markdown").expect("markdown grammar");
// Line 0: ```rust ; line 1: fn demo() {} ; line 2: ```
let src = b"```rust\nfn demo() {}\n```\n";
let mut buf = Buffer::new(BufferId::next(), "doc.md");
buf.apply_edit(EditOp::Insert { pos: 0, bytes: src })
.unwrap();
let view = ParseView::new(&buf, language, "markdown".to_owned());
let handle = view.handle();
let _vid = buf.attach_view(Box::new(view));
let mut req = handle.make_request();
req.injection_aliases = reg.injection_alias_snapshot();
let bundle = crate::syntax::run_parse(req).expect("markdown parse");
handle.install(reg.resolve_layer_queries(&bundle));
let mut hv = SyntaxHighlightView::new(handle, reg.theme());
let cols = 40usize;
let rows = 3usize;
let mut backing: Vec<Cell> = vec![Cell::default(); rows * cols];
let mut grid = CellGrid {
cells: &mut backing,
stride: cols as u32,
size: CellSize::new(rows as u32, cols as u32),
};
let viewport = Viewport {
buffer_start: 0,
buffer_end: u64::MAX,
cell_origin: CellCoord::new(0, 0),
cell_size: CellSize::new(rows as u32, cols as u32),
gutter_w: 0,
};
let registry = buf; // keep buf alive
hv.render(&registry, viewport, &mut grid);
// `fn` is at line 1, cols 0..2. default_dark styles `keyword` bold.
let c = grid.get(CellCoord::new(1, 0));
assert!(
c.style.bold,
"the injected rust `fn` keyword is painted (bold) inside the fence"
);
}
}

50
src/lua_bindings/mod.rs
View File

@ -6362,7 +6362,7 @@ pub struct ParseNodeLua {
impl ParseNodeLua {
fn resolve(&self) -> Option<tree_sitter::Node<'_>> {
let mut node = self.bundle.tree.root_node();
let mut node = self.bundle.root_tree().root_node();
for &idx in &self.path {
node = node.child(idx)?;
}
@ -6395,7 +6395,9 @@ impl UserData for ParseTreeLua {
methods.add_method("text", |lua, this, ()| {
lua.create_string(this.0.source.as_ref())
});
methods.add_method("sexp", |_, this, ()| Ok(this.0.tree.root_node().to_sexp()));
methods.add_method("sexp", |_, this, ()| {
Ok(this.0.root_tree().root_node().to_sexp())
});
}
}
@ -6617,6 +6619,21 @@ pub fn install_parse(
)?;
}
// Injection alias write-through (framing Q#IJ4). `syntax.lua` wraps
// this in a `pmacs.parse.injection_aliases` proxy table so users write
// `pmacs.parse.injection_aliases.mylang = "rust"`. The registry holds
// the merged map (defaults + overrides); each dispatch snapshots it.
{
let s = syntax.clone();
parse_mod.set(
"_register_injection_alias",
lua.create_function(move |_, (alias, lang): (String, String)| {
s.register_injection_alias(alias, lang);
Ok(())
})?,
)?;
}
{
let s = syntax.clone();
parse_mod.set(
@ -6658,7 +6675,10 @@ pub fn install_parse(
let handle = get_or_create_parse_view(&s, &reg, id.0, &lang)?;
let req = handle.make_request();
let bundle = syntax::run_parse(req).map_err(mlua::Error::external)?;
let arc = Arc::new(bundle);
// Resolve each layer's highlight query from the registry
// cache before install so producers can style every layer
// (framing Q#IJ2 stage 2).
let arc = s.resolve_layer_queries(&bundle);
handle.install(arc.clone());
Ok(ParseTreeLua(arc))
})?,
@ -6676,7 +6696,10 @@ pub fn install_parse(
"_dispatch",
lua.create_function(move |_, (id, lang): (BufferIdLua, String)| {
let handle = get_or_create_parse_view(&s, &reg, id.0, &lang)?;
let req = handle.make_request();
let mut req = handle.make_request();
// Snapshot the alias map into the request so the worker can
// resolve dynamic fence names off the main thread (Q#IJ4).
req.injection_aliases = s.injection_alias_snapshot();
let job_id = rt.dispatch_parse(req, None);
s.record_parse_job(job_id, id.0);
Ok(job_id)
@ -6707,7 +6730,10 @@ pub fn install_parse(
return Ok(false);
};
if let Some(handle) = s.view(buf_id) {
handle.install(bundle);
// Stage 2 (framing Q#IJ2): resolve each layer's highlight
// query on the main thread before install.
let resolved = s.resolve_layer_queries(&bundle);
handle.install(resolved);
Ok(true)
} else {
Ok(false)
@ -6737,12 +6763,14 @@ pub fn install_parse(
id.0
)));
};
let Some(query) = s.highlights_query(&lang) else {
// No highlights query for this language --- treat as
// a benign no-op so callers don't need to special-case
// grammars without highlights bundled.
if s.highlights_query(&lang).is_none() {
// Root language ships no highlights --- treat as a benign
// no-op so callers don't need to special-case grammars
// without highlights bundled. (Injected child layers still
// resolve their own queries at settle; a root-highlight-less
// injector is out of v1 scope.)
return Ok(false);
};
}
let theme = s.theme();
let core = lua
.app_data_ref::<SharedCore>()
@ -6755,7 +6783,7 @@ pub fn install_parse(
id.0
)));
}
let overlay = SyntaxHighlightView::new(handle, query, theme);
let overlay = SyntaxHighlightView::new(handle, theme);
win.push_overlay(Box::new(overlay));
Ok(true)
})?,

222
src/semantic_render.rs
View File

@ -1743,12 +1743,6 @@ fn scoped_style_spans(state: &EditorState, vp: &DeclaredViewport) -> Vec<StyleSp
let Some(bundle) = handle.current() else {
return Vec::new();
};
let Some(query) = state
.syntax_registry
.highlights_query(&bundle.language_name)
else {
return Vec::new();
};
let theme = state
.syntax_registry
.theme()
@ -1756,39 +1750,121 @@ fn scoped_style_spans(state: &EditorState, vp: &DeclaredViewport) -> Vec<StyleSp
.expect("theme mutex poisoned")
.clone();
let source_len = bundle.source.len() as u64;
let source: &[u8] = bundle.source.as_ref();
let source_len = source.len() as u64;
let vis_start = vp.visible.start.min(source_len);
let vis_end = vp.visible.end.min(source_len);
if vis_end <= vis_start {
return Vec::new();
}
let capture_names = query.capture_names();
// Scope the tree-sitter capture walk to the visible byte range so
// re-styling on each edit is O(visible), not O(file) — the typing
// bottleneck on large files (framing Q#S6). Captures whose nodes
// intersect the range are returned, then clipped exactly below.
let highlights = crate::syntax::compute_highlight_spans_in_range(
&query,
&bundle,
Some(vis_start as usize..vis_end as usize),
);
let mut out = Vec::new();
for hs in highlights {
let s = u64::from(hs.start_byte).max(vis_start);
let e = u64::from(hs.end_byte).min(vis_end);
if e <= s {
continue; // No overlap with the viewport.
}
let Some(name) = capture_names.get(hs.capture_index as usize) else {
// Collect the styled spans from every injection layer, scoping each
// capture walk to the visible byte range so re-styling on each edit is
// O(visible), not O(file) (framing Q#S6). Each span carries a priority
// `(depth, order)`: a deeper layer wins over a shallower one, and
// within a layer the wider-first order lets narrower captures override
// (framing Q#IJ6). Fully-default styles are dropped (they fold as
// identity anyway).
let mut styled: Vec<StyledLayerSpan> = Vec::new();
for layer in &bundle.layers {
let Some(query) = layer.highlight_query.as_ref() else {
continue;
};
let style = theme.lookup(name);
let names = query.capture_names();
let highlights = crate::syntax::compute_highlight_spans_for(
query,
&layer.tree,
source,
Some(vis_start as usize..vis_end as usize),
);
for (order, hs) in highlights.iter().enumerate() {
let s = u64::from(hs.start_byte).max(vis_start);
let e = u64::from(hs.end_byte).min(vis_end);
if e <= s {
continue;
}
let Some(name) = names.get(hs.capture_index as usize) else {
continue;
};
let style = theme.lookup(name);
if style == Style::default() {
continue;
}
styled.push(StyledLayerSpan {
start: s,
end: e,
style,
priority: (layer.depth, order as u32),
});
}
}
flatten_layer_spans(&styled)
}
/// One styled span from a single injection layer, tagged with a priority
/// used to resolve overlaps: `(depth, order)`, higher wins. `order` is the
/// index in the layer's wider-first list, so a narrower capture (later)
/// overrides a wider one within the same layer.
struct StyledLayerSpan {
start: u64,
end: u64,
style: Style,
priority: (u16, u32),
}
/// Flatten possibly-overlapping per-layer styled spans into **disjoint**
/// `StyleSpan`s whose per-byte style is the priority-ordered fold of every
/// covering span (framing Q#IJ6). Emitting disjoint spans makes the result
/// robust to the GPU wire re-sorting spans by start (`replace_style_spans`
/// / `merge_style_spans`), which would otherwise destroy producer order.
/// A boundary sweep over the (viewport-bounded) span endpoints; adjacent
/// equal-style runs are merged for wire economy.
fn flatten_layer_spans(styled: &[StyledLayerSpan]) -> Vec<StyleSpan> {
if styled.is_empty() {
return Vec::new();
}
let mut bounds: Vec<u64> = Vec::with_capacity(styled.len() * 2);
for sp in styled {
bounds.push(sp.start);
bounds.push(sp.end);
}
bounds.sort_unstable();
bounds.dedup();
let mut out: Vec<StyleSpan> = Vec::new();
for win in bounds.windows(2) {
let (a, b) = (win[0], win[1]);
if b <= a {
continue;
}
// Fold every span covering [a, b) in ascending priority so a
// deeper/narrower span overrides, while a span that only sets a
// non-color attribute still composes (matches the semantic-client
// `effective_style_at` contract).
let mut covering: Vec<&StyledLayerSpan> = styled
.iter()
.filter(|sp| sp.start <= a && sp.end >= b)
.collect();
if covering.is_empty() {
continue;
}
covering.sort_by_key(|sp| sp.priority);
let mut style = Style::default();
for sp in covering {
style = crate::overlay::merge_styles(style, sp.style);
}
if style == Style::default() {
continue; // Nothing to render — skip the wire byte.
continue;
}
if let Some(last) = out.last_mut()
&& last.range.end == a
&& last.style == style
{
last.range.end = b;
continue;
}
out.push(StyleSpan {
range: ByteRange { start: s, end: e },
range: ByteRange { start: a, end: b },
style,
});
}
@ -3300,7 +3376,9 @@ mod tests {
let handle = parse_view.handle();
let req = handle.make_request();
let bundle = crate::syntax::run_parse(req).expect("initial rust parse");
handle.install(std::sync::Arc::new(bundle));
// Mirror the production settle path: resolve each layer's highlight
// query before install so the producer can style it (framing Q#IJ2).
handle.install(state.syntax_registry.resolve_layer_queries(&bundle));
buf.attach_view(Box::new(parse_view));
drop(registry);
core.set_buffer_path(buffer_id, Some(std::path::PathBuf::from("/tmp/x.rs")));
@ -3309,6 +3387,90 @@ mod tests {
handle
}
fn seed_markdown_parse_view(
state: &EditorState,
buffer_id: BufferId,
text: &[u8],
) -> crate::syntax::ParseViewHandle {
let language = state
.syntax_registry
.language("markdown")
.expect("markdown grammar");
let mut core = state.core.borrow_mut();
let registry_handle = core.registry.clone();
let mut registry = registry_handle.borrow_mut();
let buf = registry.get_mut(buffer_id).expect("active buffer");
if !text.is_empty() {
buf.apply_edit(crate::buffer::EditOp::Insert {
pos: 0,
bytes: text,
})
.expect("seed markdown text");
}
let parse_view = crate::syntax::ParseView::new(buf, language, "markdown".to_owned());
let handle = parse_view.handle();
let mut req = handle.make_request();
req.injection_aliases = state.syntax_registry.injection_alias_snapshot();
let bundle = crate::syntax::run_parse(req).expect("markdown parse");
handle.install(state.syntax_registry.resolve_layer_queries(&bundle));
buf.attach_view(Box::new(parse_view));
drop(registry);
core.set_buffer_path(buffer_id, Some(std::path::PathBuf::from("/tmp/x.md")));
drop(core);
state.syntax_registry.attach_view(buffer_id, handle.clone());
handle
}
#[test]
fn wire_producer_emits_disjoint_child_spans_in_fence() {
// Framing acceptance #8: `scoped_style_spans` over a ```rust fence
// emits a styled span on the `fn` keyword INSIDE the fence — which
// only the injected rust layer can produce (the markdown root has
// no keyword styling there). The pre-injection single-layer producer
// fails this. Emitted spans are disjoint (framing Q#IJ6 flatten).
let state = empty_state();
let bid = active_buffer(&state);
let src = b"# T\n\n```rust\nfn demo() {}\n```\n";
seed_markdown_parse_view(&state, bid, src);
let vp = DeclaredViewport {
buffer_id: bid,
visible: ByteRange {
start: 0,
end: src.len() as u64,
},
frontend_generation: 0,
};
let spans = scoped_style_spans(&state, &vp);
assert!(!spans.is_empty(), "layered producer emits style spans");
// Disjoint (flattened): no two spans overlap.
let mut sorted = spans.clone();
sorted.sort_by_key(|s| s.range.start);
for w in sorted.windows(2) {
assert!(
w[0].range.end <= w[1].range.start,
"flattened spans must be disjoint: {:?} then {:?}",
w[0].range,
w[1].range
);
}
// The `fn` keyword inside the fence carries a non-default style.
let fn_off = src
.windows(2)
.position(|w| w == b"fn")
.expect("`fn` present in source") as u64;
let covering = spans
.iter()
.find(|s| s.range.start <= fn_off && fn_off < s.range.end)
.expect("a span covers the `fn` keyword inside the fence");
assert_ne!(
covering.style,
Style::default(),
"the injected rust keyword is styled"
);
}
#[test]
fn cpp_style_comes_from_lsp_when_no_tree_sitter_grammar() {
let state = empty_state();
@ -3402,7 +3564,7 @@ mod tests {
);
let bundle = crate::syntax::run_parse(req).expect("settled rust parse");
handle.install(std::sync::Arc::new(bundle));
handle.install(state.syntax_registry.resolve_layer_queries(&bundle));
assert_eq!(state.syntax_registry.take_parse_job(9001), Some(bid));
let settled = s.render_frame(&state);
assert!(

852
src/syntax.rs
View File

File diff suppressed because it is too large Load Diff

165
tests/injection_acceptance.rs Normal file
View File

@ -0,0 +1,165 @@
// injection_acceptance.rs --- multi-language injection acceptance gates.
//! Multi-language injection acceptance gates that need the public API
//! surface (the Lua async path and a settle-time perf budget). The
//! layer-structure, alias-resolution, producer, and GPU gates live next to
//! the code they exercise (`syntax.rs` / `semantic_render.rs` /
//! `highlight.rs` / `pmacs-gpu`), where `run_parse`, `scoped_style_spans`,
//! and `source_color_at` are reachable. See the framing acceptance list in
//! `docs/multi-language-injections-framing.md`.
use std::fmt::Write as _;
use std::time::{Duration, Instant};
use pmacs::buffer::{Buffer, BufferId, EditOp};
use pmacs::editor::EditorState;
use pmacs::lua_bindings::BufferIdLua;
use pmacs::syntax::{self, ParseView, SyntaxRegistry};
/// Drive `tick_async` until `predicate` holds or a deadline passes.
fn pump_async<F: Fn(&EditorState) -> bool>(state: &mut EditorState, predicate: F) {
let deadline = Instant::now() + Duration::from_secs(3);
while !predicate(state) {
assert!(Instant::now() < deadline, "async pump deadline exceeded");
state.tick_async();
std::thread::sleep(Duration::from_millis(2));
}
}
/// Framing acceptance #5 (full Lua async path): an alias added from Lua via
/// `pmacs.parse.injection_aliases` must reach the parse worker through the
/// dispatch snapshot, so an *asynchronously* dispatched injected parse
/// resolves a fence named by the new alias. Testing only the static
/// resolver would leave the Lua write-through + snapshot bridge unproven.
#[test]
fn lua_alias_override_resolves_on_async_parse() {
let mut state = EditorState::new();
// Add a bespoke fence alias from Lua (write-through to the registry).
state
.lua_host
.lua()
.load(r#"pmacs.parse.injection_aliases.mydsl = "rust""#)
.exec()
.expect("set injection alias from Lua");
// A markdown buffer whose fence uses the new alias.
let src = b"# Doc\n\n```mydsl\nfn injected() { let x = 1; }\n```\n";
let buf_id = state
.lua_host
.registry()
.borrow_mut()
.create_from_bytes("doc.md".to_owned(), src);
state
.lua_host
.lua()
.globals()
.set("BUF", BufferIdLua(buf_id))
.expect("bind BUF");
// Dispatch asynchronously (the wrapped `_dispatch` records the job; the
// per-tick settle path installs the resolved bundle).
state
.lua_host
.lua()
.load("pmacs.parse._dispatch(BUF, 'markdown')")
.exec()
.expect("async dispatch");
pump_async(&mut state, |s| {
s.syntax_registry
.view(buf_id)
.and_then(|h| h.current())
.is_some()
});
let bundle = state
.syntax_registry
.view(buf_id)
.and_then(|h| h.current())
.expect("settled bundle");
assert!(
bundle.layers.iter().any(|l| l.language_name == "rust"),
"the Lua-set `mydsl` alias resolved the fence to a rust child layer; \
layers: {:?}",
bundle
.layers
.iter()
.map(|l| l.language_name.as_str())
.collect::<Vec<_>>()
);
}
/// Framing acceptance #12: a large all-inline markdown buffer settles
/// (root + one cold inline layer per paragraph) within a comfortable
/// budget, and the FINAL paragraph still receives an inline layer — the
/// tail is not silently dropped. This is the measured guard that keeps
/// child-incrementality (Q#IJ8) out of v1.
#[test]
fn many_paragraph_settle_under_budget_with_tail_covered() {
let reg = SyntaxRegistry::new();
let n = 200usize;
let mut src = String::new();
for i in 0..n {
// A blank line separates paragraphs; each carries emphasis + a link
// so the block grammar injects a markdown_inline layer for it.
writeln!(
src,
"Paragraph {i} with *emphasis* and a [link](http://x).\n"
)
.expect("write");
}
// A distinctly-marked final paragraph.
let marker = "FINALPARAGRAPH";
writeln!(src, "{marker} with *stress*.\n").expect("write");
let language = reg.language("markdown").expect("markdown grammar");
let mut buf = Buffer::new(BufferId::next(), "big.md");
buf.apply_edit(EditOp::Insert {
pos: 0,
bytes: src.as_bytes(),
})
.expect("seed markdown");
let view = ParseView::new(&buf, language, "markdown".to_owned());
let handle = view.handle();
let _vid = buf.attach_view(Box::new(view));
let mut req = handle.make_request();
req.injection_aliases = reg.injection_alias_snapshot();
let start = Instant::now();
let bundle = syntax::run_parse(req).expect("layered markdown parse");
let elapsed = start.elapsed();
// Catastrophic-regression guard: cold-parsing ~200 tiny inline layers
// is milliseconds of work; a generous ceiling avoids debug/CI flakiness
// while still catching a blow-up (e.g. accidental O(n^2) layer work).
assert!(
elapsed < Duration::from_secs(2),
"many-paragraph settle took {elapsed:?}, exceeds the budget"
);
let inline_count = bundle
.layers
.iter()
.filter(|l| l.language_name == "markdown_inline")
.count();
assert!(
inline_count >= 100,
"most paragraphs produce an inline layer; got {inline_count}"
);
// Tail coverage: an inline layer's tree spans the final paragraph.
let marker_off = src.find(marker).expect("marker present");
let tail_covered = bundle
.layers
.iter()
.filter(|l| l.language_name == "markdown_inline")
.any(|l| {
let r = l.tree.root_node();
(r.start_byte()..r.end_byte()).contains(&marker_off)
});
assert!(
tail_covered,
"the final paragraph still receives an inline layer (no tail loss)"
);
}

9
tests/m4_acceptance.rs
View File

@ -125,9 +125,10 @@ fn m4_1_initial_parse_of_5000_line_file_under_100ms() {
language_name: "rust".to_owned(),
prior_tree: None,
edits: Vec::new(),
injection_aliases: Arc::new(std::collections::HashMap::new()),
};
let bundle = syntax::run_parse(req).expect("parse succeeds");
assert_eq!(bundle.tree.root_node().kind(), "source_file");
assert_eq!(bundle.root_tree().root_node().kind(), "source_file");
assert!(
bundle.parse_duration < Duration::from_millis(100),
"5000-line cold parse took {:?}, exceeds 100 ms budget",
@ -204,7 +205,7 @@ fn m4_1_incremental_parse_under_5ms() {
let req = handle.make_request();
let bundle = syntax::run_parse(req).expect("incremental parse");
assert_eq!(bundle.tree.root_node().kind(), "source_file");
assert_eq!(bundle.root_tree().root_node().kind(), "source_file");
assert!(
bundle.parse_duration < Duration::from_millis(5),
"incremental parse took {:?}, exceeds 5 ms budget",
@ -226,6 +227,7 @@ fn m4_1_dispatch_parse_round_trips_via_runtime() {
language_name: "rust".to_owned(),
prior_tree: None,
edits: Vec::new(),
injection_aliases: Arc::new(std::collections::HashMap::new()),
};
let id = rt.dispatch_parse(req, None);
let deadline = Instant::now() + Duration::from_secs(5);
@ -243,7 +245,7 @@ fn m4_1_dispatch_parse_round_trips_via_runtime() {
}
other => panic!("unexpected outcome: {other:?}"),
}
assert_eq!(bundle.tree.root_node().kind(), "source_file");
assert_eq!(bundle.root_tree().root_node().kind(), "source_file");
assert_eq!(bundle.language_name, "rust");
}
@ -542,6 +544,7 @@ fn m4_3_open_rust_file_highlights_under_100ms() {
language_name: "rust".to_owned(),
prior_tree: None,
edits: Vec::new(),
injection_aliases: Arc::new(std::collections::HashMap::new()),
};
let bundle = syntax::run_parse(req).expect("parse succeeds");
let spans = syntax::compute_highlight_spans(&query, &bundle);