fix(injections): PR #122 round 1 — sweep flatten, sync aliases, real multi-range, surfaced cap
Four review findings + a cleanup bundle. [P1] Wire flattener was O(spans²) and ran over the WHOLE buffer (the file-style summary uses a whole-buffer viewport, not the visible one). Replaced the per-interval full scan with an ordered active-set event sweep (activate on start, expire on end, fold the active set) — linear in practice. Added full_buffer_summary_scales_on_large_grammar_file (1500-line rust) as the perf gate. [P2] _parse_now used the empty alias map from make_request while _dispatch snapshotted the registry map, so a `py` fence injected async but not sync. Snapshot aliases on both paths; pinned by sync_parse_now_resolves_alias. [P2] The multi-range inline test used a one-line paragraph, whose block inline node has no named children (link/emphasis are child-grammar structures) — one range, so it couldn't falsify multi-range. Replaced with a multi-line blockquote whose inline node carries a named block_continuation: content_node_ranges now asserts >1 collected range and emphasis parses on both lines. [P2] The layer backstop dropped regions silently; the framing requires a surfaced warning. run_parse now sets ParseTreeBundle::injection_capped; syntax.lua's settle tick raises it once per buffer via pmacs.error (_injection_capped). Added injection_layer_cap_surfaces_and_preserves_root (drives >4096 fences, asserts the flag + bounded count + intact root). Cleanup: - The GPU acceptance test now drives the real StyleSpans full-frame transform (spans_from_segments, extracted from replace_style_spans) instead of a hand-rolled sort. - content_node_ranges excludes NAMED children (documented as a round-1 refinement); framing mechanic #3 / Q#IJ5 updated to match. - parse_duration doc now says root parse; the markdown entry no longer describes inline as unhighlighted/future. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
This commit is contained in:
parent
4282b1c333
commit
79d75a29e0
|
|
@ -22,6 +22,10 @@ local inflight_parse_by_buffer = {}
|
|||
local parse_buffer_by_key = {}
|
||||
local parse_lang_by_buffer = {}
|
||||
local reparse_requested_by_buffer = {}
|
||||
-- Buffers already warned about hitting the injection layer cap (Q#IJ3);
|
||||
-- keyed like the others so we warn once, and re-arm if the file stops
|
||||
-- capping (an edit removed the excess regions).
|
||||
local injection_cap_warned = {}
|
||||
|
||||
local raw_dispatch = pmacs.parse._dispatch
|
||||
|
||||
|
|
@ -338,6 +342,19 @@ pmacs._async.tick = function(...)
|
|||
if pmacs._async._is_complete(job_id) then
|
||||
local key = parse_job_buffer_keys[job_id]
|
||||
pmacs.parse._install_settled(job_id)
|
||||
-- Surface the injection layer backstop (Q#IJ3) once per buffer rather
|
||||
-- than dropping embedded regions silently. Best-effort: a missing
|
||||
-- buffer or error here must not stall the settle loop.
|
||||
local capped_buf = key and parse_buffer_by_key[key]
|
||||
if capped_buf and pmacs.parse._injection_capped(capped_buf) then
|
||||
if not injection_cap_warned[key] and pmacs.error then
|
||||
injection_cap_warned[key] = true
|
||||
pmacs.error(
|
||||
"syntax: injection layer cap reached; some embedded regions are unhighlighted")
|
||||
end
|
||||
elseif key then
|
||||
injection_cap_warned[key] = nil
|
||||
end
|
||||
pending_parse_jobs[job_id] = nil
|
||||
parse_job_buffer_keys[job_id] = nil
|
||||
if key and inflight_parse_by_buffer[key] == job_id then
|
||||
|
|
|
|||
|
|
@ -83,15 +83,17 @@ docs):
|
|||
@injection.language)` + `(code_fence_content) @injection.content`, and
|
||||
static `((inline) @injection.content (#set! injection.language
|
||||
"markdown_inline"))` (also `html`/`yaml`/`toml`).
|
||||
3. **The generic injection contract excludes child ranges and intersects
|
||||
with the parent.** Unless `#set! injection.include-children` is set,
|
||||
the injected ranges are the content node's extent **minus its direct
|
||||
children's ranges**, then **intersected with the parent layer's own
|
||||
included ranges** so a nested injection cannot reintroduce bytes its
|
||||
parent excluded (tree-sitter highlighting docs; the bundled markdown
|
||||
convenience parser shows a markdown-specialized form of the child
|
||||
split at `parser.rs:406-425`). This makes `markdown_inline` a genuine
|
||||
multi-range case (Q#IJ5).
|
||||
3. **The injection contract excludes child ranges and intersects with the
|
||||
parent.** Unless `#set! injection.include-children` is set, the injected
|
||||
ranges are the content node's extent **minus its NAMED children's
|
||||
ranges**, then **intersected with the parent layer's own included
|
||||
ranges** so a nested injection cannot reintroduce bytes its parent
|
||||
excluded. *(Implementation note, round 1: excluding ALL children — not
|
||||
just named — shreds a markdown block `inline` node, whose children are
|
||||
anonymous text tokens, into unparseable fragments. Excluding only named
|
||||
children matches `tree-sitter-md`'s own inline splitter,
|
||||
`parser.rs:406-425`, and is correct for every real injection site.)*
|
||||
This makes `markdown_inline` a genuine multi-range case (Q#IJ5).
|
||||
4. `LanguageEntry.loader` (`fn() -> Language`) and query-source `&'static
|
||||
str` consts are `Send` and touch no grammar C-object until called — a
|
||||
worker resolves injected languages **lazily** by indexing `&'static
|
||||
|
|
@ -154,9 +156,13 @@ in v1. Every headline case is bundled.
|
|||
- **max depth** (default 3),
|
||||
- **max total layer count** — a *runaway backstop* set well above any
|
||||
real document (default **4096**), decoupled from performance; the real
|
||||
perf bound is the Q#IJ10 settle-time guard. Hitting it is **logged, not
|
||||
silent** (surfaced via `pmacs.error`/`*workers*`), and a modest markdown
|
||||
doc's inline layers (one per paragraph/heading) sit far under it.
|
||||
perf bound is the Q#IJ10 settle-time guard. Hitting it is **surfaced, not
|
||||
silent**: `run_parse` sets a `ParseTreeBundle::injection_capped` flag,
|
||||
and `syntax.lua`'s settle tick raises it once per buffer via
|
||||
`pmacs.error` (`_injection_capped`). A modest markdown doc's inline
|
||||
layers (one per paragraph/heading) sit far under the backstop. A real
|
||||
boundary test drives just over 4096 fences and asserts the flag + capped
|
||||
count + intact root.
|
||||
- a **visited set on `(language_name, ranges)`** so a same-language
|
||||
self-injection over a non-shrinking region can't reproduce itself.
|
||||
|
||||
|
|
@ -190,14 +196,18 @@ A layer's ranges are built from its `@injection.content` node(s):
|
|||
|
||||
- `#set! injection.include-children` → `[node.range]`;
|
||||
- otherwise (default; the markdown-inline case) → the node's extent
|
||||
**minus its direct children's ranges**, **then intersected with the
|
||||
parent layer's included ranges** (mechanic #3). Ranges come out ordered
|
||||
and non-overlapping; empty ranges dropped.
|
||||
**minus its NAMED children's ranges** (anonymous token children are the
|
||||
injected text itself and are kept — mechanic #3), **then intersected
|
||||
with the parent layer's included ranges**. Ranges come out ordered and
|
||||
non-overlapping; empty ranges dropped.
|
||||
|
||||
Fenced code (`code_fence_content`, no children) → one range; an inline
|
||||
paragraph with a link and emphasis → several. Core, not deferred —
|
||||
`markdown_inline` needs it. Acceptance parses an inline paragraph with a
|
||||
link **and** emphasis, not just plain text.
|
||||
Fenced code (`code_fence_content`, no children) → one range; a **multi-line**
|
||||
container (a blockquote/list whose inline node carries a named
|
||||
`block_continuation` child) → several. Core, not deferred —
|
||||
`markdown_inline` needs it. Acceptance asserts `content_node_ranges`
|
||||
returns **>1 range** for a multi-line blockquote and that both sides parse
|
||||
and highlight (a one-line paragraph would give a single range and could
|
||||
not falsify multi-range support).
|
||||
|
||||
### Q#IJ6 — The wire producer flattens layers into disjoint effective spans
|
||||
|
||||
|
|
@ -308,27 +318,39 @@ half-styled frame. `grammar_style_parse_not_ready` unchanged.
|
|||
offsets matching its position in the **full** markdown source.
|
||||
4. `dynamic_alias_resolves` — ` ```py `→python, ` ```JS `→javascript
|
||||
(case-folded); ` ```nonsense ` → no child, no error, root intact.
|
||||
5. `alias_override_from_lua_async` — mutate `injection_aliases` in Lua,
|
||||
then an **async** injected parse resolves the new alias (Q#IJ4 bridge).
|
||||
6. `inline_layer_multi_range_link_and_emphasis` — an inline paragraph
|
||||
with a link **and** emphasis becomes a `markdown_inline` layer whose
|
||||
ranges exclude the direct children and highlight correctly (Q#IJ5).
|
||||
7. `recursion_bounds_hold` — depth cap, layer-count backstop, and
|
||||
`(language, ranges)` visited guard each terminate; a failing child
|
||||
drops only itself, root installs (Q#IJ3).
|
||||
8. `wire_producer_emits_child_spans` — `scoped_style_spans` over a
|
||||
` ```rust ` fence emits disjoint `StyleSpan`s covering a rust keyword
|
||||
**inside** the fence. **Bite-verified** vs the single-layer producer.
|
||||
9. `gpu_overlapping_child_color_wins` — parent-red / child-green overlap
|
||||
applied through `replace_style_spans` then rendered: the child (green)
|
||||
wins. **Bite-verified** vs the current first-span-wins path (Q#IJ6).
|
||||
10. `grid_producer_paints_child_span` — `SyntaxHighlightView` paints a
|
||||
5. `lua_alias_override_resolves_on_async_parse` — mutate
|
||||
`injection_aliases` in Lua, then an **async** injected parse resolves
|
||||
the new alias (Q#IJ4 bridge). Plus `sync_parse_now_resolves_alias`
|
||||
(round 1): the same must hold on the **synchronous** `_parse_now` path.
|
||||
6. `inline_layer_multi_range_excludes_block_continuation` — a **multi-line**
|
||||
blockquote's inline node carries a named `block_continuation`;
|
||||
`content_node_ranges` returns **>1 range** and both sides parse +
|
||||
highlight (Q#IJ5). (A one-line paragraph gives a single range and can't
|
||||
falsify multi-range.)
|
||||
7. `recursion_bounds_terminate` — rust macro self-injection terminates
|
||||
within the depth bound; `injection_layer_cap_surfaces_and_preserves_root`
|
||||
drives >4096 fences and asserts the surfaced `injection_capped` flag,
|
||||
the bounded count, and an intact root; a failing child drops only itself
|
||||
(Q#IJ3).
|
||||
8. `wire_producer_emits_disjoint_child_spans_in_fence` — `scoped_style_spans`
|
||||
over a ` ```rust ` fence emits disjoint `StyleSpan`s covering a rust
|
||||
keyword **inside** the fence. **Bite-verified** vs the single-layer
|
||||
producer. Plus `full_buffer_summary_scales_on_large_grammar_file`
|
||||
(round 1): the whole-buffer summary path stays ~linear under the event
|
||||
sweep (a quadratic flatten regresses it).
|
||||
9. `source_color_folds_overlapping_child_over_parent` — parent-red /
|
||||
child-green overlap driven through the real `spans_from_segments`
|
||||
(`replace_style_spans` body): the child (green) wins the fold.
|
||||
**Bite-verified** vs the first-span-wins path (Q#IJ6).
|
||||
10. `grid_paints_injected_child_keyword` — `SyntaxHighlightView` paints a
|
||||
rust-keyword cell inside the fence.
|
||||
11. `incremental_edit_reflects_in_child` — editing inside a fence shows in
|
||||
child spans after reparse; a **new** fence adds a layer (Q#IJ8).
|
||||
12. `many_paragraph_settle_under_budget_tail_covered` — a large all-inline
|
||||
markdown buffer settles within a comfortable budget **and the final
|
||||
paragraph receives a layer/capture** (Q#IJ3 cap + Q#IJ10 guard).
|
||||
11. `incremental_edit_reflects_in_child_and_new_fence_adds_layer` — editing
|
||||
inside a fence shows in child spans after reparse; a **new** fence adds
|
||||
a layer (Q#IJ8).
|
||||
12. `many_paragraph_settle_under_budget_with_tail_covered` — a large
|
||||
all-inline markdown buffer settles within a comfortable budget **and
|
||||
the final paragraph receives an inline layer** (Q#IJ3 cap + Q#IJ10
|
||||
guard).
|
||||
13. `non_injecting_buffer_single_layer` — a plain `.rs` buffer still
|
||||
yields exactly one layer (regression guard).
|
||||
|
||||
|
|
|
|||
|
|
@ -4126,11 +4126,7 @@ impl State {
|
|||
/// `full = true` path: discard prior styling, take the segments'
|
||||
/// spans as authoritative for the declared viewport.
|
||||
fn replace_style_spans(&mut self, segments: Vec<StyleSegment>) {
|
||||
self.current_spans.clear();
|
||||
for seg in segments {
|
||||
self.current_spans.extend(seg.spans);
|
||||
}
|
||||
self.current_spans.sort_by_key(|s| s.range.start);
|
||||
self.current_spans = spans_from_segments(segments);
|
||||
}
|
||||
|
||||
/// `full = false` path: each segment's `range` authoritatively
|
||||
|
|
@ -6731,6 +6727,21 @@ fn adornment_text_color(fg: CellColor) -> glyphon::Color {
|
|||
cell_color_to_glyphon(fg).unwrap_or_else(|| glyphon::Color::rgb(130, 130, 140))
|
||||
}
|
||||
|
||||
/// The `StyleSpans { full: true }` transform `replace_style_spans` applies:
|
||||
/// flatten every segment's spans and start-sort. Extracted as a free
|
||||
/// function so it (and `source_color_at`) can be exercised without a live
|
||||
/// `State` — the start-sort is the step that makes producer depth-order
|
||||
/// irrelevant on the wire, which is why the daemon flattens to disjoint
|
||||
/// spans and this consumer folds (framing Q#IJ6).
|
||||
fn spans_from_segments(segments: Vec<StyleSegment>) -> Vec<StyleSpan> {
|
||||
let mut spans: Vec<StyleSpan> = Vec::new();
|
||||
for seg in segments {
|
||||
spans.extend(seg.spans);
|
||||
}
|
||||
spans.sort_by_key(|s| s.range.start);
|
||||
spans
|
||||
}
|
||||
|
||||
fn source_color_at(byte: u64, spans: &[StyleSpan]) -> Option<glyphon::Color> {
|
||||
// Fold every covering span in order, matching the semantic-client
|
||||
// `effective_style_at` contract (last covering span with a non-default
|
||||
|
|
@ -8985,24 +8996,29 @@ mod tests {
|
|||
#[test]
|
||||
fn source_color_folds_overlapping_child_over_parent() {
|
||||
// Framing acceptance #9 (GPU consumer): a styled markdown parent
|
||||
// span [0,10) (red) with an injected child span [3,6) (green) on top.
|
||||
// `replace_style_spans` sorts incoming spans by start, so we mirror
|
||||
// that here; `source_color_at` must FOLD the covering spans (child
|
||||
// green wins at a shared byte), not return the first (parent) span.
|
||||
// Bite: the pre-fix first-covering-span code returns red at byte 4.
|
||||
// span [0,10) (red) with an injected child span [3,6) (green) on top,
|
||||
// driven through the ACTUAL `StyleSpans { full: true }` transform
|
||||
// (`spans_from_segments`, the body of `replace_style_spans`) rather
|
||||
// than a hand-rolled sort. `source_color_at` must FOLD the covering
|
||||
// spans (child green wins at a shared byte), not return the first
|
||||
// (parent) span. Bite: the pre-fix first-covering-span code returns
|
||||
// red at byte 4.
|
||||
let red = style_with_fg(CellColor::Rgb(200, 0, 0));
|
||||
let green = style_with_fg(CellColor::Rgb(0, 200, 0));
|
||||
let mut spans = vec![
|
||||
StyleSpan {
|
||||
range: ByteRange { start: 0, end: 10 },
|
||||
style: red,
|
||||
},
|
||||
StyleSpan {
|
||||
range: ByteRange { start: 3, end: 6 },
|
||||
style: green,
|
||||
},
|
||||
];
|
||||
spans.sort_by_key(|s| s.range.start); // as replace_style_spans does
|
||||
let segments = vec![StyleSegment {
|
||||
range: ByteRange { start: 0, end: 10 },
|
||||
spans: vec![
|
||||
StyleSpan {
|
||||
range: ByteRange { start: 0, end: 10 },
|
||||
style: red,
|
||||
},
|
||||
StyleSpan {
|
||||
range: ByteRange { start: 3, end: 6 },
|
||||
style: green,
|
||||
},
|
||||
],
|
||||
}];
|
||||
let spans = spans_from_segments(segments); // full-frame message application
|
||||
|
||||
// Byte 4 is covered by both: the child (green) wins the fold.
|
||||
assert_eq!(
|
||||
|
|
|
|||
|
|
@ -6652,6 +6652,21 @@ pub fn install_parse(
|
|||
)?;
|
||||
}
|
||||
|
||||
// True if the buffer's installed bundle hit the injection layer backstop
|
||||
// (framing Q#IJ3). `syntax.lua` reads this after settle to surface the
|
||||
// cap once via `pmacs.error` rather than dropping regions silently.
|
||||
{
|
||||
let s = syntax.clone();
|
||||
parse_mod.set(
|
||||
"_injection_capped",
|
||||
lua.create_function(move |_, id: BufferIdLua| {
|
||||
Ok(s.view(id.0)
|
||||
.and_then(|h| h.current())
|
||||
.is_some_and(|b| b.injection_capped))
|
||||
})?,
|
||||
)?;
|
||||
}
|
||||
|
||||
{
|
||||
let s = syntax.clone();
|
||||
parse_mod.set(
|
||||
|
|
@ -6673,7 +6688,11 @@ pub fn install_parse(
|
|||
"_parse_now",
|
||||
lua.create_function(move |_, (id, lang): (BufferIdLua, String)| {
|
||||
let handle = get_or_create_parse_view(&s, ®, id.0, &lang)?;
|
||||
let req = handle.make_request();
|
||||
let mut req = handle.make_request();
|
||||
// Snapshot the alias map on the sync path too (framing Q#IJ4)
|
||||
// — otherwise a `py` fence or a Lua-added alias would resolve
|
||||
// asynchronously but not through `_parse_now`.
|
||||
req.injection_aliases = s.injection_alias_snapshot();
|
||||
let bundle = syntax::run_parse(req).map_err(mlua::Error::external)?;
|
||||
// Resolve each layer's highlight query from the registry
|
||||
// cache before install so producers can style every layer
|
||||
|
|
|
|||
|
|
@ -1823,6 +1823,14 @@ fn flatten_layer_spans(styled: &[StyledLayerSpan]) -> Vec<StyleSpan> {
|
|||
if styled.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
// Boundary sweep. Every unique span endpoint is a boundary; between
|
||||
// consecutive boundaries the covering set is constant. An ordered
|
||||
// active-set (activate on start, expire on end) keeps each interval's
|
||||
// fold O(active) rather than O(all spans), so the whole pass is
|
||||
// O(spans·log spans + Σ active) — linear in practice (active is bounded
|
||||
// by overlap depth, not the total span count). This matters because the
|
||||
// file-style summary runs this over the *entire* buffer, not just the
|
||||
// viewport.
|
||||
let mut bounds: Vec<u64> = Vec::with_capacity(styled.len() * 2);
|
||||
for sp in styled {
|
||||
bounds.push(sp.start);
|
||||
|
|
@ -1831,27 +1839,36 @@ fn flatten_layer_spans(styled: &[StyledLayerSpan]) -> Vec<StyleSpan> {
|
|||
bounds.sort_unstable();
|
||||
bounds.dedup();
|
||||
|
||||
// Span indices ordered by start; activated as the sweep reaches them.
|
||||
let mut by_start: Vec<usize> = (0..styled.len()).collect();
|
||||
by_start.sort_by_key(|&i| styled[i].start);
|
||||
let mut next = 0usize;
|
||||
|
||||
// Active covering spans, kept in ascending-priority order so the fold
|
||||
// is a single in-order pass.
|
||||
let mut active: Vec<usize> = Vec::new();
|
||||
|
||||
let mut out: Vec<StyleSpan> = Vec::new();
|
||||
for win in bounds.windows(2) {
|
||||
let (a, b) = (win[0], win[1]);
|
||||
if b <= a {
|
||||
// Activate spans starting at or before `a` (each activates once).
|
||||
while next < by_start.len() && styled[by_start[next]].start <= a {
|
||||
let idx = by_start[next];
|
||||
let pos = active.partition_point(|&j| styled[j].priority < styled[idx].priority);
|
||||
active.insert(pos, idx);
|
||||
next += 1;
|
||||
}
|
||||
// Expire spans that ended at or before `a` (ranges are half-open).
|
||||
active.retain(|&j| styled[j].end > a);
|
||||
if active.is_empty() {
|
||||
continue;
|
||||
}
|
||||
// Fold every span covering [a, b) in ascending priority so a
|
||||
// deeper/narrower span overrides, while a span that only sets a
|
||||
// non-color attribute still composes (matches the semantic-client
|
||||
// `effective_style_at` contract).
|
||||
let mut covering: Vec<&StyledLayerSpan> = styled
|
||||
.iter()
|
||||
.filter(|sp| sp.start <= a && sp.end >= b)
|
||||
.collect();
|
||||
if covering.is_empty() {
|
||||
continue;
|
||||
}
|
||||
covering.sort_by_key(|sp| sp.priority);
|
||||
// Fold the active set in ascending priority: a deeper/narrower span
|
||||
// overrides, an attribute-only span still composes (matches the
|
||||
// semantic-client `effective_style_at` contract).
|
||||
let mut style = Style::default();
|
||||
for sp in covering {
|
||||
style = crate::overlay::merge_styles(style, sp.style);
|
||||
for &j in &active {
|
||||
style = crate::overlay::merge_styles(style, styled[j].style);
|
||||
}
|
||||
if style == Style::default() {
|
||||
continue;
|
||||
|
|
@ -3471,6 +3488,36 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_buffer_summary_scales_on_large_grammar_file() {
|
||||
// Perf gate (round-2 finding 1): the file-style summary runs the
|
||||
// flattener over the WHOLE buffer, not the viewport. The ordered
|
||||
// active-set sweep must keep that roughly linear — the pre-sweep
|
||||
// O(spans^2) flatten stalled a large grammar-backed file here.
|
||||
use std::fmt::Write as _;
|
||||
let state = empty_state();
|
||||
let bid = active_buffer(&state);
|
||||
let mut src = String::new();
|
||||
for i in 0..1500 {
|
||||
writeln!(
|
||||
src,
|
||||
"pub fn f_{i}(x: u32) -> u32 {{ let y = x + {i}; y * 2 }}"
|
||||
)
|
||||
.expect("write");
|
||||
}
|
||||
seed_rust_parse_view(&state, bid, src.as_bytes());
|
||||
|
||||
let start = std::time::Instant::now();
|
||||
let summary = scoped_file_summary(&state, bid);
|
||||
let elapsed = start.elapsed();
|
||||
assert!(!summary.is_empty(), "summary produced for a styled buffer");
|
||||
assert!(
|
||||
elapsed < std::time::Duration::from_secs(1),
|
||||
"full-buffer summary took {elapsed:?}; the sweep must stay ~linear \
|
||||
(a quadratic flatten regresses here)"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cpp_style_comes_from_lsp_when_no_tree_sitter_grammar() {
|
||||
let state = empty_state();
|
||||
|
|
|
|||
156
src/syntax.rs
156
src/syntax.rs
|
|
@ -103,10 +103,16 @@ pub struct ParseTreeBundle {
|
|||
/// and the `*workers*` buffer can ask "what grammar produced this?"
|
||||
/// without indexing the layer vec.
|
||||
pub language_name: String,
|
||||
/// Wall-clock duration of the whole layered parse (excludes dispatch
|
||||
/// queueing, source materialization, and bus delivery). T M4.1
|
||||
/// acceptance criteria are stated in this metric.
|
||||
/// Wall-clock duration of the **root** parse (excludes injection layer
|
||||
/// building and dispatch/materialization/bus overhead). The M4.1
|
||||
/// acceptance perf gates are stated in this metric, so it stays the
|
||||
/// single-tree cost even as injection layers are added on top.
|
||||
pub parse_duration: Duration,
|
||||
/// True if injection expansion hit the total-layer backstop (framing
|
||||
/// Q#IJ3) and dropped some regions. Surfaced (not silent) at settle via
|
||||
/// `pmacs.error`; only a pathological file (thousands of embedded
|
||||
/// regions) can set it.
|
||||
pub injection_capped: bool,
|
||||
}
|
||||
|
||||
/// One injection layer within a [`ParseTreeBundle`] (framing Q#IJ1). A
|
||||
|
|
@ -179,12 +185,14 @@ pub fn run_parse(req: ParseRequest) -> Result<ParseTreeBundle, String> {
|
|||
depth: 0,
|
||||
highlight_query: None,
|
||||
}];
|
||||
build_injection_layers(&mut layers, req.source.as_ref(), &req.injection_aliases);
|
||||
let injection_capped =
|
||||
build_injection_layers(&mut layers, req.source.as_ref(), &req.injection_aliases);
|
||||
Ok(ParseTreeBundle {
|
||||
layers,
|
||||
source: req.source,
|
||||
language_name: req.language_name,
|
||||
parse_duration,
|
||||
injection_capped,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -251,17 +259,20 @@ struct InjectionMatch {
|
|||
/// appending children in depth-ascending order (Q#IJ1, Q#IJ6 rely on this
|
||||
/// ordering). Bounded by depth, total layer count, and a
|
||||
/// `(language, ranges)` visited guard (Q#IJ3). BFS by depth so siblings
|
||||
/// at a level are grouped before descending.
|
||||
/// at a level are grouped before descending. Returns `true` if the
|
||||
/// total-layer backstop was hit and some regions were dropped (surfaced at
|
||||
/// settle, framing Q#IJ3).
|
||||
fn build_injection_layers(
|
||||
layers: &mut Vec<Layer>,
|
||||
source: &[u8],
|
||||
aliases: &HashMap<String, String>,
|
||||
) {
|
||||
) -> bool {
|
||||
let mut query_cache: HashMap<String, Option<Arc<tree_sitter::Query>>> = HashMap::new();
|
||||
let mut visited: HashSet<(String, Vec<(usize, usize)>)> = HashSet::new();
|
||||
// Frontier entries are (layer index, that layer's included ranges).
|
||||
let mut frontier: Vec<(usize, Vec<Range>)> = vec![(0, vec![whole_source_range(source)])];
|
||||
let mut depth: u16 = 0;
|
||||
let mut capped = false;
|
||||
|
||||
while depth < MAX_INJECTION_DEPTH && !frontier.is_empty() {
|
||||
// Children discovered this level: (layer, its ranges) to append and
|
||||
|
|
@ -274,6 +285,7 @@ fn build_injection_layers(
|
|||
};
|
||||
for m in collect_injection_matches(&query, &layers[*parent_idx].tree, source) {
|
||||
if layers.len() + children.len() >= MAX_INJECTION_LAYERS {
|
||||
capped = true;
|
||||
break 'parents; // runaway backstop; tail dropped
|
||||
}
|
||||
let Some(child_lang) = resolve_injected_language(&m.language, aliases) else {
|
||||
|
|
@ -314,6 +326,7 @@ fn build_injection_layers(
|
|||
frontier = next_frontier;
|
||||
depth += 1;
|
||||
}
|
||||
capped
|
||||
}
|
||||
|
||||
/// Compile (once, cached) the `injections.scm` for `lang` from the static
|
||||
|
|
@ -797,16 +810,12 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[
|
|||
highlights_query: &[tree_sitter_lua::HIGHLIGHTS_QUERY],
|
||||
injections_query: &[],
|
||||
},
|
||||
// T M9.7: markdown grammar for prompt-result buffers with
|
||||
// `_meta.format = "markdown"`. Uses only the block grammar
|
||||
// (`tree_sitter_md::LANGUAGE`) — block-level highlighting (headers,
|
||||
// lists, fenced code blocks, blockquotes) is the v0.1 floor.
|
||||
// Inline highlighting (emphasis, links inside running text) would
|
||||
// require the dual-grammar `MarkdownParser` and is M9.8+ work.
|
||||
// The `markdown_inline` fixture prompt + matching acceptance test
|
||||
// pin this floor: an `**emphasis**` span must not crash, and is
|
||||
// expected to render unhighlighted; any future expansion that
|
||||
// adds inline coverage is additive, not a regression.
|
||||
// T M9.7: markdown block grammar (`tree_sitter_md::LANGUAGE`) — headers,
|
||||
// lists, fenced code blocks, blockquotes. Its `injections.scm` (framing
|
||||
// Q#IJ10) drives two layer kinds: fenced code blocks inject the fence's
|
||||
// named language, and paragraph/heading text injects `markdown_inline`
|
||||
// (the entry below) — so inline emphasis/links are now highlighted, and
|
||||
// the former M9.7 "block-only, inline unhighlighted" floor is retired.
|
||||
// Note the constant name: `HIGHLIGHT_QUERY_BLOCK` (singular) is
|
||||
// the markdown crate's idiom; `tree-sitter-rust` and
|
||||
// `tree-sitter-lua` use `HIGHLIGHTS_QUERY` (plural).
|
||||
|
|
@ -1281,6 +1290,7 @@ impl SyntaxRegistry {
|
|||
source: raw.source.clone(),
|
||||
language_name: raw.language_name.clone(),
|
||||
parse_duration: raw.parse_duration,
|
||||
injection_capped: raw.injection_capped,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -1597,44 +1607,80 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn inline_layer_multi_range_link_and_emphasis() {
|
||||
// Framing acceptance #6: a paragraph with a link AND emphasis
|
||||
// becomes a markdown_inline layer whose included ranges exclude the
|
||||
// block grammar's named children — the inline grammar still parses
|
||||
// the emphasis run in a surviving text range.
|
||||
fn inline_layer_multi_range_excludes_block_continuation() {
|
||||
// Framing acceptance #6 (round-2 finding 3): the multi-range path.
|
||||
// A one-line paragraph would give a SINGLE range (a link/emphasis
|
||||
// are child-grammar structures, not block-grammar named children),
|
||||
// so it can't prove multi-range. A MULTI-LINE blockquote's inline
|
||||
// node carries a named `block_continuation` child (the `> ` marker),
|
||||
// which `content_node_ranges` excludes — yielding MORE THAN ONE
|
||||
// included range, the genuine path markdown_inline depends on.
|
||||
let reg = SyntaxRegistry::new();
|
||||
let src = b"See [the docs](http://example.com) and *emphasis* here.\n";
|
||||
let bundle = parse_layered(®, "markdown", src);
|
||||
let src = b"> first *one*\n> second *two*\n";
|
||||
let language = reg.language("markdown").expect("markdown");
|
||||
let mut buf = fresh_buffer("doc");
|
||||
buf.apply_edit(EditOp::Insert { pos: 0, bytes: src })
|
||||
.unwrap();
|
||||
let view = ParseView::new(&buf, language, "markdown".to_owned());
|
||||
let handle = view.handle();
|
||||
let _vid = buf.attach_view(Box::new(view));
|
||||
let mut req = handle.make_request();
|
||||
req.injection_aliases = reg.injection_alias_snapshot();
|
||||
let raw = run_parse(req).expect("parse");
|
||||
|
||||
// The inline injection match collects more than one included range.
|
||||
let mut cache = HashMap::new();
|
||||
let query = injection_query_cached(&mut cache, "markdown").expect("md injections");
|
||||
let inline_match = collect_injection_matches(&query, &raw.layers[0].tree, src)
|
||||
.into_iter()
|
||||
.find(|m| m.language == "markdown_inline")
|
||||
.expect("an inline injection match");
|
||||
assert!(
|
||||
inline_match.ranges.len() >= 2,
|
||||
"the multi-line inline node yields >1 range (block_continuation \
|
||||
excluded); got {:?}",
|
||||
inline_match
|
||||
.ranges
|
||||
.iter()
|
||||
.map(|r| (r.start_byte, r.end_byte))
|
||||
.collect::<Vec<_>>()
|
||||
);
|
||||
|
||||
// Both ranges parse and highlight: emphasis is recognized on BOTH
|
||||
// lines, and the resolved layer produces spans.
|
||||
let bundle = reg.resolve_layer_queries(&raw);
|
||||
let inline = bundle
|
||||
.layers
|
||||
.iter()
|
||||
.find(|l| l.language_name == "markdown_inline")
|
||||
.expect("inline paragraph becomes a markdown_inline layer");
|
||||
assert_eq!(
|
||||
inline.depth, 1,
|
||||
"inline is a depth-1 injection of the block grammar"
|
||||
);
|
||||
.expect("inline layer");
|
||||
let sexp = inline.tree.root_node().to_sexp();
|
||||
assert!(
|
||||
sexp.contains("emphasis"),
|
||||
"the inline layer parsed the *emphasis* run: {sexp}"
|
||||
sexp.matches("emphasis").count() >= 2,
|
||||
"the inline grammar parsed emphasis in both ranges: {sexp}"
|
||||
);
|
||||
let query = inline
|
||||
let hquery = inline
|
||||
.highlight_query
|
||||
.as_ref()
|
||||
.expect("inline highlights resolved at settle");
|
||||
let spans = compute_highlight_spans_for(query, &inline.tree, &bundle.source, None);
|
||||
let spans = compute_highlight_spans_for(hquery, &inline.tree, &bundle.source, None);
|
||||
assert!(
|
||||
!spans.is_empty(),
|
||||
"the inline layer produces highlight spans"
|
||||
"the inline layer produces highlight spans across both ranges"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recursion_bounds_terminate() {
|
||||
// Framing acceptance #7: rust self-injects into macro token-trees.
|
||||
// Nested injections must terminate (depth cap + visited guard),
|
||||
// never loop — if the guard failed this test would hang.
|
||||
// Framing acceptance #7: rust self-injects into macro token-trees, so
|
||||
// nested injections recurse. This proves the depth bound and, most
|
||||
// importantly, TERMINATION — a completing test (vs a hang) is the
|
||||
// observable guarantee. (The total-layer backstop is exercised
|
||||
// separately by `injection_layer_cap_surfaces_and_preserves_root`;
|
||||
// the `(lang, ranges)` visited guard is a defensive early-out that
|
||||
// no bundled grammar's self-injection-over-a-fixed-range can trip,
|
||||
// so it is covered by inspection + this termination check, not an
|
||||
// isolated positive case.)
|
||||
let reg = SyntaxRegistry::new();
|
||||
let src =
|
||||
b"macro_rules! m { () => { println!(\"{}\", vec![1, 2, 3]); }; }\nfn f() { m!(); }\n";
|
||||
|
|
@ -1655,6 +1701,44 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn injection_layer_cap_surfaces_and_preserves_root() {
|
||||
// Round-2 finding 4: hitting the total-layer backstop must set the
|
||||
// surfaced `injection_capped` flag (not drop silently), bound the
|
||||
// layer count, and keep the root intact.
|
||||
let reg = SyntaxRegistry::new();
|
||||
let fences = MAX_INJECTION_LAYERS + 8; // just over the backstop
|
||||
let mut src = String::with_capacity(fences * 15);
|
||||
for _ in 0..fences {
|
||||
src.push_str("```rust\nx\n```\n\n");
|
||||
}
|
||||
let language = reg.language("markdown").expect("markdown");
|
||||
let mut buf = fresh_buffer("doc");
|
||||
buf.apply_edit(EditOp::Insert {
|
||||
pos: 0,
|
||||
bytes: src.as_bytes(),
|
||||
})
|
||||
.unwrap();
|
||||
let view = ParseView::new(&buf, language, "markdown".to_owned());
|
||||
let handle = view.handle();
|
||||
let _vid = buf.attach_view(Box::new(view));
|
||||
let mut req = handle.make_request();
|
||||
req.injection_aliases = reg.injection_alias_snapshot();
|
||||
let bundle = run_parse(req).expect("root parse");
|
||||
|
||||
assert!(
|
||||
bundle.injection_capped,
|
||||
"hitting the backstop sets the surfaced flag"
|
||||
);
|
||||
assert!(
|
||||
bundle.layers.len() <= MAX_INJECTION_LAYERS,
|
||||
"layer count is bounded by the backstop; got {}",
|
||||
bundle.layers.len()
|
||||
);
|
||||
assert_eq!(bundle.layers[0].language_name, "markdown", "root intact");
|
||||
assert_eq!(bundle.root_tree().root_node().kind(), "document");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_injecting_buffer_single_layer() {
|
||||
// Framing acceptance #13: a plain rust file with no macros produces
|
||||
|
|
|
|||
|
|
@ -90,6 +90,43 @@ fn lua_alias_override_resolves_on_async_parse() {
|
|||
);
|
||||
}
|
||||
|
||||
/// Round-2 finding: the synchronous parse path (`_parse_now`) must snapshot
|
||||
/// injection aliases too — otherwise a `py` fence (or a Lua-added alias)
|
||||
/// injects asynchronously but not synchronously. The default `py`→python
|
||||
/// alias discriminates the fix: with the empty map it would not resolve.
|
||||
#[test]
|
||||
fn sync_parse_now_resolves_alias() {
|
||||
let state = EditorState::new();
|
||||
let src = b"# Doc\n\n```py\nx = 1\n```\n";
|
||||
let buf_id = state
|
||||
.lua_host
|
||||
.registry()
|
||||
.borrow_mut()
|
||||
.create_from_bytes("doc.md".to_owned(), src);
|
||||
state
|
||||
.lua_host
|
||||
.lua()
|
||||
.globals()
|
||||
.set("BUF", BufferIdLua(buf_id))
|
||||
.expect("bind BUF");
|
||||
state
|
||||
.lua_host
|
||||
.lua()
|
||||
.load("pmacs.parse._parse_now(BUF, 'markdown')")
|
||||
.exec()
|
||||
.expect("synchronous parse");
|
||||
|
||||
let bundle = state
|
||||
.syntax_registry
|
||||
.view(buf_id)
|
||||
.and_then(|h| h.current())
|
||||
.expect("installed bundle");
|
||||
assert!(
|
||||
bundle.layers.iter().any(|l| l.language_name == "python"),
|
||||
"the `py` fence resolved to python on the synchronous `_parse_now` path"
|
||||
);
|
||||
}
|
||||
|
||||
/// Framing acceptance #12: a large all-inline markdown buffer settles
|
||||
/// (root + one cold inline layer per paragraph) within a comfortable
|
||||
/// budget, and the FINAL paragraph still receives an inline layer — the
|
||||
|
|
|
|||
Loading…
Reference in New Issue