diff --git a/docs/active-work.md b/docs/active-work.md index 54b68e9..559df74 100644 --- a/docs/active-work.md +++ b/docs/active-work.md @@ -14,7 +14,7 @@ backlog. machine-local: `origin` may name this canonical URL, a release mirror, or something else, and therefore has no authority by name alone. - Canonical base at this snapshot: - `githubsucks/main` @ `1dd47fc` (modeline detection #132 merged; protocol v18). + `githubsucks/main` @ `8bd8298` (landed-state handoff #133; protocol v18). - On the transfer source, `origin/main` named a release mirror at `d3fa632` and lagged badly. On the current destination, `origin` names the canonical URL. This difference is why all recovery begins by @@ -48,9 +48,43 @@ git worktree list git status --short --branch ``` -The first command must expose `1dd47fc` or a newer intentional main. +The first command must expose `8bd8298` or a newer intentional main. If it does not, stop and repair the remote/fetch configuration. +## Locals-query processing lane + +- Portable branch: `githubsucks/locals-query-processing` +- Base: canonical `main` @ `8bd8298`. +- Framing head: `42a28b9` (`docs/locals-query-processing-framing.md`). +- Implementation head: `47ffe5d`. +- PR: #134, , open for review. + Implementation is complete; protocol remains v18. +- Scope: bundled Lua/JavaScript/TypeScript locals metadata, lexical + scope/definition/value/reference resolution cached on each settled layer, + and shared `#is?`/`#is-not? local` filtering in the TUI and semantic/GPU + highlight producers. +- Focused verification: + - default and Lua 5.4 lexical suites: 35 passed each; + - default and Lua 5.4 end-to-end render/edit acceptance: 1 passed each. +- Full pre-PR verification: + - `cargo fmt --check`: clean; + - strict workspace Clippy: clean; + - default library: 1,757 passed (3 ignored); + - CRDT library: 1,933 passed (3 ignored); + - M4 acceptance excluding basedpyright: 121 passed, 3 ignored, 1 filtered; + - required GPU: 109 passed; + - workspace: 2,893 passed across 82 suites, 19 ignored, 1 filtered; + - `git diff --check`: clean. + +Recovery worktree on a machine that does not already own the branch: + +```sh +git worktree add --track \ + -b locals-query-processing \ + ../pmacs-locals-query \ + githubsucks/locals-query-processing +``` + ## Parked lane: kill-ring browser + persistence - Portable branch: `githubsucks/kill-ring-browser` diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index bf6c013..c2124f3 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1,9 +1,9 @@ # Agent handoff — cross-machine continuity -**Last updated: 2026-07-22, after modeline language detection (#132) and -Vterm Stage 2 (#130) landed on `main`, atop mode system wiring (#129/#131), -config registry (#127), Vterm Stage 1 terminal core (#126), and completed -Themes Arc 4 (#120/#124/#125). Vterm Stage 3 is not implemented.** +**Last updated: 2026-07-22, with locals-query processing (#134) implemented +and awaiting review, after modeline language +detection (#132), Vterm Stage 2 (#130), and mode system wiring (#129/#131) +landed on `main`. Vterm Stage 3 is not implemented.** This file is the bridge between development machines. If you are an agent reading this on a fresh clone: this document plus the `docs/*-framing.md` @@ -17,7 +17,8 @@ commands, read `docs/active-work.md` immediately after this file. ## 1. Where the project stands (2026-07-22) -- `main` @ `1dd47fc` (modeline detection #132), protocol **v18** +- `main` @ `8bd8298` (landed-state handoff #133; latest runtime merge + `1dd47fc`, modeline detection #132), protocol **v18** (`SUPPORTED=[6..18]`; v16 = `ThemeFacts`, v17 = `FontFacts`, v18 = `StatuslineSegments`). - **Config registry LANDED — #127** (`docs/config-registry-framing.md` @@ -117,12 +118,16 @@ commands, read `docs/active-work.md` immediately after this file. cmake (`cmake-language-server`, config via `init_options.buildDirectory="build"` — it does NOT pull `workspace/configuration`). Make has no server. - - **Substrate**: `LanguageEntry.highlights_query` is now - `&[&'static str]` — fragments joined base-first, for grammars whose - bundled highlights are a `; inherits:` delta (cuda over c/cpp; ts - over js/jsx). `compute_highlight_spans` FAILS CLOSED on the - `#is?`/`#is-not? local` property predicate (no locals processing) — - drops those captures so shadowed builtins aren't mis-styled. + - **Substrate**: `LanguageEntry.highlights_query` and `.locals_query` are + `&[&'static str]` fragments joined base-first (cuda over c/cpp; ts over + js/jsx). On `locals-query-processing` @ `47ffe5d`, settle compiles the + grammar's `LOCALS_QUERY`, resolves Tree-sitter's scope/definition/value/ + reference conventions into sorted `LocalFacts`, and stores them beside + each layer's tree/query. Work runs once per fresh bundle and only when the + highlight query asks about `local`; viewport rendering remains bounded. + Both TUI and semantic/GPU producers evaluate `#is?`/`#is-not? local` + through the shared capture walk. Non-shadowed JS/TS builtins are restored; + shadowed definitions/references keep ordinary variable styling. - **Multi-language injections (#122) LANDED** — the direct continuation of the #114–#118 highlight arc; four review rounds, framing `docs/multi-language-injections-framing.md` (Q#IJ1–IJ11). A buffer can @@ -142,7 +147,8 @@ commands, read `docs/active-work.md` immediately after this file. worker never touches the `Rc` registry or Lua); `ParseTreeBundle.injection_capped` (the 4096-layer backstop, surfaced once/buffer via `pmacs.error` at settle); - `compute_highlight_spans_for(query, tree, source, range)` (per-layer); + `compute_highlight_spans_for(query, tree, source, local_facts, range)` + (per-layer); the wire `flatten_layer_spans` event-sweep → DISJOINT effective spans (deeper / later-sibling / narrower wins, keyed by `(layer_index, capture_order)`); GPU `spans_from_segments` + `source_color_at` fold. @@ -335,6 +341,10 @@ commands, read `docs/active-work.md` immediately after this file. editing/indent/comment items that were config-blocked. - **Mode system wiring COMPLETE (#129)** — major-mode keymaps, introspection, lifecycle initialization, and statusline display shipped. + - **Locals-query processing IN REVIEW — #134** — grammar locals metadata, + lexical resolution, settled per-layer facts, and shared TUI/GPU + local-predicate filtering are implemented on `locals-query-processing` + @ `47ffe5d`. - Remaining ranked arcs: 6 folding, 7 DAP, 8 GPU splits, plus the `.ipynb` arc (its JSON-grammar prerequisite shipped in #123). diff --git a/docs/locals-query-processing-framing.md b/docs/locals-query-processing-framing.md new file mode 100644 index 0000000..4cba836 --- /dev/null +++ b/docs/locals-query-processing-framing.md @@ -0,0 +1,306 @@ +# Locals-query processing - syntax-highlight completion + +**Status:** Revision 1, implementation active by user direction, 2026-07-22. + +**Base:** `githubsucks/main` at `8bd8298` (mode system #129, Vterm Stage 2 +#130, modeline detection #132, and landed-state handoff #133). Protocol remains +v18. + +## Problem + +pmacs runs each grammar's `highlights.scm` directly through a +`tree_sitter::QueryCursor`. That cursor evaluates text predicates such as +`#eq?`, `#match?`, and `#any-of?`, but it does not assign or evaluate semantic +properties. In particular, JavaScript's bundled highlight query contains: + +```scheme +((identifier) @variable.builtin + (#match? @variable.builtin "^(arguments|module|console|window|document)$") + (#is-not? local)) + +((identifier) @function.builtin + (#eq? @function.builtin "require") + (#is-not? local)) +``` + +A correct highlighter must first run the grammar's `LOCALS_QUERY`, resolve +lexical definitions and references, and then apply `#is? local` / +`#is-not? local` while selecting highlight captures. Without those facts, +a local `console` or `require` can be styled as a builtin. + +The current implementation fails closed by dropping every highlight pattern +that carries a `local` property predicate. That prevents the false-positive +shadowed-builtin style, but it also drops legitimate builtin highlighting for +non-shadowed `console`, `window`, `require`, and peers. This is the first +remaining item in the side-quest north-star list at +`docs/side-quest-backlog.md:244-249`. + +## Goal + +Run the bundled locals query as part of each settled syntax layer, retain a +compact lexical-local map, and use that map to evaluate `#is? local` and +`#is-not? local` in both syntax-highlight producers: + +1. `highlight::SyntaxHighlightView` (TUI / overlay path), and +2. `semantic_render::scoped_style_spans` (semantic/GPU path). + +A shadowed builtin must retain its ordinary fallback capture but lose the +builtin refinement. A non-shadowed builtin must regain the builtin capture. +The result must stay correct for nested scopes, TypeScript's inherited locals +query, injected-language layers, viewport-limited rendering, and edits that +settle a new parse bundle. + +## Scope + +### In + +- Add bundled `locals.scm` fragments to syntax grammar metadata. +- Compose inherited locals fragments base-first, exactly as highlight fragments + already compose. +- Implement Tree-sitter's local-scope conventions: + - `@local.scope` + - `@local.definition` + - `@local.definition-value` + - `@local.reference` + - `#set! local.scope-inherits false` +- Evaluate positive and negative `local` property predicates, including the + optional capture-qualified form accepted by Tree-sitter's query parser. +- Cache local facts on each settled parse layer. +- Feed those facts to both highlight producers. +- Cover lexical behavior and end-to-end rendering under both supported Lua + backends. + +### Out + +- No new grammar, parser, or language-detection behavior. +- No Lua API, command, config key, theme key, protocol field, or frontend-only + state. +- No LSP semantic-token changes. +- No user-defined query registration surface. +- No cross-language name resolution between a host layer and an injected + child layer. +- No Tree-sitter-highlight `reference_highlight` propagation from a local + definition's syntactic capture to all references. The target is property + predicate correctness. Adding propagation would change ordinary variable + styling beyond the reported builtin bug and needs separate framing. +- No general engine for arbitrary `#is?` property keys. This side quest owns + the Tree-sitter-defined boolean `local` property only; other property + predicates retain their current behavior. + +## Existing contracts to preserve + +1. **Layering:** every injection `Layer` owns its grammar tree and highlight + query. Deeper layers and later same-depth siblings retain their existing + precedence. +2. **Viewport work:** the semantic producer's highlight capture walk remains + restricted with `QueryCursor::set_byte_range`; rendering a frame must not + launch a whole-file locals walk. +3. **Settle freshness:** a new parse bundle replaces the old bundle + atomically. Local facts must travel with the same bundle, never in a side + cache that can pair old scopes with a new tree. +4. **Incremental edits:** a completed reparse produces new local facts before + the bundle becomes visible. Until settle, producers continue to use the + previous internally consistent bundle. +5. **Query inheritance:** fragments are newline-joined base-first. Bare + concatenation can extend a trailing Scheme comment and is forbidden. +6. **Fallback styling:** suppressing a builtin refinement must not suppress + an independent ordinary-variable capture for the same identifier. +7. **No protocol change:** all work is internal render state. + +## Decisions + +### Q#LQ1 - Grammar metadata carries locals fragments + +`LanguageEntry` gains `locals_query: &'static [&'static str]`, parallel to +`highlights_query` and `injections_query`. + +The bundled crates currently exposing `LOCALS_QUERY` are wired as follows: + +| pmacs language | effective locals fragments | +| --- | --- | +| `lua` | `tree_sitter_lua::LOCALS_QUERY` | +| `javascript` | `tree_sitter_javascript::LOCALS_QUERY` | +| `javascriptreact` | `tree_sitter_javascript::LOCALS_QUERY` | +| `typescript` | JavaScript locals, then TypeScript locals | +| `typescriptreact` | JavaScript locals, then TypeScript locals | + +All other entries use an empty slice. TypeScript's locals query is a small +parameter-definition delta, so compiling it alone would omit JavaScript's +scopes, declarations, and references. JSX uses the JavaScript grammar and +therefore the JavaScript locals query. TSX uses the TypeScript crate's TSX +language with the same base-plus-delta locals composition. + +`SyntaxRegistry` lazily compiles and caches one effective locals query per +language, including cached failure/no-query results, matching the existing +highlight-query policy. + +### Q#LQ2 - Local facts follow Tree-sitter lexical semantics + +A locals capture walk maintains a stack of scopes. The stack begins with one +non-inheriting root scope covering the layer. Captures are processed in query +order and source order: + +1. `@local.scope` pushes a scope covering that node. It inherits outer + definitions unless its pattern sets `local.scope-inherits` to `false`. +2. `@local.definition` records the identifier in the innermost scope and marks + that identifier range local. +3. A sibling `@local.definition-value` capture records the initializer/value + range. That new definition is not visible while resolving references inside + its value, so an outer definition with the same name can still win there. +4. `@local.reference` searches definitions newest-first, then scopes + innermost-first. Search stops at a non-inheriting scope. A resolved + reference range is marked local; an unresolved reference remains non-local. +5. Scopes whose end precedes the next capture are popped. + +Definition names are compared as borrowed source byte slices. No identifier +strings are allocated. Invalid or out-of-bounds ranges do not produce local +facts. + +The result is an opaque `LocalFacts` value containing sorted, deduplicated +`(start_byte, end_byte)` ranges. Highlight predicate checks use binary search; +they do not hash, copy source text, or rebuild scope state. + +### Q#LQ3 - Predicate evaluation is per emitted capture + +Before emitting a highlight capture, inspect +`Query::property_predicates(pattern_index)`: + +- `#is? local` passes only when the selected node is in `LocalFacts`. +- `#is-not? local` passes only when the selected node is not in `LocalFacts`. +- If the property names a capture, test that capture's node. +- If it does not name a capture, test the highlight capture currently being + considered, matching Tree-sitter-highlight's per-node behavior. +- Multiple `local` predicates on one pattern are conjunctive. +- A missing or failed locals query yields no local ranges: negative predicates + pass and positive predicates fail. This preserves useful non-local + highlighting without inventing local classifications. +- Predicates with keys other than `local` remain ignored, preserving the + current query engine's scope. + +Text predicates remain the query cursor's responsibility. Property settings +such as `local.scope-inherits` are read only by local analysis; they are not +mistaken for highlight predicates. + +### Q#LQ4 - Facts are computed once at settle and owned by the layer + +`SyntaxRegistry::resolve_layer_queries` remains the main-thread Stage 2 handoff. +For each raw worker layer it: + +1. resolves the cached highlight query; +2. checks whether that query contains any `local` property predicate; +3. only when needed, resolves the cached locals query and walks the layer tree; +4. stores `Arc` beside the layer's tree and highlight query. + +This is a single whole-layer analysis per completed parse, not per render. +Languages whose highlights never ask about `local` pay only the cheap predicate +scan and carry no facts. This includes Lua today even though its locals query +is correctly registered for future local-sensitive highlight patterns. + +Putting facts on `Layer` makes the consistency invariant structural: + +```text +settled Layer = tree + grammar + highlight query + local facts +``` + +A producer cannot accidentally retrieve facts for another buffer revision. +Child injection layers compute facts against their own tree and grammar; local +bindings never cross layer boundaries. A deliberately combined injection +layer shares one tree and therefore one lexical environment, matching its +combined parse semantics. + +### Q#LQ5 - Both producers share one capture-selection function + +`compute_highlight_spans_for` accepts the layer's optional local facts and +owns predicate evaluation. The whole-buffer overlay path and the +viewport-limited semantic path both call this function. There is no second +predicate implementation in a frontend. + +`compute_highlight_spans_in_range` passes the root layer's facts. Injection +producers pass each layer's facts. Existing wider-first ordering and +cross-layer merge precedence remain unchanged after captures are selected. + +### Q#LQ6 - Performance boundary + +The locals-query capture walk is linear for one settled layer. Like upstream, +resolving references scans visible definitions newest-first, so the worst case +is $O(\text{captures} + \text{references} \times \text{definitions})$. +It runs only for a language whose highlight query actually contains a `local` +predicate, and only once when a fresh parse bundle settles. + +The render hot paths remain: + +- TUI: cached whole-layer highlight spans, rebuilt only when the bundle pointer + changes; +- semantic/GPU: viewport-bounded highlight query plus binary-search local + checks. + +No frame performs a whole-file locals query. Memory is two `u32` offsets per +local definition/resolved reference plus vector capacity; ranges are sorted +and deduplicated before storage. + +Incremental locals invalidation is intentionally bundle-granular. A local edit +can alter all later name resolution in a scope, so attempting to splice only +changed ranges without a scope dependency graph risks stale classifications. +The parse itself remains incremental; this bounded lexical pass is the boring, +correct cutover. + +## Data flow + +```text +worker parse + -> raw ParseTreeBundle { Layer { tree, language, no queries/facts } } + -> main-thread resolve_layer_queries + -> cached highlights query + -> cached locals query (only if highlights uses local predicates) + -> lexical capture walk -> sorted LocalFacts + -> settled ParseTreeBundle + -> TUI SyntaxHighlightView cache rebuild + -> semantic/GPU viewport capture walk + -> shared local predicate filter + -> existing span ordering/merge/theme lookup +``` + +## Acceptance criteria + +1. **Non-shadowed builtin restored:** JavaScript `console`, `window`, or + `require` with no matching lexical definition emits its bundled + `*.builtin` capture. +2. **Shadowed builtin suppressed:** a parameter or local declaration named + `console`/`require` and references resolved to it do not emit a builtin + capture; their ordinary variable captures remain. +3. **Scope correctness:** shadowing is confined to its lexical scope. A builtin + before/after the scope remains builtin, while the definition and references + inside are local. +4. **Positive predicate:** a focused custom highlight query using `#is? local` + emits resolved definitions/references and rejects an unresolved identifier. +5. **TypeScript inheritance:** parameter shadowing in TypeScript and TSX uses + the JavaScript base locals query plus the TypeScript delta; query compilation + and classification succeed for both grammars. +6. **End-to-end render:** opening a JavaScript buffer through the shipped + runtime and applying a theme that styles only `variable.builtin` renders an + unshadowed builtin with that style and a shadowed occurrence without it. +7. **Edit freshness:** after changing a shadowing identifier and settling the + new parse, the next render reflects the new local/non-local classification; + no stale local facts survive. +8. **Producer coverage:** both the overlay and semantic/GPU callsites pass the + corresponding layer facts to the shared capture walk; injected layers keep + their own facts. +9. **Backend parity:** the focused acceptance test passes under default Luau + and `--no-default-features --features lua54`. +10. **No regressions:** formatting, Clippy, default library tests, CRDT library + tests, M4 acceptance (excluding the machine-broken basedpyright case), + required GPU tests, workspace sweep, and `git diff --check` pass. + +## Expected files + +- `src/syntax.rs` - grammar metadata, locals-query cache, lexical analysis, + layer facts, predicate selection, unit coverage. +- `src/highlight.rs` - pass per-layer facts to the overlay producer. +- `src/semantic_render.rs` - pass per-layer facts to the viewport producer. +- `tests/m4_acceptance.rs` - end-to-end local/non-local rendering and edit + freshness. +- `docs/agent-handoff.md` and `docs/active-work.md` - updated only after the + implementation is proven and published according to their protocols. + +No other runtime, Lua, frontend, protocol, or theme file should need a behavior +change. diff --git a/src/highlight.rs b/src/highlight.rs index 487afb9..18740ca 100644 --- a/src/highlight.rs +++ b/src/highlight.rs @@ -374,7 +374,13 @@ impl SyntaxHighlightView { let Some(query) = layer.highlight_query.as_ref() else { continue; }; - let spans = compute_highlight_spans_for(query, &layer.tree, source, None); + let spans = compute_highlight_spans_for( + query, + &layer.tree, + source, + layer.local_facts.as_deref(), + None, + ); if spans.is_empty() { continue; } diff --git a/src/semantic_render.rs b/src/semantic_render.rs index 2d3ec0a..3ca2aa7 100644 --- a/src/semantic_render.rs +++ b/src/semantic_render.rs @@ -1964,6 +1964,7 @@ fn scoped_style_spans(state: &EditorState, vp: &DeclaredViewport) -> Vec crate::syntax::ParseViewHandle { let language = state .syntax_registry - .language("rust") - .expect("rust language"); + .language(language_name) + .unwrap_or_else(|| panic!("{language_name} language")); let mut core = state.core.borrow_mut(); let registry_handle = core.registry.clone(); let mut registry = registry_handle.borrow_mut(); @@ -3936,23 +3939,31 @@ mod tests { pos: 0, bytes: text, }) - .expect("seed rust text"); + .expect("seed syntax text"); } - let parse_view = crate::syntax::ParseView::new(buf, language, "rust".to_owned()); + let parse_view = crate::syntax::ParseView::new(buf, language, language_name.to_owned()); let handle = parse_view.handle(); let req = handle.make_request(); - let bundle = crate::syntax::run_parse(req).expect("initial rust parse"); - // Mirror the production settle path: resolve each layer's highlight - // query before install so the producer can style it (framing Q#IJ2). + let bundle = crate::syntax::run_parse(req).expect("initial syntax parse"); + // Mirror the production settle path: queries and lexical facts travel + // with the same bundle the producer reads. handle.install(state.syntax_registry.resolve_layer_queries(&bundle)); buf.attach_view(Box::new(parse_view)); drop(registry); - core.set_buffer_path(buffer_id, Some(std::path::PathBuf::from("/tmp/x.rs"))); + core.set_buffer_path(buffer_id, Some(std::path::PathBuf::from(path))); drop(core); state.syntax_registry.attach_view(buffer_id, handle.clone()); handle } + fn seed_rust_parse_view( + state: &EditorState, + buffer_id: BufferId, + text: &[u8], + ) -> crate::syntax::ParseViewHandle { + seed_parse_view(state, buffer_id, text, "rust", "/tmp/x.rs") + } + fn seed_markdown_parse_view( state: &EditorState, buffer_id: BufferId, @@ -4037,6 +4048,57 @@ mod tests { ); } + #[test] + fn viewport_style_producer_uses_settled_local_facts() { + let state = empty_state(); + let buffer_id = active_buffer(&state); + let source = b"console;\nfunction f(console) { console; }\n"; + seed_parse_view(&state, buffer_id, source, "javascript", "/tmp/locals.js"); + state.syntax_registry.theme().lock().expect("theme").insert( + "variable.builtin", + Style { + fg: crate::cell::Color::Indexed(6), + ..Style::default() + }, + ); + + let style_at = |visible: ByteRange, offset: u64| { + scoped_style_spans( + &state, + &DeclaredViewport { + buffer_id, + visible, + frontend_generation: 0, + }, + ) + .into_iter() + .find(|span| span.range.start <= offset && offset < span.range.end) + .map_or_else(Style::default, |span| span.style) + }; + + assert_eq!( + style_at(ByteRange { start: 0, end: 7 }, 0).fg, + crate::cell::Color::Indexed(6), + "unresolved outer `console` receives the builtin capture" + ); + let inner = source + .windows("console".len()) + .rposition(|window| window == b"console") + .expect("inner console") as u64; + assert_ne!( + style_at( + ByteRange { + start: inner, + end: inner + "console".len() as u64, + }, + inner, + ) + .fg, + crate::cell::Color::Indexed(6), + "viewport-only highlighting still sees the parameter definition outside the viewport" + ); + } + #[test] fn full_buffer_summary_flatten_scales_on_large_grammar_file() { // Perf gate (round-1 finding 1): the file-style summary runs the diff --git a/src/syntax.rs b/src/syntax.rs index cdd6d56..6c91115 100644 --- a/src/syntax.rs +++ b/src/syntax.rs @@ -115,6 +115,14 @@ pub struct ParseTreeBundle { pub injection_capped: bool, } +/// Lexically-local identifier ranges derived from a grammar's bundled +/// `locals.scm` query. Ranges are sorted and deduplicated so highlight +/// predicate checks are allocation-free binary searches. +#[derive(Debug, Default)] +pub struct LocalFacts { + ranges: Box<[(u32, u32)]>, +} + /// One injection layer within a [`ParseTreeBundle`] (framing Q#IJ1). A /// layer pairs a parse tree with the language that produced it and the /// injection-nesting depth (root = 0). `highlight_query` is resolved on @@ -133,6 +141,10 @@ pub struct Layer { /// `None` when the language ships no highlights, or on the worker /// (pre-settle). Producers read it to style this layer. pub highlight_query: Option>, + /// Lexically-local definitions and resolved references for this tree. + /// Present only when the highlight query asks about the `local` + /// property; computed once when the bundle settles. + pub local_facts: Option>, } impl ParseTreeBundle { @@ -184,6 +196,7 @@ pub fn run_parse(req: ParseRequest) -> Result { tree: root_tree, depth: 0, highlight_query: None, + local_facts: None, }]; let injection_capped = build_injection_layers(&mut layers, req.source.as_ref(), &req.injection_aliases); @@ -309,6 +322,7 @@ fn build_injection_layers( tree, depth: depth + 1, highlight_query: None, + local_facts: None, }, ranges, )); @@ -773,6 +787,10 @@ pub struct LanguageEntry { /// slice (or all-empty fragments) means no highlights: the view /// runs but emits nothing. pub highlights_query: &'static [&'static str], + /// Bundled `locals.scm` query fragments, composed base-first like + /// [`Self::highlights_query`]. The query supplies lexical scopes, + /// definitions, values, and references for `local` property predicates. + pub locals_query: &'static [&'static str], /// Bundled `injections.scm` fragments (framing Q#IJ2), joined with a /// newline and compiled on the parse worker to find embedded-language /// regions. Empty for the many grammars that ship none (or don't @@ -801,6 +819,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["rs"], loader: || tree_sitter_rust::LANGUAGE.into(), highlights_query: &[tree_sitter_rust::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[tree_sitter_rust::INJECTIONS_QUERY], }, LanguageEntry { @@ -808,6 +827,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["lua"], loader: || tree_sitter_lua::LANGUAGE.into(), highlights_query: &[tree_sitter_lua::HIGHLIGHTS_QUERY], + locals_query: &[tree_sitter_lua::LOCALS_QUERY], injections_query: &[], }, // T M9.7: markdown block grammar (`tree_sitter_md::LANGUAGE`) — headers, @@ -824,6 +844,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["md", "markdown"], loader: || tree_sitter_md::LANGUAGE.into(), highlights_query: &[tree_sitter_md::HIGHLIGHT_QUERY_BLOCK], + locals_query: &[], injections_query: &[tree_sitter_md::INJECTION_QUERY_BLOCK], }, // markdown_inline (framing Q#IJ10) — the inline grammar the block @@ -837,6 +858,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &[], loader: || tree_sitter_md::INLINE_LANGUAGE.into(), highlights_query: &[tree_sitter_md::HIGHLIGHT_QUERY_INLINE], + locals_query: &[], injections_query: &[tree_sitter_md::INJECTION_QUERY_INLINE], }, // T M_B3 — C / C++. Lexical highlighting (keywords / strings / @@ -859,6 +881,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["c", "h"], loader: || tree_sitter_c::LANGUAGE.into(), highlights_query: &[tree_sitter_c::HIGHLIGHT_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -866,6 +889,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["cpp", "cc", "cxx", "hpp", "hh", "hxx", "ipp", "inl", "cppm"], loader: || tree_sitter_cpp::LANGUAGE.into(), highlights_query: &[tree_sitter_cpp::HIGHLIGHT_QUERY], + locals_query: &[], injections_query: &[], }, // CUDA (`.cu` source, `.cuh` header). A dedicated grammar rather @@ -897,6 +921,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ tree_sitter_cpp::HIGHLIGHT_QUERY, tree_sitter_cuda::HIGHLIGHTS_QUERY, ], + locals_query: &[], injections_query: &[], }, // Shell / bash. Lexical highlighting for the shell family; the LSP @@ -915,6 +940,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["sh", "bash", "zsh", "ksh", "ash", "bats"], loader: || tree_sitter_bash::LANGUAGE.into(), highlights_query: &[tree_sitter_bash::HIGHLIGHT_QUERY], + locals_query: &[], injections_query: &[], }, // Filename-identified languages. These files usually have no useful @@ -931,6 +957,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["dockerfile", "containerfile"], loader: || tree_sitter_containerfile::LANGUAGE.into(), highlights_query: &[tree_sitter_containerfile::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -938,6 +965,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["mk", "make"], loader: || tree_sitter_make::LANGUAGE.into(), highlights_query: &[tree_sitter_make::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -945,6 +973,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["cmake"], loader: || tree_sitter_cmake::LANGUAGE.into(), highlights_query: &[tree_sitter_cmake::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, // Grammar-gap languages — these already had LSP configs but no @@ -958,6 +987,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["py", "pyi"], loader: || tree_sitter_python::LANGUAGE.into(), highlights_query: &[tree_sitter_python::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -965,6 +995,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["go"], loader: || tree_sitter_go::LANGUAGE.into(), highlights_query: &[tree_sitter_go::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, // JavaScript / TypeScript. One `tree-sitter-javascript` grammar parses @@ -980,6 +1011,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["js", "mjs", "cjs"], loader: || tree_sitter_javascript::LANGUAGE.into(), highlights_query: &[tree_sitter_javascript::HIGHLIGHT_QUERY], + locals_query: &[tree_sitter_javascript::LOCALS_QUERY], injections_query: &[], }, LanguageEntry { @@ -990,6 +1022,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ tree_sitter_javascript::HIGHLIGHT_QUERY, tree_sitter_javascript::JSX_HIGHLIGHT_QUERY, ], + locals_query: &[tree_sitter_javascript::LOCALS_QUERY], injections_query: &[], }, LanguageEntry { @@ -1000,6 +1033,10 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ tree_sitter_javascript::HIGHLIGHT_QUERY, tree_sitter_typescript::HIGHLIGHTS_QUERY, ], + locals_query: &[ + tree_sitter_javascript::LOCALS_QUERY, + tree_sitter_typescript::LOCALS_QUERY, + ], injections_query: &[], }, LanguageEntry { @@ -1011,6 +1048,10 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ tree_sitter_javascript::JSX_HIGHLIGHT_QUERY, tree_sitter_typescript::HIGHLIGHTS_QUERY, ], + locals_query: &[ + tree_sitter_javascript::LOCALS_QUERY, + tree_sitter_typescript::LOCALS_QUERY, + ], injections_query: &[], }, LanguageEntry { @@ -1018,6 +1059,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["toml"], loader: || tree_sitter_toml_ng::LANGUAGE.into(), highlights_query: &[tree_sitter_toml_ng::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -1025,6 +1067,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["zig", "zon"], loader: || tree_sitter_zig::LANGUAGE.into(), highlights_query: &[tree_sitter_zig::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, // JSON + YAML — config formats, both self-contained highlights and no @@ -1039,6 +1082,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["json"], loader: || tree_sitter_json::LANGUAGE.into(), highlights_query: &[tree_sitter_json::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, LanguageEntry { @@ -1046,6 +1090,7 @@ pub const BUILTIN_LANGUAGES: &[LanguageEntry] = &[ extensions: &["yaml", "yml"], loader: || tree_sitter_yaml::LANGUAGE.into(), highlights_query: &[tree_sitter_yaml::HIGHLIGHTS_QUERY], + locals_query: &[], injections_query: &[], }, ]; @@ -1076,6 +1121,9 @@ pub struct SyntaxRegistry { /// compilation failure (e.g. grammar / query ABI skew) is /// cached as `Err(message)` so we don't burn cycles re-trying. queries: RefCell, String>>>, + /// Compiled `locals.scm` query per language. Like `queries`, both + /// compilation failures and absent query sources are cached. + local_queries: RefCell, String>>>, /// Fence-name → canonical-language alias map (framing Q#IJ4). Seeded /// with [`default_injection_aliases`]; Lua adds to it through /// [`Self::register_injection_alias`]. Snapshotted into each @@ -1107,6 +1155,7 @@ impl SyntaxRegistry { parse_jobs: RefCell::new(HashMap::new()), extra_extensions: RefCell::new(HashMap::new()), queries: RefCell::new(HashMap::new()), + local_queries: RefCell::new(HashMap::new()), injection_aliases: RefCell::new(default_injection_aliases()), theme: Arc::new(Mutex::new(Theme::default_dark())), } @@ -1273,6 +1322,32 @@ impl SyntaxRegistry { result } + /// Lazy-compile and cache the bundled `locals.scm` query for + /// `lang_name`. Empty sources and compilation failures are cached. + #[must_use] + pub fn locals_query(&self, lang_name: &str) -> Option> { + if let Some(slot) = self.local_queries.borrow().get(lang_name) { + return slot.as_ref().ok().cloned(); + } + let language = self.language(lang_name)?; + let entry = BUILTIN_LANGUAGES.iter().find(|e| e.name == lang_name); + let source = entry.map_or_else(String::new, |e| e.locals_query.join("\n")); + if source.trim().is_empty() { + self.local_queries + .borrow_mut() + .insert(lang_name.to_owned(), Err("no locals query".to_owned())); + return None; + } + let compiled = tree_sitter::Query::new(&language, &source) + .map(Arc::new) + .map_err(|e| format!("compile {lang_name} locals: {e:?}")); + let result = compiled.as_ref().ok().cloned(); + self.local_queries + .borrow_mut() + .insert(lang_name.to_owned(), compiled); + result + } + /// Add or override a fence-name → language alias (framing Q#IJ4). The /// alias key is case-folded to match the resolver. Called from Lua via /// `pmacs.parse.injection_aliases`. @@ -1299,11 +1374,26 @@ impl SyntaxRegistry { let layers = raw .layers .iter() - .map(|l| Layer { - language_name: l.language_name.clone(), - tree: l.tree.clone(), - depth: l.depth, - highlight_query: self.highlights_query(&l.language_name), + .map(|layer| { + let highlight_query = self.highlights_query(&layer.language_name); + let local_facts = highlight_query + .as_deref() + .filter(|query| query_uses_local_predicates(query)) + .and_then(|_| self.locals_query(&layer.language_name)) + .map(|query| { + Arc::new(compute_local_facts( + &query, + &layer.tree, + raw.source.as_ref(), + )) + }); + Layer { + language_name: layer.language_name.clone(), + tree: layer.tree.clone(), + depth: layer.depth, + highlight_query, + local_facts, + } }) .collect(); Arc::new(ParseTreeBundle { @@ -1316,6 +1406,148 @@ impl SyntaxRegistry { } } +fn query_uses_local_predicates(query: &tree_sitter::Query) -> bool { + (0..query.pattern_count()).any(|pattern| { + query + .property_predicates(pattern) + .iter() + .any(|(property, _)| property.key.as_ref() == "local") + }) +} + +#[derive(Debug)] +struct LocalDefinition { + name_range: std::ops::Range, + value_range: std::ops::Range, +} + +#[derive(Debug)] +struct LocalScope { + inherits: bool, + range: std::ops::Range, + definitions: Vec, +} + +impl LocalFacts { + fn contains(&self, start_byte: usize, end_byte: usize) -> bool { + let (Ok(start_byte), Ok(end_byte)) = (u32::try_from(start_byte), u32::try_from(end_byte)) + else { + return false; + }; + self.ranges.binary_search(&(start_byte, end_byte)).is_ok() + } +} + +/// Resolve lexical definitions and references according to Tree-sitter's +/// standard `locals.scm` capture conventions. +fn compute_local_facts( + query: &tree_sitter::Query, + tree: &tree_sitter::Tree, + source: &[u8], +) -> LocalFacts { + let scope_capture = query.capture_index_for_name("local.scope"); + let definition_capture = query.capture_index_for_name("local.definition"); + let value_capture = query.capture_index_for_name("local.definition-value"); + let reference_capture = query.capture_index_for_name("local.reference"); + + let mut scopes = vec![LocalScope { + inherits: false, + range: 0..source.len(), + definitions: Vec::new(), + }]; + let mut ranges = Vec::new(); + let mut cursor = tree_sitter::QueryCursor::new(); + let mut captures = cursor.captures(query, tree.root_node(), source); + + while let Some((query_match, capture_index)) = captures.next() { + let capture = query_match.captures[*capture_index]; + let node_range = capture.node.byte_range(); + while scopes.len() > 1 + && node_range.start > scopes.last().expect("root scope exists").range.end + { + scopes.pop(); + } + + if Some(capture.index) == scope_capture { + let mut inherits = true; + for property in query.property_settings(query_match.pattern_index) { + if property.key.as_ref() == "local.scope-inherits" { + inherits = property + .value + .as_deref() + .is_none_or(|value| value == "true"); + } + } + scopes.push(LocalScope { + inherits, + range: node_range, + definitions: Vec::new(), + }); + continue; + } + + if Some(capture.index) == definition_capture { + let Some(_) = source.get(node_range.clone()) else { + continue; + }; + let value_range = query_match + .captures + .iter() + .find(|candidate| Some(candidate.index) == value_capture) + .map_or(0..0, |candidate| candidate.node.byte_range()); + scopes + .last_mut() + .expect("root scope exists") + .definitions + .push(LocalDefinition { + name_range: node_range.clone(), + value_range, + }); + if let (Ok(start), Ok(end)) = ( + u32::try_from(node_range.start), + u32::try_from(node_range.end), + ) { + ranges.push((start, end)); + } + continue; + } + + if Some(capture.index) != reference_capture { + continue; + } + let Some(name) = source.get(node_range.clone()) else { + continue; + }; + let mut resolved = false; + for scope in scopes.iter().rev() { + if scope.definitions.iter().rev().any(|definition| { + node_range.start >= definition.value_range.end + && source.get(definition.name_range.clone()) == Some(name) + }) { + resolved = true; + break; + } + if !scope.inherits { + break; + } + } + if resolved + && let (Ok(start), Ok(end)) = ( + u32::try_from(node_range.start), + u32::try_from(node_range.end), + ) + { + ranges.push((start, end)); + } + } + + ranges.sort_unstable(); + ranges.dedup(); + LocalFacts { + ranges: ranges.into_boxed_slice(), + } +} + impl Default for SyntaxRegistry { fn default() -> Self { Self::new() @@ -1418,20 +1650,22 @@ pub fn compute_highlight_spans_in_range( query, bundle.root_tree(), bundle.source.as_ref(), + bundle.layers[0].local_facts.as_deref(), byte_range, ) } /// Like [`compute_highlight_spans_in_range`] but over an explicit -/// `(tree, source)` — the per-layer form the producers call for each -/// injection layer (framing Q#IJ7). `source` is the whole buffer; a -/// child layer's tree carries absolute offsets into it, so the same +/// `(tree, source, local_facts)` layer tuple — the form producers call for +/// each injection layer (framing Q#IJ7 and Q#LQ5). `source` is the whole +/// buffer; a child layer's tree carries absolute offsets into it, so the same /// capture walk works unchanged. #[must_use] pub fn compute_highlight_spans_for( query: &tree_sitter::Query, tree: &tree_sitter::Tree, source: &[u8], + local_facts: Option<&LocalFacts>, byte_range: Option>, ) -> Vec { let mut spans = Vec::new(); @@ -1441,30 +1675,33 @@ pub fn compute_highlight_spans_for( } let root = tree.root_node(); let mut iter = cursor.captures(query, root, source); - while let Some((qmatch, capture_idx)) = iter.next() { - // Fail-closed on the locals property predicate. The capture - // iterator already applies text predicates (`#eq?`/`#match?`/ - // `#any-of?`), but `#is? local` / `#is-not? local` are *property* - // predicates (`Query::property_predicates`) that need a scope map - // built from the grammar's LOCALS_QUERY, which pmacs does not run. - // Applying such a capture regardless mis-styles shadowed locals — - // e.g. a local `console`/`require` in JS/TS would still capture as - // `@variable.builtin`/`@function.builtin`. Until locals processing - // exists, drop captures whose pattern carries one; the identifier - // falls back to its non-builtin capture. `#set!` (property - // *settings*) is a different API and is not consulted here. - if query - .property_predicates(qmatch.pattern_index) + while let Some((query_match, capture_index)) = iter.next() { + let capture = query_match.captures[*capture_index]; + let local_predicates_match = query + .property_predicates(query_match.pattern_index) .iter() - .any(|(prop, _)| &*prop.key == "local") - { + .filter(|(property, _)| property.key.as_ref() == "local") + .all(|(property, positive)| { + let node = property.capture_id.map_or(Some(capture.node), |target| { + query_match + .captures + .iter() + .find(|candidate| candidate.index as usize == target) + .map(|candidate| candidate.node) + }); + let is_local = node.is_some_and(|node| { + local_facts + .is_some_and(|facts| facts.contains(node.start_byte(), node.end_byte())) + }); + is_local == *positive + }); + if !local_predicates_match { continue; } - let cap = qmatch.captures[*capture_idx]; spans.push(HighlightSpan { - start_byte: cap.node.start_byte() as u32, - end_byte: cap.node.end_byte() as u32, - capture_index: cap.index, + start_byte: capture.node.start_byte() as u32, + end_byte: capture.node.end_byte() as u32, + capture_index: capture.index, }); } // Wider-first ordering at equal start: later writes (the @@ -1684,7 +1921,13 @@ mod tests { .highlight_query .as_ref() .expect("inline highlights resolved at settle"); - let spans = compute_highlight_spans_for(hquery, &inline.tree, &bundle.source, None); + let spans = compute_highlight_spans_for( + hquery, + &inline.tree, + &bundle.source, + inline.local_facts.as_deref(), + None, + ); assert!( !spans.is_empty(), "the inline layer produces highlight spans across both ranges" @@ -2221,7 +2464,13 @@ mod tests { .highlight_query .as_ref() .expect("yaml highlights resolved"); - let spans = compute_highlight_spans_for(query, &yaml.tree, &bundle.source, None); + let spans = compute_highlight_spans_for( + query, + &yaml.tree, + &bundle.source, + yaml.local_facts.as_deref(), + None, + ); assert!(!spans.is_empty(), "the yaml frontmatter layer highlights"); } @@ -2418,54 +2667,296 @@ mod tests { } #[test] - fn javascript_shadowed_builtin_is_not_mislabeled() { - // `#is-not? local` (JS/TS use it for console/require/etc.) needs a - // scope map from the LOCALS_QUERY we don't run, so - // `compute_highlight_spans` drops captures guarded by it. Here - // `console` is a LOCAL declaration — it must not surface as a - // `*.builtin` capture (which is what a naive run of the shared JS - // query would produce). - let reg = SyntaxRegistry::new(); - let language = reg.language("javascript").expect("javascript loads"); - let query = reg - .highlights_query("javascript") - .expect("javascript highlights compile"); - let mut buf = fresh_buffer("shadow.js"); - buf.apply_edit(EditOp::Insert { - pos: 0, - bytes: b"const console = 5;\nconsole;\n", - }) - .unwrap(); - let view = ParseView::new(&buf, language, "javascript".to_owned()); - let handle = view.handle(); - let _vid = buf.attach_view(Box::new(view)); - let bundle = parse_synchronously(&handle); - let spans = compute_highlight_spans(&query, &bundle); - assert!(!spans.is_empty(), "the JS query produced highlight spans"); - let names = query.capture_names(); - let builtin: Vec<&str> = spans - .iter() - .map(|s| names[s.capture_index as usize]) - .filter(|n| n.contains("builtin")) - .collect(); - assert!( - builtin.is_empty(), - "a locally-shadowed `console` must not get a *.builtin capture; got {builtin:?}" - ); - // ...and dropping the builtin pattern must not strip *all* styling: - // each `console` occurrence still keeps its ordinary `@variable` - // capture (the fallback), so the loss is only the `.builtin` refine. - let src = "const console = 5;\nconsole;\n"; - for (pos, _) in src.match_indices("console") { - let (start, end) = (pos as u32, (pos + "console".len()) as u32); - let caps: Vec<&str> = spans - .iter() - .filter(|s| s.start_byte == start && s.end_byte == end) - .map(|s| names[s.capture_index as usize]) - .collect(); + fn local_sensitive_builtin_highlights_have_compilable_locals_queries() { + let registry = SyntaxRegistry::new(); + for entry in BUILTIN_LANGUAGES { + let Some(highlights) = registry.highlights_query(entry.name) else { + continue; + }; + if !query_uses_local_predicates(&highlights) { + continue; + } assert!( - caps.iter().any(|n| n.starts_with("variable")), - "`console` at byte {pos} keeps a variable capture; got {caps:?}" + entry + .locals_query + .iter() + .any(|fragment| !fragment.trim().is_empty()), + "`{}` highlights use a local predicate but ship no locals query", + entry.name + ); + assert!( + registry.locals_query(entry.name).is_some(), + "`{}` highlights use a local predicate but its locals query does not compile", + entry.name + ); + } + } + + #[test] + fn javascript_local_predicates_distinguish_lexical_scope() { + let registry = SyntaxRegistry::new(); + let source = b"console.log('outer');\n\ + require('outer');\n\ + function f(console, require) {\n\ + console.log('inner');\n\ + require('inner');\n\ + }\n\ + window.alert('outer');\n"; + let bundle = parse_layered(®istry, "javascript", source); + let layer = &bundle.layers[0]; + let query = layer + .highlight_query + .as_deref() + .expect("javascript highlights compile"); + assert!( + layer.local_facts.is_some(), + "a local-sensitive highlight query must settle lexical facts" + ); + let spans = compute_highlight_spans(query, &bundle); + let names = query.capture_names(); + let captures_at = |start: usize, len: usize| -> Vec<&str> { + spans + .iter() + .filter(|span| { + span.start_byte == start as u32 && span.end_byte == (start + len) as u32 + }) + .map(|span| names[span.capture_index as usize]) + .collect() + }; + + for identifier in ["console", "require"] { + let positions: Vec = std::str::from_utf8(source) + .expect("fixture is UTF-8") + .match_indices(identifier) + .map(|(position, _)| position) + .collect(); + assert_eq!(positions.len(), 3, "fixture has three `{identifier}` uses"); + assert!( + captures_at(positions[0], identifier.len()) + .iter() + .any(|name| name.ends_with(".builtin")), + "unshadowed outer `{identifier}` keeps its builtin refinement" + ); + for position in &positions[1..] { + let captures = captures_at(*position, identifier.len()); + assert!( + !captures.iter().any(|name| name.ends_with(".builtin")), + "local `{identifier}` at byte {position} is not builtin: {captures:?}" + ); + assert!( + captures.iter().any(|name| name.starts_with("variable")), + "local `{identifier}` keeps an ordinary variable capture: {captures:?}" + ); + } + } + + let window = std::str::from_utf8(source) + .expect("fixture is UTF-8") + .find("window") + .expect("window fixture"); + assert!( + captures_at(window, "window".len()) + .iter() + .any(|name| name == &"variable.builtin"), + "an unresolved builtin after the function remains builtin" + ); + } + + #[test] + fn positive_and_capture_qualified_local_predicates_use_resolved_facts() { + let registry = SyntaxRegistry::new(); + let source = b"let f = () => {};\nf();\ng();\n"; + let bundle = parse_layered(®istry, "javascript", source); + let language = registry.language("javascript").expect("javascript loads"); + let facts = bundle.layers[0] + .local_facts + .as_deref() + .expect("javascript local facts settle"); + + let positive = tree_sitter::Query::new(&language, "((identifier) @local-id (#is? local))") + .expect("positive local predicate compiles"); + let positive_spans = + compute_highlight_spans_for(&positive, bundle.root_tree(), source, Some(facts), None); + let f_positions: Vec = std::str::from_utf8(source) + .expect("fixture is UTF-8") + .match_indices('f') + .map(|(position, _)| position) + .collect(); + assert_eq!(f_positions.len(), 2); + for position in f_positions { + assert!( + positive_spans.iter().any(|span| { + span.start_byte == position as u32 && span.end_byte == (position + 1) as u32 + }), + "definition/reference `f` at byte {position} is local" + ); + } + let g_position = std::str::from_utf8(source) + .expect("fixture is UTF-8") + .find("g()") + .expect("g call"); + assert!( + positive_spans + .iter() + .all(|span| span.start_byte != g_position as u32), + "unresolved `g` does not satisfy #is? local" + ); + + let qualified = tree_sitter::Query::new( + &language, + "((call_expression function: (identifier) @callee) @call \ + (#is? @callee local))", + ) + .expect("capture-qualified local predicate compiles"); + let qualified_spans = + compute_highlight_spans_for(&qualified, bundle.root_tree(), source, Some(facts), None); + let qualified_names = qualified.capture_names(); + assert!( + qualified_spans.iter().any(|span| { + qualified_names[span.capture_index as usize] == "call" + && span.start_byte + == source + .windows(4) + .position(|window| window == b"f();") + .expect("f call") as u32 + }), + "the call whose @callee is local satisfies the qualified predicate" + ); + assert!( + qualified_spans + .iter() + .all(|span| span.start_byte != g_position as u32), + "the call whose @callee is unresolved fails the qualified predicate" + ); + } + + #[test] + fn local_definition_value_and_scope_inheritance_control_resolution() { + let registry = SyntaxRegistry::new(); + let language = registry.language("javascript").expect("javascript loads"); + + let value_source = b"let x = x;\nx;\n"; + let value_tree = { + let mut parser = tree_sitter::Parser::new(); + parser + .set_language(&language) + .expect("set javascript language"); + parser + .parse(value_source, None) + .expect("parse value fixture") + }; + let value_locals = tree_sitter::Query::new( + &language, + "(variable_declarator \ + name: (identifier) @local.definition \ + value: (identifier) @local.definition-value) \ + (identifier) @local.reference", + ) + .expect("definition-value locals query compiles"); + let value_facts = compute_local_facts(&value_locals, &value_tree, value_source); + let x_positions: Vec = std::str::from_utf8(value_source) + .expect("fixture is UTF-8") + .match_indices('x') + .map(|(position, _)| position) + .collect(); + assert_eq!(x_positions.len(), 3); + assert!(value_facts.contains(x_positions[0], x_positions[0] + 1)); + assert!( + !value_facts.contains(x_positions[1], x_positions[1] + 1), + "a definition is not visible inside its own value" + ); + assert!(value_facts.contains(x_positions[2], x_positions[2] + 1)); + + let scope_source = b"let x = 1;\nfunction f() { x; }\nx;\n"; + let scope_tree = { + let mut parser = tree_sitter::Parser::new(); + parser + .set_language(&language) + .expect("set javascript language"); + parser + .parse(scope_source, None) + .expect("parse scope fixture") + }; + let scope_locals = tree_sitter::Query::new( + &language, + "((function_declaration) @local.scope \ + (#set! local.scope-inherits false)) \ + (variable_declarator name: (identifier) @local.definition) \ + (identifier) @local.reference", + ) + .expect("non-inheriting locals query compiles"); + let scope_facts = compute_local_facts(&scope_locals, &scope_tree, scope_source); + let x_positions: Vec = std::str::from_utf8(scope_source) + .expect("fixture is UTF-8") + .match_indices('x') + .map(|(position, _)| position) + .collect(); + assert_eq!(x_positions.len(), 3); + assert!(scope_facts.contains(x_positions[0], x_positions[0] + 1)); + assert!( + !scope_facts.contains(x_positions[1], x_positions[1] + 1), + "a non-inheriting scope cannot see the outer `x`" + ); + assert!( + scope_facts.contains(x_positions[2], x_positions[2] + 1), + "leaving the scope restores outer resolution" + ); + } + + #[test] + fn typescript_locals_compose_javascript_scopes_and_parameter_delta() { + let registry = SyntaxRegistry::new(); + for (language_name, source) in [ + ( + "typescript", + &b"function f(console: string) { console.log('x'); }\n\ + window.alert('x');\n"[..], + ), + ( + "typescriptreact", + &b"function F(console: string) { return
{console}
; }\n\ + window.alert('x');\n"[..], + ), + ] { + let locals = registry + .locals_query(language_name) + .unwrap_or_else(|| panic!("{language_name} locals compile")); + assert!( + locals.capture_index_for_name("local.scope").is_some() + && locals.capture_index_for_name("local.definition").is_some() + && locals.capture_index_for_name("local.reference").is_some(), + "{language_name} includes JavaScript's scopes and references" + ); + + let bundle = parse_layered(®istry, language_name, source); + let layer = &bundle.layers[0]; + let query = layer + .highlight_query + .as_deref() + .expect("highlights compile"); + let spans = compute_highlight_spans(query, &bundle); + let names = query.capture_names(); + let text = std::str::from_utf8(source).expect("fixture is UTF-8"); + for (position, _) in text.match_indices("console") { + assert!( + spans + .iter() + .filter(|span| { + span.start_byte == position as u32 + && span.end_byte == (position + "console".len()) as u32 + }) + .all(|span| !names[span.capture_index as usize].ends_with(".builtin")), + "{language_name} parameter/reference `console` is local" + ); + } + let window = text.find("window").expect("window fixture"); + assert!( + spans.iter().any(|span| { + span.start_byte == window as u32 + && span.end_byte == (window + "window".len()) as u32 + && names[span.capture_index as usize] == "variable.builtin" + }), + "{language_name} unresolved `window` remains builtin" ); } } diff --git a/tests/m4_acceptance.rs b/tests/m4_acceptance.rs index 753aeec..69c8500 100644 --- a/tests/m4_acceptance.rs +++ b/tests/m4_acceptance.rs @@ -771,6 +771,98 @@ fn m4_3_theming_via_lua_color_scheme() { ); } +/// Locals-query acceptance: the shipped JavaScript grammar must distinguish an +/// unresolved builtin from a lexically-shadowed parameter, then replace the +/// classification with the fresh parse bundle after an edit removes the +/// shadow. The theme maps only `variable.builtin`, making the classification +/// observable in rendered cells rather than through an internal scope map. +#[test] +fn m4_locals_query_shadowing_and_edit_freshness() { + use pmacs::cell::Color; + + const COLS: usize = 40; + const BUILTIN: Color = Color::Indexed(6); + let initial = "console;\nfunction f(console) {\n console;\n}\n"; + let edited = "console;\nfunction f(logger) {\n console;\n}\n"; + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("locals.js"); + std::fs::write(&path, initial.as_bytes()).expect("write JavaScript fixture"); + + let mut state = open_and_wait_for_parse(path); + state + .lua_host + .lua() + .load( + r#" + pmacs.theme.set { + ["variable.builtin"] = { fg = 6, bold = true }, + } + "#, + ) + .exec() + .expect("apply builtin-only theme"); + + let before = render_active_window_to_grid(&mut state, 5, COLS as u32); + for (col, cell) in before.iter().take(7).enumerate() { + assert_eq!( + cell.style.fg, BUILTIN, + "unshadowed row-0 `console` byte {col} is builtin" + ); + assert!( + cell.style.bold, + "builtin-only theme reaches row-0 `console` byte {col}" + ); + } + for col in 11..18 { + assert_ne!( + before[COLS + col].style.fg, + BUILTIN, + "parameter `console` byte {col} is lexically local" + ); + } + for col in 2..9 { + assert_ne!( + before[2 * COLS + col].style.fg, + BUILTIN, + "reference to shadowing parameter at row 2, col {col} is not builtin" + ); + } + + state + .lua_host + .lua() + .load( + r" + local buf = pmacs.window.buffer() + buf:replace(20, 27, 'logger') + pmacs.parse._dispatch(buf, 'javascript') + ", + ) + .exec() + .expect("rename shadowing parameter"); + pump_async(&mut state, |s| { + current_tree_text(s).as_deref() == Some(edited) + }); + assert_eq!( + current_tree_text(&state).as_deref(), + Some(edited), + "the edited JavaScript parse settled" + ); + + let after = render_active_window_to_grid(&mut state, 5, COLS as u32); + for col in 2..9 { + assert_eq!( + after[2 * COLS + col].style.fg, + BUILTIN, + "fresh local facts restore builtin styling at row 2, col {col}" + ); + assert!( + after[2 * COLS + col].style.bold, + "fresh builtin capture reaches the rendered cell at row 2, col {col}" + ); + } +} + /// Sanity: the bundled `default_dark` theme produces a non-empty /// capture map and resolves common captures to non-default styles. /// Catches accidental regressions to a literally empty theme that