From 42a28b90c71a6c70e16b4747f50f995684d42db6 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 22 Jul 2026 12:03:37 -0400 Subject: [PATCH] docs: frame locals-query processing Define grammar locals metadata, lexical resolution semantics, shared local predicate filtering, settle-time caching, producer coverage, and acceptance criteria for shadowed versus non-shadowed builtins. --- docs/locals-query-processing-framing.md | 304 ++++++++++++++++++++++++ 1 file changed, 304 insertions(+) create mode 100644 docs/locals-query-processing-framing.md diff --git a/docs/locals-query-processing-framing.md b/docs/locals-query-processing-framing.md new file mode 100644 index 0000000..79013b6 --- /dev/null +++ b/docs/locals-query-processing-framing.md @@ -0,0 +1,304 @@ +# Locals-query processing - syntax-highlight completion + +**Status:** Revision 1, implementation active by user direction, 2026-07-22. + +**Base:** `githubsucks/main` at `8bd8298` (mode system #129, Vterm Stage 2 +#130, modeline detection #132, and landed-state handoff #133). Protocol remains +v18. + +## Problem + +pmacs runs each grammar's `highlights.scm` directly through a +`tree_sitter::QueryCursor`. That cursor evaluates text predicates such as +`#eq?`, `#match?`, and `#any-of?`, but it does not assign or evaluate semantic +properties. In particular, JavaScript's bundled highlight query contains: + +```scheme +((identifier) @variable.builtin + (#match? @variable.builtin "^(arguments|module|console|window|document)$") + (#is-not? local)) + +((identifier) @function.builtin + (#eq? @function.builtin "require") + (#is-not? local)) +``` + +A correct highlighter must first run the grammar's `LOCALS_QUERY`, resolve +lexical definitions and references, and then apply `#is? local` / +`#is-not? local` while selecting highlight captures. Without those facts, +a local `console` or `require` can be styled as a builtin. + +The current implementation fails closed by dropping every highlight pattern +that carries a `local` property predicate. That prevents the false-positive +shadowed-builtin style, but it also drops legitimate builtin highlighting for +non-shadowed `console`, `window`, `require`, and peers. This is the first +remaining item in the side-quest north-star list at +`docs/side-quest-backlog.md:244-249`. + +## Goal + +Run the bundled locals query as part of each settled syntax layer, retain a +compact lexical-local map, and use that map to evaluate `#is? local` and +`#is-not? local` in both syntax-highlight producers: + +1. `highlight::SyntaxHighlightView` (TUI / overlay path), and +2. `semantic_render::scoped_style_spans` (semantic/GPU path). + +A shadowed builtin must retain its ordinary fallback capture but lose the +builtin refinement. A non-shadowed builtin must regain the builtin capture. +The result must stay correct for nested scopes, TypeScript's inherited locals +query, injected-language layers, viewport-limited rendering, and edits that +settle a new parse bundle. + +## Scope + +### In + +- Add bundled `locals.scm` fragments to syntax grammar metadata. +- Compose inherited locals fragments base-first, exactly as highlight fragments + already compose. +- Implement Tree-sitter's local-scope conventions: + - `@local.scope` + - `@local.definition` + - `@local.definition-value` + - `@local.reference` + - `#set! local.scope-inherits false` +- Evaluate positive and negative `local` property predicates, including the + optional capture-qualified form accepted by Tree-sitter's query parser. +- Cache local facts on each settled parse layer. +- Feed those facts to both highlight producers. +- Cover lexical behavior and end-to-end rendering under both supported Lua + backends. + +### Out + +- No new grammar, parser, or language-detection behavior. +- No Lua API, command, config key, theme key, protocol field, or frontend-only + state. +- No LSP semantic-token changes. +- No user-defined query registration surface. +- No cross-language name resolution between a host layer and an injected + child layer. +- No Tree-sitter-highlight `reference_highlight` propagation from a local + definition's syntactic capture to all references. The target is property + predicate correctness. Adding propagation would change ordinary variable + styling beyond the reported builtin bug and needs separate framing. +- No general engine for arbitrary `#is?` property keys. This side quest owns + the Tree-sitter-defined boolean `local` property only; other property + predicates retain their current behavior. + +## Existing contracts to preserve + +1. **Layering:** every injection `Layer` owns its grammar tree and highlight + query. Deeper layers and later same-depth siblings retain their existing + precedence. +2. **Viewport work:** the semantic producer's highlight capture walk remains + restricted with `QueryCursor::set_byte_range`; rendering a frame must not + launch a whole-file locals walk. +3. **Settle freshness:** a new parse bundle replaces the old bundle + atomically. Local facts must travel with the same bundle, never in a side + cache that can pair old scopes with a new tree. +4. **Incremental edits:** a completed reparse produces new local facts before + the bundle becomes visible. Until settle, producers continue to use the + previous internally consistent bundle. +5. **Query inheritance:** fragments are newline-joined base-first. Bare + concatenation can extend a trailing Scheme comment and is forbidden. +6. **Fallback styling:** suppressing a builtin refinement must not suppress + an independent ordinary-variable capture for the same identifier. +7. **No protocol change:** all work is internal render state. + +## Decisions + +### Q#LQ1 - Grammar metadata carries locals fragments + +`LanguageEntry` gains `locals_query: &'static [&'static str]`, parallel to +`highlights_query` and `injections_query`. + +The bundled crates currently exposing `LOCALS_QUERY` are wired as follows: + +| pmacs language | effective locals fragments | +| --- | --- | +| `lua` | `tree_sitter_lua::LOCALS_QUERY` | +| `javascript` | `tree_sitter_javascript::LOCALS_QUERY` | +| `javascriptreact` | `tree_sitter_javascript::LOCALS_QUERY` | +| `typescript` | JavaScript locals, then TypeScript locals | +| `typescriptreact` | JavaScript locals, then TypeScript locals | + +All other entries use an empty slice. TypeScript's locals query is a small +parameter-definition delta, so compiling it alone would omit JavaScript's +scopes, declarations, and references. JSX uses the JavaScript grammar and +therefore the JavaScript locals query. TSX uses the TypeScript crate's TSX +language with the same base-plus-delta locals composition. + +`SyntaxRegistry` lazily compiles and caches one effective locals query per +language, including cached failure/no-query results, matching the existing +highlight-query policy. + +### Q#LQ2 - Local facts follow Tree-sitter lexical semantics + +A locals capture walk maintains a stack of scopes. The stack begins with one +non-inheriting root scope covering the layer. Captures are processed in query +order and source order: + +1. `@local.scope` pushes a scope covering that node. It inherits outer + definitions unless its pattern sets `local.scope-inherits` to `false`. +2. `@local.definition` records the identifier in the innermost scope and marks + that identifier range local. +3. A sibling `@local.definition-value` capture records the initializer/value + range. That new definition is not visible while resolving references inside + its value, so an outer definition with the same name can still win there. +4. `@local.reference` searches definitions newest-first, then scopes + innermost-first. Search stops at a non-inheriting scope. A resolved + reference range is marked local; an unresolved reference remains non-local. +5. Scopes whose end precedes the next capture are popped. + +Definition names are compared as borrowed source byte slices. No identifier +strings are allocated. Invalid or out-of-bounds ranges do not produce local +facts. + +The result is an opaque `LocalFacts` value containing sorted, deduplicated +`(start_byte, end_byte)` ranges. Highlight predicate checks use binary search; +they do not hash, copy source text, or rebuild scope state. + +### Q#LQ3 - Predicate evaluation is per emitted capture + +Before emitting a highlight capture, inspect +`Query::property_predicates(pattern_index)`: + +- `#is? local` passes only when the selected node is in `LocalFacts`. +- `#is-not? local` passes only when the selected node is not in `LocalFacts`. +- If the property names a capture, test that capture's node. +- If it does not name a capture, test the highlight capture currently being + considered, matching Tree-sitter-highlight's per-node behavior. +- Multiple `local` predicates on one pattern are conjunctive. +- A missing or failed locals query yields no local ranges: negative predicates + pass and positive predicates fail. This preserves useful non-local + highlighting without inventing local classifications. +- Predicates with keys other than `local` remain ignored, preserving the + current query engine's scope. + +Text predicates remain the query cursor's responsibility. Property settings +such as `local.scope-inherits` are read only by local analysis; they are not +mistaken for highlight predicates. + +### Q#LQ4 - Facts are computed once at settle and owned by the layer + +`SyntaxRegistry::resolve_layer_queries` remains the main-thread Stage 2 handoff. +For each raw worker layer it: + +1. resolves the cached highlight query; +2. checks whether that query contains any `local` property predicate; +3. only when needed, resolves the cached locals query and walks the layer tree; +4. stores `Arc` beside the layer's tree and highlight query. + +This is a single whole-layer analysis per completed parse, not per render. +Languages whose highlights never ask about `local` pay only the cheap predicate +scan and carry no facts. This includes Lua today even though its locals query +is correctly registered for future local-sensitive highlight patterns. + +Putting facts on `Layer` makes the consistency invariant structural: + +```text +settled Layer = tree + grammar + highlight query + local facts +``` + +A producer cannot accidentally retrieve facts for another buffer revision. +Child injection layers compute facts against their own tree and grammar; local +bindings never cross layer boundaries. A deliberately combined injection +layer shares one tree and therefore one lexical environment, matching its +combined parse semantics. + +### Q#LQ5 - Both producers share one capture-selection function + +`compute_highlight_spans_for` accepts the layer's optional local facts and +owns predicate evaluation. The whole-buffer overlay path and the +viewport-limited semantic path both call this function. There is no second +predicate implementation in a frontend. + +`compute_highlight_spans_in_range` passes the root layer's facts. Injection +producers pass each layer's facts. Existing wider-first ordering and +cross-layer merge precedence remain unchanged after captures are selected. + +### Q#LQ6 - Performance boundary + +Local analysis is linear in the locals-query captures for one settled layer. +It runs only for a language whose highlight query actually contains a `local` +predicate, and only once when a fresh parse bundle settles. + +The render hot paths remain: + +- TUI: cached whole-layer highlight spans, rebuilt only when the bundle pointer + changes; +- semantic/GPU: viewport-bounded highlight query plus binary-search local + checks. + +No frame performs a whole-file locals query. Memory is two `u32` offsets per +local definition/resolved reference plus vector capacity; ranges are sorted +and deduplicated before storage. + +Incremental locals invalidation is intentionally bundle-granular. A local edit +can alter all later name resolution in a scope, so attempting to splice only +changed ranges without a scope dependency graph risks stale classifications. +The parse itself remains incremental; this bounded lexical pass is the boring, +correct cutover. + +## Data flow + +```text +worker parse + -> raw ParseTreeBundle { Layer { tree, language, no queries/facts } } + -> main-thread resolve_layer_queries + -> cached highlights query + -> cached locals query (only if highlights uses local predicates) + -> lexical capture walk -> sorted LocalFacts + -> settled ParseTreeBundle + -> TUI SyntaxHighlightView cache rebuild + -> semantic/GPU viewport capture walk + -> shared local predicate filter + -> existing span ordering/merge/theme lookup +``` + +## Acceptance criteria + +1. **Non-shadowed builtin restored:** JavaScript `console`, `window`, or + `require` with no matching lexical definition emits its bundled + `*.builtin` capture. +2. **Shadowed builtin suppressed:** a parameter or local declaration named + `console`/`require` and references resolved to it do not emit a builtin + capture; their ordinary variable captures remain. +3. **Scope correctness:** shadowing is confined to its lexical scope. A builtin + before/after the scope remains builtin, while the definition and references + inside are local. +4. **Positive predicate:** a focused custom highlight query using `#is? local` + emits resolved definitions/references and rejects an unresolved identifier. +5. **TypeScript inheritance:** parameter shadowing in TypeScript and TSX uses + the JavaScript base locals query plus the TypeScript delta; query compilation + and classification succeed for both grammars. +6. **End-to-end render:** opening a JavaScript buffer through the shipped + runtime and applying a theme that styles only `variable.builtin` renders an + unshadowed builtin with that style and a shadowed occurrence without it. +7. **Edit freshness:** after changing a shadowing identifier and settling the + new parse, the next render reflects the new local/non-local classification; + no stale local facts survive. +8. **Producer coverage:** both the overlay and semantic/GPU callsites pass the + corresponding layer facts to the shared capture walk; injected layers keep + their own facts. +9. **Backend parity:** the focused acceptance test passes under default Luau + and `--no-default-features --features lua54`. +10. **No regressions:** formatting, Clippy, default library tests, CRDT library + tests, M4 acceptance (excluding the machine-broken basedpyright case), + required GPU tests, workspace sweep, and `git diff --check` pass. + +## Expected files + +- `src/syntax.rs` - grammar metadata, locals-query cache, lexical analysis, + layer facts, predicate selection, unit coverage. +- `src/highlight.rs` - pass per-layer facts to the overlay producer. +- `src/semantic_render.rs` - pass per-layer facts to the viewport producer. +- `tests/m4_acceptance.rs` - end-to-end local/non-local rendering and edit + freshness. +- `docs/agent-handoff.md` and `docs/active-work.md` - updated only after the + implementation is proven and published according to their protocols. + +No other runtime, Lua, frontend, protocol, or theme file should need a behavior +change.