pmacs/docs/locals-query-processing-framing.md

Locals-query processing - syntax-highlight completion

Status: Revision 1 implemented and landed as PR #134 on 2026-07-22 (8cbb9f4). Protocol remains v18.

Base: githubsucks/main at 8bd8298 (mode system #129, Vterm Stage 2 #130, modeline detection #132, and landed-state handoff #133). Protocol remains v18.

Problem

pmacs runs each grammar's highlights.scm directly through a tree_sitter::QueryCursor. That cursor evaluates text predicates such as #eq?, #match?, and #any-of?, but it does not assign or evaluate semantic properties. In particular, JavaScript's bundled highlight query contains:

((identifier) @variable.builtin
 (#match? @variable.builtin "^(arguments|module|console|window|document)$")
 (#is-not? local))

((identifier) @function.builtin
 (#eq? @function.builtin "require")
 (#is-not? local))

A correct highlighter must first run the grammar's LOCALS_QUERY, resolve lexical definitions and references, and then apply #is? local / #is-not? local while selecting highlight captures. Without those facts, a local console or require can be styled as a builtin.

The current implementation fails closed by dropping every highlight pattern that carries a local property predicate. That prevents the false-positive shadowed-builtin style, but it also drops legitimate builtin highlighting for non-shadowed console, window, require, and peers. This is the first remaining item in the side-quest north-star list at docs/side-quest-backlog.md:244-249.

Goal

Run the bundled locals query as part of each settled syntax layer, retain a compact lexical-local map, and use that map to evaluate #is? local and #is-not? local in both syntax-highlight producers:

1. highlight::SyntaxHighlightView (TUI / overlay path), and
2. semantic_render::scoped_style_spans (semantic/GPU path).

A shadowed builtin must retain its ordinary fallback capture but lose the builtin refinement. A non-shadowed builtin must regain the builtin capture. The result must stay correct for nested scopes, TypeScript's inherited locals query, injected-language layers, viewport-limited rendering, and edits that settle a new parse bundle.

Scope

In

• Add bundled locals.scm fragments to syntax grammar metadata.
• Compose inherited locals fragments base-first, exactly as highlight fragments already compose.
• Implement Tree-sitter's local-scope conventions:
  • @local.scope
  • @local.definition
  • @local.definition-value
  • @local.reference
  • #set! local.scope-inherits false
• Evaluate positive and negative local property predicates, including the optional capture-qualified form accepted by Tree-sitter's query parser.
• Cache local facts on each settled parse layer.
• Feed those facts to both highlight producers.
• Cover lexical behavior and end-to-end rendering under both supported Lua backends.

Out

• No new grammar, parser, or language-detection behavior.
• No Lua API, command, config key, theme key, protocol field, or frontend-only state.
• No LSP semantic-token changes.
• No user-defined query registration surface.
• No cross-language name resolution between a host layer and an injected child layer.
• No Tree-sitter-highlight reference_highlight propagation from a local definition's syntactic capture to all references. The target is property predicate correctness. Adding propagation would change ordinary variable styling beyond the reported builtin bug and needs separate framing.
• No general engine for arbitrary #is? property keys. This side quest owns the Tree-sitter-defined boolean local property only; other property predicates retain their current behavior.

Existing contracts to preserve

1. Layering: every injection Layer owns its grammar tree and highlight query. Deeper layers and later same-depth siblings retain their existing precedence.
2. Viewport work: the semantic producer's highlight capture walk remains restricted with QueryCursor::set_byte_range; rendering a frame must not launch a whole-file locals walk.
3. Settle freshness: a new parse bundle replaces the old bundle atomically. Local facts must travel with the same bundle, never in a side cache that can pair old scopes with a new tree.
4. Incremental edits: a completed reparse produces new local facts before the bundle becomes visible. Until settle, producers continue to use the previous internally consistent bundle.
5. Query inheritance: fragments are newline-joined base-first. Bare concatenation can extend a trailing Scheme comment and is forbidden.
6. Fallback styling: suppressing a builtin refinement must not suppress an independent ordinary-variable capture for the same identifier.
7. No protocol change: all work is internal render state.

Decisions

Q#LQ1 - Grammar metadata carries locals fragments

LanguageEntry gains locals_query: &'static [&'static str], parallel to highlights_query and injections_query.

The bundled crates currently exposing LOCALS_QUERY are wired as follows:

pmacs language	effective locals fragments
lua	tree_sitter_lua::LOCALS_QUERY
javascript	tree_sitter_javascript::LOCALS_QUERY
javascriptreact	tree_sitter_javascript::LOCALS_QUERY
typescript	JavaScript locals, then TypeScript locals
typescriptreact	JavaScript locals, then TypeScript locals

All other entries use an empty slice. TypeScript's locals query is a small parameter-definition delta, so compiling it alone would omit JavaScript's scopes, declarations, and references. JSX uses the JavaScript grammar and therefore the JavaScript locals query. TSX uses the TypeScript crate's TSX language with the same base-plus-delta locals composition.

SyntaxRegistry lazily compiles and caches one effective locals query per language, including cached failure/no-query results, matching the existing highlight-query policy.

Q#LQ2 - Local facts follow Tree-sitter lexical semantics

A locals capture walk maintains a stack of scopes. The stack begins with one non-inheriting root scope covering the layer. Captures are processed in query order and source order:

1. @local.scope pushes a scope covering that node. It inherits outer definitions unless its pattern sets local.scope-inherits to false.
2. @local.definition records the identifier in the innermost scope and marks that identifier range local.
3. A sibling @local.definition-value capture records the initializer/value range. That new definition is not visible while resolving references inside its value, so an outer definition with the same name can still win there.
4. @local.reference searches definitions newest-first, then scopes innermost-first. Search stops at a non-inheriting scope. A resolved reference range is marked local; an unresolved reference remains non-local.
5. Scopes whose end precedes the next capture are popped.

Definition names are compared as borrowed source byte slices. No identifier strings are allocated. Invalid or out-of-bounds ranges do not produce local facts.

The result is an opaque LocalFacts value containing sorted, deduplicated (start_byte, end_byte) ranges. Highlight predicate checks use binary search; they do not hash, copy source text, or rebuild scope state.

Q#LQ3 - Predicate evaluation is per emitted capture

Before emitting a highlight capture, inspect Query::property_predicates(pattern_index):

• #is? local passes only when the selected node is in LocalFacts.
• #is-not? local passes only when the selected node is not in LocalFacts.
• If the property names a capture, test that capture's node.
• If it does not name a capture, test the highlight capture currently being considered, matching Tree-sitter-highlight's per-node behavior.
• Multiple local predicates on one pattern are conjunctive.
• A missing or failed locals query yields no local ranges: negative predicates pass and positive predicates fail. This preserves useful non-local highlighting without inventing local classifications.
• Predicates with keys other than local remain ignored, preserving the current query engine's scope.

Text predicates remain the query cursor's responsibility. Property settings such as local.scope-inherits are read only by local analysis; they are not mistaken for highlight predicates.

Q#LQ4 - Facts are computed once at settle and owned by the layer

SyntaxRegistry::resolve_layer_queries remains the main-thread Stage 2 handoff. For each raw worker layer it:

1. resolves the cached highlight query;
2. checks whether that query contains any local property predicate;
3. only when needed, resolves the cached locals query and walks the layer tree;
4. stores Arc<LocalFacts> beside the layer's tree and highlight query.

This is a single whole-layer analysis per completed parse, not per render. Languages whose highlights never ask about local pay only the cheap predicate scan and carry no facts. This includes Lua today even though its locals query is correctly registered for future local-sensitive highlight patterns.

Putting facts on Layer makes the consistency invariant structural:

settled Layer = tree + grammar + highlight query + local facts

A producer cannot accidentally retrieve facts for another buffer revision. Child injection layers compute facts against their own tree and grammar; local bindings never cross layer boundaries. A deliberately combined injection layer shares one tree and therefore one lexical environment, matching its combined parse semantics.

Q#LQ5 - Both producers share one capture-selection function

compute_highlight_spans_for accepts the layer's optional local facts and owns predicate evaluation. The whole-buffer overlay path and the viewport-limited semantic path both call this function. There is no second predicate implementation in a frontend.

compute_highlight_spans_in_range passes the root layer's facts. Injection producers pass each layer's facts. Existing wider-first ordering and cross-layer merge precedence remain unchanged after captures are selected.

Q#LQ6 - Performance boundary

The locals-query capture walk is linear for one settled layer. Like upstream, resolving references scans visible definitions newest-first, so the worst case is O(\text{captures} + \text{references} \times \text{definitions}). It runs only for a language whose highlight query actually contains a local predicate, and only once when a fresh parse bundle settles.

The render hot paths remain:

• TUI: cached whole-layer highlight spans, rebuilt only when the bundle pointer changes;
• semantic/GPU: viewport-bounded highlight query plus binary-search local checks.

No frame performs a whole-file locals query. Memory is two u32 offsets per local definition/resolved reference plus vector capacity; ranges are sorted and deduplicated before storage.

Incremental locals invalidation is intentionally bundle-granular. A local edit can alter all later name resolution in a scope, so attempting to splice only changed ranges without a scope dependency graph risks stale classifications. The parse itself remains incremental; this bounded lexical pass is the boring, correct cutover.

Data flow

worker parse
  -> raw ParseTreeBundle { Layer { tree, language, no queries/facts } }
  -> main-thread resolve_layer_queries
       -> cached highlights query
       -> cached locals query (only if highlights uses local predicates)
       -> lexical capture walk -> sorted LocalFacts
  -> settled ParseTreeBundle
       -> TUI SyntaxHighlightView cache rebuild
       -> semantic/GPU viewport capture walk
            -> shared local predicate filter
            -> existing span ordering/merge/theme lookup

Acceptance criteria

1. Non-shadowed builtin restored: JavaScript console, window, or require with no matching lexical definition emits its bundled *.builtin capture.
2. Shadowed builtin suppressed: a parameter or local declaration named console/require and references resolved to it do not emit a builtin capture; their ordinary variable captures remain.
3. Scope correctness: shadowing is confined to its lexical scope. A builtin before/after the scope remains builtin, while the definition and references inside are local.
4. Positive predicate: a focused custom highlight query using #is? local emits resolved definitions/references and rejects an unresolved identifier.
5. TypeScript inheritance: parameter shadowing in TypeScript and TSX uses the JavaScript base locals query plus the TypeScript delta; query compilation and classification succeed for both grammars.
6. End-to-end render: opening a JavaScript buffer through the shipped runtime and applying a theme that styles only variable.builtin renders an unshadowed builtin with that style and a shadowed occurrence without it.
7. Edit freshness: after changing a shadowing identifier and settling the new parse, the next render reflects the new local/non-local classification; no stale local facts survive.
8. Producer coverage: both the overlay and semantic/GPU callsites pass the corresponding layer facts to the shared capture walk; injected layers keep their own facts.
9. Backend parity: the focused acceptance test passes under default Luau and --no-default-features --features lua54.
10. No regressions: formatting, Clippy, default library tests, CRDT library tests, M4 acceptance (excluding the machine-broken basedpyright case), required GPU tests, workspace sweep, and git diff --check pass.

Expected files

• src/syntax.rs - grammar metadata, locals-query cache, lexical analysis, layer facts, predicate selection, unit coverage.
• src/highlight.rs - pass per-layer facts to the overlay producer.
• src/semantic_render.rs - pass per-layer facts to the viewport producer.
• tests/m4_acceptance.rs - end-to-end local/non-local rendering and edit freshness.
• docs/agent-handoff.md and docs/active-work.md - updated only after the implementation is proven and published according to their protocols.

No other runtime, Lua, frontend, protocol, or theme file should need a behavior change.