Completes Arc 8 Stage 1: the Lua-side tables that turn a recognized
grammar into a usable mode, plus the acceptance suite for all twelve
framing criteria.
comment.lua -- `lean4 = "--"` (Q#LN5). Line comments only; Lean's block
comment `/- -/` and docstring `/-- -/` belong to the comment arc's own
named deferral and this lane does not front-run it.
pair.lua -- `⟨⟩`, `⦃⦄`, `⟮⟯` alongside the ASCII brackets (Q#LN6). The
anonymous constructor is among the most-typed constructs in Lean;
omitting it would make the pair set feel broken. The other two ride along
because the Stage 4 input method can produce them, and a bracket the pair
set does not understand is worse than one it does. All three sit outside
the nine built-in pair chars, so per Q#AP1 their undo is
cross-peer-degraded -- the documented, pre-existing limitation of
user-extended pairs. No `''`: Lean uses the prime as an identifier suffix
(`h'`, `foo'`), the same reason Rust excludes it.
syntax.lua -- the `lean` -> `lean4` modeline alias (Q#LN2), so an Emacs
`-*- mode: lean -*-` or a Vim `ft=lean` line is not stranded by the entry
being named `lean4`.
syntax.rs -- the `lean` -> `lean4` injection alias (Q#LN17), so both
```lean and ```lean4 fences highlight. The Lean 3 spelling is mapped
forward deliberately: a ```lean fence is overwhelmingly Lean 4 in
practice.
highlight.rs -- `warning` moves from bold red to bold BRIGHT red. Writing
the test found the collision: `number` is plain `fg(1)`, so `sorry` and
the literal `42` beside it were the same colour, differing only in the
bold flag. `sorry` means "admitted, not proved" and is the one token in a
proof file a reader must never skim past, so it now gets the loudest
entry in the table and the test asserts the full style rather than the
colour.
Twelve criteria, seventeen tests. Notes on the ones that could have been
vacuous:
* acc4 uses a `.txt` fixture, not `.lean` -- on a `.lean` path the
extension alone yields `lean4` and the assertion would pass with the
alias table empty. acc4b removes the alias and pins that the raw name
survives, so acc4 cannot silently stop testing anything.
* acc11 goes through the real `_parse_now` injection path and asserts a
`lean4` CHILD LAYER appears. `pmacs.parse.injection_aliases` is a
documented write-only proxy, so an alias-table read would have proven
nothing about the parser; acc11b pins that a misspelled fence still
resolves to nothing.
* acc12 asserts through the process supervisor and the server list that
opening a Lean buffer spawns nothing. This is not decorative: the
machine this arc was scouted on has elan installed with no default
toolchain, where `lake --version` itself fails, and Stage 1 must be
unaffected by that.
Gates: fmt and strict workspace clippy clean; 1,826 default + 2,003 CRDT
library tests; lean4 Stage 1 9, comment toggle 14, auto-pair 45,
injection 4; M4 121; required GPU 152; isolated-config workspace sweep
3,150 across 90 suites; `git diff --check` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parse bounded Emacs and Vim modelines, normalize common aliases, and give
explicit file metadata precedence over inferred language. Pin one fresh-load
language decision for syntax, LSP, pairing, comments, and initial major mode,
while preserving the LSP path guard and explicit mode overrides.
Cover supported forms, rejection boundaries, precedence, unknown modes,
shebang and modeline pinning, reopen behavior, and pathless buffers.
Store a detected major mode on each buffer and expose it through Lua.
Resolve mode-scoped bindings in dispatch, describe-key, and help links,
including exact encoded mode context after entering the help buffer.
Initialize modes once at buffer load, preserve explicit overrides and
clears across switches, and publish the mode through a per-window
statusline provider. Add daemon acceptance for the complete mode lifecycle.
Four review findings + a cleanup bundle.
[P1] Wire flattener was O(spans²) and ran over the WHOLE buffer (the
file-style summary uses a whole-buffer viewport, not the visible one).
Replaced the per-interval full scan with an ordered active-set event
sweep (activate on start, expire on end, fold the active set) — linear
in practice. Added full_buffer_summary_scales_on_large_grammar_file
(1500-line rust) as the perf gate.
[P2] _parse_now used the empty alias map from make_request while
_dispatch snapshotted the registry map, so a `py` fence injected async
but not sync. Snapshot aliases on both paths; pinned by
sync_parse_now_resolves_alias.
[P2] The multi-range inline test used a one-line paragraph, whose block
inline node has no named children (link/emphasis are child-grammar
structures) — one range, so it couldn't falsify multi-range. Replaced
with a multi-line blockquote whose inline node carries a named
block_continuation: content_node_ranges now asserts >1 collected range
and emphasis parses on both lines.
[P2] The layer backstop dropped regions silently; the framing requires
a surfaced warning. run_parse now sets ParseTreeBundle::injection_capped;
syntax.lua's settle tick raises it once per buffer via pmacs.error
(_injection_capped). Added injection_layer_cap_surfaces_and_preserves_root
(drives >4096 fences, asserts the flag + bounded count + intact root).
Cleanup:
- The GPU acceptance test now drives the real StyleSpans full-frame
transform (spans_from_segments, extracted from replace_style_spans)
instead of a hand-rolled sort.
- content_node_ranges excludes NAMED children (documented as a round-1
refinement); framing mechanic #3 / Q#IJ5 updated to match.
- parse_duration doc now says root parse; the markdown entry no longer
describes inline as unhighlighted/future.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
Teach the syntax engine that one buffer can hold more than one
language. After the root parse, run the grammar's injections.scm, parse
each embedded region with the injected language, and merge every
layer's highlight spans. First consumer: markdown fenced code + inline
(zero new grammars — the block grammar already ships an injection query
and the injected langs already have grammars from #118).
Engine (src/syntax.rs):
- ParseTreeBundle now holds Vec<Layer> (root layer 0 + injected
children, depth-ascending); installed atomically so the existing
Arc::ptr_eq style gate and highlight cache keep working (Q#IJ1).
- run_parse builds layers on the worker: run injections.scm, resolve
the injected language, compute Vec<Range> (exclude NAMED children,
intersect the parent's ranges), set_included_ranges cold-parse,
recurse — bounded by depth (3), a layer backstop (4096), and a
(lang,ranges) visited guard; any child failure drops that child only
(Q#IJ3/IJ5). LanguageEntry gains injections_query; markdown_inline is
registered (retires the M9.7 block-only floor); markdown/rust carry
injection queries.
- Injected languages resolve off the static BUILTIN_LANGUAGES table
(Send loaders + query sources), preserving lazy loading. Dynamic
fence names go through a case-folded alias map seeded with defaults
and Lua-extensible via pmacs.parse.injection_aliases, snapshotted into
ParseRequest at dispatch so the worker never touches the Rc registry
or a Lua table (Q#IJ2/IJ4). Highlight queries are resolved at settle
(resolve_layer_queries), keeping query compilation main-thread/cached.
Producers:
- SyntaxHighlightView (grid) iterates layers shallow-to-deep so a
deeper layer's styling wins within its region (Q#IJ6/IJ7).
- scoped_style_spans (wire) flattens all layers into DISJOINT effective
spans via a boundary sweep, since the GPU re-sorts spans by start
(replace_style_spans / merge_style_spans) and would otherwise destroy
producer order. The GPU source_color_at consumer is fixed to fold all
covering spans (matching semantic_client's effective_style_at) rather
than returning the first.
Named-children exclusion: content ranges exclude only NAMED children
(matching tree-sitter-md's own inline splitter) — excluding a block
inline node's anonymous text tokens would shred the paragraph into
unparseable fragments.
13 acceptance gates (framing docs/multi-language-injections-framing.md):
layer structure, absolute child offsets, alias resolution (static +
case-folded dynamic + unknown-skip + Lua-async override), multi-range
inline, recursion bounds, wire + grid + GPU producers, incremental edit
/ new fence, many-paragraph settle budget with tail coverage, and the
single-layer regression guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
[P2] The shared JavaScript highlights query guards its builtin captures
(console, require, …) with `#is-not? local`, a PROPERTY predicate
(`Query::property_predicates`) that needs a scope map from the grammar's
LOCALS_QUERY — which pmacs does not run. `compute_highlight_spans` took
every capture, so a locally-shadowed `console`/`require` still surfaced
as `@variable.builtin`/`@function.builtin`; a theme distinguishing
`.builtin` would mis-style the shadowed local.
Full locals processing is substrate work; conservatively fail-closed
instead: drop captures whose pattern carries an `#is?`/`#is-not? local`
property predicate (the identifier falls back to its non-builtin
capture). The text predicates (`#eq?`/`#match?`/`#any-of?`, already
applied by the capture iterator) and `#set!` settings are untouched.
This is a general engine fix — it corrects the same latent mis-styling
for any grammar using the locals predicate, not just JS/TS.
- javascript_shadowed_builtin_is_not_mislabeled: a local `const console`
produces no `*.builtin` capture (directly observed to fail — two
`variable.builtin` captures — before the fix).
[P3] Comments this PR invalidated: `lsp.lua` no longer claims Python has
no grammar; `syntax.lua`'s `_has_language` gate comment uses a
still-grammarless example (an init.lua `shebangs.ruby`) instead of
python/javascript; and the rewritten Ruby shebang test's doc no longer
describes it as a Python test.
Gates: fmt; clippy -D warnings; test --lib; --features crdt;
m4_acceptance --skip basedpyright; GPU; full workspace sweep;
git diff --check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
Files identified by their whole basename — Dockerfile, Makefile,
CMakeLists.txt, rc dotfiles — had no detection path (extension-, then
shebang-keyed). Add a filename layer and the three grammars behind it.
- **Grammars** (BUILTIN_LANGUAGES): dockerfile via `tree-sitter-containerfile`
(the ABI-current grammar; the old `tree-sitter-dockerfile` pins
`tree-sitter ^0.20` and would fork the graph — containerfile rides
`tree-sitter-language 0.1`, tree-sitter dev-only, like the others),
make via `tree-sitter-make`, cmake via `tree-sitter-cmake`. All ship
self-contained highlights (single fragment). Extensions:
`.dockerfile`/`.containerfile`, `.mk`/`.make`, `.cmake`.
- **Filename layer**: `pmacs.parse.language_from_filename(name)` backed by
an extensible `pmacs.parse.filenames` map, wired into the precedence
chain in both syntax.lua (grammar) and lsp.lua (LSP): grammar-ext →
filetype map → filename → shebang. A recognized extension still wins;
the basename map only fires when the extension misses. Seeds the three
filenames plus shell rc dotfiles (`.bashrc`/`.zshrc`/`PKGBUILD`/… →
bash) — highlighting them against the grammar shipped in #115.
- **LSP**: `config.dockerfile` (docker-langserver --stdio) and
`config.cmake` (cmake-language-server). Make has no server, so no
`config.make` — grammar highlight only. Extension filetype fallbacks
added for id stability.
Bite-verified acceptance:
- filename_grammars_load_and_parse — each grammar's ABI is accepted by
the tree-sitter 0.26 core and parses a representative snippet without
error (dockerfile/cmake root at source_file, make at makefile).
- builtin_languages_include_dockerfile_make_cmake /
language_for_path_resolves_dockerfile_make_cmake_extensions — entry
presence and extension detection.
- m4_filename_map_resolves_special_files — the basename map (incl. path
form and dotfiles→bash), config.dockerfile/cmake commands, and no
config.make. Bite-verified against pre-feature syntax.lua.
- m4_filename_extensionless_dockerfile_highlights — an extensionless
`Dockerfile` resolves to dockerfile for LSP and gets a dockerfile parse
tree; reachable only via the filename map. Bite-verified.
Gates: fmt; clippy -D warnings; test --lib; --features crdt;
m4_acceptance --skip basedpyright; GPU; full workspace sweep;
git diff --check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
The attached split-string payload (`-Spython3`, `--split-string=...`) can
itself begin with env options or VAR=value assignments before the
interpreter: `-S-i python3`, `-SFOO=bar python3`,
`--split-string=-u FOO python3`. Rather than taking the payload's first
word as the interpreter, re-inject the attached payload into the token
stream so it flows through the same option / operand / assignment state
machine as a separated payload. Adds the three cases as resolver tests.
Two follow-ups from review, both in builtin/runtime/syntax.lua.
1. [P2] Buffer switching bypassed the pinned grammar. after-edit already
reparsed the pinned language, but the after-switch reattach path
(attach_for_active_buffer) re-resolved from scratch — so open an
extensionless `#!/bin/sh` (bash), edit its shebang to lua, switch away
and back, and the grammar flipped to lua while the LSP side kept its
bash attachment (lsp.lua's after-switch reuses the existing record).
attach_for_active_buffer now reuses the language pinned at first attach
whenever a parse view already exists; only a first-seen buffer
resolves. A language change still needs a close/reopen, matching both
the after-edit behavior and how extensions work.
2. [P2] Attached `env -S`/`--split-string` forms failed. The walk skipped
the whole option token, but for split-string the interpreter rides
inside it: `-Spython3`, `-vSpython3` (after no-operand short flags
i/v/0), and `--split-string=python3` all resolved to nil (the last was
also eaten by the earlier `=` branch). The env walk now extracts the
interpreter from the attached value (`^-[iv0]*S(.+)$` /
`^--split-string=(.+)$`); the separated forms (`-S python3`) still work
by walking on to the next token.
Tests (bite-verified against the round-1 syntax.lua — both fail there;
scripts/bite HEAD builtin/runtime/syntax.lua):
- m4_shebang_edit_keeps_pinned_grammar now adds a switch-away/back cycle
(via pmacs.window.switch_buffer, which fires after-switch
synchronously) and asserts the tree stays bash.
- m4_shebang_resolver_maps_interpreters adds the attached split-string
cases (`-Spython3`, `--split-string=python3`, `-vSpython3`).
Gates: fmt; clippy -D warnings; m4_acceptance --skip basedpyright; GPU;
git diff --check green. Only-known-flake caveat as round 1
(editor::composition_overhead_under_ten_percent perf microbenchmark,
unrelated to this Lua change). Change is Lua-only plus the acceptance
tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
Three review findings, all in builtin/runtime/syntax.lua.
1. [P1] Syntax bypassed extension precedence and grammar availability.
attach_for_active_buffer resolved `language_for_path or shebang`, but
language_for_path knows only grammar-backed extensions — so a `.py`
file opening with `#!/bin/sh` fell through to the shebang and got a
bash parse tree, and an extensionless `#!/usr/bin/env python3` script
dispatched "python" (no grammar) and raised "unknown language". A new
resolve_active_language walks the full precedence chain — grammar
extension -> LSP filetype map -> shebang — consulting the shebang only
when the extension is unrecognized (a recognized non-grammar extension
like .py is authoritative). Dispatch is then gated on
pmacs.parse._has_language(lang), so grammarless languages are skipped
silently. The extension parts stay keyed on buf:name() (unchanged from
before), so path-less buffers that resolve a grammar by name — e.g.
generated markdown buffers — are unaffected.
2. [P2] Editing an open script's shebang left parsing/highlighting stale.
The after-edit path re-sniffed the mutable shebang: sh -> python
raised "unknown language" while leaving the old bash tree, and
sh -> lua swapped the parse tree under a highlight overlay still
holding the original grammar's query. Reparse now uses the language
pinned at first attach (parse_lang_by_buffer), never re-resolving —
a language change needs a close/reopen, as it does for extensions.
3. [P2] `env` options with operands were mistaken for interpreters.
`#!/usr/bin/env -u FOO python3` skipped `-u` but took `FOO`. The env
walk now skips the operand of the operand-consuming GNU-env options
(-u/--unset, -C/--chdir, -a/--argv0) before selecting the interpreter.
-S/--split-string stays excluded (its string carries the interpreter).
Tests (bite-verified against pre-fix syntax.lua — each fails without its
fix; scripts/bite HEAD builtin/runtime/syntax.lua):
- m4_shebang_does_not_override_extension now also asserts _has_view is
false (no bash grammar tree for a `.py` + `#!/bin/sh`), not only the
LSP language.
- m4_shebang_extensionless_grammarless_language_is_silent — extensionless
python resolves for LSP, gets no grammar view, and records no error.
- m4_shebang_edit_keeps_pinned_grammar — rewriting a `#!/bin/sh` script's
shebang to lua keeps the bash tree and reports no error.
- m4_shebang_resolver_maps_interpreters — added the env-operand cases
(`-u FOO`, `-C /tmp`, combined).
Gates: fmt; clippy -D warnings; m4_acceptance --skip basedpyright; GPU;
git diff --check all green. The only sweep failure is the pre-existing
editor::composition_overhead_under_ten_percent render microbenchmark
(ratio hovers at the 1.10 cutoff; flakes ~1/3 even isolated single-
threaded, already asserted-off on macOS) — a pure-Rust render loop this
Lua-only change cannot touch. Change is Lua-only plus the acceptance
tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
Extension detection missed extensionless scripts — `scripts/deploy`, git
hooks, `configure`, and `scripts/bite` itself — so they got neither
highlighting nor an LSP server. Add a first-line shebang fallback.
- New `pmacs.parse.language_from_shebang(buf)` (builtin/runtime/syntax.lua):
sniffs the first line (capped at 256 bytes), maps the interpreter's
basename to a language, and resolves the `#!/usr/bin/env python3`
indirection (skipping env's own `-S`/flags and `VAR=val` assignments).
Backed by `pmacs.parse.shebangs`, a user-extensible map seeded with the
interpreters pmacs can act on: sh-family -> bash, python* -> python,
node -> javascript, lua* -> lua.
- Wired as a strict *fallback* on both resolution paths: syntax.lua's
grammar attach (`language_for_path or language_from_shebang`) and
lsp.lua's `buffer_language` (grammar -> filetypes -> shebang). A
recognized extension always wins, so a `.py`/`.sh` file is never
re-classified by a stray shebang.
- Cross-language, not shell-only: `#!/usr/bin/env python` /`node` /`lua`
resolve too. Special filenames (`.bashrc`, `Dockerfile`, `Makefile`)
are intentionally deferred until there are grammars behind them.
Bite-verified acceptance (tests/m4_acceptance.rs):
- m4_shebang_resolver_maps_interpreters — the mapping incl. env
indirection and `env -S`; non-shebangs and unmapped interpreters
(ruby) resolve to nil.
- m4_shebang_extensionless_script_resolves_bash — opening an
extensionless `#!/bin/sh` script resolves to bash on BOTH paths:
lsp.lua's `active_buffer_language()` and a settled bash parse tree
(grammar attach). Reachable only via the shebang, since the file has
no extension.
- m4_shebang_does_not_override_extension — a `.py` file opening with
`#!/bin/sh` still resolves to python (extension precedence).
Gates green: fmt; clippy -D warnings; test --lib; --features crdt;
m4_acceptance --skip basedpyright; GPU; full workspace sweep;
git diff --check. Change is Lua-only plus the acceptance tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan
Two PR #94 validation findings.
1. (High, blocking) GPU stuck after leaving a panel: the GPU only
swaps its displayed buffer on BufferSnapshot, and the daemon only
sent one on the first CRDT upgrade (F29's ensure returns None for
an already-backed buffer). A panel's q / RET switched the daemon's
active buffer back to the already-known source and sent nothing --
the GPU kept rendering the panel while input targeted the source: a
typing-into-a-buffer-you-can't-see hazard. Fix: the per-tick loop
now FOLLOWS each replica frontend's own active buffer -- when it
differs from the last snapshot sent to that frontend, ship that
buffer's snapshot to that frontend only (the F29 broadcast records
itself so the upgrade tick doesn't double-send). First-tick send
also repairs the attach-time last-snapshot-wins ambiguity. Snapshot
export extracted and shared with the F29 broadcast; per-fid state
cleaned on both detach paths.
2. (High, wider than reported) 'LSP doesn't activate on navigate':
switch_active_buffer clears the window's overlays, and the runtime
dedup tables (highlighted_buffers, styled_buffers,
diag_viewed_buffers) blocked re-attachment -- so EVERY buffer
switch (plain C-x b included, long-latent) permanently stripped
syntax color, LSP semantic style, and diagnostic underlines;
verified: overlay kinds [syntax-highlight, lsp-style, diagnostic]
-> [] after one away-and-back. Fix: a new additive
buffer.after-switch hook, fired by the window.switch_buffer binding
and find_or_open's existing-buffer branch; syntax.lua and lsp.lua
subscribe and re-push their views (the just-cleared window makes
that exactly-once per switch; fresh loads keep firing after-load).
Regression: tests/overlay_reattach_acceptance.rs (double round-trip
counts exactly one highlight overlay; panel q restores styling).
The daemon follow path is validated live (daemon + GPU) -- its unit
seam is the shared export helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Levi Neuwirth