diff --git a/docs/modeline-detection-framing.md b/docs/modeline-detection-framing.md new file mode 100644 index 0000000..70d120c --- /dev/null +++ b/docs/modeline-detection-framing.md @@ -0,0 +1,355 @@ +# Modeline language detection — side quest + +**Status:** Draft for user review, 2026-07-22. No implementation yet. +**Base:** `githubsucks/main` at `d5d9b9c`; protocol v18. + +## Problem + +Language inference currently uses four signals, in order: + +1. bundled grammar extension; +2. `pmacs.lsp.filetypes` extension; +3. exact basename; +4. shebang. + +That chain cannot classify a deliberately misleading or extensionless file when +its author supplied editor metadata such as `-*- mode: python -*-` or +`vim: set ft=python:`. Mode-system wiring #129 gives the result a persistent +per-buffer home and makes it observable through key dispatch, help, and the +statusline, but no modeline parser exists. + +The goal is one bounded, non-executing modeline detector whose result drives the +same initial language used by syntax, LSP, and `Buffer.major_mode`. This is a +language-detection feature, not a general file-local settings system. + +## Ground truth + +### Detection is duplicated today + +`builtin/runtime/syntax.lua::resolve_active_language` and +`builtin/runtime/lsp.lua::buffer_language` independently implement the same +extension → filetype → filename → shebang chain. They already have small but +important differences: + +- syntax uses `buf:name()`, preserving historical grammar-by-name behavior for + pathless buffers; +- LSP requires `buf:path()`, because it cannot construct a URI or project root + for a pathless buffer; +- syntax pins the grammar in `parse_lang_by_buffer` after dispatch, while the + public LSP language query re-sniffs mutable shebang text on every call. + +Adding the fifth signal to both copies would create a third convention and let +syntax, LSP, mode initialization, auto-pairing, and comment commands disagree. +The implementation must consolidate the actual inference in `syntax.lua` and +leave only the LSP path-eligibility guard in `lsp.lua`. + +### Hook order is already useful + +`src/editor.rs` loads `syntax.lua` before `lsp.lua`. Their `buffer.after-load` +callbacks therefore run in that registration order: + +1. syntax detects the language, initializes the major mode, and dispatches a + grammar when one exists; +2. LSP resolves the same buffer language and attaches a server when configured. + +The modeline result must be computed and pinned by step 1 so step 2 cannot +independently reinterpret the file. + +### Major mode and parser language have different later lifecycles + +At first load, the detected language is the initial major-mode name. Afterward: + +- explicit `pmacs.buffer.set_major_mode` changes dispatch/statusline only; +- syntax and LSP remain attached to their initially selected language; +- `buffer.after-switch` restores views without re-detecting; +- edits do not change the selected grammar. + +Modelines follow that same load-time contract. Editing a cookie is not a live +mode or parser switch. Close/reopen re-evaluates it. A future true reload that +fires `buffer.after-load` re-evaluates it as a fresh load; explicit mode +overrides and clears are not reload-persistent, matching the #129 framing. + +### Existing Lua reads are sufficient + +`BufferIdLua` exposes byte-length and byte-slice operations. Detection can read +bounded prefix/suffix windows without copying the whole buffer or adding a Rust +binding. The rope is byte-addressed, so a bounded slice does not need UTF-8 +boundary repair before Lua pattern matching. + +### Compatibility references + +The supported subset follows the documented, non-evaluating pieces of: + +- GNU Emacs, “Specifying File Variables”: + +- Vim, `:help modeline` and `:help 'modelines'`: + + +Compatibility is deliberately bounded below. pmacs does not become an Emacs +file-variable evaluator or a Vim `:set` interpreter. + +## Scope + +In scope: + +- Emacs `-*- mode: NAME -*-` and mode-only `-*- NAME -*-` cookies; +- Vim/Vi `ft=NAME` and `filetype=NAME` modelines; +- first/last-line scanning with fixed byte and line limits; +- canonical aliases for common external filetype names; +- modeline precedence over inferred path/shebang language; +- one shared, pinned per-buffer language decision; +- initial major mode, grammar, LSP, and language-aware Lua consumers agreeing; +- focused parser and end-to-end acceptance on both Lua backends. + +Out of scope: + +- Emacs `Local Variables:` tail blocks; +- variables other than Emacs `mode` or Vim `ft`/`filetype`; +- `eval`, Vim commands, option mutation, directory-local variables, or project + trust prompts; +- Vim `ex:` markers, version predicates, escaped option values, and combined + dotted filetypes; +- live re-detection after edits, saves, renames, or buffer switches; +- `buffer.after-mode-change`, minor modes, mode-scoped settings, `describe-mode`, + or session persistence of explicit mode overrides; +- a protocol change or frontend-specific work. + +## Decisions + +### Q#MD1 — Scan only bounded edge lines + +Read at most 8 KiB from each end of the buffer and inspect: + +- Emacs: line 1, or line 2 only when line 1 begins with `#!`; +- Vim/Vi: the first five and last five logical lines, matching Vim's default + `'modelines'=5` behavior. + +When the prefix and suffix overlap, deduplicate lines before parsing. Strip one +trailing `\r` so CRLF and LF behave identically. A candidate line truncated by +the 8 KiB boundary is not parsed. Detection therefore allocates at most 16 KiB +per fresh load, independent of file size, and an adversarial giant edge line +cannot force a whole-buffer copy. + +The Emacs 3000-character tail `Local Variables:` mechanism is a separate parser +with comment-prefix/suffix rules and is excluded. + +### Q#MD2 — Recognize a conservative syntax subset + +Emacs: + +- require a complete pair of `-*-` delimiters on the eligible line; +- accept `mode: NAME` in a semicolon-separated property list; +- accept a mode-only payload such as `-*- Lisp -*-`; +- ignore every property except `mode`; +- when a cookie contains multiple valid `mode` properties, the last wins. + +Vim/Vi: + +- accept `vim:`, `vi:`, and `Vim:` at line start or preceded by ASCII space or + tab; `Vim:` requires the `set` form, matching Vim; +- accept optional `set ` / `se ` and exact `ft=NAME`, `ft:NAME`, + `filetype=NAME`, or `filetype:NAME` assignments; +- ignore all other option tokens rather than interpreting them; +- require a terminating colon for the `set`/`se` form, so a comment suffix is + never consumed as an option value; +- reject `ex:`, Vim version predicates, and marker substrings embedded in a + word. + +Across all eligible lines, the last valid mode assignment in document order +wins. This matches Emacs's “final defined mode” behavior and ordinary sequential +option assignment. A footer Vim modeline can intentionally override a header +Emacs cookie; conflicting metadata does not depend on Lua table iteration. + +### Q#MD3 — Modeline names are data, never code + +Trim ASCII edge whitespace, ASCII-lowercase the name, require +`[a-z0-9][a-z0-9+_-]*`, and cap it at 128 bytes. Empty, non-ASCII, control, +whitespace-containing, or overlong values are ignored silently. + +The restriction is intentionally narrower than `pmacs.buffer.set_major_mode`, +which continues accepting arbitrary Lua strings for trusted configuration. +Untrusted file content receives no path to control characters, huge statusline +values, Lua evaluation, or option mutation. + +### Q#MD4 — Normalize common external names through one alias table + +Expose a user-extensible Lua table: + +```lua +pmacs.parse.modeline_aliases = { + ["c++"] = "cpp", + cxx = "cpp", + sh = "bash", + shell = "bash", + ["shell-script"] = "bash", + py = "python", + js = "javascript", + jsx = "javascriptreact", + ts = "typescript", + tsx = "typescriptreact", + yml = "yaml", + makefile = "make", +} +``` + +A normalized name absent from the table passes through unchanged. Users may +add, replace, or remove aliases in `init.lua`; defaults use `or`-style seeding +so preconfigured entries are not overwritten. Alias outputs must satisfy the +same 128-byte token rule before use. + +This keeps canonical pmacs names stable without pretending that extension maps +and editor-mode names are the same namespace. Do not strip a trailing `-mode`: +GNU Emacs explicitly specifies the value without that suffix, and silent +stripping would make custom names ambiguous. + +### Q#MD5 — Explicit modelines override inference + +The final fresh-load order is: + +1. modeline; +2. bundled grammar extension; +3. `pmacs.lsp.filetypes` extension; +4. exact basename; +5. shebang. + +A modeline is explicit file metadata; the remaining signals are inference. Thus +a `template.txt` containing `vim: set ft=python:` selects `python`, and a +misnamed `.py` file containing `-*- mode: lua -*-` selects `lua` consistently +for mode, parser, and LSP. + +This interprets the backlog's “fifth layer after extension → filetype → filename +→ shebang” as a layer added after that work, not as a lowest-priority fallback. +Making explicit metadata lose to a suffix would defeat the feature's primary +use case. + +### Q#MD6 — One resolver owns the effective language + +`syntax.lua` owns: + +- `pmacs.parse.language_from_modeline(buf)`: parse current content and return + the normalized modeline language or nil; +- the private fresh inference chain; +- `pmacs.parse.buffer_language(buf)`: return the language pinned for this + buffer's current load, resolving once only for an unseen buffer. + +The syntax `buffer.after-load` path forces a fresh inference, records either the +language or an explicit “resolved none” sentinel, then uses that same value for +mode initialization and grammar dispatch. `buffer.after-switch` consumes the +pin and resolves only the pre-existing hidden-buffer case that never received +an after-load event. + +`pmacs.lsp.buffer_language(buf)` keeps its current path requirement, then +delegates to `pmacs.parse.buffer_language(buf)`. The active-buffer wrapper stays +unchanged. This preserves pathless LSP behavior while deleting the duplicate +extension/filetype/filename/shebang chain. + +The pin also closes an existing shebang inconsistency: editing `#!/bin/sh` to +`#!/usr/bin/env lua` no longer leaves a bash parse tree while making later +auto-pair/comment queries report Lua. Raw parser tests may call +`language_from_modeline`; behavior-driving consumers use the pin. + +### Q#MD7 — Initial mode, syntax, and LSP share one value + +On `buffer.after-load`, set the buffer's major mode to the freshly resolved +language, including nil when no signal resolves. This replaces the current +“only if nil” guard and makes the already-documented reload contract exact: +a fresh after-load decision replaces an earlier explicit override or clear. + +Then: + +- dispatch a grammar only if `pmacs.parse._has_language(lang)`; +- let LSP attach only if the same language has a configured server and a real + file path; +- retain a valid unknown language as the major mode, while silently skipping + grammar/LSP attachment. + +After load, explicit `set_major_mode` remains independent: it immediately +changes mode key dispatch and statusline display but does not rewrite the +pinned parser/LSP language. Switches preserve both values. + +### Q#MD8 — Malformed or unsupported metadata is fail-closed and quiet + +A malformed marker, invalid name, unsupported form, truncated candidate, or +unknown language never raises from `buffer.after-load` and never emits an +`*errors*` entry. The detector returns nil and the existing inference chain +continues. + +A syntactically valid unknown name is different: it is a legitimate major mode +and is pinned, but `_has_language` and LSP config gates prevent a bogus parser +or server launch. This preserves #129's custom-mode capability without +executing file content. + +No enable/disable setting is added. The supported input can only select a +bounded string already consumed as passive mode/language identity; it cannot +run hooks or set options. If a future `buffer.after-mode-change` hook makes mode +selection executable, modeline trust must be revisited in that feature's +framing. + +### Q#MD9 — No Rust or protocol surface is required + +Expected implementation touch set: + +- `builtin/runtime/syntax.lua` — bounded parser, aliases, shared resolver, pin, + and after-load initialization; +- `builtin/runtime/lsp.lua` — delegate language inference while retaining the + path guard; +- `tests/m4_acceptance.rs` — parser, precedence, lifecycle, and end-to-end + regression coverage; +- this framing and the side-quest/handoff state documents when the feature + lands. + +No changes are expected in `src/buffer.rs`, Lua bindings, frontends, +`pmacs-protocol`, or protocol version 18. + +## Bets + +1. **Sixteen KiB of edge text is sufficient.** Real modelines are short; files + with an 8 KiB first/last candidate line are better treated as malformed than + copied wholesale during load. +2. **One canonical language should drive mode, parser, and LSP initially.** A + future distinction between editor mode and parser language needs a real + consumer and an explicit mapping contract, not accidental divergence. +3. **ASCII-lowercased identifiers cover interoperable modelines.** Trusted Lua + remains available for arbitrary UTF-8 custom mode names. +4. **Load-time detection is enough.** Live cookie edits would require orderly + parser teardown, LSP `didClose`/`didOpen`, overlay replacement, mode-change + notification, and failure rollback; that is not a one-shot detector. +5. **Pathless LSP buffers stay ineligible.** Syntax may still infer from a + buffer name, but spawning a server without a URI/project root remains wrong. + +## Acceptance + +All end-to-end fixtures clear `pmacs.lsp.config` unless the case intentionally +observes server selection, so opening a test file never starts a machine-local +language server. + +1. **Emacs forms:** first-line property and shorthand cookies resolve; a cookie + on line 2 resolves only after a shebang; unrelated properties are ignored; + the last `mode` property wins. +2. **Vim forms:** `vim:`/`vi:` direct and `set` forms resolve `ft` and + `filetype` in the first/last five lines; CRLF works; `Vim:` requires `set`. +3. **Boundary rejection:** sixth-line, sixth-from-end, middle-of-file, + word-embedded, unterminated, truncated, invalid-character, and overlong + candidates do not resolve and do not log errors. +4. **Conflict order:** overlapping edge windows are deduplicated and the last + valid assignment in document order wins deterministically. +5. **Alias behavior:** seeded aliases map `sh → bash`, `c++ → cpp`, and + `tsx → typescriptreact`; a user override wins; an invalid alias output is + ignored. +6. **Explicit precedence:** a `.py` file with a Lua modeline yields `lua` from + `pmacs.parse.buffer_language`, `pmacs.lsp.active_buffer_language`, the parse + tree, and `pmacs.buffer.major_mode`. +7. **Unknown valid mode:** a `.txt` file with `mode: prose` receives major mode + `prose`, creates no parse view, starts no LSP server, and produces no error. +8. **No-modeline regression:** extension, filetype, filename, and shebang cases + retain their present precedence and outputs. +9. **Pinned lifecycle:** changing a loaded modeline does not change the pinned + language, parser, or major mode; switch-away/back remains stable; close and + reopen re-evaluates the changed on-disk cookie. +10. **Explicit override independence:** `set_major_mode` after load changes + dispatch/statusline but not `pmacs.parse.buffer_language`; switches preserve + the override. +11. **Pathless preservation:** syntax-by-buffer-name behavior remains, while + `pmacs.lsp.buffer_language` still returns nil without a backing path. +12. **Backend parity:** focused acceptance passes with default LuaJIT and + `--no-default-features --features lua54`; protocol remains v18.