docs: frame modeline language detection
Define bounded Emacs and Vim modeline parsing, alias normalization, explicit precedence, one pinned language decision shared by syntax and LSP, and the acceptance contract for load-time mode detection.
This commit is contained in:
parent
d5d9b9c1ca
commit
47a7f4cefd
|
|
@ -0,0 +1,355 @@
|
|||
# Modeline language detection — side quest
|
||||
|
||||
**Status:** Draft for user review, 2026-07-22. No implementation yet.
|
||||
**Base:** `githubsucks/main` at `d5d9b9c`; protocol v18.
|
||||
|
||||
## Problem
|
||||
|
||||
Language inference currently uses four signals, in order:
|
||||
|
||||
1. bundled grammar extension;
|
||||
2. `pmacs.lsp.filetypes` extension;
|
||||
3. exact basename;
|
||||
4. shebang.
|
||||
|
||||
That chain cannot classify a deliberately misleading or extensionless file when
|
||||
its author supplied editor metadata such as `-*- mode: python -*-` or
|
||||
`vim: set ft=python:`. Mode-system wiring #129 gives the result a persistent
|
||||
per-buffer home and makes it observable through key dispatch, help, and the
|
||||
statusline, but no modeline parser exists.
|
||||
|
||||
The goal is one bounded, non-executing modeline detector whose result drives the
|
||||
same initial language used by syntax, LSP, and `Buffer.major_mode`. This is a
|
||||
language-detection feature, not a general file-local settings system.
|
||||
|
||||
## Ground truth
|
||||
|
||||
### Detection is duplicated today
|
||||
|
||||
`builtin/runtime/syntax.lua::resolve_active_language` and
|
||||
`builtin/runtime/lsp.lua::buffer_language` independently implement the same
|
||||
extension → filetype → filename → shebang chain. They already have small but
|
||||
important differences:
|
||||
|
||||
- syntax uses `buf:name()`, preserving historical grammar-by-name behavior for
|
||||
pathless buffers;
|
||||
- LSP requires `buf:path()`, because it cannot construct a URI or project root
|
||||
for a pathless buffer;
|
||||
- syntax pins the grammar in `parse_lang_by_buffer` after dispatch, while the
|
||||
public LSP language query re-sniffs mutable shebang text on every call.
|
||||
|
||||
Adding the fifth signal to both copies would create a third convention and let
|
||||
syntax, LSP, mode initialization, auto-pairing, and comment commands disagree.
|
||||
The implementation must consolidate the actual inference in `syntax.lua` and
|
||||
leave only the LSP path-eligibility guard in `lsp.lua`.
|
||||
|
||||
### Hook order is already useful
|
||||
|
||||
`src/editor.rs` loads `syntax.lua` before `lsp.lua`. Their `buffer.after-load`
|
||||
callbacks therefore run in that registration order:
|
||||
|
||||
1. syntax detects the language, initializes the major mode, and dispatches a
|
||||
grammar when one exists;
|
||||
2. LSP resolves the same buffer language and attaches a server when configured.
|
||||
|
||||
The modeline result must be computed and pinned by step 1 so step 2 cannot
|
||||
independently reinterpret the file.
|
||||
|
||||
### Major mode and parser language have different later lifecycles
|
||||
|
||||
At first load, the detected language is the initial major-mode name. Afterward:
|
||||
|
||||
- explicit `pmacs.buffer.set_major_mode` changes dispatch/statusline only;
|
||||
- syntax and LSP remain attached to their initially selected language;
|
||||
- `buffer.after-switch` restores views without re-detecting;
|
||||
- edits do not change the selected grammar.
|
||||
|
||||
Modelines follow that same load-time contract. Editing a cookie is not a live
|
||||
mode or parser switch. Close/reopen re-evaluates it. A future true reload that
|
||||
fires `buffer.after-load` re-evaluates it as a fresh load; explicit mode
|
||||
overrides and clears are not reload-persistent, matching the #129 framing.
|
||||
|
||||
### Existing Lua reads are sufficient
|
||||
|
||||
`BufferIdLua` exposes byte-length and byte-slice operations. Detection can read
|
||||
bounded prefix/suffix windows without copying the whole buffer or adding a Rust
|
||||
binding. The rope is byte-addressed, so a bounded slice does not need UTF-8
|
||||
boundary repair before Lua pattern matching.
|
||||
|
||||
### Compatibility references
|
||||
|
||||
The supported subset follows the documented, non-evaluating pieces of:
|
||||
|
||||
- GNU Emacs, “Specifying File Variables”:
|
||||
<https://www.gnu.org/software/emacs/manual/html_node/emacs/Specifying-File-Variables.html>
|
||||
- Vim, `:help modeline` and `:help 'modelines'`:
|
||||
<https://vimhelp.org/options.txt.html#modeline>
|
||||
|
||||
Compatibility is deliberately bounded below. pmacs does not become an Emacs
|
||||
file-variable evaluator or a Vim `:set` interpreter.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- Emacs `-*- mode: NAME -*-` and mode-only `-*- NAME -*-` cookies;
|
||||
- Vim/Vi `ft=NAME` and `filetype=NAME` modelines;
|
||||
- first/last-line scanning with fixed byte and line limits;
|
||||
- canonical aliases for common external filetype names;
|
||||
- modeline precedence over inferred path/shebang language;
|
||||
- one shared, pinned per-buffer language decision;
|
||||
- initial major mode, grammar, LSP, and language-aware Lua consumers agreeing;
|
||||
- focused parser and end-to-end acceptance on both Lua backends.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Emacs `Local Variables:` tail blocks;
|
||||
- variables other than Emacs `mode` or Vim `ft`/`filetype`;
|
||||
- `eval`, Vim commands, option mutation, directory-local variables, or project
|
||||
trust prompts;
|
||||
- Vim `ex:` markers, version predicates, escaped option values, and combined
|
||||
dotted filetypes;
|
||||
- live re-detection after edits, saves, renames, or buffer switches;
|
||||
- `buffer.after-mode-change`, minor modes, mode-scoped settings, `describe-mode`,
|
||||
or session persistence of explicit mode overrides;
|
||||
- a protocol change or frontend-specific work.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Q#MD1 — Scan only bounded edge lines
|
||||
|
||||
Read at most 8 KiB from each end of the buffer and inspect:
|
||||
|
||||
- Emacs: line 1, or line 2 only when line 1 begins with `#!`;
|
||||
- Vim/Vi: the first five and last five logical lines, matching Vim's default
|
||||
`'modelines'=5` behavior.
|
||||
|
||||
When the prefix and suffix overlap, deduplicate lines before parsing. Strip one
|
||||
trailing `\r` so CRLF and LF behave identically. A candidate line truncated by
|
||||
the 8 KiB boundary is not parsed. Detection therefore allocates at most 16 KiB
|
||||
per fresh load, independent of file size, and an adversarial giant edge line
|
||||
cannot force a whole-buffer copy.
|
||||
|
||||
The Emacs 3000-character tail `Local Variables:` mechanism is a separate parser
|
||||
with comment-prefix/suffix rules and is excluded.
|
||||
|
||||
### Q#MD2 — Recognize a conservative syntax subset
|
||||
|
||||
Emacs:
|
||||
|
||||
- require a complete pair of `-*-` delimiters on the eligible line;
|
||||
- accept `mode: NAME` in a semicolon-separated property list;
|
||||
- accept a mode-only payload such as `-*- Lisp -*-`;
|
||||
- ignore every property except `mode`;
|
||||
- when a cookie contains multiple valid `mode` properties, the last wins.
|
||||
|
||||
Vim/Vi:
|
||||
|
||||
- accept `vim:`, `vi:`, and `Vim:` at line start or preceded by ASCII space or
|
||||
tab; `Vim:` requires the `set` form, matching Vim;
|
||||
- accept optional `set ` / `se ` and exact `ft=NAME`, `ft:NAME`,
|
||||
`filetype=NAME`, or `filetype:NAME` assignments;
|
||||
- ignore all other option tokens rather than interpreting them;
|
||||
- require a terminating colon for the `set`/`se` form, so a comment suffix is
|
||||
never consumed as an option value;
|
||||
- reject `ex:`, Vim version predicates, and marker substrings embedded in a
|
||||
word.
|
||||
|
||||
Across all eligible lines, the last valid mode assignment in document order
|
||||
wins. This matches Emacs's “final defined mode” behavior and ordinary sequential
|
||||
option assignment. A footer Vim modeline can intentionally override a header
|
||||
Emacs cookie; conflicting metadata does not depend on Lua table iteration.
|
||||
|
||||
### Q#MD3 — Modeline names are data, never code
|
||||
|
||||
Trim ASCII edge whitespace, ASCII-lowercase the name, require
|
||||
`[a-z0-9][a-z0-9+_-]*`, and cap it at 128 bytes. Empty, non-ASCII, control,
|
||||
whitespace-containing, or overlong values are ignored silently.
|
||||
|
||||
The restriction is intentionally narrower than `pmacs.buffer.set_major_mode`,
|
||||
which continues accepting arbitrary Lua strings for trusted configuration.
|
||||
Untrusted file content receives no path to control characters, huge statusline
|
||||
values, Lua evaluation, or option mutation.
|
||||
|
||||
### Q#MD4 — Normalize common external names through one alias table
|
||||
|
||||
Expose a user-extensible Lua table:
|
||||
|
||||
```lua
|
||||
pmacs.parse.modeline_aliases = {
|
||||
["c++"] = "cpp",
|
||||
cxx = "cpp",
|
||||
sh = "bash",
|
||||
shell = "bash",
|
||||
["shell-script"] = "bash",
|
||||
py = "python",
|
||||
js = "javascript",
|
||||
jsx = "javascriptreact",
|
||||
ts = "typescript",
|
||||
tsx = "typescriptreact",
|
||||
yml = "yaml",
|
||||
makefile = "make",
|
||||
}
|
||||
```
|
||||
|
||||
A normalized name absent from the table passes through unchanged. Users may
|
||||
add, replace, or remove aliases in `init.lua`; defaults use `or`-style seeding
|
||||
so preconfigured entries are not overwritten. Alias outputs must satisfy the
|
||||
same 128-byte token rule before use.
|
||||
|
||||
This keeps canonical pmacs names stable without pretending that extension maps
|
||||
and editor-mode names are the same namespace. Do not strip a trailing `-mode`:
|
||||
GNU Emacs explicitly specifies the value without that suffix, and silent
|
||||
stripping would make custom names ambiguous.
|
||||
|
||||
### Q#MD5 — Explicit modelines override inference
|
||||
|
||||
The final fresh-load order is:
|
||||
|
||||
1. modeline;
|
||||
2. bundled grammar extension;
|
||||
3. `pmacs.lsp.filetypes` extension;
|
||||
4. exact basename;
|
||||
5. shebang.
|
||||
|
||||
A modeline is explicit file metadata; the remaining signals are inference. Thus
|
||||
a `template.txt` containing `vim: set ft=python:` selects `python`, and a
|
||||
misnamed `.py` file containing `-*- mode: lua -*-` selects `lua` consistently
|
||||
for mode, parser, and LSP.
|
||||
|
||||
This interprets the backlog's “fifth layer after extension → filetype → filename
|
||||
→ shebang” as a layer added after that work, not as a lowest-priority fallback.
|
||||
Making explicit metadata lose to a suffix would defeat the feature's primary
|
||||
use case.
|
||||
|
||||
### Q#MD6 — One resolver owns the effective language
|
||||
|
||||
`syntax.lua` owns:
|
||||
|
||||
- `pmacs.parse.language_from_modeline(buf)`: parse current content and return
|
||||
the normalized modeline language or nil;
|
||||
- the private fresh inference chain;
|
||||
- `pmacs.parse.buffer_language(buf)`: return the language pinned for this
|
||||
buffer's current load, resolving once only for an unseen buffer.
|
||||
|
||||
The syntax `buffer.after-load` path forces a fresh inference, records either the
|
||||
language or an explicit “resolved none” sentinel, then uses that same value for
|
||||
mode initialization and grammar dispatch. `buffer.after-switch` consumes the
|
||||
pin and resolves only the pre-existing hidden-buffer case that never received
|
||||
an after-load event.
|
||||
|
||||
`pmacs.lsp.buffer_language(buf)` keeps its current path requirement, then
|
||||
delegates to `pmacs.parse.buffer_language(buf)`. The active-buffer wrapper stays
|
||||
unchanged. This preserves pathless LSP behavior while deleting the duplicate
|
||||
extension/filetype/filename/shebang chain.
|
||||
|
||||
The pin also closes an existing shebang inconsistency: editing `#!/bin/sh` to
|
||||
`#!/usr/bin/env lua` no longer leaves a bash parse tree while making later
|
||||
auto-pair/comment queries report Lua. Raw parser tests may call
|
||||
`language_from_modeline`; behavior-driving consumers use the pin.
|
||||
|
||||
### Q#MD7 — Initial mode, syntax, and LSP share one value
|
||||
|
||||
On `buffer.after-load`, set the buffer's major mode to the freshly resolved
|
||||
language, including nil when no signal resolves. This replaces the current
|
||||
“only if nil” guard and makes the already-documented reload contract exact:
|
||||
a fresh after-load decision replaces an earlier explicit override or clear.
|
||||
|
||||
Then:
|
||||
|
||||
- dispatch a grammar only if `pmacs.parse._has_language(lang)`;
|
||||
- let LSP attach only if the same language has a configured server and a real
|
||||
file path;
|
||||
- retain a valid unknown language as the major mode, while silently skipping
|
||||
grammar/LSP attachment.
|
||||
|
||||
After load, explicit `set_major_mode` remains independent: it immediately
|
||||
changes mode key dispatch and statusline display but does not rewrite the
|
||||
pinned parser/LSP language. Switches preserve both values.
|
||||
|
||||
### Q#MD8 — Malformed or unsupported metadata is fail-closed and quiet
|
||||
|
||||
A malformed marker, invalid name, unsupported form, truncated candidate, or
|
||||
unknown language never raises from `buffer.after-load` and never emits an
|
||||
`*errors*` entry. The detector returns nil and the existing inference chain
|
||||
continues.
|
||||
|
||||
A syntactically valid unknown name is different: it is a legitimate major mode
|
||||
and is pinned, but `_has_language` and LSP config gates prevent a bogus parser
|
||||
or server launch. This preserves #129's custom-mode capability without
|
||||
executing file content.
|
||||
|
||||
No enable/disable setting is added. The supported input can only select a
|
||||
bounded string already consumed as passive mode/language identity; it cannot
|
||||
run hooks or set options. If a future `buffer.after-mode-change` hook makes mode
|
||||
selection executable, modeline trust must be revisited in that feature's
|
||||
framing.
|
||||
|
||||
### Q#MD9 — No Rust or protocol surface is required
|
||||
|
||||
Expected implementation touch set:
|
||||
|
||||
- `builtin/runtime/syntax.lua` — bounded parser, aliases, shared resolver, pin,
|
||||
and after-load initialization;
|
||||
- `builtin/runtime/lsp.lua` — delegate language inference while retaining the
|
||||
path guard;
|
||||
- `tests/m4_acceptance.rs` — parser, precedence, lifecycle, and end-to-end
|
||||
regression coverage;
|
||||
- this framing and the side-quest/handoff state documents when the feature
|
||||
lands.
|
||||
|
||||
No changes are expected in `src/buffer.rs`, Lua bindings, frontends,
|
||||
`pmacs-protocol`, or protocol version 18.
|
||||
|
||||
## Bets
|
||||
|
||||
1. **Sixteen KiB of edge text is sufficient.** Real modelines are short; files
|
||||
with an 8 KiB first/last candidate line are better treated as malformed than
|
||||
copied wholesale during load.
|
||||
2. **One canonical language should drive mode, parser, and LSP initially.** A
|
||||
future distinction between editor mode and parser language needs a real
|
||||
consumer and an explicit mapping contract, not accidental divergence.
|
||||
3. **ASCII-lowercased identifiers cover interoperable modelines.** Trusted Lua
|
||||
remains available for arbitrary UTF-8 custom mode names.
|
||||
4. **Load-time detection is enough.** Live cookie edits would require orderly
|
||||
parser teardown, LSP `didClose`/`didOpen`, overlay replacement, mode-change
|
||||
notification, and failure rollback; that is not a one-shot detector.
|
||||
5. **Pathless LSP buffers stay ineligible.** Syntax may still infer from a
|
||||
buffer name, but spawning a server without a URI/project root remains wrong.
|
||||
|
||||
## Acceptance
|
||||
|
||||
All end-to-end fixtures clear `pmacs.lsp.config` unless the case intentionally
|
||||
observes server selection, so opening a test file never starts a machine-local
|
||||
language server.
|
||||
|
||||
1. **Emacs forms:** first-line property and shorthand cookies resolve; a cookie
|
||||
on line 2 resolves only after a shebang; unrelated properties are ignored;
|
||||
the last `mode` property wins.
|
||||
2. **Vim forms:** `vim:`/`vi:` direct and `set` forms resolve `ft` and
|
||||
`filetype` in the first/last five lines; CRLF works; `Vim:` requires `set`.
|
||||
3. **Boundary rejection:** sixth-line, sixth-from-end, middle-of-file,
|
||||
word-embedded, unterminated, truncated, invalid-character, and overlong
|
||||
candidates do not resolve and do not log errors.
|
||||
4. **Conflict order:** overlapping edge windows are deduplicated and the last
|
||||
valid assignment in document order wins deterministically.
|
||||
5. **Alias behavior:** seeded aliases map `sh → bash`, `c++ → cpp`, and
|
||||
`tsx → typescriptreact`; a user override wins; an invalid alias output is
|
||||
ignored.
|
||||
6. **Explicit precedence:** a `.py` file with a Lua modeline yields `lua` from
|
||||
`pmacs.parse.buffer_language`, `pmacs.lsp.active_buffer_language`, the parse
|
||||
tree, and `pmacs.buffer.major_mode`.
|
||||
7. **Unknown valid mode:** a `.txt` file with `mode: prose` receives major mode
|
||||
`prose`, creates no parse view, starts no LSP server, and produces no error.
|
||||
8. **No-modeline regression:** extension, filetype, filename, and shebang cases
|
||||
retain their present precedence and outputs.
|
||||
9. **Pinned lifecycle:** changing a loaded modeline does not change the pinned
|
||||
language, parser, or major mode; switch-away/back remains stable; close and
|
||||
reopen re-evaluates the changed on-disk cookie.
|
||||
10. **Explicit override independence:** `set_major_mode` after load changes
|
||||
dispatch/statusline but not `pmacs.parse.buffer_language`; switches preserve
|
||||
the override.
|
||||
11. **Pathless preservation:** syntax-by-buffer-name behavior remains, while
|
||||
`pmacs.lsp.buffer_language` still returns nil without a backing path.
|
||||
12. **Backend parity:** focused acceptance passes with default LuaJIT and
|
||||
`--no-default-features --features lua54`; protocol remains v18.
|
||||
Loading…
Reference in New Issue