From f233765f742b8cb4e454b0df2dbd1da53ea65f4d Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 15 Jul 2026 17:00:45 +0100 Subject: [PATCH 1/5] docs(json-yaml): framing for JSON + YAML grammars Frame the JSON/YAML grammar-gap addition (tree-sitter-json 0.24 + tree-sitter-yaml 0.7, both ABI-current) plus their language servers, and the free frontmatter/fence highlighting that falls out of the #122 injection engine (the markdown block query already injects yaml for `---` frontmatter and toml for `+++`). Four decisions (Q#JY1-JY4), nine acceptance tests. JSON is also the standing prerequisite for the Jupyter .ipynb arc. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan --- docs/json-yaml-framing.md | 177 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/json-yaml-framing.md diff --git a/docs/json-yaml-framing.md b/docs/json-yaml-framing.md new file mode 100644 index 0000000..1ab3a34 --- /dev/null +++ b/docs/json-yaml-framing.md @@ -0,0 +1,177 @@ +# JSON + YAML grammars — framing (side quest, highlight family) + +**Intent.** Add `tree-sitter-json` and `tree-sitter-yaml` grammars (plus +their language servers) to the bundle. Two config formats that pmacs +currently renders as plain text, and — the reason this is the natural +next side quest — the **honest gate on the Jupyter `.ipynb` path** (JSON) +and an **immediate payoff from the injection engine just shipped (#122)**: +the markdown block grammar's `injections.scm` already sets +`injection.language "yaml"` for `---` frontmatter and `"toml"` for `+++` +frontmatter, so registering YAML lights up YAML frontmatter highlighting +with zero extra wiring (TOML frontmatter already works — `toml` landed in +#118 and injections in #122). This is a mostly-additive grammar-gap-style +change, following the #118 pattern, with the frontmatter/fence synergy as +the demonstrable headline. + +--- + +## Ground truth (as of `main` @ `56eb67e`, #121) + +- **Adding a grammar** is a one-line `LanguageEntry` in + `crate::syntax::BUILTIN_LANGUAGES` (`name`, `extensions`, `loader`, + `highlights_query`, `injections_query`) + a `tree-sitter-foo` dep. The + Lua `buffer.after-load` path picks it up automatically; detection is + extension → LSP filetype → filename → shebang + (`resolve_active_language`). +- **Grammar name MUST equal the `pmacs.lsp.config.` key** — grammar + detection wins over the filetype map, so the name it resolves is the id + the LSP client keys off (the #118 invariant; there's an acceptance test + that pins every grammar-gap language to its config key). +- **LSP configs** are `pmacs.lsp.config. = … or { command, args, + [settings|init_options] }` (`builtin/runtime/lsp.lua`); no json/yaml + config today. `pmacs.lsp.filetypes` is the LSP-only extension fallback + (consulted only when `language_for_path` misses). +- **Injection synergy (#122).** The bundled `tree_sitter_md:: + INJECTION_QUERY_BLOCK` contains: + - `((minus_metadata) @injection.content (#set! injection.language + "yaml"))` — `---`-fenced frontmatter, + - `((plus_metadata) @injection.content (#set! injection.language + "toml"))` — `+++`-fenced frontmatter, + - fenced code blocks via the dynamic info-string. + So a registered `yaml` grammar is injected into markdown frontmatter + automatically, and ` ```json `/` ```yaml `/` ```yml ` fences resolve + (`yml`→yaml is already in `default_injection_aliases`; `json` is the + bundled name). + +**Confirmed crate facts** (probed against the registry + a build under +tree-sitter 0.26): + +1. `tree-sitter-json` **0.24.8** — `pub const LANGUAGE: LanguageFn` (via + `tree-sitter-language`, the modern shared ABI) + `HIGHLIGHTS_QUERY`. + Compiles and links under our tree-sitter 0.26. No `INJECTIONS_QUERY` + (JSON embeds nothing). +2. `tree-sitter-yaml` **0.7.2** — same shape (`LANGUAGE: LanguageFn`, + `HIGHLIGHTS_QUERY`, `tree-sitter-language` dep). Compiles under 0.26. + No `INJECTIONS_QUERY`. +3. Both are the ABI-current crates — **not** a `tree-sitter ^0.20` fork + (the dockerfile trap from #118). A single build confirmed link + + compile; runtime `set_language` is pinned by the ABI acceptance test. + +--- + +## Decisions + +### Q#JY1 — Two `LanguageEntry`s, self-contained highlights, no injections + +Add `json` and `yaml` to `BUILTIN_LANGUAGES`, each +`highlights_query: &[…::HIGHLIGHTS_QUERY]` (self-contained, no +`; inherits:` delta), `injections_query: &[]`. Extensions: + +- **json:** `.json`. (`.jsonc`/`.json5` — comment/trailing-comma variants + the plain JSON grammar rejects — are **deferred**; a `.jsonc` grammar + or a lenient mode is a separate call.) +- **yaml:** `.yaml`, `.yml`. + +Root kinds (pinned by the ABI test): json `document`, yaml `stream`. + +### Q#JY2 — LSP configs: `vscode-json-language-server` + `yaml-language-server` + +- **json:** `vscode-json-language-server --stdio` from the **maintained + `vscode-langservers-extracted` bundle** — NOT the stale standalone + `vscode-json-languageserver` npm package. Schema-driven + diagnostics/completion; auto-associates well-known files + (`package.json`, `tsconfig.json`) via its schema store. The underlying + Microsoft JSON server is **MIT-licensed with no telemetry path**; its + only relevant outbound behavior is fetching remote `$schema` content. + **Remote schema retrieval stays enabled by default** — setting + `handledSchemaProtocols = {"file"}` would disable server-side HTTP but + then remote schemas fail unless pmacs implements the `vscode/content` + request (out of scope), so we leave it on and document the behavior. +- **yaml:** `yaml-language-server --stdio` (Red Hat). + +Both servers stay **external** (installed by the user), adding **no +licensing payload** to pmacs; if either is ever bundled, retain its MIT + +dependency notices. Both **pull `workspace/configuration`** (the +mechanism that bit CMake in #117), so the config ships **non-nil empty +settings** matching the taplo/cmake precedent. **Implementation verifies +the exact `workspace/configuration` sections each real server requests +and pins those sections in the config + tests** — not merely that a +non-null settings table exists (if a server is not installed on the +build machine, the sections are derived from its source/docs and the +gap is noted, per the basedpyright machine-caveat precedent). Servers +activate only if installed; the grammar is the always-on value. + +### Q#JY3 — Filetype fallback + alias entries + +Add `pmacs.lsp.filetypes` entries (`json`→json, `yaml`/`yml`→yaml) as the +stable-id fallback (grammar detection wins in practice, same role as the +`cuda`/`lua` entries). `default_injection_aliases` already has `yml`→yaml; +`json`/`yaml` are bundled names needing no alias. Special *filenames* +(`.prettierrc`, `docker-compose.yml` is already `.yml`, extensionless +CI/config yaml) are **deferred** to the filename map as a follow-up. + +### Q#JY4 — Frontmatter/fence highlighting is the headline, and it's free + +No new injection wiring: registering `yaml` makes the existing markdown +`minus_metadata`→yaml injection resolve, and ` ```json `/` ```yaml ` +fences resolve through the #122 engine. Acceptance proves both end to end +(this is the demonstrable payoff and the tie-back to injections). + +--- + +## Bets + +1. The two crates are ABI-current and drop in like the #118 grammar-gap + languages — verified by a build; the ABI test is the runtime pin. +2. The frontmatter/fence synergy needs zero engine changes — it falls out + of #122 + the markdown injection query. +3. The LSP `workspace/configuration` shape is the only real unknown; + empty-but-present settings + implementation-time verification against + the real servers is the mitigation (the #117 CMake lesson). + +## Deferred (named) + +- `.jsonc` / `.json5` (comments / trailing commas) — needs a lenient + grammar or variant entry. +- Special-filename detection for extensionless config files (`.prettierrc`, + CI yaml) via the filename map. +- JSON **schema** wiring (custom `json.schemas` / `yaml.schemas` settings) + beyond the servers' built-in schema stores. +- The Jupyter `.ipynb` arc itself (JSON is its prerequisite, not its + delivery). + +## Acceptance + +1. `builtin_languages_include_json_and_yaml` — entries present, claim + their extensions, ship non-empty highlights. +2. `json_grammar_loads_and_parses` — ABI: `set_language` + parse a JSON + object; root `document`, no error (the runtime ABI pin). +3. `yaml_grammar_loads_and_parses` — ABI: parse a YAML mapping; root + `stream`, no error. +4. `json_yaml_highlights_compile` — both highlights queries compile and + resolve several capture classes. +5. `language_for_path_resolves_json_yaml` — `.json`→json, `.yaml`/`.yml`→ + yaml. +6. `json_yaml_align_with_lsp_configs` — grammar name == the + `pmacs.lsp.config.` key (the #118 invariant). +7. **`yaml_frontmatter_injects_in_markdown`** — a markdown doc with a + `---\nkey: val\n---` frontmatter yields a `yaml` child layer that + highlights; **the headline synergy with #122**. +8. `json_fence_injects_in_markdown` — a ` ```json ` fence yields a `json` + child layer. +9. LSP config tests (m4-style): `pmacs.lsp.config.json`/`.yaml` present + with the expected command/args (json uses + `vscode-json-language-server`), and the settings table carries + **exactly the `workspace/configuration` sections each server + requests** — pinned, not merely asserted non-nil. + +## Risks / interactions + +- **LSP `workspace/configuration`** (Q#JY2) — the real risk; verified + against the servers at implementation time, empty-present settings as + the safe default. +- **Themes / injections** — untouched. This is pure grammar+detection + addition; it consumes the #122 engine, doesn't change it. No protocol + bump. +- **`.yml` vs `.yaml`** — both map to `yaml`; no collision with any + existing entry. From 9ce6f1abf3790b73e5153156a46a317067a030d4 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 15 Jul 2026 17:44:59 +0100 Subject: [PATCH 2/5] feat(json-yaml): JSON + YAML grammars and language servers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add tree-sitter-json (0.24) and tree-sitter-yaml (0.7) to BUILTIN_LANGUAGES (both ABI-current via tree-sitter-language, verified compiling under tree-sitter 0.26), each self-contained highlights, no injections of their own. Extensions json=.json, yaml=.yaml/.yml; root kinds json `document`, yaml `stream`. The payoff from the #122 injection engine is free: the markdown block injection query already sets injection.language "yaml" for `---` frontmatter (minus_metadata) and "toml" for `+++` (plus_metadata), so registering yaml lights up YAML frontmatter highlighting with no extra wiring, and ```json / ```yaml / ```yml fences resolve through the engine (yml->yaml alias already present). Two acceptance tests pin this synergy. LSP (builtin/runtime/lsp.lua): pmacs.lsp.config.json uses the maintained extracted-bundle binary `vscode-json-language-server --stdio` (NOT the stale standalone vscode-json-languageserver); MIT, no telemetry, remote $schema fetch left enabled (no handledSchemaProtocols). pmacs.lsp.config .yaml uses `yaml-language-server --stdio` with Red Hat telemetry disabled by default. Both ship the exact workspace/configuration sections each server pulls (json+http; yaml+http+redhat.telemetry) present-not-null so the servers get defaults rather than erroring — the CMake #117 lesson. Sections derived from server source/docs (neither binary installed on this build machine to observe live; verify where present). Filetype fallback entries added. JSON is the standing prerequisite for the Jupyter .ipynb arc; handoff §6 updated. Nine acceptance tests (grammar ABI, highlights compile, detection, grammar<->LSP-key alignment, the two frontmatter/fence synergy proofs, and the pinned LSP-config sections). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan --- Cargo.lock | 22 +++++ Cargo.toml | 7 ++ builtin/runtime/lsp.lua | 39 +++++++++ docs/agent-handoff.md | 12 ++- src/syntax.rs | 182 ++++++++++++++++++++++++++++++++++++++++ tests/m4_acceptance.rs | 69 +++++++++++++++ 6 files changed, 327 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 15707b8..dd5b071 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2558,6 +2558,7 @@ dependencies = [ "tree-sitter-cuda", "tree-sitter-go", "tree-sitter-javascript", + "tree-sitter-json", "tree-sitter-lua", "tree-sitter-make", "tree-sitter-md", @@ -2565,6 +2566,7 @@ dependencies = [ "tree-sitter-rust", "tree-sitter-toml-ng", "tree-sitter-typescript", + "tree-sitter-yaml", "tree-sitter-zig", "unicode-width", ] @@ -3807,6 +3809,16 @@ dependencies = [ "tree-sitter-language", ] +[[package]] +name = "tree-sitter-json" +version = "0.24.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d727acca406c0020cffc6cf35516764f36c8e3dc4408e5ebe2cb35a947ec471" +dependencies = [ + "cc", + "tree-sitter-language", +] + [[package]] name = "tree-sitter-language" version = "0.1.7" @@ -3883,6 +3895,16 @@ dependencies = [ "tree-sitter-language", ] +[[package]] +name = "tree-sitter-yaml" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53c223db85f05e34794f065454843b0668ebc15d240ada63e2b5939f43ce7c97" +dependencies = [ + "cc", + "tree-sitter-language", +] + [[package]] name = "tree-sitter-zig" version = "1.1.2" diff --git a/Cargo.toml b/Cargo.toml index 98fec84..c3e9ea9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -196,6 +196,13 @@ tree-sitter-javascript = "0.25" tree-sitter-typescript = "0.23" tree-sitter-toml-ng = "0.7" tree-sitter-zig = "1.1" +# JSON + YAML — config formats + the honest gate on the Jupyter path +# (JSON). Both are ABI-current (`LANGUAGE: LanguageFn` via +# `tree-sitter-language`), NOT a `tree-sitter ^0.20` fork. Registering +# yaml also lights up markdown `---` frontmatter through the #122 +# injection engine (the block injection query already sets yaml for it). +tree-sitter-json = "0.24" +tree-sitter-yaml = "0.7" # T M9.7: markdown grammar so prompt result buffers with # `_meta.format = "markdown"` get structured highlighting through the # same M4 path as rust/lua — no special-case painter in Lua. diff --git a/builtin/runtime/lsp.lua b/builtin/runtime/lsp.lua index eb4f1e4..74c3866 100644 --- a/builtin/runtime/lsp.lua +++ b/builtin/runtime/lsp.lua @@ -200,6 +200,40 @@ pmacs.lsp.config.zig = pmacs.lsp.config.zig or { args = {}, } +-- JSON via the maintained `vscode-langservers-extracted` bundle +-- (`vscode-json-language-server`, NOT the stale standalone +-- `vscode-json-languageserver` npm package). Schema-driven +-- diagnostics/completion; auto-associates well-known files +-- (`package.json`, `tsconfig.json`) via its built-in schema store. It +-- pulls `workspace/configuration` for the `json` and `http` sections, so +-- both ship present-not-null (empty ⇒ server defaults). The underlying +-- Microsoft server is MIT with no telemetry path; its only outbound +-- behavior is fetching remote `$schema` content, LEFT ENABLED by default +-- (disabling via `handledSchemaProtocols = {"file"}` would break remote +-- schemas without a `vscode/content` implementation). Sections are +-- derived from the server source/docs — the binary is not installed on +-- this build machine to observe live; verify where it is present. +pmacs.lsp.config.json = pmacs.lsp.config.json or { + command = "vscode-json-language-server", + args = { "--stdio" }, + settings = { json = {}, http = {} }, +} + +-- YAML via Red Hat `yaml-language-server`. It pulls +-- `workspace/configuration` for `yaml`, `http`, and `redhat.telemetry`; +-- all three ship present-not-null, with Red Hat telemetry disabled by +-- default (privacy-respecting; inert if the server never asks). Sections +-- derived from source/docs — verify against an installed binary. +pmacs.lsp.config.yaml = pmacs.lsp.config.yaml or { + command = "yaml-language-server", + args = { "--stdio" }, + settings = { + yaml = {}, + http = {}, + redhat = { telemetry = { enabled = false } }, + }, +} + -- LSP-side extension → language map, deliberately independent of the -- tree-sitter detection in `pmacs.parse`. Consulted only when -- `pmacs.parse.language_for_path` finds nothing (an extension with a @@ -267,6 +301,11 @@ pmacs.lsp.filetypes.toml = pmacs.lsp.filetypes.toml or "toml" -- Zig (zls). `.zon` is Zig Object Notation, handled by the same server. pmacs.lsp.filetypes.zig = pmacs.lsp.filetypes.zig or "zig" pmacs.lsp.filetypes.zon = pmacs.lsp.filetypes.zon or "zig" +-- JSON / YAML. Both ship grammars, so `language_for_path` already resolves +-- these and the map is the stable-id fallback (same role as `lua`/`cuda`). +pmacs.lsp.filetypes.json = pmacs.lsp.filetypes.json or "json" +pmacs.lsp.filetypes.yaml = pmacs.lsp.filetypes.yaml or "yaml" +pmacs.lsp.filetypes.yml = pmacs.lsp.filetypes.yml or "yaml" -- Per-buffer attachment record: { language, server, uri, version }. -- Keyed by `tostring(BufferIdLua)` because BufferIdLua hands out fresh diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index d9986c7..d54cba0 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -323,11 +323,15 @@ runtime/Lua-registered languages (v1 resolves only against `BUILTIN_LANGUAGES`), and the next injection *consumers* gated on new grammars — HTML/CSS/GraphQL/SQL (`