Merge pull request #121 from levineuwirth/side-quest-backlog

docs: side-quest-backlog — non-themes deferral inventory

docs/agent-handoff.md

@@ -1,7 +1,7 @@
 # Agent handoff — cross-machine continuity
 
-**Last updated: 2026-07-15, by the themes stage-1 session (#120
-post-merge snapshot).** This file is the
+**Last updated: 2026-07-15, after multi-language injections (#122)
+merged; also carries the #120 themes-stage-1 snapshot.** This file is the
 bridge between development machines. If you are an agent reading
 this on a fresh clone: this document plus the `docs/*-framing.md`
 files ARE your memory. Read this fully before taking on work, seed
@@ -11,8 +11,9 @@ reads it the way you just did.
 
 ## 1. Where the project stands (2026-07-15)
 
-- `main` @ `8ce2e9c` (themes stage 1 #120 merged), protocol **v16**
-  (`SUPPORTED=[6..16]`; v15→16 shipped the `ThemeFacts` channel).
+- `main` @ `5e73966` (multi-language injections #122 merged; #120
+  themes stage 1 below it), protocol **v16** (`SUPPORTED=[6..16]`;
+  v15→16 shipped the `ThemeFacts` channel — injections added no wire).
 - **Syntax-highlight / language-detection side-quest (#114–#118)
   LANDED** — a one-shot arc built in sibling worktrees off main while
   the user's themes lane (`theme-faces`) ran concurrently in the shared
@@ -41,6 +42,34 @@ reads it the way you just did.
     over js/jsx). `compute_highlight_spans` FAILS CLOSED on the
     `#is?`/`#is-not? local` property predicate (no locals processing) —
     drops those captures so shadowed builtins aren't mis-styled.
+- **Multi-language injections (#122) LANDED** — the direct continuation
+  of the #114–#118 highlight arc; four review rounds, framing
+  `docs/multi-language-injections-framing.md` (Q#IJ1–IJ11). A buffer can
+  now hold more than one language: `ParseTreeBundle` holds `Vec<Layer>`
+  (root + injected children, depth-ascending, installed atomically so the
+  existing `StyleGate` + highlight-cache Arc gates still work). The parse
+  worker builds child trees off the static `BUILTIN_LANGUAGES` table
+  (lazy load preserved) via `set_included_ranges` — child node offsets
+  are ABSOLUTE, so injected spans are buffer-coordinate-native; settle
+  resolves each layer's highlight query
+  (`SyntaxRegistry::resolve_layer_queries`). First consumers: markdown
+  fenced code + `markdown_inline` (retires the M9.7 block-only floor).
+  New substrate: `LanguageEntry.injections_query`;
+  `default_injection_aliases` + `SyntaxRegistry::injection_alias_snapshot`
+  (case-folded fence names, Lua-extensible via
+  `pmacs.parse.injection_aliases`, snapshotted into `ParseRequest` so the
+  worker never touches the `Rc` registry or Lua);
+  `ParseTreeBundle.injection_capped` (the 4096-layer backstop, surfaced
+  once/buffer via `pmacs.error` at settle);
+  `compute_highlight_spans_for(query, tree, source, range)` (per-layer);
+  the wire `flatten_layer_spans` event-sweep → DISJOINT effective spans
+  (deeper / later-sibling / narrower wins, keyed by `(layer_index,
+  capture_order)`); GPU `spans_from_segments` + `source_color_at` fold.
+  Two findings to keep: injection ranges exclude only NAMED children
+  (anonymous tokens ARE the injected text — excluding them shreds a block
+  `inline` node; matches tree-sitter-md's own splitter), and the wire
+  flattener runs over the WHOLE buffer via the file-style summary, so it
+  must be an event sweep, not O(spans²).
 - **Compile-mode (Arc 5 stage 1, #113) LANDED** (2026-07-14, 7 rounds;
   framing `docs/compile-mode-framing.md` rev 13). `compile.run` streams
   `/bin/sh -c "exec 2>&1; <cmd>"` into an intercept-read-only
@@ -282,17 +311,23 @@ intercept changes what later callbacks — LSP, completion — observe).
 LSP/persistence: hidden-buffer LSP attach, daemon desktop-restore, the
 *warning* half of external-change detection (verify-visited-file-
 modtime), config registry (no unified config surface yet).
-Highlight/detection (from the #114–#118 side-quest): locals-query
-processing (run each grammar's LOCALS_QUERY so `#is?`/`#is-not? local`
-is honored instead of the current fail-closed drop — restores `.builtin`
-styling for non-shadowed console/require etc.); multi-language
-injections (one-grammar-per-buffer today — blocks per-cell notebook
-highlighting, markdown fenced code blocks, HTML-in-JS); modeline
-detection as a 5th layer (`-*- mode: … -*-` / `# vim: ft=…`); JSON/YAML
-grammars+LSP; byte-accurate multibyte cursor placement in `move_active
-_cursor_to` (still steps one codepoint per LSP byte column). A full
-Jupyter `.ipynb` setup (reader → editable → kernel execution) is a real
-arc gated on JSON + injections, NOT a one-shot.
+Highlight/detection (from the #114–#118 side-quest + injections #122):
+locals-query processing (run each grammar's LOCALS_QUERY so
+`#is?`/`#is-not? local` is honored instead of the current fail-closed
+drop — restores `.builtin` styling for non-shadowed console/require
+etc.); **injection follow-ups now the engine landed (#122)** —
+`injection.combined` (many matches → one shared parse; PHP-in-HTML, some
+comment schemes), child-tree incrementality + range-scoped layer rebuild
+(child layers cold-reparse on every settle today), injectable
+runtime/Lua-registered languages (v1 resolves only against
+`BUILTIN_LANGUAGES`), and the next injection *consumers* gated on new
+grammars — HTML/CSS/GraphQL/SQL (`<script>`/`<style>`, JS/TS template
+literals, doc-comment code); modeline detection as a 5th layer
+(`-*- mode: … -*-` / `# vim: ft=…`); JSON/YAML grammars+LSP;
+byte-accurate multibyte cursor placement in `move_active_cursor_to`
+(still steps one codepoint per LSP byte column). A full Jupyter `.ipynb`
+setup (reader → editable → kernel execution) is a real arc now gated on
+**JSON** (injections shipped in #122), NOT a one-shot.
 GPU: auto-reconnect after daemon restart, splits/multi-buffer, gutter
 riders (whitespace guides, folding, git markers).
 Themes (full list in theme-faces framing rev 9 "Deferred (named)"):
@@ -307,6 +342,9 @@ surface, chrome bold/italic/underline re-shaping).
 Housekeeping: F-016 `lua_bindings/mod.rs` split paused mid-way
 (tranches 0–2 landed, ~5–8 PRs left; see
 `docs/lua-bindings-split-framing.md`).
+Full cross-cutting index of the non-themes backlog (this list + every
+framing doc's Deferred section + a code sweep, themes excluded, with a
+prioritization north star): `docs/side-quest-backlog.md`.
 
 ## 7. Machine-local facts (desktop) that do NOT travel
 

docs/side-quest-backlog.md

@@ -0,0 +1,240 @@
+# Side-quest backlog — the non-themes deferral inventory
+
+**Compiled 2026-07-14** from an exhaustive sweep of every framing doc's
+"Deferred (named)" section, the handoff §6 consolidated list, the
+`docs/roadmap-2026-07.md` arcs, and a code-level marker sweep
+(src/, builtin/, pmacs-gpu/, pmacs-protocol/). **Updated 2026-07-15:**
+multi-language injections shipped (#122) — item pruned to its remaining
+follow-ups; north star and Jupyter gate revised.
+
+**Scope.** This is the *side-quest* backlog: self-contained,
+frontend-agnostic-ish work that does **not** belong to the **themes /
+faces main quest** (Arc 4, `docs/theme-faces-framing.md`). Anything about
+colors, faces, syntax-highlight *styling*, or semantic-token→color
+mapping is excluded here and lives with the theme workstream — those are
+listed once, at the bottom, so the reader knows they were seen.
+
+Framing: **side quests** are one-shot-ish and mostly enabling; **raid
+bosses** are big multi-PR arcs. Each framing doc's own Deferred section
+remains the authoritative per-feature source; this file is the
+cross-cutting index. Keep it pruned as items ship.
+
+---
+
+## Side quests — highlighting & language detection
+
+The direct continuation of the #114–#118 grammar/detection stack.
+
+- **Locals-query processing** — run each grammar's `LOCALS_QUERY` so
+  `#is?`/`#is-not? local` is honored (restores `.builtin` styling for
+  *non-shadowed* console/require etc.), replacing the current
+  fail-closed drop in `compute_highlight_spans`.
+- **Multi-language injections — SHIPPED (#122).** The load-bearing
+  engine landed: `ParseTreeBundle` holds `Vec<Layer>`, the worker builds
+  child trees off the static grammar table, settle resolves per-layer
+  highlight queries; markdown fenced code + `markdown_inline` are the
+  first consumers. Remaining follow-ups: `injection.combined` (many
+  matches → one shared parse; PHP-in-HTML, some comment schemes),
+  child-tree incrementality + range-scoped layer rebuild (children
+  cold-reparse on every settle today), injectable runtime/Lua-registered
+  languages (v1 resolves only against `BUILTIN_LANGUAGES`), and new
+  injection *consumers* gated on grammars — HTML/CSS/GraphQL/SQL
+  (`<script>`/`<style>`, template literals, doc-comment code).
+- **Modeline detection** — a 5th detection layer (`-*- mode: … -*-`,
+  `# vim: ft=…`) after extension → filetype → filename → shebang.
+- **JSON + YAML** grammars + LSP (`tree-sitter-json` +
+  vscode-json-languageserver; yaml-language-server). JSON is also the
+  prerequisite for the notebook path.
+- **More grammars for languages with neither grammar nor LSP** — ruby,
+  php, html, css, sql, etc.
+- **Grapheme / combining-mark awareness** in the text view
+  (`text_view.rs` skips zero-width marks).
+- **Byte-accurate multibyte cursor placement** — `move_active_cursor_to`
+  (and go-to-definition) step one codepoint per LSP byte-column,
+  landing short on multibyte lines.
+- **Daemon query-by-range / incremental highlight** — `scoped_style_spans`
+  runs the query over the whole tree even for a scoped viewport (perf
+  ceiling on very large files).
+
+## Side quests — LSP feature surface
+
+- **Multi-root scoping** — one server *per project root* instead of the
+  current first-file-wins single-root-per-language.
+- **Hidden-buffer LSP attach** — a restored/registry-only buffer attaches
+  LSP only on first visit (no `after-load` fires).
+- **Inlay-hint inline rendering** — hint data exists but nothing renders;
+  needs the virtual-text renderer (the cell-overlay model can't reflow
+  glyphs around inserted columns yet). Plus hint-part interactivity.
+- **Completion depth** — `completionItem/resolve` + `additionalTextEdits`
+  (auto-import), snippet tabstops/placeholders, a doc panel beside the
+  popup, fuzzy matching, persisted frequency ranking.
+- **Panels** — reference-row line snippets, in-panel fuzzy filtering,
+  peek/preview on n/p, call-hierarchy & workspace-symbols panels,
+  migrate `*buffer-list*` onto `listview`.
+- **Quick-fix titles streamed into the context menu** (vs one generic
+  "Quick Fix" item).
+- **Attach transports** — TLS / Custom (SSH landed; these still
+  parse-and-defer).
+
+## Side quests — editing (Emacs table-stakes not yet built)
+
+- **Kill-ring / motion:** word kills (`M-d`/`M-BS`/`C-BS`/`C-h`/`C-DEL`,
+  need bytes-returning deleters), `C-SPC` set-mark, `C-u C-y`, `C-M-w`
+  append-next-kill, kill-ring browser, ring persistence, clipboard
+  watching.
+- **Comments:** block comments + mid-line spans, `comment-dwim`
+  append-at-EOL, doc-comment continuation on newline, per-language
+  padding.
+- **Auto-indent:** language-aware/electric indent (`indents.scm`),
+  TAB-as-reindent, `C-o` open-line, `C-j`, strip abandoned-line trailing
+  whitespace on split.
+- **Auto-pairing:** wrap-region on opener, pair-aware backspace,
+  RET-inside-pair closer-on-own-line, in-string/in-comment inhibit
+  (needs a node-at-byte `pmacs.parse` binding), undo amalgamation,
+  balance-aware quotes, per-buffer toggle.
+- **Editops:** recenter (`C-l`, blocked on viewport facts), Unicode
+  case/word classes, region-spanning move/duplicate, locale/numeric
+  sort-lines, ensure-final-newline on save, punctuation-aware
+  `fixup-whitespace`, chords for the M-x-only commands.
+- **Query-replace:** capture-group refs (`\1`), `,`/`^`/`?` keys,
+  backward & whole-buffer, single undo-group per run, smart default
+  from-string, lazy-highlight of remaining matches, non-interactive
+  `replace-string`/`replace-regexp`.
+
+## Side quests — search
+
+- **Regex + Unicode case-folding** in search (`search.rs` is ASCII-fold,
+  literal-only today).
+- **occur-mode**; **lazy-highlight** of all matches; **search-match
+  background rendering** on GPU (producer + `SearchMatch` quads).
+
+---
+
+## Cross-cutting substrate (unblocks whole clusters — high leverage)
+
+- **Config/settings registry** (`pmacs.config` / defcustom) — blocks
+  language-aware indent, the five hardcoded tab-width sites, per-buffer
+  auto-pair toggle, per-language comment padding, per-project compile
+  commands.
+- **Real `read_only` buffer flag** on both edit paths — true immutability
+  for panels / REPL / generated buffers.
+- **Mode system wiring** — dispatch passes an empty mode list (`&[]`);
+  mode-scoped keybindings unresolved everywhere.
+- **Buffer-aware edit epoch + origin-pinned `after-edit` fan-out** — a
+  command that edits buffer A then switches to B currently evades
+  `didChange` / reparse / autosave observers.
+- **Undo-group boundaries (`begin/end_undo_group`) + cross-peer
+  chronological undo arbiter** — coherent multi-edit undo and mixed
+  source/daemon history.
+- **Viewport facts on the wire** — the GPU never consumes daemon
+  `view_top`; blocks recenter and any scroll command.
+
+---
+
+## Raid bosses (big multi-PR arcs)
+
+- **Arc 5 stage 2 — vterm:** extend `ansi.rs` into a 2D grid model
+  (alt-screen, cursor addressing, scrollback), grid-backed buffer view,
+  GPU grid rendering.
+- **Arc 6 — Folding:** fold engine (tree-sitter fold ranges) +
+  `FoldState` wire family + gutter fold markers.
+- **Arc 7 — Debugging (DAP):** DAP client (JSON-RPC/stdio, same shape as
+  the LSP client), breakpoint gutter signs, stack/variables panels
+  (reuse the Arc 1b listview).
+- **Arc 8 — GPU splits / multi-buffer** (largest unscoped design
+  problem).
+- **Jupyter `.ipynb`** — reader → editable → kernel execution; now gated
+  on JSON only (injections shipped in #122).
+- **Git-diff gutter markers** (needs a diff source; rides the gutter).
+
+---
+
+## Compile-mode / process follow-ups
+
+PTY-mode variant, auto-scroll option, severity threshold for error-only
+stepping, per-language/project default commands, echo-area short output
++ distinct `M-&`, error parsing in `*shell-command*`, split-window
+display, persist last compile command, next-error across historical
+result buffers, occur-mode, configurable `GROUP_TERM_GRACE`, unify the
+poll-based cancellable readers across REPL/LSP, fully tick-driven final
+drain (kill the ~600 ms teardown stall).
+
+## Persistence & files
+
+Desktop daemon/GPU-attach restore, save unsaved *content* in the desktop,
+multiple named desktops, window-local minor state, remote/cross-machine
+desktops, non-file buffers in the desktop, saveplace for non-file
+buffers, recentf as a `listview` panel, per-project `init.lua` desktop
+scoping, newline-in-path handling. **Autosave:** idle-gated timing,
+non-file buffers, orphan-recovery GC + browser, offload large writes to a
+worker, **external-change guard on `save()` itself** (a real adjacent
+bug), a `y_or_n` confirm helper, harden the whole `$XDG_STATE_HOME/pmacs`
+root to `0700`, fire `after-load` for `[new file]` buffers. **Files:**
+byte-preserving (round-trip-exact) save paths; the *warning* half of
+external-change detection (verify-modtime-while-open, revert?).
+
+## GPU frontend mechanics (non-theme)
+
+- **Input:** full command/minibuffer chord forwarding to the GUI,
+  Meta/Super chords, rebindable local `Ctrl-V`/`Escape`, middle-click
+  paste, right-click context menu, frontend-local provisional selection.
+- **Minibuffer:** `i/total` hint (already on the wire), Telescope-style
+  preview pane, candidate kind/doc annotations, unify TUI inline vs GPU
+  dropdown, multibyte-exact band caret, the nav highlight-wrap bug.
+- **Scrolling:** scrollbar scroll, pixel-smooth sub-line scroll,
+  horizontal scroll / soft-wrap.
+- **Rendering:** peer caret glyph + name label, own-vs-peer cursor merge,
+  `SelectionSnapshot` vs `Decorations::Selection` reconciliation,
+  background-kind decorations actually painted, inline adornment
+  placements beyond `AtOffset`.
+- **Robustness:** auto-reconnect + "reconnecting…" banner, capability
+  renegotiation (offer to relaunch the daemon `--features crdt`), the
+  `AttachRequest.initial_size` cell-grid assumption.
+- **Perf / harness:** glyphon full-buffer `prepare` ceiling, golden-PNG
+  comparison harness, `Renderer` sub-struct extraction.
+
+## Packages / protocol / CLI
+
+Forge aliases (`codeberg:` / `forgejo:`, post-v1.0), namespace-preserving
+package layout + install-time cross-resolve collision detection,
+project-local `init.lua` wiring + after-init calls, multiple files on the
+CLI, `describe-job` full-message surfacing.
+
+## Housekeeping / refactor
+
+`lua_bindings/mod.rs` split (~4 tranches / ~14 k lines left; the `theme`
+domain excluded — see below), `editor.rs` split (7 k lines),
+`pmacs-gpu/src/main.rs` split (6.7 k lines), narrow re-exported
+`install_*` from `pub` → `pub(crate)`.
+
+---
+
+## Excluded — themes / faces main quest (seen, routed elsewhere)
+
+`pmacs.gpu.set_font` + the statusline-segment API; background/selection
+theming; per-peer stable presence colors; exact quad colors per
+decoration kind; GPU gutter background layer / wash recolor / chrome
+bold-italic-underline; current-line highlight refinements; multi-server
+semantic-token *style* blending; the compile-mode **severity→color**
+classifier (the classification is behavioral, but the coloring is theme
+work); the `theme` domain + style/color converters in the lua-bindings
+split. Borderline, kept above as mechanics: whitespace glyphs and indent
+guides (visual, not color).
+
+---
+
+## North star (highest-leverage first)
+
+**Multi-language injections shipped (#122)** — one of the two original
+north-star items is done, so the highest-leverage board is now:
+
+1. **Config registry** — frees ~5 editing/indent/comment features at once.
+2. **JSON (+ YAML) grammar** — the one remaining gate on the Jupyter
+   `.ipynb` path now that injections exist, and a clean highlight one-shot.
+3. **Locals-query processing** — restores `.builtin` styling for
+   non-shadowed builtins, the last rough edge of the highlight stack.
+
+Beyond those, the cleanest one-shots in the highlight family are
+**modeline detection** and the HTML/CSS grammars that light up more
+injection *consumers*; the most-missed editing table-stakes remain
+**word-kills + `C-SPC` set-mark**.