From c51c88cac08aa88ac0d7aa36bb35e393693eb041 Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Tue, 14 Jul 2026 19:10:58 +0100 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20side-quest-backlog=20=E2=80=94=20no?= =?UTF-8?q?n-themes=20deferral=20inventory?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidate the non-themes backlog into docs/side-quest-backlog.md: an exhaustive, deduplicated index compiled from every framing doc's "Deferred (named)" section, the handoff §6 list, the roadmap arcs, and a code-level marker sweep. Organized as side quests (one-shot-ish, led by the highlighting/detection/LSP/editing families that continue the #114–#118 stack), raid bosses (vterm, folding, DAP, GPU splits, Jupyter), cross-cutting substrate blockers, and a prioritization north star. Themes/faces/color items are excluded (they're the Arc 4 main quest) and listed once so they're visibly accounted for. Handoff §6 gains a pointer to it. Docs only. git diff --check clean. --- docs/agent-handoff.md | 3 + docs/side-quest-backlog.md | 227 +++++++++++++++++++++++++++++++++++++ 2 files changed, 230 insertions(+) create mode 100644 docs/side-quest-backlog.md diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index 6f0f899..416f484 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -307,6 +307,9 @@ surface, chrome bold/italic/underline re-shaping). Housekeeping: F-016 `lua_bindings/mod.rs` split paused mid-way (tranches 0–2 landed, ~5–8 PRs left; see `docs/lua-bindings-split-framing.md`). +Full cross-cutting index of the non-themes backlog (this list + every +framing doc's Deferred section + a code sweep, themes excluded, with a +prioritization north star): `docs/side-quest-backlog.md`. ## 7. Machine-local facts (desktop) that do NOT travel diff --git a/docs/side-quest-backlog.md b/docs/side-quest-backlog.md new file mode 100644 index 0000000..39f713b --- /dev/null +++ b/docs/side-quest-backlog.md @@ -0,0 +1,227 @@ +# Side-quest backlog — the non-themes deferral inventory + +**Compiled 2026-07-14** from an exhaustive sweep of every framing doc's +"Deferred (named)" section, the handoff §6 consolidated list, the +`docs/roadmap-2026-07.md` arcs, and a code-level marker sweep +(src/, builtin/, pmacs-gpu/, pmacs-protocol/). + +**Scope.** This is the *side-quest* backlog: self-contained, +frontend-agnostic-ish work that does **not** belong to the **themes / +faces main quest** (Arc 4, `docs/theme-faces-framing.md`). Anything about +colors, faces, syntax-highlight *styling*, or semantic-token→color +mapping is excluded here and lives with the theme workstream — those are +listed once, at the bottom, so the reader knows they were seen. + +Framing: **side quests** are one-shot-ish and mostly enabling; **raid +bosses** are big multi-PR arcs. Each framing doc's own Deferred section +remains the authoritative per-feature source; this file is the +cross-cutting index. Keep it pruned as items ship. + +--- + +## Side quests — highlighting & language detection + +The direct continuation of the #114–#118 grammar/detection stack. + +- **Locals-query processing** — run each grammar's `LOCALS_QUERY` so + `#is?`/`#is-not? local` is honored (restores `.builtin` styling for + *non-shadowed* console/require etc.), replacing the current + fail-closed drop in `compute_highlight_spans`. +- **Multi-language injections** — the load-bearing enabler. One grammar + per buffer today (`syntax.lua`, "first language wins"); injections + unlock markdown fenced code blocks, HTML-in-JS/CSS, embedded langs, + and per-cell notebooks. +- **Modeline detection** — a 5th detection layer (`-*- mode: … -*-`, + `# vim: ft=…`) after extension → filetype → filename → shebang. +- **JSON + YAML** grammars + LSP (`tree-sitter-json` + + vscode-json-languageserver; yaml-language-server). JSON is also the + prerequisite for the notebook path. +- **More grammars for languages with neither grammar nor LSP** — ruby, + php, html, css, sql, etc. +- **Grapheme / combining-mark awareness** in the text view + (`text_view.rs` skips zero-width marks). +- **Byte-accurate multibyte cursor placement** — `move_active_cursor_to` + (and go-to-definition) step one codepoint per LSP byte-column, + landing short on multibyte lines. +- **Daemon query-by-range / incremental highlight** — `scoped_style_spans` + runs the query over the whole tree even for a scoped viewport (perf + ceiling on very large files). + +## Side quests — LSP feature surface + +- **Multi-root scoping** — one server *per project root* instead of the + current first-file-wins single-root-per-language. +- **Hidden-buffer LSP attach** — a restored/registry-only buffer attaches + LSP only on first visit (no `after-load` fires). +- **Inlay-hint inline rendering** — hint data exists but nothing renders; + needs the virtual-text renderer (the cell-overlay model can't reflow + glyphs around inserted columns yet). Plus hint-part interactivity. +- **Completion depth** — `completionItem/resolve` + `additionalTextEdits` + (auto-import), snippet tabstops/placeholders, a doc panel beside the + popup, fuzzy matching, persisted frequency ranking. +- **Panels** — reference-row line snippets, in-panel fuzzy filtering, + peek/preview on n/p, call-hierarchy & workspace-symbols panels, + migrate `*buffer-list*` onto `listview`. +- **Quick-fix titles streamed into the context menu** (vs one generic + "Quick Fix" item). +- **Attach transports** — TLS / Custom (SSH landed; these still + parse-and-defer). + +## Side quests — editing (Emacs table-stakes not yet built) + +- **Kill-ring / motion:** word kills (`M-d`/`M-BS`/`C-BS`/`C-h`/`C-DEL`, + need bytes-returning deleters), `C-SPC` set-mark, `C-u C-y`, `C-M-w` + append-next-kill, kill-ring browser, ring persistence, clipboard + watching. +- **Comments:** block comments + mid-line spans, `comment-dwim` + append-at-EOL, doc-comment continuation on newline, per-language + padding. +- **Auto-indent:** language-aware/electric indent (`indents.scm`), + TAB-as-reindent, `C-o` open-line, `C-j`, strip abandoned-line trailing + whitespace on split. +- **Auto-pairing:** wrap-region on opener, pair-aware backspace, + RET-inside-pair closer-on-own-line, in-string/in-comment inhibit + (needs a node-at-byte `pmacs.parse` binding), undo amalgamation, + balance-aware quotes, per-buffer toggle. +- **Editops:** recenter (`C-l`, blocked on viewport facts), Unicode + case/word classes, region-spanning move/duplicate, locale/numeric + sort-lines, ensure-final-newline on save, punctuation-aware + `fixup-whitespace`, chords for the M-x-only commands. +- **Query-replace:** capture-group refs (`\1`), `,`/`^`/`?` keys, + backward & whole-buffer, single undo-group per run, smart default + from-string, lazy-highlight of remaining matches, non-interactive + `replace-string`/`replace-regexp`. + +## Side quests — search + +- **Regex + Unicode case-folding** in search (`search.rs` is ASCII-fold, + literal-only today). +- **occur-mode**; **lazy-highlight** of all matches; **search-match + background rendering** on GPU (producer + `SearchMatch` quads). + +--- + +## Cross-cutting substrate (unblocks whole clusters — high leverage) + +- **Config/settings registry** (`pmacs.config` / defcustom) — blocks + language-aware indent, the five hardcoded tab-width sites, per-buffer + auto-pair toggle, per-language comment padding, per-project compile + commands. +- **Real `read_only` buffer flag** on both edit paths — true immutability + for panels / REPL / generated buffers. +- **Mode system wiring** — dispatch passes an empty mode list (`&[]`); + mode-scoped keybindings unresolved everywhere. +- **Buffer-aware edit epoch + origin-pinned `after-edit` fan-out** — a + command that edits buffer A then switches to B currently evades + `didChange` / reparse / autosave observers. +- **Undo-group boundaries (`begin/end_undo_group`) + cross-peer + chronological undo arbiter** — coherent multi-edit undo and mixed + source/daemon history. +- **Viewport facts on the wire** — the GPU never consumes daemon + `view_top`; blocks recenter and any scroll command. + +--- + +## Raid bosses (big multi-PR arcs) + +- **Arc 5 stage 2 — vterm:** extend `ansi.rs` into a 2D grid model + (alt-screen, cursor addressing, scrollback), grid-backed buffer view, + GPU grid rendering. +- **Arc 6 — Folding:** fold engine (tree-sitter fold ranges) + + `FoldState` wire family + gutter fold markers. +- **Arc 7 — Debugging (DAP):** DAP client (JSON-RPC/stdio, same shape as + the LSP client), breakpoint gutter signs, stack/variables panels + (reuse the Arc 1b listview). +- **Arc 8 — GPU splits / multi-buffer** (largest unscoped design + problem). +- **Jupyter `.ipynb`** — reader → editable → kernel execution; gated on + JSON + injections. +- **Git-diff gutter markers** (needs a diff source; rides the gutter). + +--- + +## Compile-mode / process follow-ups + +PTY-mode variant, auto-scroll option, severity threshold for error-only +stepping, per-language/project default commands, echo-area short output ++ distinct `M-&`, error parsing in `*shell-command*`, split-window +display, persist last compile command, next-error across historical +result buffers, occur-mode, configurable `GROUP_TERM_GRACE`, unify the +poll-based cancellable readers across REPL/LSP, fully tick-driven final +drain (kill the ~600 ms teardown stall). + +## Persistence & files + +Desktop daemon/GPU-attach restore, save unsaved *content* in the desktop, +multiple named desktops, window-local minor state, remote/cross-machine +desktops, non-file buffers in the desktop, saveplace for non-file +buffers, recentf as a `listview` panel, per-project `init.lua` desktop +scoping, newline-in-path handling. **Autosave:** idle-gated timing, +non-file buffers, orphan-recovery GC + browser, offload large writes to a +worker, **external-change guard on `save()` itself** (a real adjacent +bug), a `y_or_n` confirm helper, harden the whole `$XDG_STATE_HOME/pmacs` +root to `0700`, fire `after-load` for `[new file]` buffers. **Files:** +byte-preserving (round-trip-exact) save paths; the *warning* half of +external-change detection (verify-modtime-while-open, revert?). + +## GPU frontend mechanics (non-theme) + +- **Input:** full command/minibuffer chord forwarding to the GUI, + Meta/Super chords, rebindable local `Ctrl-V`/`Escape`, middle-click + paste, right-click context menu, frontend-local provisional selection. +- **Minibuffer:** `i/total` hint (already on the wire), Telescope-style + preview pane, candidate kind/doc annotations, unify TUI inline vs GPU + dropdown, multibyte-exact band caret, the nav highlight-wrap bug. +- **Scrolling:** scrollbar scroll, pixel-smooth sub-line scroll, + horizontal scroll / soft-wrap. +- **Rendering:** peer caret glyph + name label, own-vs-peer cursor merge, + `SelectionSnapshot` vs `Decorations::Selection` reconciliation, + background-kind decorations actually painted, inline adornment + placements beyond `AtOffset`. +- **Robustness:** auto-reconnect + "reconnecting…" banner, capability + renegotiation (offer to relaunch the daemon `--features crdt`), the + `AttachRequest.initial_size` cell-grid assumption. +- **Perf / harness:** glyphon full-buffer `prepare` ceiling, golden-PNG + comparison harness, `Renderer` sub-struct extraction. + +## Packages / protocol / CLI + +Forge aliases (`codeberg:` / `forgejo:`, post-v1.0), namespace-preserving +package layout + install-time cross-resolve collision detection, +project-local `init.lua` wiring + after-init calls, multiple files on the +CLI, `describe-job` full-message surfacing. + +## Housekeeping / refactor + +`lua_bindings/mod.rs` split (~4 tranches / ~14 k lines left; the `theme` +domain excluded — see below), `editor.rs` split (7 k lines), +`pmacs-gpu/src/main.rs` split (6.7 k lines), narrow re-exported +`install_*` from `pub` → `pub(crate)`. + +--- + +## Excluded — themes / faces main quest (seen, routed elsewhere) + +`pmacs.gpu.set_font` + the statusline-segment API; background/selection +theming; per-peer stable presence colors; exact quad colors per +decoration kind; GPU gutter background layer / wash recolor / chrome +bold-italic-underline; current-line highlight refinements; multi-server +semantic-token *style* blending; the compile-mode **severity→color** +classifier (the classification is behavioral, but the coloring is theme +work); the `theme` domain + style/color converters in the lua-bindings +split. Borderline, kept above as mechanics: whitespace glyphs and indent +guides (visual, not color). + +--- + +## North star (highest-leverage first) + +Two substrate items unlock the most side quests: + +1. **Config registry** — frees ~5 editing/indent/comment features at once. +2. **Multi-language injections** — frees notebooks, markdown code, + embedded langs, and is the honest gate on Jupyter. + +After those, the cleanest one-shots in the highlight family are +**JSON/YAML + modeline detection**, and the most-missed editing +table-stakes are **word-kills + `C-SPC` set-mark**. From afe2690158fed4c5a1f71ee2d9e9eedc1054657a Mon Sep 17 00:00:00 2001 From: Levi Neuwirth Date: Wed, 15 Jul 2026 16:21:43 +0100 Subject: [PATCH 2/2] docs: sync handoff for injections (#122); prune backlog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-merge handoff sync for multi-language injections (#122), folded into the side-quest-backlog PR (rebased onto current main, which now carries #120 themes stage 1 and #122 injections). agent-handoff.md: - §1: main head → 5e73966; add the #122 injection-engine entry (layered ParseTreeBundle, worker-side layer building off the static grammar table, settle-time query resolution, the new substrate — injections_query, the alias snapshot, injection_capped, compute_highlight_spans_for, the flatten event-sweep, the GPU fold — and the two hard-won findings). - §6: the "multi-language injections" deferral is retired; replaced with the real follow-ups (injection.combined, child-tree incrementality, injectable Lua-registered languages, HTML/CSS/GraphQL/SQL consumer grammars). Jupyter is now gated on JSON only. side-quest-backlog.md: injections marked SHIPPED with its remaining follow-ups; Jupyter gate and north star revised (JSON grammar and locals-query processing rise to the top now that injections landed). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01YJ9FQ832QwftJXCD9LeFan --- docs/agent-handoff.md | 65 +++++++++++++++++++++++++++++--------- docs/side-quest-backlog.md | 39 +++++++++++++++-------- 2 files changed, 76 insertions(+), 28 deletions(-) diff --git a/docs/agent-handoff.md b/docs/agent-handoff.md index 416f484..d9986c7 100644 --- a/docs/agent-handoff.md +++ b/docs/agent-handoff.md @@ -1,7 +1,7 @@ # Agent handoff — cross-machine continuity -**Last updated: 2026-07-15, by the themes stage-1 session (#120 -post-merge snapshot).** This file is the +**Last updated: 2026-07-15, after multi-language injections (#122) +merged; also carries the #120 themes-stage-1 snapshot.** This file is the bridge between development machines. If you are an agent reading this on a fresh clone: this document plus the `docs/*-framing.md` files ARE your memory. Read this fully before taking on work, seed @@ -11,8 +11,9 @@ reads it the way you just did. ## 1. Where the project stands (2026-07-15) -- `main` @ `8ce2e9c` (themes stage 1 #120 merged), protocol **v16** - (`SUPPORTED=[6..16]`; v15→16 shipped the `ThemeFacts` channel). +- `main` @ `5e73966` (multi-language injections #122 merged; #120 + themes stage 1 below it), protocol **v16** (`SUPPORTED=[6..16]`; + v15→16 shipped the `ThemeFacts` channel — injections added no wire). - **Syntax-highlight / language-detection side-quest (#114–#118) LANDED** — a one-shot arc built in sibling worktrees off main while the user's themes lane (`theme-faces`) ran concurrently in the shared @@ -41,6 +42,34 @@ reads it the way you just did. over js/jsx). `compute_highlight_spans` FAILS CLOSED on the `#is?`/`#is-not? local` property predicate (no locals processing) — drops those captures so shadowed builtins aren't mis-styled. +- **Multi-language injections (#122) LANDED** — the direct continuation + of the #114–#118 highlight arc; four review rounds, framing + `docs/multi-language-injections-framing.md` (Q#IJ1–IJ11). A buffer can + now hold more than one language: `ParseTreeBundle` holds `Vec` + (root + injected children, depth-ascending, installed atomically so the + existing `StyleGate` + highlight-cache Arc gates still work). The parse + worker builds child trees off the static `BUILTIN_LANGUAGES` table + (lazy load preserved) via `set_included_ranges` — child node offsets + are ABSOLUTE, so injected spans are buffer-coordinate-native; settle + resolves each layer's highlight query + (`SyntaxRegistry::resolve_layer_queries`). First consumers: markdown + fenced code + `markdown_inline` (retires the M9.7 block-only floor). + New substrate: `LanguageEntry.injections_query`; + `default_injection_aliases` + `SyntaxRegistry::injection_alias_snapshot` + (case-folded fence names, Lua-extensible via + `pmacs.parse.injection_aliases`, snapshotted into `ParseRequest` so the + worker never touches the `Rc` registry or Lua); + `ParseTreeBundle.injection_capped` (the 4096-layer backstop, surfaced + once/buffer via `pmacs.error` at settle); + `compute_highlight_spans_for(query, tree, source, range)` (per-layer); + the wire `flatten_layer_spans` event-sweep → DISJOINT effective spans + (deeper / later-sibling / narrower wins, keyed by `(layer_index, + capture_order)`); GPU `spans_from_segments` + `source_color_at` fold. + Two findings to keep: injection ranges exclude only NAMED children + (anonymous tokens ARE the injected text — excluding them shreds a block + `inline` node; matches tree-sitter-md's own splitter), and the wire + flattener runs over the WHOLE buffer via the file-style summary, so it + must be an event sweep, not O(spans²). - **Compile-mode (Arc 5 stage 1, #113) LANDED** (2026-07-14, 7 rounds; framing `docs/compile-mode-framing.md` rev 13). `compile.run` streams `/bin/sh -c "exec 2>&1; "` into an intercept-read-only @@ -282,17 +311,23 @@ intercept changes what later callbacks — LSP, completion — observe). LSP/persistence: hidden-buffer LSP attach, daemon desktop-restore, the *warning* half of external-change detection (verify-visited-file- modtime), config registry (no unified config surface yet). -Highlight/detection (from the #114–#118 side-quest): locals-query -processing (run each grammar's LOCALS_QUERY so `#is?`/`#is-not? local` -is honored instead of the current fail-closed drop — restores `.builtin` -styling for non-shadowed console/require etc.); multi-language -injections (one-grammar-per-buffer today — blocks per-cell notebook -highlighting, markdown fenced code blocks, HTML-in-JS); modeline -detection as a 5th layer (`-*- mode: … -*-` / `# vim: ft=…`); JSON/YAML -grammars+LSP; byte-accurate multibyte cursor placement in `move_active -_cursor_to` (still steps one codepoint per LSP byte column). A full -Jupyter `.ipynb` setup (reader → editable → kernel execution) is a real -arc gated on JSON + injections, NOT a one-shot. +Highlight/detection (from the #114–#118 side-quest + injections #122): +locals-query processing (run each grammar's LOCALS_QUERY so +`#is?`/`#is-not? local` is honored instead of the current fail-closed +drop — restores `.builtin` styling for non-shadowed console/require +etc.); **injection follow-ups now the engine landed (#122)** — +`injection.combined` (many matches → one shared parse; PHP-in-HTML, some +comment schemes), child-tree incrementality + range-scoped layer rebuild +(child layers cold-reparse on every settle today), injectable +runtime/Lua-registered languages (v1 resolves only against +`BUILTIN_LANGUAGES`), and the next injection *consumers* gated on new +grammars — HTML/CSS/GraphQL/SQL (`