pmacs/docs/pmacs-gpu-mouse-framing.md

pmacs-gpu mouse input — framing pass

Date: 2026-06-10. Follows the Phase B framing (pmacs-gpu-phase-b-framing.md), which deferred the mouse wire decision as Q#B5. The optimistic-typing arc is merged (PR #60, main @ 7fdae76); this is the next capability session.

What exists today (fact-checked)

• Wire: FrontendEvent::Mouse(MouseEvent) carries a CellCoord (pmacs-protocol/src/message.rs:210-219) — a cell-grid coordinate. PROTOCOL_VERSION = 4 (message.rs:970); the DispatchIdle variant set the precedent for gating new variants on negotiated_protocol_version (daemon.rs:998).
• Daemon semantics: EditorState::dispatch_mouse owns the full gesture state machine — Down activates the window, positions the cursor, begins a selection; Drag grows the region; Up collapses an empty region; a second Down in the same cell within 500ms is a double-click and calls select_word_at_cursor; ScrollUp/Down move the viewport (src/editor.rs, CUA arc). For a semantic session, apply_semantic_input_event already routes FrontendEvent::Mouse → mouse_to_crossterm → dispatch_mouse (daemon.rs:1983-1986) — against the session's placeholder 24×80 cell geometry, which pmacs-gpu does not have.
• GPU layout: the shaped buffer holds the visible slice of the source (current_text[vstart..vend]) with inline adornment text injected by projected_rich_chunks (pmacs-gpu/src/main.rs:3202) — so shaped-buffer byte offsets ≠ source bytes whenever inlay hints are visible. Text is drawn at TEXT_LEFT/TEXT_TOP pixel offsets.
• Hit testing: cosmic_text::Buffer::hit(x, y) -> Option<Cursor> exists (cosmic-text 0.18.2, buffer.rs:923) and returns a cursor in the shaped buffer's (line, byte-index-within-line) space — the same line-relative space QB3 taught us to rebase via the slice line table.
• Scroll wheel: the GPU owns viewport + scroll state (S1: scroll_top / slice reshape / viewport re-declaration), but no winit MouseWheel handler exists today (fact-check: the window_event catch-all swallows it; scrolling is keyboard-driven). M-2 adds the handler — it composes from existing pieces (scroll_top adjust → reshape → viewport_send_if_changed) and needs no wire.

Q#M1 — the wire shape (resolves Q#B5)

How does a pixel frontend tell the instance where a click landed?

• (α) Reuse FrontendEvent::Mouse with synthesized cell coords. The GPU would approximate (row, col) from pixels. REJECTED: the daemon's cell math runs against window geometry the GPU doesn't share (placeholder 24×80), inline adornments shift visual columns with no cell-space representation, and the design doc's contract is explicit — the instance never learns a pixel, and there is no hit-test round trip. α is broken-by-design, not merely lossy.
• (β1) Local hit-test → byte-position pointer events. The GPU resolves pixels to a source byte offset locally (it owns the layout) and ships gesture-level events; the daemon replays its existing mouse state machine in byte space. Selection semantics (CUA, double-click word select, future triple-click) stay instance-side, exactly like the Key round-trip philosophy.
• (β2) Frontend-computed selection states. The GPU computes cursor+selection itself and ships final states. REJECTED: forks the gesture semantics into a second implementation the TUI doesn't share, and fights the daemon-authoritative selection model the CUA arc just consolidated.

Stance: β1. New wire variant:

FrontendEvent::Pointer {
    frontend_id: FrontendId,
    buffer_id: BufferId,
    /// Source byte offset the frontend hit-tested locally.
    byte: u64,
    kind: PointerKind, // Down | Drag | Up | DoubleDown
    mods: Modifiers,
}

DoubleDown is detected frontend-side (the frontend knows pixel proximity and its own double-click interval; the daemon's cell-based double-click detection cannot see pixels). The daemon maps: Down → activate semantic window + set cursor + begin_selection; Drag → cursor move (region grows); Up → collapse-if-empty; DoubleDown → select_word_at_cursor. This reuses the same core primitives dispatch_mouse calls today — no new selection logic.

Protocol: bump PROTOCOL_VERSION to 5; the GPU sends Pointer only when hello.protocol_version >= 5; the daemon ignores the variant for sessions that lack semantic_render.

Q#M2 — projected→source byte mapping

Buffer::hit answers in shaped coordinates: line index + byte offset within the shaped line, where shaped text = source slice with adornment text spliced in. Two-step mapping:

1. Shaped (line, index) → shaped-buffer byte offset via the shaped line table (the QB3 rebase, but over the projected text).
2. Projected byte → source byte via a run map built during reshape. This is new work, not a current by-product (fact-check: RichChunk today carries only text + color, no source range — main.rs:2269): projected_rich_chunks walks (source-run | adornment-run) chunks in order, so M-2 extends it to also emit Vec<ProjectedRun { projected_start, len, source_start: Option<u64> }>. A hit inside a source run maps linearly; a hit inside an adornment run snaps to the adornment's source anchor.

Stance: build the run map in reshape (it is O(chunks), walked there anyway) rather than re-deriving adornment offsets at click time. Clicks land between keystrokes, so the map being rebuilt per reshape costs nothing new.

Q#M3 — gesture scope for the first session

In: left-click cursor placement, left-drag selection, double-click word select, wheel scroll (local-only, no wire). Deferred: triple-click line select (needs a core select_line_at_cursor), middle-click paste (needs clipboard session), right-click menu (needs GUI chrome), drag auto-scroll at window edges (needs a repeat timer; record as a known gap), minimap click-to-jump (local-only; natural follow-up).

Q#M4 — interaction with optimistic state

A pointer event is a round-trip input: it must (a) mark cursor_fresh = false until the daemon's CursorByte answers, and (b) defer behind the optimistic-cursor floor exactly like round-trip keys do (defer_round_trip_key_if_needed shape) — a click landing between unconfirmed keystrokes would otherwise race the cursor confirmation. Deferred pointer events should keep only the latest Down/Drag (coalescing, like the TUI's frame-boundary mouse coalescing) rather than queueing every drag sample.

Predicted findings (categorical bets, scored at session close)

1. Coordinate-space bug (the QB3 family): some offset in the pixel→shaped→projected→source chain is wrong on first implementation — most likely the adornment run snap or the TEXT_LEFT/TEXT_TOP subtraction. Surfaces only in manual validation with inlay hints visible.
2. Window-activation gap (the B1 family): the daemon-side Pointer handler forgets some part of what activate_and_position does for grid windows (window focus, goal_col reset), so a click works but a subsequent arrow key moves from the wrong anchor.
3. Selection-wash latency: the wash arrives a round trip after the drag (Decorations cadence), reading as rubber-band lag. If it surfaces, the fix is frontend-local provisional selection — out of scope, record it.
4. Coalescing: drag streams at pixel rate (hundreds of events/sec) flood the socket or the dispatcher. Mitigation in-scope: send Drag only when the hit byte changes.

Session plan

• M-1 (wire + daemon): PointerKind/Pointer variant, version bump, daemon handler reusing core primitives, unit tests through handle_dispatcher_event. Compile-green alone.
• M-2 (GPU): winit mouse events → hit test → run map (new, from projected_rich_chunks) → Pointer sends; drag coalescing on byte change; NEW local wheel-scroll handler; optimistic-floor deferral. Manual validation gate: click placement with inlay hints on the same line, drag selection, double-click, wheel — in both pmacs-gpu and a TUI peer simultaneously.