pmacs/docs/lsp-file-watch-d3-framing.md

# LSP file watcher D3 — the polling cost — framing

**Status: revision 2 — DRAFT, awaiting review. No implementation may
begin from this document.**

Continues issue #233, which stays open until this lane closes it. D1
and D2 — matching correctness and the re-registration leak — merged as
**#234** (`ae84d58`); this frames the remainder the user ruled next on
2026-08-11: the watcher is now *correct* and still walks everything,
every tick, forever.

## Review round 1 — five findings, and what each changed

Revision 1 was reviewed 2026-08-11 and did not survive it. Recorded
here because three of the five are cases where the framing reasoned
from the wrong mechanism, which is exactly what a framing review
exists to catch before code does.

- **P1 — the promised idle state was impossible.** Revision 1
  promised "one brief `walk_tree` blip every four seconds"; but
  `pmacs.workers.sleep` allocates a **running job for its whole
  duration** (`dispatch_sleep`, `src/async_runtime.rs:1022-1037` —
  the job sleeps in 1 ms slices on a **pool thread**), and
  `activity_summary` counts every `Running` job
  (`src/async_runtime.rs:1570`). A 4 s backoff sleep would render as a
  *constant* `⋯1 sleep 4000ms`, and filtering sleeps from the
  indicator would touch the instrument this lane declares out of
  scope. **Revision 2 removes the sleep from the design entirely**
  (see "The cadence" below) — the fix is the codebase's own idiom,
  not a new mechanism.
- **P1 — the scan root must be the server's, not a freshly detected
  one.** Revision 1 proposed `pmacs.project.detect` for the
  string-form base. Server rooting honours **configured strings and
  custom resolvers first** (`project_root_for`,
  `builtin/runtime/lsp.lua:783`), and the bundled texlab entry
  documents why its root *cannot* be `project.detect` (Q#LX2,
  `lsp.lua:284` — texlab wants the document root, not the repository
  root). The server's own `root_uri` and `cwd` are already exposed
  (`pmacs.lsp.list`, `src/lua_bindings/mod.rs:10985-11002`;
  `root_uri` is the spec field verbatim, nil when the server never
  asked for a root). **Revision 2 roots the scan at what the
  registering server actually serves**: `root_uri` → `cwd` →
  attached-file directory, in that order.
- **P1 — coalescing needs registration-epoch semantics.** Revision 1
  said "route the shared diff through every watcher" without defining
  which watcher set owns a diff. A watcher registered between two
  snapshots would receive a false CREATED for a file that predates
  its registration; membership changing during a walk recreates
  either #234's stale-watcher batch or the same pre-registration
  event. **Revision 2 defines shared snapshots with per-watcher
  baselines** (below), plus two witnesses the existing six tests do
  not cover.
- **P1 — "VCS-only exclusion is safest" was wrong.** A hard skip
  silently ignores a server that legitimately registers `.git/HEAD`
  or `**/.git/**`, and glob semantics mean even `**/*.rs` *can* match
  under `.git/` — so any unconditional exclusion is a deviation from
  the registered contract, not a safe default. Revision 1 also
  overweighted the win: exclusion was a **job-count** lever when every
  directory was a separate job, and the walk primitive removes that
  economics. **Revision 2 defaults to no unconditional exclusion**
  and reframes Q#D3-2 around the full option set.
- **P2 — the arithmetic used the issue's machine, not this one.**
  With D = 220 on this checkout it is 220 `read_dir` jobs **plus one
  sleep** per watcher per tick — 221, or **1,326 across
  rust-analyzer's six watchers** — and revision 1's proposed steady
  state was itself two jobs (sleep + walk), not one. Corrected
  throughout; the revised design's steady state is **zero jobs at
  idle** and one `walk_tree` job while a scan runs.

## Verified against the tree at `add0ba1`

Every claim below was read or measured this session (revision 2
re-verified the round-1 corrections against the code).

- Each registered watcher is its own coroutine looping
  `sleep(FILE_WATCH_INTERVAL_MS)` → `scan_tree` (`lsp.lua:1924`,
  `:2074-2083`); the interval is 250 ms.
- `scan_tree` awaits `pmacs.fs.read_dir` once **per directory**
  (`:2038-2041`), one async job each. `walk` recurses
  unconditionally; `matches` gates only whether an entry is
  *recorded*.
- **A sleep is a pool-occupying running job.** `dispatch_sleep`
  dispatches `run_sleep` onto the worker pool
  (`async_runtime.rs:1022-1037`), so every sleeping watcher holds one
  of the pool's `available_parallelism - 1` threads for the full
  interval — the hazard `autosave.lua:133-139` documents in so many
  words. Twelve pre-#234 rust-analyzer watchers were twelve
  mostly-sleeping pool threads.
- **Jobs per tick per watcher = 1 sleep + D read_dirs.** On this
  checkout D = 220, so 221 per watcher and **1,326 per tick for
  rust-analyzer's six watchers**.
- **250 ms is a floor, not a period.** The awaits are sequential, so
  a tree whose walk takes longer than the interval makes the
  effective period scan-bound — the loop never idles. The issue
  measured ~270 ms on a two-directory tree; D round trips dominate on
  real trees.
- Measured on this checkout: **220 directories, of which `.git` is
  177 — over 80 % of the walk**. (The issue's machine carried an
  in-tree `target/`: 187 directories, 46 outside `.git`+`target`.
  This machine exports an external `CARGO_TARGET_DIR` and still pays
  `.git`.)
- **The string-form base is nondeterministic, found while framing:**
  `resolve_watcher`'s string arm takes the directory of the FIRST
  attachment `pairs()` happens to yield (`:2150-2173`) — table
  order, not a chosen root. #234 made matching correct *per base*;
  **which** base is still accidental.
- **The codebase already has a no-job cadence idiom with five
  adopters.** `process.after-tick` fires every frame, including idle
  frames (the run loops tick on a frame *timeout*), and
  `pmacs.editor.monotonic_ms` is the clock built for exactly such
  loops (`lua_bindings/mod.rs:13833`). `autosave.lua`'s Q#AS2 sweep
  is the model: one clock read and one compare per frame, no job, no
  pool thread.
- **No `notify`/inotify dependency in the tree** and **no
  ignore-list infrastructure to reuse** — both re-verified, both
  carried from the D1/D2 framing.

## What §9 asks of this lane

Not "quiet the modeline." The indicator is the instrument that found
this, and the churn it shows is real; quieting it is explicitly out
of bounds. The lane's job is to make the background work **small,
attributable, and honest**: at idle there should *be* no running
background work to report, and while a scan runs it should be one job
named for its root.

## The cadence — after-tick deadlines, not sleeps (review P1)

The per-watcher sleep loop is replaced by the Q#AS2 idiom: one
`process.after-tick` subscription owns every scan group's schedule.
Per frame it reads `pmacs.editor.monotonic_ms` once and compares each
group's `next_scan_at`; a due group gets its scan started (a
coroutine that runs the walk and diff). Waiting therefore allocates
**no job and no pool thread** and renders **no indicator segment** —
`activity_summary` returns `None` at zero by contract. While a scan
runs, the indicator honestly shows its one job.

This also retires a defect revision 1 did not name: today's sleeps
hold pool threads, so N watchers subtract N threads from a pool of
`available_parallelism - 1`. The after-tick cadence gives them all
back.

## The scan root (review P1)

For a string-form (bare `*.txt` / absolute) registration the base
becomes, in order: the server's **`root_uri`** (spec verbatim — nil
when the server never asked for a root), the server's **`cwd`**, and
only then the attached-file directory. These describe the workspace
the registering server actually serves — including configured roots
and custom resolvers like texlab's, which `pmacs.project.detect` can
never reproduce (Q#LX2). This replaces the `pairs()`-order accident
with a deterministic, server-owned answer. `RelativePattern`s keep
their own `baseUri`, unchanged.

## Coalescing, with registration epochs (review P1)

One scan group per (server, base). The group's scanner records
**all** files (the matcher moves from scan time to diff time); each
completed scan increments the group's **snapshot epoch**.

Delivery semantics, stated precisely because revision 1 did not:

- Each watcher records the epoch current when it **joined** the
  group. Its **baseline is the first snapshot completed after it
  joined**; it receives diffs only between snapshots it has a
  baseline for. A file created after the group's previous snapshot
  but before a new watcher registered therefore produces **no event
  for that watcher** — it is folded into the watcher's baseline,
  exactly as the initial scan folds pre-existing files today.
- **Membership is captured at scan start**; a watcher joining
  mid-walk waits for the next snapshot.
- **Cancellation is rechecked per watcher at emit time** — #234's P2
  rule, now applied per member: a watcher superseded or unregistered
  during the walk emits nothing, and its replacement (a fresh join)
  has no baseline yet, so it emits nothing either. Both halves of the
  round-1 hazard close on the same two rules.
- Changes passing a watcher's matcher and kind mask are deduped by
  `(uri, type)` into the server's single
  `workspace/didChangeWatchedFiles` notification, as today.

## The walk primitive

`pmacs.fs.walk_tree(base)` — the whole recursive walk as **one job**
instead of one per directory: 220 `read_dir` jobs per scan on this
repo become 1. The indicator shows one purpose (`walk_tree <root>`).
An additive fs binding plus its async-runtime job; **no wire change**
(fs bindings are not the frontend protocol) and no new crate. Symlink
non-traversal (`scan_tree`'s loop-safety) moves into the primitive's
contract, witnessed by its own Rust tests.

## Exclusions (review P1) — none by default

Glob semantics make any unconditional skip a contract deviation:
`**/*.rs` compiles with a separator-spanning prefix, so it *can*
match under `.git/`, and a server may register `.git/HEAD` outright
(branch-watching tools do). The only semantics-preserving default is
**no unconditional exclusion**, and with the walk primitive the
economics support it: exclusion was worth 80 % of the *job count*
when every directory was a job; inside one `walk_tree` job it is only
readdir syscalls, and the whole 220-directory walk is a few
milliseconds of one pool thread every backoff interval.

The option space, for Q#D3-2: (a) no unconditional exclusion — the
proposed default; (b) **opt-in** exclusion through configuration, for
users with pathological trees, framed explicitly as a watcher-contract
trade; (c) matcher-aware pruning — skip a subtree only when *no*
active watcher's pattern can match under it — which is sound but
almost never fires against real registrations, because
`**/`-leading globs can match anywhere; (d) a hard built-in VCS skip,
which revision 1 called "safest" and is not: it is (b) without the
opt-in.

## Idle backoff

The interval doubles while consecutive scans observe no change,
capped at 4 s; any change batch resets it to 250 ms. Under the
after-tick cadence a longer interval costs *nothing* while waiting —
backoff now bounds **scan frequency**, not sleep-job length.
Worst-case latency for an external change at idle equals the cap;
LSP imposes no latency bound, and edits made through pmacs never
depended on the watcher (the server sees `didChange`). The watcher
exists for git checkouts, generated files, and other editors.

## Deliberately staged separately — kernel notification

`notify` (inotify / FSEvents / kqueue) eliminates polling. Also: a
new dependency, a new Rust subsystem, a Lua binding, a platform
matrix, and an interaction with §9's ownership model. Staged as its
own framing — not because it is wrong but because everything above is
a pure win it does not obsolete (a kernel watcher still needs the
initial scan and a polling fallback), and a new-crate decision
deserves its own review.

## Proposed shape — Stage 1

After-tick cadence + walk primitive + coalescing-with-epochs +
backoff; no exclusions by default; server-owned scan root.

At rest on this repo with rust-analyzer attached: **from 1,326 jobs
per scan-bound tick (six of them pool-thread-holding sleeps) to zero
jobs at idle**, with one `walk_tree` job for the few milliseconds a
scan actually runs, at most every 250 ms under activity and every 4 s
at rest.

## Open rulings — each blocks implementation

- **Q#D3-1 — the acceptance bar.** With Stage 1 the indicator is
  **absent at idle** (no running job exists — `activity_summary`'s
  `None`-at-zero contract) and shows `⋯1 walk_tree <root>` for the
  duration of each scan. Is that the bar — an honest blip per scan,
  absence otherwise — with true event-driven silence deferred to the
  kernel-notification framing?
- **Q#D3-2 — exclusions.** Proposed: none by default, with opt-in
  exclusion as a documented contract trade (option b) if a user asks.
  Confirm, or rule for one of (b)/(c)/(d) above.
- **Q#D3-3 — the scan root.** Proposed: server `root_uri` → server
  `cwd` → attached-file directory. This widens the watched tree for
  servers with a real root (today it is one attached file's
  directory, chosen by hash order) — a behavioural change to a path
  real servers exercise. Confirm the order, or rule otherwise.
- **Q#D3-4 — interval, cap, and backoff curve: constants or config
  keys.** The D1/D2 framing refused a knob for a defect; with D3 the
  cadence becomes a designed mechanism, so keys are defensible — but
  more registry surface is coherence cost. Proposed: constants until
  someone asks.

## Verification sketch

- **Idle witness:** with a server attached, watchers registered, and
  no file activity, `activity_summary` settles to `None` (the absent
  segment) between scans — the strongest form of the job-count claim,
  and unwritable under the sleep design.
- **Scan-cost witness:** one scan allocates O(1) jobs, not
  O(directories), on a tree with enough directories to discriminate.
- **Epoch witness (registration between snapshots):** create a file
  after the group's snapshot, then register a second watcher, then
  let a scan complete — the old watcher receives CREATED, the new one
  receives **nothing** for that file, and does receive events for
  files created after its baseline.
- **Epoch witness (replacement during a scan):** through the group's
  scan seam (the `_after_scan_for_tests` device, lifted to the
  group), re-register mid-scan — the superseded watcher emits
  nothing (P2's rule, per member) and the replacement emits nothing
  until its own baseline exists.
- **Backoff witness:** quiet scans lengthen the gap between scans
  and one change resets it — observed through scan timestamps at the
  seam, not through sleep purposes (there are none).
- **Root witness:** a server with a configured root watches that
  root, not the attached file's directory; texlab's resolver shape is
  the fixture model.
- **Contract preservation:** all six existing `m4_24` watcher tests
  stay **byte-unchanged** and green.
- `walk_tree` Rust unit tests: symlinks recorded-not-traversed,
  signature parity with the Lua walk it replaces.
- Each new behaviour is mutation-tested against the defect it guards.

## Coherence impact (§20)

- **Journey steps:** none added; step 5 is unchanged.
- **Interaction islands:** none.
- **Config registry:** none by default; Q#D3-2/Q#D3-4 could add keys
  and are flagged as such.
- **Background-work attribution (§9):** at idle there is genuinely no
  running background work, and the indicator's absence is then a true
  statement rather than a filtered one; each scan is one job named
  for its root. The ownership *model* remains §9 Stage 2's work. As a
  side effect the watcher stops holding pool threads while waiting.

## Gates

`./scripts/gate --acceptance m4_acceptance`. No `--protocol` — no
wire change, no `PROTOCOL_VERSION` bump; `walk_tree` is an fs
binding, not a protocol message.