diff --git a/ARCHIVE.md b/ARCHIVE.md index 0c1d546..dcde6d6 100644 --- a/ARCHIVE.md +++ b/ARCHIVE.md @@ -32,7 +32,8 @@ native viewer), symmetric with HTML snapshots — see the Display — PDF decisi appends an archive affordance to body links whose target is archived; `archive.py wayback` (+ `make archive-wayback`) backfills Wayback captures; `visibility: private` keeps an entry's artifact in-repo but undeployed. -Bibliography annotation is documented as a `Citations.hs` follow-up. +Bibliography annotation is documented as a `Citations.hs` follow-up +*(implemented 2026-06-10 — see the Bibliography note under Link annotation)*. **Phase 4 complete (2026-05-22).** Backlinks + similar-pages: `Backlinks.hs` keeps archived external links and canonicalises them to their `/archive//` @@ -49,11 +50,13 @@ needs 3 fails over ≥14 days; a single success recovers immediately). each archive page surfaces its link status (provenance row, header note, Pagefind `status` filter tag); `/archive/` flags rotted entries; `/build/` gains a "Link archive" telemetry section. The search-UI `status` filter wiring -in `search-filters.js` is deliberately partial — see the Phase 5 Met note. +in `search-filters.js` was deliberately partial in this phase *(completed +2026-06-10 — see the Phase 5 Met note)*. **All five phases done.** Refinements next; see the Phase 5 Met note for the -documented deferrals (search-UI status filter; bibliography annotation from -Phase 3; pull-from-Wayback at fetch time). +documented deferrals (search-UI status filter, bibliography annotation from +Phase 3, and pull-from-Wayback at fetch time — all three implemented +2026-06-10). **Refinements (2026-05-22).** A code-review pass found and fixed several correctness and posture issues across the system: @@ -103,7 +106,8 @@ Three smaller items remain documented and deferred: own design pass (archive pages aren't `match`ed Hakyll items in the normal way). - **`archive.py suggest`** (bibliography discovery — diff `.bib` URLs - against the manifest) is documented but not implemented. + against the manifest) is documented but not implemented. *(Implemented + 2026-06-10 — see the status note below.)* - **The controlled-host end-to-end link-rot test** (reserve `archive-test.levineuwirth.org`, run it through a 14-day-spanning fail streak, watch the flip happen) is inherently a multi-week real-world @@ -237,6 +241,77 @@ failure-closed paths: The genuinely-open questions that remain are collected at the end — the list is short. +**`archive.py suggest` implemented (2026-06-10).** The bibliography-discovery +subcommand from the design is built: `make archive-suggest` scans `data/*.bib` +for `url` / `doi` fields (per entry the `url` field wins; a DOI-only entry +resolves to `https://doi.org/{doi}`), diffs them against `manifest.yaml` and +`removed.yaml` under the same `normalize_url` equivalence the link-annotation +filter uses, and prints manifest-ready `- url:` lines, each preceded by +`# cited by {bibfile}:{key}` comments (a work cited from several papers prints +once, with every citer). Read-only and offline: it never edits the manifest +and performs no network I/O. Known equivalence gap, surfaced by the first real +run: a work archived under its landing URL but cited via its DOI (e.g. +FIPS 203 — archived as the `nvlpubs.nist.gov` PDF, cited as +`doi.org/10.6028/NIST.FIPS.203`) is re-suggested, because nothing offline can +equate the two forms. That is the same gap that denies those bibliography +links an archive affordance, so the suggestion is honest — the fix, if wanted, +is alias-level (manifest or `url_aliases`), not suppression in `suggest`. + +**Manifest `aliases:` field implemented (2026-06-10).** The alias-level fix +above: an optional authored `aliases:` list per manifest entry for equivalent +URLs that normalisation cannot derive (the DOI ↔ landing-URL case). Aliases +are matching metadata, not identity — editing them rewrites the index on the +next fetch, no `refresh` involved. Enforcement mirrors canonical URLs on both +sides of the build: the `archive.py fetch` pre-scan and `Archive.hs`'s +`validateManifestEntries` each reject an alias that collides with another +entry (directly or via aliases) or that matches a `removed.yaml` takedown, +and `ArchiveIndex.flatIndex` additionally drops alias keys matching recorded +takedowns so a stale index cannot keep annotating a removed work. `suggest` +counts aliases as covered. Applied: `nist-fips-203` now carries +`https://doi.org/10.6028/NIST.FIPS.203`, so the simd paper's DOI citation +resolves to the archived PDF and `suggest` no longer re-suggests it. + +**Link-rot scan scheduled (2026-06-10).** The scan now runs unattended: a +systemd user timer (`systemd/archive-check.{service,timer}`, symlink-installed +on the build machine) fires `make archive-check` daily. Going unattended +exposed a hysteresis hazard hand-run scans masked — a machine left offline +would record a `fail` against every entry, and three such scans spanning two +weeks would flip the whole archive to `rotted` — so `cmd_check` gained a +canary guard: `https://levineuwirth.org/` unreachable → scan inconclusive, +state untouched, exit 0. Details in the Phase 5 section. + +**Fetch-time Wayback fallback implemented (2026-06-10).** The design's +"original already dead at first fetch → pull the most recent existing Wayback +capture" promise is now real. `fetch_pdf`/`fetch_html` classify their +failures (`ok` / `dead` / `skip`), and only a *dead* original — not a +`noarchive` refusal, cap skip, or tooling failure — falls through to +`fetch_from_wayback`, which pulls the raw `id_` capture bytes through the +normal pipeline, honours a preserved `X-Archive-Orig-X-Robots-Tag: +noarchive`, re-detects the artifact type against the capture (the dead +original's Content-Type probe degrades to the html default), and records +`fetched-from` + `wayback` in `PROVENANCE.json`. `refresh` deliberately +excludes the fallback: a dead original fails the refresh and restores the +prior first-hand snapshot. Mechanics in Wayback Machine — non-blocking. + +**Bibliography annotation implemented (2026-06-10).** The Phase 3 follow-up +is done: bibliography entries now carry the same archive affordance (and +rotted-link flip) as body links. The gating CSL-URL check passed — +citeproc renders entry URLs as `Link` inlines — so `Filters.Archive` exports +`annotateBlock` and `Citations.hs` applies it after `enhanceEntry` in both +`renderBibDiv` (essay bibliographies) and `renderBibliographyHtml` +(`/bibliography/` pages). Verified on the rendered SIMD essay, including the +DOI-cited FIPS 203 entry resolving through its manifest alias. Details under +Link annotation — Bibliography. + +**Search-UI archive filter implemented (2026-06-10).** The last deferred +refinement with a code-shaped fix: the search page's filter panel gains an +"archive" mode (exclude / only) and a "link status" multi-select (live / +moved / rotted / error), backed by a new `data/archive-meta.json` emitted by +`Archive.hs` — the archive analogue of `epistemic-meta.json`. The collision +with the epistemic `status` filter is resolved by scoping (own state fields, +button classes, and labels), not renaming. Verified in headless Chrome. +Details in the Phase 5 Met note. + --- ## Motivation @@ -388,6 +463,8 @@ auto-derived. # slug: auto-derived → arxiv-2403-12345 (override only to disambiguate) # title: auto-derived from the artifact / popup-proxy metadata # type: auto-detected (pdf | html) + aliases: # optional — equivalent URLs that URL + - "https://doi.org/10.48550/arXiv.2403.12345" # normalisation cannot derive tags: [research/ml] # optional — same slash-hierarchy as content note: > # optional — why this is referenced Cited in the scaling-laws essay; section 4 is the load-bearing part. @@ -407,6 +484,7 @@ auto-derived. | `slug` | no | Override the auto-derived slug. Must be unique. | | `title` | no | Override the auto-derived title. | | `type` | no | `pdf` \| `html`. Auto-detected from `Content-Type` / extension. | +| `aliases` | no | Equivalent URLs of the same work that normalisation cannot derive — above all a DOI form vs. the landing URL it resolves to. Matching metadata, not identity: edit freely, no `refresh` needed. Each must be unique across the manifest and absent from `removed.yaml`; both `archive.py` and the direct-build validator enforce this. | | `tags` | no | Slash-hierarchy tags (`Tags.hs`). Place the work on tag indexes. | | `note` | no | Author's reason for archiving; shown on the archive page. | | `visibility` | no | `public` (default) or `private`. | @@ -456,6 +534,12 @@ immediately-prior snapshot's hash, so the last prior snapshot is reachable the artifact**, not in a rolling global file, so the immutable claim is genuinely immutable in git history. +One optional field: `fetched-from`, present only when the original was +already dead at first fetch and the snapshot's bytes came from a Wayback +capture instead (see Wayback Machine — non-blocking). It holds the exact raw +(`id_`) capture URL that was fetched; its absence means a first-hand fetch +from the original. + ### Mutable state — `data/archive-state.json` Written **only** by `tools/archive.py check`. Holds the volatile link-rot @@ -745,7 +829,33 @@ separately, POSTs the outstanding URLs to `https://web.archive.org/save/` returned timestamped URL into each `PROVENANCE.json`. This second, independent copy means a rotted entry whose local artifact is somehow lost still has a fallback. If the original is *already* dead at first fetch, `archive.py fetch` -pulls the most recent existing Wayback capture instead. +pulls the most recent existing Wayback capture instead (implemented +2026-06-10): + +- **Dead means dead** — the fallback fires only when the document itself + could not be retrieved (DNS failure, refused connection, timeout, HTTP + error). A `noarchive` refusal, the size cap, or a local/tooling failure + never falls through to a third-party copy. +- **Raw bytes via `id_`** — the capture is fetched in its + `…/web/{timestamp}id_/{url}` form, which replays the original response + bytes without the Wayback toolbar or link rewriting; the normal pipeline + (size cap, CSP + noindex injection, quality classification) applies. +- **Preserved directives are honoured** — Wayback replays the original's + response headers as `X-Archive-Orig-*`; a capture whose + `X-Archive-Orig-X-Robots-Tag` carries `noarchive` is refused, since the + dead original can no longer be asked directly. +- **Honest provenance** — `PROVENANCE.json` gains an optional + `fetched-from` field holding the exact raw capture URL, so the record + never implies a first-hand fetch that did not happen; `wayback` is set + to the capture immediately (so `archive-wayback` correctly skips the + entry — a dead URL cannot be re-submitted). +- **Lookup only** — the fallback queries the availability API; it never + creates third-party state. Note the API has blind spots (it reports no + capture for some URLs that arguably have one, e.g. certain large PDFs); + a "no capture" skip is retried on the next build like any other skip. +- **Never during `refresh`** — a deliberate re-snapshot of a dead original + fails and restores the prior first-hand snapshot rather than silently + downgrading it to third-party bytes. ### Politeness & safety @@ -866,6 +976,15 @@ So `archive.py` computes the equivalent-URL set per entry and stores it as `Backlinks.hs` matches an incoming link against any alias before keying it to the archive URL. +Forms that no offline normalisation can derive — above all a **DOI vs. the +landing URL it resolves to** — are *authored*: the manifest's `aliases:` field +lists equivalent URLs per entry. Authored aliases join the entry's alias set +in `archive-index.json` (each with its own generated expansions, so a DOI +alias's `http://` form matches like the canonical's would), they count as +"covered" in `archive.py suggest`, and they are validated like canonical URLs: +unique across the manifest and absent from `removed.yaml`, enforced by both +the `archive.py fetch` pre-scan and `Archive.hs`'s direct-build validator. + ### Granular backlinks (Phase 4 refinement) If a citation targets a fragment — `…/abs/2403.12345#section-4`, or a PDF page @@ -967,13 +1086,25 @@ This does **not** put the broken popup layer on the critical path, as the draft feared. `Citations.hs` already performs AST surgery on each bibliography entry (`enhanceEntry` — it wraps `file:` PDF links and appends keyword strips), so the realistic annotation hook is `enhanceEntry`, reusing `Filters.Archive`'s -index lookup — no popup dependency. That is **deferred to a Phase 3 follow-up**: -it first needs a check that `chicago-notes.csl` renders a cited work's +index lookup — no popup dependency. That was **deferred to a Phase 3 +follow-up** pending a check that `chicago-notes.csl` renders a cited work's `url`/`doi` as a `Link` node (a CSL style that omits URLs would leave nothing -to match). Phase 3 ships prose-link annotation; bibliography annotation is -documented as in-scope and hookable via `enhanceEntry`, pending that check. A -future popup rewrite may *also* consult `archive-index.json`, but the archive -system depends on neither the current nor a future popup implementation. +to match). A future popup rewrite may *also* consult `archive-index.json`, but +the archive system depends on neither the current nor a future popup +implementation. + +**Implemented (2026-06-10).** The CSL-URL check passed empirically: citeproc +with `chicago-notes.csl` renders entry URLs as real `Link` inlines. +`Filters.Archive` now exports `annotateBlock` — the same annotation pass +(affordance when live, primary-link flip when `rotted`), minus the header +protection bibliography entries don't need — and `Citations.hs` composes it +after `enhanceEntry` at both rendering sites: essay bibliographies +(`renderBibDiv`) and the synthetic `/bibliography/` pages +(`renderBibliographyHtml`). It runs *after* `enhanceEntry` so the PDF-link +title wrap sees the entry's original inline shape. Verified on the rendered +SIMD essay: the FIPS 203 entry (cited via its DOI — resolved through the +manifest `aliases:` field) and the `cr.yp.to/aes-speed.html` entry both carry +the affordance, correctly relativized. --- @@ -982,6 +1113,15 @@ system depends on neither the current nor a future popup implementation. `tools/archive.py check` issues a `HEAD` (falling back to a ranged `GET`) to every original URL in the manifest and updates `data/archive-state.json`. +**Offline guard (canary).** Before probing any target, `check` probes +`https://levineuwirth.org/` itself. If the canary is unreachable, the machine +is offline (or DNS is down) and no probe result would be evidence about the +*targets* — the scan is declared inconclusive, the state file is left +untouched, and the run exits 0. Without this, an unattended scheduled scan on +a machine left offline would record a `fail` against every entry, and three +such scans spanning two weeks would flip the entire archive to `rotted` at +once. An empty manifest skips the canary too: no network I/O at all. + **Hysteresis is asymmetric.** Rotting is slow; recovery is fast. - *Rotting.* A failed probe increments `consecutive-failures` and sets @@ -1011,10 +1151,30 @@ clicks through to a working local snapshot instead of a 404, with no manual intervention — and only after the rot is confirmed, not guessed. `check` is a slow network job, not something every `make build` should pay for. -It runs on its own cadence — a periodic local `make archive-check`, or a -scheduled remote agent. It is decoupled from the main build: the build consumes +It runs on its own cadence, decoupled from the main build: the build consumes whatever `archive-state.json` exists. +**Scheduling (installed 2026-06-10).** A systemd *user* timer runs the scan +daily on the build machine — the state file is local and gitignored, so the +scan must run where builds happen, not on the VPS. The units live in the repo +at `systemd/archive-check.{service,timer}` and are installed as symlinks, so +the repo stays the source of truth (edits apply after +`systemctl --user daemon-reload`): + +``` +systemctl --user link ~/Repos/levineuwirth.org/systemd/archive-check.service +systemctl --user enable --now ~/Repos/levineuwirth.org/systemd/archive-check.timer +``` + +Cadence: daily at 12:00 (±30 min jitter) gives the hysteresis its minimum +detection latency — 14 days — at one polite `HEAD` per cited host per day. +`Persistent=true` fires a missed scan at the next login; the canary guard +makes an offline fire harmless. The timer only runs while a session is up +(`Linger=no`); `loginctl enable-linger` would lift that, but with +`Persistent=true` and a 14-day rot floor it is not needed. Rendering remains +pull-based by design: a status flip reaches the live site at the next +`make deploy` (always a clean build), and `/build/` shows the current tallies. + --- ## Build-pipeline integration @@ -1351,7 +1511,9 @@ field; the hook is `Citations.hs`'s `enhanceEntry`, pending a CSL-URL check — not popup-gated) and **pull-from-Wayback when the original is dead at fetch time** (it belongs with Phase 5 link-rot detection, where a dead URL is the central case and a Wayback-sourced artifact's provenance can be handled -properly). The live `make archive-wayback` run is author-initiated — it submits +properly). *(Both implemented 2026-06-10 — see the Bibliography note under +Link annotation, and Wayback Machine — non-blocking.)* The live +`make archive-wayback` run is author-initiated — it submits public captures to a third-party service. ### Phase 4 — Backlinks & similar-pages indexing @@ -1470,17 +1632,25 @@ verification the author runs (or a CI cron); the hysteresis logic itself is unit-tested deterministically in `next_state`, and the rendering side is verified by the hand-crafted `rotted` state file. -**Search-UI filter (`search-filters.js`) — partial.** The data-side is in -place: every archive page carries `data-pagefind-filter="type:archive, -status:$status$"`, so Pagefind's filter index now distinguishes archive hits -by rot status and (when @pagefind-ui@ is configured to show filters) lists -them as a filterable facet. The remaining work — wiring a custom UI control -into `search-filters.js` — is a deliberate refinement, not done in Phase 5: -its existing `status` filter is reserved for *epistemic* status (working -model / drafting / etc.) sourced from `data/epistemic-meta.json`, so adding an -archive `status` dimension needs a name to avoid the collision plus new -filter-panel buttons. Search-UX best iterated with the live page in front of -the author. +**Search-UI filter (`search-filters.js`) — implemented (2026-06-10).** The +data-side was in place since Phase 5 (every archive page carries +`data-pagefind-filter="type:archive, status:$status$"`); the deferred UI +work is now done, with the naming collision resolved by *scoping, not +renaming*: the epistemic `status` filter keeps its name and namespace, and +the archive dimension lives in its own state fields (`archiveMode`, +`archiveStatus`), button classes (`filter-archive-mode-btn`, +`filter-archive-status-btn`), and panel rows ("archive", "link status"). +Mechanics follow the established epistemic pattern rather than the Pagefind +filter API: `Archive.hs` emits `data/archive-meta.json` (routed page path → +link-rot status, the analogue of `data/epistemic-meta.json`), which +`search-filters.js` fetches lazily and applies to rendered results — both +Pagefind and semantic. Semantics: `archiveMode` is a single-select toggle — +`exclude` hides `/archive/` results, `only` shows nothing else; `link +status` is a multi-select that further restricts *archive* results to the +chosen statuses and never affects native content; both compose freely with +the epistemic filters. Verified in headless Chrome against a harness with a +simulated `rotted` entry: exclude / only / rotted-only / rotted+draft all +behave per the table above. --- diff --git a/Makefile b/Makefile index 7bf8350..7367220 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: build deploy sign download-model download-pdfjs download-leaflet compress-assets convert-images pdf-thumbs pdfs watch clean dev audit-marks archive-gc archive-wayback archive-check +.PHONY: build deploy sign download-model download-pdfjs download-leaflet compress-assets convert-images pdf-thumbs pdfs watch clean dev audit-marks archive-gc archive-wayback archive-check archive-suggest # deploy's prerequisite order (clean -> build -> sign) is only correct # serially; under `make -j` they could interleave. This build has no @@ -218,6 +218,17 @@ archive-wayback: python3 tools/archive.py wayback; \ fi +# Print works cited in data/*.bib but not yet archived, as manifest-ready +# lines the author copies by hand. Read-only — it never edits the manifest +# (bibliography auto-seeding is rejected by design; see ARCHIVE.md). +# Offline: scans local files only, no network. +archive-suggest: + @if [ -d .venv ]; then \ + uv run python tools/archive.py suggest; \ + else \ + python3 tools/archive.py suggest; \ + fi + # Probe every archived URL for link rot, updating data/archive-state.json. # A slow network job — opt-in, never run by `make build`. Asymmetric # hysteresis: `rotted` needs 3 consecutive failures over >=14 days; a diff --git a/archive/manifest.yaml b/archive/manifest.yaml index 781b853..d03701a 100644 --- a/archive/manifest.yaml +++ b/archive/manifest.yaml @@ -11,6 +11,10 @@ slug: nist-fips-203 title: "FIPS 203 — Module-Lattice-Based Key-Encapsulation Mechanism Standard" type: pdf + aliases: + # The DOI form the simd paper cites; URL normalisation cannot equate + # a DOI with the landing URL it resolves to, so it is authored here. + - "https://doi.org/10.6028/NIST.FIPS.203" tags: [research] note: > The ML-KEM standard. Cited in the SIMD / post-quantum systems work; diff --git a/build/Archive.hs b/build/Archive.hs index 9718011..30a423b 100644 --- a/build/Archive.hs +++ b/build/Archive.hs @@ -27,7 +27,7 @@ module Archive (archiveRules, archiveBuildStats) where import Control.Exception (SomeException, catch) -import Control.Monad (filterM, forM, when) +import Control.Monad (filterM, forM, forM_, when) import Data.Function (on) import Data.List (groupBy, intercalate, sort, sortBy) import qualified Data.Map.Strict as Map @@ -39,6 +39,7 @@ import Data.Time (Day, diffDays, fromGregorian, getCurrentTime, utctDay) import qualified Data.Aeson as A import Data.Aeson ((.:), (.:?)) +import qualified Data.ByteString.Lazy.Char8 as LBS import qualified Data.Yaml as Y import System.Directory (doesDirectoryExist, doesFileExist, listDirectory) @@ -63,6 +64,9 @@ import ArchiveIndex (ArchiveStatus (..), statusName, -- Phase 4) and need no Haskell-side binding. data ManifestEntry = ManifestEntry { meUrl :: String + , meAliases :: [String] -- ^ authored equivalent URLs of the + -- same work (e.g. its DOI form); + -- matching metadata, not identity , meNote :: Maybe String , mePaywalled :: Bool , meVisibility :: String -- ^ "public" (default) | "private" @@ -71,6 +75,7 @@ data ManifestEntry = ManifestEntry instance A.FromJSON ManifestEntry where parseJSON = A.withObject "ManifestEntry" $ \o -> do url <- o .: "url" + aliases <- fromMaybe [] <$> o .:? "aliases" note <- o .:? "note" paywalled <- fromMaybe False <$> o .:? "paywalled" visibility <- fromMaybe "public" <$> o .:? "visibility" @@ -81,7 +86,7 @@ instance A.FromJSON ManifestEntry where "manifest entry " ++ url ++ ": visibility must be \"public\" or \"private\", got " ++ show visibility - return (ManifestEntry url note paywalled visibility) + return (ManifestEntry url aliases note paywalled visibility) newtype RemovedEntry = RemovedEntry { reUrl :: String } @@ -197,21 +202,30 @@ validateManifestEntries manifest removed = go Map.empty manifest where go _ [] = return () go seen (entry : rest) = do - let url = meUrl entry - norm = normalizeUrl (T.pack url) - when (norm `Set.member` removed) $ do - hPutStrLn stderr $ - "[archive] FATAL: manifest URL " ++ show url - ++ " is also recorded in removed.yaml; refusing to publish " - ++ "a deliberately removed work." - exitFailure - case Map.lookup norm seen of - Just prior -> do + -- The canonical URL and every authored alias must be unique + -- across the manifest and absent from removed.yaml — the same + -- equivalence the Python pre-scan enforces, so a direct Hakyll + -- build fails just as closed. Within-entry duplicates (an alias + -- normalising to its own entry's URL) are deduped, not errors. + let url = meUrl entry + norms = Set.toList . Set.fromList $ + map (normalizeUrl . T.pack) (url : meAliases entry) + forM_ norms $ \norm -> do + when (norm `Set.member` removed) $ do hPutStrLn stderr $ - "[archive] FATAL: manifest URLs " ++ show prior ++ " and " - ++ show url ++ " normalise to the same archive target." + "[archive] FATAL: manifest entry " ++ show url + ++ " matches removed.yaml (directly or via `aliases:`); " + ++ "refusing to publish a deliberately removed work." exitFailure - Nothing -> go (Map.insert norm url seen) rest + case Map.lookup norm seen of + Just prior -> do + hPutStrLn stderr $ + "[archive] FATAL: manifest entries " ++ show prior + ++ " and " ++ show url ++ " normalise to the same " + ++ "archive target (directly or via `aliases:`)." + exitFailure + Nothing -> return () + go (foldr (\n m -> Map.insert n url m) seen norms) rest -- | Scan @archive//PROVENANCE.json@ into a @url -> (slug, Provenance)@ -- map. The directory name is the slug; the join key is the URL. @@ -349,6 +363,26 @@ archiveRules = do mapM_ archiveEntryRule entries archiveIndexRule entries + archiveMetaRule entries + +-- | @data/archive-meta.json@ — routed page path -> link-rot status, the +-- client-side manifest behind the search page's archive filter (same +-- pattern as @data/epistemic-meta.json@). Keys use the routed +-- @.../index.html@ form to match @search-filters.js@'s @normUrl@. +-- Named @archive-meta@, and its field @status@ scoped under it, so the +-- epistemic @status@ filter namespace is untouched. +archiveMetaRule :: [ArchiveEntry] -> Rules () +archiveMetaRule entries = + create ["data/archive-meta.json"] $ do + route idRoute + compile $ do + _ <- loadAll "archive/*/PROVENANCE.json" :: Compiler [Item String] + let metaMap = Map.fromList + [ ( "/archive/" ++ pvSlug (aeProv e) ++ "/index.html" + , Map.singleton ("status" :: String) + (statusName (aeStatus e)) ) + | e <- entries ] + makeItem (LBS.unpack (A.encode metaMap)) -- | One @/archive//@ page. archiveEntryRule :: ArchiveEntry -> Rules () diff --git a/build/ArchiveIndex.hs b/build/ArchiveIndex.hs index ad810e6..6156ef4 100644 --- a/build/ArchiveIndex.hs +++ b/build/ArchiveIndex.hs @@ -124,6 +124,13 @@ readUrlSet path = do Left e -> ioError . userError $ "[archive] FATAL: " ++ path ++ ": " ++ show e +-- | Normalised URLs recorded as deliberate takedowns in @removed.yaml@. +-- Consulted independently of 'activeUrls': a takedown must also knock +-- out any *alias* key a stale index still carries for another entry. +{-# NOINLINE removedUrls #-} +removedUrls :: Set Text +removedUrls = unsafePerformIO (readUrlSet removedPath) + -- | Canonical URLs still permitted to participate in link annotation. -- Filtering the generated index at build time makes a direct Hakyll build -- respect authored manifest/removal state even when archive.py did not run. @@ -131,8 +138,7 @@ readUrlSet path = do activeUrls :: Set Text activeUrls = unsafePerformIO $ do manifest <- readUrlSet manifestPath - removed <- readUrlSet removedPath - return (manifest `Set.difference` removed) + return (manifest `Set.difference` removedUrls) -- | @canonical-url -> entry@. Absent/malformed file -> empty; entries no -- longer permitted by the authored manifest/removal state are removed. @@ -164,13 +170,18 @@ rawState = unsafePerformIO $ do -- @archive-index.json@, each fed through 'normalizeUrl'. Both keys and -- lookups are normalised, so a citation form the alias set cannot -- enumerate (e.g. an unbounded arXiv version, or any tracking-laden --- variant of a clean manifest URL) still resolves. +-- variant of a clean manifest URL) still resolves. Alias keys that +-- match a recorded takedown are dropped, so a stale index (archive.py +-- not run since removed.yaml changed) cannot keep annotating a removed +-- work through an alias of a still-active entry. {-# NOINLINE flatIndex #-} flatIndex :: Map Text String flatIndex = Map.fromList - [ (normalizeUrl key, ieSlug e) + [ (nkey, ieSlug e) | (canon, e) <- Map.toList rawIndex , key <- canon : ieAliases e + , let nkey = normalizeUrl key + , nkey `Set.notMember` removedUrls ] -- | @slug -> status@: each entry's status, looked up by its canonical URL diff --git a/build/Citations.hs b/build/Citations.hs index 3850936..bc14131 100644 --- a/build/Citations.hs +++ b/build/Citations.hs @@ -44,6 +44,7 @@ import Text.Pandoc.Citeproc (processCitations) import Text.Pandoc.Walk import BibExtras (BibExtra (..), emptyBibExtra, parseBibExtras) +import qualified Filters.Archive (annotateBlock) -- --------------------------------------------------------------------------- @@ -96,7 +97,7 @@ renderBibliographyHtml bibPaths extras keys = do processed <- runIOorExplode $ processCitations doc let refsDivs = concatMap unwrapRefs (pandocBlocks processed) ordered = reorderByKeys keys refsDivs - enhanced = map (enhanceEntry extras) ordered + enhanced = map (annotateArchive . enhanceEntry extras) ordered return (renderEntries "csl-bib-body" enhanced) where pandocBlocks (Pandoc _ bs) = bs @@ -263,7 +264,7 @@ extractBibliography extras citeOrder frKeys blocks = -- @\
@ appended when @keywords:@ is set. renderBibDiv :: Map String BibExtra -> [Text] -> [Text] -> Block -> (Text, Text) renderBibDiv extras citeOrder _frKeys (Div _ children) = - let enhanced = map (enhanceEntry extras) children + let enhanced = map (annotateArchive . enhanceEntry extras) children keyIndex = Map.fromList (zip citeOrder [0 :: Int ..]) (citedEntries, furtherEntries) = partition (isCited keyIndex) enhanced @@ -281,6 +282,16 @@ renderBibDiv _ _ _ _ = ("", "") -- Bib entry enhancement (Phase 6a) -- --------------------------------------------------------------------------- +-- | Bibliography-side archive annotation: the same affordance (and +-- rotted-link flip) 'Filters.Archive' gives body links, applied to a +-- rendered CSL entry. The bibliography never travels the body filter +-- chain — it is extracted and rendered to an HTML string for the +-- template's @$bibliography$@ field — so the entry Divs are annotated +-- here, after 'enhanceEntry' (whose PDF-link title wrap must see the +-- entry's original inline shape). +annotateArchive :: Block -> Block +annotateArchive = Filters.Archive.annotateBlock + -- | Augment a single @csl-entry@ Div with the custom fields we parsed -- from the .bib file. Other Blocks pass through unchanged. enhanceEntry :: Map String BibExtra -> Block -> Block diff --git a/build/Filters/Archive.hs b/build/Filters/Archive.hs index 872b0b4..91bb90d 100644 --- a/build/Filters/Archive.hs +++ b/build/Filters/Archive.hs @@ -22,7 +22,12 @@ -- -- No-op when @data/archive-index.json@ is absent. When no rot scan has -- run, every entry is 'Live' — no link is ever flipped. -module Filters.Archive (apply) where +-- +-- 'annotateBlock' exposes the same pass for rendered blocks that never +-- travel the body filter chain — @Citations@ applies it to each +-- CSL-rendered bibliography entry, so a bibliography URL gets the same +-- affordance (and the same rotted-link flip) as a body link. +module Filters.Archive (apply, annotateBlock) where import qualified Data.Text as T import Text.Pandoc.Definition @@ -42,6 +47,15 @@ apply doc | otherwise = walk unprotectLink . walk annotateInlines . walk protectHeader $ doc +-- | The annotation pass for a single already-rendered block, outside the +-- body filter chain. No header protection — the callers' blocks +-- (CSL bibliography entries) contain none. Identity when the index is +-- absent. +annotateBlock :: Block -> Block +annotateBlock b + | archiveIndexIsEmpty = b + | otherwise = walk annotateInlines b + -- | Sentinel class marking a link the annotation walk must skip. It -- only exists between the protect and unprotect walks inside 'apply'. skipClass :: T.Text diff --git a/content/search.md b/content/search.md index b07f7f1..d9e6e93 100644 --- a/content/search.md +++ b/content/search.md @@ -102,6 +102,22 @@ Filters
+
+archive +
+ + +
+
+
+link status +
+ + + + +
+
diff --git a/static/js/search-filters.js b/static/js/search-filters.js index 83d03a7..51a4b4b 100644 --- a/static/js/search-filters.js +++ b/static/js/search-filters.js @@ -1,9 +1,15 @@ -/* search-filters.js — Epistemic effort filters for the search page. +/* search-filters.js — Epistemic effort + archive filters for the search page. * - * Loads /data/epistemic-meta.json (a map of URL → epistemic fields) + * Loads /data/epistemic-meta.json (a map of URL → epistemic fields) and + * /data/archive-meta.json (a map of /archive/ page URL → link-rot status) * and hides search results whose source page doesn't match the active * filters. Works for both Pagefind keyword results and semantic results. * + * Naming: the epistemic `status` filter (draft / working model / …) and + * the archive link status (live / moved / rotted / error) are distinct + * dimensions — the latter lives under `archiveMode` / `archiveStatus` + * and its own button classes, never touching the epistemic namespace. + * * Reuses the same CSS classes and filter-panel markup as library.html * so the two pages look and behave identically. */ @@ -28,10 +34,13 @@ scope: null, novelty: null, practicality: null, - stability: null + stability: null, + archiveMode: null, /* null | 'exclude' | 'only' */ + archiveStatus: [] /* link-rot statuses; archive results only */ }; var epistemicMeta = null; /* URL → {status, confidence, …} loaded lazily */ + var archiveMeta = null; /* /archive/ URL → {status: live|moved|…} */ /* ---- Persistence ---- */ @@ -54,8 +63,9 @@ /* ---- Metadata loading ---- */ var metaPromise = null; + var archiveMetaPromise = null; - function loadMeta() { + function loadEpistemicMeta() { if (epistemicMeta) return Promise.resolve(epistemicMeta); if (metaPromise) return metaPromise; metaPromise = fetch('/data/epistemic-meta.json') @@ -65,6 +75,22 @@ return metaPromise; } + function loadArchiveMeta() { + if (archiveMeta) return Promise.resolve(archiveMeta); + if (archiveMetaPromise) return archiveMetaPromise; + archiveMetaPromise = fetch('/data/archive-meta.json') + .then(function (r) { return r.ok ? r.json() : {}; }) + .catch(function () { return {}; }) + .then(function (data) { archiveMeta = data; return data; }); + return archiveMetaPromise; + } + + /* Both maps load together: epistemicMeta doubles as the + "metadata is ready" marker in the apply/observer paths. */ + function loadMeta() { + return Promise.all([loadEpistemicMeta(), loadArchiveMeta()]); + } + /* ---- Filtering logic ---- */ function passes(meta) { @@ -100,8 +126,29 @@ return true; } + /* Archive dimension. `archiveMode` governs whether /archive/ pages + appear at all (exclude) or alone (only); `archiveStatus` further + restricts archive results to the selected link-rot statuses and + never affects non-archive results. */ + function isArchiveUrl(p) { + return !!p && p.indexOf('/archive/') === 0; + } + + function passesArchive(url) { + var isArch = isArchiveUrl(url); + if (state.archiveMode === 'exclude' && isArch) return false; + if (state.archiveMode === 'only' && !isArch) return false; + if (state.archiveStatus.length && isArch && archiveMeta) { + var rec = archiveMeta[url]; + var s = rec && rec.status; + if (!s || state.archiveStatus.indexOf(s) === -1) return false; + } + return true; + } + function hasActiveFilters() { if (state.status.length) return true; + if (state.archiveMode !== null || state.archiveStatus.length) return true; var fields = ['confidence', 'importance', 'evidence', 'score', 'scope', 'novelty', 'practicality', 'stability']; for (var i = 0; i < fields.length; i++) { @@ -144,7 +191,8 @@ if (!link) return; var url = normUrl(link.getAttribute('href')); var meta = url ? epistemicMeta[url] : null; - el.classList.toggle('search-filtered', !passes(meta)); + el.classList.toggle('search-filtered', + !(passesArchive(url) && passes(meta))); }); } @@ -160,7 +208,8 @@ if (!link) return; var url = normUrl(link.getAttribute('href')); var meta = url ? epistemicMeta[url] : null; - el.classList.toggle('search-filtered', !passes(meta)); + el.classList.toggle('search-filtered', + !(passesArchive(url) && passes(meta))); }); } @@ -176,6 +225,8 @@ function activeCount() { var n = 0; if (state.status.length) n++; + if (state.archiveMode !== null) n++; + if (state.archiveStatus.length) n++; var fields = ['confidence', 'importance', 'evidence', 'score', 'scope', 'novelty', 'practicality', 'stability']; for (var i = 0; i < fields.length; i++) { @@ -193,6 +244,14 @@ btn.classList.toggle('is-active', state.status.indexOf(btn.dataset.value) !== -1); }); + document.querySelectorAll('.filter-archive-mode-btn').forEach(function (btn) { + btn.classList.toggle('is-active', state.archiveMode === btn.dataset.value); + }); + + document.querySelectorAll('.filter-archive-status-btn').forEach(function (btn) { + btn.classList.toggle('is-active', state.archiveStatus.indexOf(btn.dataset.value) !== -1); + }); + var ci = document.getElementById('filter-confidence'); if (ci) ci.value = state.confidence !== null ? state.confidence : ''; var si = document.getElementById('filter-score'); @@ -247,6 +306,26 @@ }); }); + /* Archive mode buttons (exclude / only) — single-select toggle */ + document.querySelectorAll('.filter-archive-mode-btn').forEach(function (btn) { + btn.addEventListener('click', function () { + var v = btn.dataset.value; + state.archiveMode = (state.archiveMode === v) ? null : v; + loadMeta().then(applyFilters); + }); + }); + + /* Archive link-status buttons — multi-select, archive results only */ + document.querySelectorAll('.filter-archive-status-btn').forEach(function (btn) { + btn.addEventListener('click', function () { + var v = btn.dataset.value; + var i = state.archiveStatus.indexOf(v); + if (i === -1) state.archiveStatus.push(v); + else state.archiveStatus.splice(i, 1); + loadMeta().then(applyFilters); + }); + }); + /* Threshold buttons (importance, evidence) */ document.querySelectorAll('.filter-threshold-btn').forEach(function (btn) { btn.addEventListener('click', function () { @@ -296,6 +375,8 @@ state.novelty = null; state.practicality = null; state.stability = null; + state.archiveMode = null; + state.archiveStatus = []; applyFilters(); }); } diff --git a/systemd/archive-check.service b/systemd/archive-check.service new file mode 100644 index 0000000..bd6afde --- /dev/null +++ b/systemd/archive-check.service @@ -0,0 +1,20 @@ +# levineuwirth.org link-rot scan — runs `make archive-check` in the repo. +# +# Installed as a symlink into this repo (so edits here take effect after +# `systemctl --user daemon-reload`): +# +# systemctl --user link ~/Repos/levineuwirth.org/systemd/archive-check.service +# systemctl --user enable --now ~/Repos/levineuwirth.org/systemd/archive-check.timer +# +# The scan only updates the gitignored data/archive-state.json; the next +# `make deploy` consumes it. See ARCHIVE.md — Link-rot detection. + +[Unit] +Description=levineuwirth.org link-rot scan (archive.py check) +Documentation=file://%h/Repos/levineuwirth.org/ARCHIVE.md + +[Service] +Type=oneshot +WorkingDirectory=%h/Repos/levineuwirth.org +ExecStart=/usr/bin/make archive-check +Nice=10 diff --git a/systemd/archive-check.timer b/systemd/archive-check.timer new file mode 100644 index 0000000..93d2a40 --- /dev/null +++ b/systemd/archive-check.timer @@ -0,0 +1,18 @@ +# Daily trigger for archive-check.service. See that unit's header for +# the install commands. + +[Unit] +Description=Daily levineuwirth.org link-rot scan + +[Timer] +# Daily: rot needs 3 consecutive fails spanning >=14 days, so a daily +# scan gives the design's minimum detection latency (14 days) at one +# polite HEAD per cited host per day. Persistent=true fires a missed +# scan at the next login; the canary guard in archive.py makes a fire +# while offline harmless (scan skipped, state untouched). +OnCalendar=*-*-* 12:00 +RandomizedDelaySec=30min +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/tools/archive.py b/tools/archive.py index 845a9b3..2357c3f 100644 --- a/tools/archive.py +++ b/tools/archive.py @@ -15,13 +15,17 @@ Two artifact types: text extracted with BeautifulSoup. Subcommands: - fetch download missing artifacts, (re)generate sidecars + index + fetch download missing artifacts, (re)generate sidecars + index; + an original already dead at first fetch falls back to its most + recent existing Wayback capture (recorded as `fetched-from`) refresh deliberately re-snapshot a single entry, recording the prior SHA in the new PROVENANCE.json's `previous-sha256` wayback submit archived URLs to the Wayback Machine as a second, independent copy; backfill the capture URL into PROVENANCE.json check HEAD/GET-probe every manifest URL for link rot, updating data/archive-state.json with asymmetric hysteresis + suggest print works cited in data/*.bib (url / doi fields) but not in + the manifest, as manifest-ready lines; never edits the manifest gc delete archive// directories listed in archive/removed.yaml Failure policy: @@ -72,6 +76,12 @@ STATE_OUT = REPO_ROOT / "data" / "archive-state.json" ROT_FAILS = 3 # consecutive failed scans before `rotted` is considered ROT_DAYS = 14 # ... and the streak must also span at least this many days +# Probed before a link-rot scan; unreachable -> the scan is inconclusive +# (offline machine, dead DNS) and state is left untouched. Guards the +# unattended timer-run case: three offline scans spanning two weeks would +# otherwise flip every entry to `rotted` at once. +CHECK_CANARY = "https://levineuwirth.org/" + SIZE_CAP = 25 * 1024 * 1024 # 25 MB per-artifact cap TIMEOUT = 60 # seconds, per network request WAYBACK_TIMEOUT = 120 # seconds — Save Page Now is slow @@ -173,6 +183,25 @@ def entry_slug(entry: dict) -> str: return slug if slug else derive_slug(entry["url"]) +def entry_aliases(entry: dict) -> list[str]: + """The authored `aliases:` list of a manifest entry, validated. An + alias is an equivalent URL of the same work that no offline + normalisation can derive — e.g. its DOI form vs. the landing URL the + artifact was fetched from. Aliases are matching metadata, not + identity: editing them never requires a refresh (the index is + rewritten from the manifest on every fetch). Malformed values are + fatal — a typo'd alias would otherwise silently never match, and the + resulting affordance gap is invisible.""" + aliases = entry.get("aliases", []) + if not isinstance(aliases, list) or not all( + isinstance(a, str) and a.startswith(("http://", "https://")) + for a in aliases): + err(f"manifest entry {entry.get('url')!r}: `aliases:` must be a " + f"list of http(s) URLs, got {aliases!r}") + sys.exit(1) + return aliases + + # --------------------------------------------------------------------------- # Hashing / type detection # --------------------------------------------------------------------------- @@ -238,9 +267,13 @@ def detect_type(url: str, override) -> str | None: # PDF fetch + text extraction # --------------------------------------------------------------------------- -def fetch_pdf(url: str, dest: Path) -> bool: - """Download `url` to `dest`, enforcing the size cap. Returns True on - success. A partial / over-cap download leaves no file behind.""" +def fetch_pdf(url: str, dest: Path) -> str: + """Download `url` to `dest`, enforcing the size cap. Returns "ok" on + success, "dead" when the document itself could not be retrieved (DNS + failure, refused connection, timeout, HTTP error status — the cases + where a Wayback fallback is legitimate), or "skip" for a policy or + local failure (noarchive directive, size cap, disk) that a fallback + must not circumvent. A partial / over-cap download leaves no file.""" req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) tmp = dest.with_suffix(dest.suffix + ".part") try: @@ -250,7 +283,7 @@ def fetch_pdf(url: str, dest: Path) -> bool: robots = (resp.headers.get("X-Robots-Tag") or "").lower() if "noarchive" in robots: err(f"{url}: response carries X-Robots-Tag: noarchive — skipped") - return False + return "skip" total = 0 with tmp.open("wb") as fh: for chunk in iter(lambda: resp.read(1 << 16), b""): @@ -260,14 +293,18 @@ def fetch_pdf(url: str, dest: Path) -> bool: tmp.unlink(missing_ok=True) err(f"{url}: exceeds {SIZE_CAP // (1024*1024)} MB cap " f"— skipped (commit deliberately with `git add -f`)") - return False + return "skip" fh.write(chunk) tmp.replace(dest) - return True + return "ok" + except (urllib.error.URLError, TimeoutError) as exc: + tmp.unlink(missing_ok=True) + err(f"{url}: fetch failed — {exc}") + return "dead" # HTTPError is a URLError subclass except Exception as exc: # noqa: BLE001 — report any failure tmp.unlink(missing_ok=True) err(f"{url}: fetch failed — {exc}") - return False + return "skip" def extract_text_pdf(pdf: Path, txt: Path) -> None: @@ -388,10 +425,13 @@ def inject_archive_metas(path: Path) -> None: path.write_text(str(soup), encoding="utf-8") -def fetch_html(url: str, dest: Path) -> bool: +def fetch_html(url: str, dest: Path) -> str: """Snapshot an HTML page with monolith into a single self-contained - file at `dest`, then inject the archive CSP. Returns True on success; - every failure path is non-fatal (warn + skip).""" + file at `dest`, then inject the archive CSP. Returns "ok" on success, + "dead" when the source document itself could not be retrieved (the + Wayback-fallback-eligible case), or "skip" for every other failure — + noarchive directives, the size cap, monolith problems — none of which + a fallback may circumvent. Every failure path is non-fatal.""" # Honour directives returned by preliminary probes before performing # the document fetch. The full document response is inspected below # and is also the exact body passed to monolith; do not let monolith @@ -400,14 +440,14 @@ def fetch_html(url: str, dest: Path) -> bool: for h in (probe_headers(url), probe_headers_get(url))): err(f"{url}: response carries X-Robots-Tag: noarchive — skipped") - return False + return "skip" mono = find_monolith() if mono is None: err(f"{url}: monolith not found — vendor the binary at " f"tools/bin/monolith (see tools/monolith-version.txt) or set " f"$MONOLITH_BIN; HTML snapshot skipped") - return False + return "skip" verify_monolith(mono) source = dest.with_suffix(dest.suffix + ".source.part") @@ -419,7 +459,7 @@ def fetch_html(url: str, dest: Path) -> bool: robots = (resp.headers.get("X-Robots-Tag") or "").lower() if "noarchive" in robots: err(f"{url}: response carries X-Robots-Tag: noarchive — skipped") - return False + return "skip" effective_url = resp.geturl() total = 0 with source.open("wb") as fh: @@ -430,17 +470,21 @@ def fetch_html(url: str, dest: Path) -> bool: source.unlink(missing_ok=True) err(f"{url}: source HTML exceeds " f"{SIZE_CAP // (1024*1024)} MB cap — skipped") - return False + return "skip" fh.write(chunk) + except (urllib.error.URLError, TimeoutError) as exc: + source.unlink(missing_ok=True) + err(f"{url}: fetch failed — {exc}") + return "dead" # HTTPError is a URLError subclass except Exception as exc: # noqa: BLE001 source.unlink(missing_ok=True) err(f"{url}: fetch failed — {exc}") - return False + return "skip" if body_noarchive(source): source.unlink(missing_ok=True) err(f"{url}: response declares noarchive — skipped") - return False + return "skip" cmd = [mono, "--no-js", "--ignore-errors", "--quiet", "--timeout", str(TIMEOUT), "--user-agent", USER_AGENT, @@ -452,12 +496,12 @@ def fetch_html(url: str, dest: Path) -> bool: source.unlink(missing_ok=True) tmp.unlink(missing_ok=True) err(f"{url}: monolith timed out — skipped") - return False + return "skip" except Exception as exc: # noqa: BLE001 source.unlink(missing_ok=True) tmp.unlink(missing_ok=True) err(f"{url}: monolith failed to run — {exc}") - return False + return "skip" finally: source.unlink(missing_ok=True) @@ -467,21 +511,21 @@ def fetch_html(url: str, dest: Path) -> bool: tail = output.decode("utf-8", errors="replace").strip().splitlines() err(f"{url}: monolith exited {proc.returncode} " f"({tail[-1] if tail else 'no output'}) — skipped") - return False + return "skip" if not tmp.exists() or tmp.stat().st_size == 0: tmp.unlink(missing_ok=True) err(f"{url}: monolith produced no output — skipped") - return False + return "skip" if tmp.stat().st_size > SIZE_CAP: size_mb = tmp.stat().st_size // (1024 * 1024) tmp.unlink(missing_ok=True) err(f"{url}: snapshot is {size_mb} MB, over the " f"{SIZE_CAP // (1024*1024)} MB cap — skipped " f"(commit deliberately with `git add -f`)") - return False + return "skip" inject_archive_metas(tmp) tmp.replace(dest) - return True + return "ok" def extract_text_html(snapshot: Path, txt: Path) -> None: @@ -667,7 +711,57 @@ def _is_tracked_and_clean(*paths: Path) -> bool: # fetch subcommand # --------------------------------------------------------------------------- -def cmd_fetch() -> int: +_WAYBACK_CAPTURE_RE = re.compile( + r"^(https?://web\.archive\.org/web/)(\d{4,14})(/.+)$") + + +def wayback_raw_url(capture: str) -> str: + """The raw-bytes form of a Wayback capture URL: the `id_` flag after + the timestamp serves the original response bytes, without the Wayback + toolbar or link rewriting. An unexpected shape passes through.""" + m = _WAYBACK_CAPTURE_RE.match(capture) + return f"{m.group(1)}{m.group(2)}id_{m.group(3)}" if m else capture + + +def fetch_from_wayback(url: str, entry: dict, slug_dir: Path + ) -> tuple[str, Path, str, str] | None: + """Fetch-time fallback for an original that is already dead: pull the + most recent *existing* Wayback capture (raw bytes via `id_`). Returns + (atype, artifact_path, capture_url, raw_url), or None when no capture + exists or the capture fetch itself fails. Lookup only — a dead URL + cannot be newly captured, so the fallback never creates third-party + state.""" + capture = wayback_lookup(url) + if capture is None: + err(f"{url}: original unreachable and the Wayback Machine has no " + f"capture — skipped") + return None + raw = wayback_raw_url(capture) + # Wayback replays the original response headers as X-Archive-Orig-*; + # a preserved noarchive directive is still the publisher's word, and + # the dead original can no longer be asked directly. + preserved = (probe_headers(raw).get("x-archive-orig-x-robots-tag") + or "").lower() + if "noarchive" in preserved: + err(f"{url}: Wayback capture preserves X-Robots-Tag: noarchive " + f"— skipped") + return None + # Re-resolve the artifact type against the raw capture: the original + # is dead, so its own Content-Type probe degraded to the html + # default; the capture replays the stored Content-Type. + atype = detect_type(raw, entry.get("type")) + if atype is None: + return None + art = slug_dir / ARTIFACT[atype] + log(f"{url}: original dead — fetching Wayback capture {capture}") + result = fetch_pdf(raw, art) if atype == "pdf" else fetch_html(raw, art) + if result != "ok": + err(f"{url}: Wayback capture fetch failed — skipped") + return None + return (atype, art, capture, raw) + + +def cmd_fetch(wayback_fallback: bool = True) -> int: manifest = load_yaml_list(MANIFEST) # Removed URLs are compared in normalised form so a tracking-laden # variant cannot bypass a takedown the author already recorded. @@ -676,19 +770,39 @@ def cmd_fetch() -> int: # Pre-scan validation: reject canonical-form duplicates *before* any # fetch I/O, so a first colliding entry never gets partially processed - # while a second's duplicate check halts. + # while a second's duplicate check halts. The canonical URL and every + # authored alias must be unique across the whole manifest — a shared + # form would route one citation under two slugs. (An alias that + # normalises to its own entry's URL is merely redundant: deduped here, + # never an error.) seen: dict[str, str] = {} for entry in manifest: url = entry.get("url") if not url: continue - norm = normalize_url(url) - if norm in seen: - err(f"manifest: {url!r} and {seen[norm]!r} normalise to the " - f"same canonical form ({norm!r}). Drop one or distinguish " - f"them; the link archive cannot route both under one slug.") + keys = {normalize_url(url)} + keys |= {normalize_url(a) for a in entry_aliases(entry)} + for norm in sorted(keys): + if norm in seen: + err(f"manifest: {url!r} and {seen[norm]!r} share the " + f"canonical form {norm!r} (directly or via `aliases:`). " + f"Drop one or distinguish them; the link archive " + f"cannot route both under one slug.") + sys.exit(1) + seen[norm] = url + # A takedown recorded in removed.yaml is enforced against aliases + # too — re-listing a removed work as an alias of another entry + # would republish it under that entry's slug. (The canonical URL + # gets the same check, with fetch-specific wording, in the + # per-entry loop below.) + hit = keys & removed_norms + if hit: + err(f"manifest entry {url!r}: canonical form {sorted(hit)[0]!r} " + f"(direct or via `aliases:`) is recorded in " + f"archive/removed.yaml as a deliberate takedown. To " + f"re-archive it, remove the corresponding line from " + f"removed.yaml first.") sys.exit(1) - seen[norm] = url index: dict[str, dict] = {} skipped = 0 @@ -757,11 +871,25 @@ def cmd_fetch() -> int: sys.exit(1) # --- fetch the artifact if it is not already present -------------- + wb_capture: str | None = None # Wayback capture used, if any + wb_raw: str | None = None # ... and its raw id_ form if not art.exists(): slug_dir.mkdir(parents=True, exist_ok=True) log(f"fetching {url} [{atype}]") - ok = fetch_pdf(url, art) if atype == "pdf" else fetch_html(url, art) - if not ok: + result = (fetch_pdf(url, art) if atype == "pdf" + else fetch_html(url, art)) + if result == "dead" and wayback_fallback: + # The original is already gone at first fetch — pull the + # most recent existing Wayback capture instead. Only for + # *dead* originals: a noarchive refusal or an over-cap + # skip must never be circumvented via a third-party copy. + fb = fetch_from_wayback(url, entry, slug_dir) + if fb is not None: + atype, art, wb_capture, wb_raw = fb + txt = slug_dir / TEXTFILE[atype] + txt_stamp = slug_dir / (TEXTFILE[atype] + ".sha256") + result = "ok" + if result != "ok": skipped += 1 continue else: @@ -795,17 +923,35 @@ def cmd_fetch() -> int: "archived": datetime.date.today().isoformat(), "source-date": entry.get("source-date"), "snapshot-quality": quality, - "wayback": None, + # When the fallback fired, the capture is already known — + # cmd_wayback (which targets `wayback: null`) skips it, + # correctly: a dead original cannot be re-submitted. + "wayback": wb_capture, } + if wb_raw is not None: + # The snapshot's bytes came from the Wayback capture, not + # the (dead) original. Recorded so the provenance never + # implies a first-hand fetch that did not happen. + prov["fetched-from"] = wb_raw atomic_write_json(prov_path, prov) - log(f"{slug}: archived [{atype}, {quality}] ({prov['bytes']} bytes)") + origin = " via Wayback" if wb_raw else "" + log(f"{slug}: archived{origin} [{atype}, {quality}] " + f"({prov['bytes']} bytes)") # --- contribute to the Hakyll index ------------------------------- + # Generated equivalents of the canonical URL, plus the authored + # `aliases:` (each with its own generated equivalents) — so a DOI + # alias's http:// form matches just like the canonical's would. + alias_set = set(url_aliases(url)) + for authored in entry_aliases(entry): + alias_set.add(authored) + alias_set.update(url_aliases(authored)) + alias_set.discard(url) index[url] = { "slug": slug, "type": prov.get("type", atype), "title": prov.get("title", slug), - "aliases": url_aliases(url), + "aliases": sorted(alias_set), } # archive-index.json is always rewritten to mirror the manifest exactly. @@ -917,7 +1063,13 @@ def cmd_refresh(argv: list[str]) -> int: succeeded = False try: - rc = cmd_fetch() + # No Wayback fallback during a refresh: the author asked for a + # fresh first-hand snapshot. If the original turns out to be dead, + # the right outcome is "refresh fails, prior snapshot restored" — + # not a silent downgrade of a committed first-hand snapshot to + # third-party bytes. Adopting a Wayback copy stays a deliberate + # act (a new entry whose original is already dead). + rc = cmd_fetch(wayback_fallback=False) # Success requires a new PROVENANCE.json *and* its declared # artifact on disk. `cmd_fetch` returns 0 even when individual @@ -1134,14 +1286,23 @@ def cmd_check() -> int: except Exception: # noqa: BLE001 old = {} + to_probe = [e["url"] for e in manifest + if e.get("url") and normalize_url(e["url"]) not in removed_norms] + + # Offline guard: when the canary is unreachable, no probe result below + # is evidence about the *targets* — record nothing rather than a + # spurious `fail` against every entry. Exit 0: an offline laptop is + # not a unit failure; the scan simply retries on the next timer fire. + if to_probe and probe_url(CHECK_CANARY)[0] == "fail": + log(f"check: canary {CHECK_CANARY} unreachable — offline or DNS " + f"down; scan inconclusive, state left untouched") + return 0 + today = datetime.date.today() state: dict[str, dict] = {} tally = {"live": 0, "moved": 0, "error": 0, "rotted": 0} - for entry in manifest: - url = entry.get("url") - if not url or normalize_url(url) in removed_norms: - continue + for url in to_probe: result, new_url = probe_url(url) rec = next_state(old.get(url, {}), result, new_url, today) state[url] = rec @@ -1198,6 +1359,84 @@ def cmd_gc(ignore_orphans: bool) -> int: return 0 +# --------------------------------------------------------------------------- +# suggest subcommand +# --------------------------------------------------------------------------- + +_BIB_ENTRY_RE = re.compile(r"@(\w+)\s*\{\s*([^,\s{}]+)\s*,") +_BIB_FIELD_RE = re.compile( + r"""^\s*(url|doi)\s*=\s*[{"]\s*([^}"]+?)\s*["}]""", + re.IGNORECASE | re.MULTILINE) + + +def bib_citations(text: str) -> list[tuple[str, str]]: + """(citation-key, URL) pairs from one .bib file's text. Per entry the + `url` field wins; a DOI-only entry resolves to https://doi.org/{doi}. + @comment/@string/@preamble blocks carry no citation and are skipped.""" + out: list[tuple[str, str]] = [] + entries = list(_BIB_ENTRY_RE.finditer(text)) + for i, m in enumerate(entries): + kind, key = m.group(1).lower(), m.group(2) + if kind in ("comment", "string", "preamble"): + continue + end = entries[i + 1].start() if i + 1 < len(entries) else len(text) + fields = {f.group(1).lower(): f.group(2) + for f in _BIB_FIELD_RE.finditer(text[m.end():end])} + url = fields.get("url") + if not url and fields.get("doi"): + url = "https://doi.org/" + fields["doi"] + if url and url.startswith(("http://", "https://")): + out.append((key, url)) + return out + + +def cmd_suggest() -> int: + """Print works cited in data/*.bib but absent from the manifest, as + manifest-ready lines. Read-only by design: tools never write + manifest.yaml — the author reviews and copies lines by hand, so the + manifest stays the *identity* of the archive, not a .bib cache.""" + manifest = load_yaml_list(MANIFEST) + removed = load_yaml_list(REMOVED) + covered = {normalize_url(e["url"]) + for e in manifest + removed if e.get("url")} + for e in manifest: + covered.update(normalize_url(a) for a in entry_aliases(e)) + + bib_files = sorted((REPO_ROOT / "data").glob("*.bib")) + if not bib_files: + log("suggest: no data/*.bib files to scan") + return 0 + + # normalized URL -> first-seen verbatim URL + every citing entry, so + # one work cited from three papers prints once, with all three citers. + suggestions: dict[str, dict] = {} + for bib in bib_files: + for key, url in bib_citations(bib.read_text(encoding="utf-8")): + norm = normalize_url(url) + if norm in covered: + continue + s = suggestions.setdefault(norm, {"url": url, "cited": []}) + s["cited"].append(f"{bib.name}:{key}") + + scanned = ", ".join(b.name for b in bib_files) + if not suggestions: + log(f"suggest: nothing to add — every URL cited in {scanned} is " + f"already archived or deliberately removed") + return 0 + + log(f"suggest: {len(suggestions)} cited work(s) not in the manifest " + f"(scanned {scanned})") + print() + print("# Cited but not archived. Review each one, then copy the") + print("# entries you want into archive/manifest.yaml.") + for s in suggestions.values(): + print() + for citer in s["cited"]: + print(f"# cited by {citer}") + print(f'- url: "{s["url"]}"') + return 0 + + # --------------------------------------------------------------------------- # Entry point # --------------------------------------------------------------------------- @@ -1212,10 +1451,12 @@ def main(argv: list[str]) -> int: return cmd_wayback() if cmd == "check": return cmd_check() + if cmd == "suggest": + return cmd_suggest() if cmd == "gc": return cmd_gc(ignore_orphans="--ignore-orphans" in argv[1:]) err(f"unknown subcommand {cmd!r} " - f"(expected: fetch | refresh | wayback | check | gc)") + f"(expected: fetch | refresh | wayback | check | suggest | gc)") return 2