Archive: close out the audit-pass refinements
Six follow-ups from the link-archive audit, all the items with a
code-shaped fix. Behaviour verified per feature (stubbed + live where a
network or browser was needed); ARCHIVE.md status notes updated alongside.
- suggest: `archive.py suggest` + `make archive-suggest` print works
cited in data/*.bib (url wins, DOI-only resolves to doi.org/...) but
absent from the manifest, as manifest-ready lines. Read-only, offline.
- aliases: optional `aliases:` manifest field for equivalent URLs no
normalisation can derive (DOI vs. landing URL). Enforced like canonical
URLs on both sides (archive.py pre-scan + Archive.hs validator);
ArchiveIndex drops alias keys matching a takedown. FIPS 203 now carries
its DOI form, so the simd paper's DOI citation resolves.
- check scheduling: systemd user timer (systemd/archive-check.{service,
timer}, symlink-installed) runs the rot scan daily. cmd_check gains an
offline canary guard so an unattended scan on an offline machine leaves
state untouched instead of mass-flipping entries to rotted.
- Wayback fallback: an original already dead at first fetch falls back to
its most recent existing Wayback capture (raw id_ bytes through the
normal pipeline), honouring a preserved X-Archive-Orig noarchive and
recording `fetched-from`. Dead-only, never during refresh.
- bibliography annotation: Filters.Archive exports annotateBlock;
Citations.hs applies it to each CSL-rendered entry, so bibliography
links get the same affordance / rotted-flip as body links.
- search-UI filter: archive "exclude/only" + "link status" filters on the
search page, backed by a new data/archive-meta.json. Scoped apart from
the epistemic `status` filter (own state, classes, labels) rather than
renamed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
76bda7af13
commit
b856911279
222
ARCHIVE.md
222
ARCHIVE.md
|
|
@ -32,7 +32,8 @@ native viewer), symmetric with HTML snapshots — see the Display — PDF decisi
|
|||
appends an archive affordance to body links whose target is archived;
|
||||
`archive.py wayback` (+ `make archive-wayback`) backfills Wayback captures;
|
||||
`visibility: private` keeps an entry's artifact in-repo but undeployed.
|
||||
Bibliography annotation is documented as a `Citations.hs` follow-up.
|
||||
Bibliography annotation is documented as a `Citations.hs` follow-up
|
||||
*(implemented 2026-06-10 — see the Bibliography note under Link annotation)*.
|
||||
|
||||
**Phase 4 complete (2026-05-22).** Backlinks + similar-pages: `Backlinks.hs`
|
||||
keeps archived external links and canonicalises them to their `/archive/<slug>/`
|
||||
|
|
@ -49,11 +50,13 @@ needs 3 fails over ≥14 days; a single success recovers immediately).
|
|||
each archive page surfaces its link status (provenance row, header note,
|
||||
Pagefind `status` filter tag); `/archive/` flags rotted entries; `/build/`
|
||||
gains a "Link archive" telemetry section. The search-UI `status` filter wiring
|
||||
in `search-filters.js` is deliberately partial — see the Phase 5 Met note.
|
||||
in `search-filters.js` was deliberately partial in this phase *(completed
|
||||
2026-06-10 — see the Phase 5 Met note)*.
|
||||
|
||||
**All five phases done.** Refinements next; see the Phase 5 Met note for the
|
||||
documented deferrals (search-UI status filter; bibliography annotation from
|
||||
Phase 3; pull-from-Wayback at fetch time).
|
||||
documented deferrals (search-UI status filter, bibliography annotation from
|
||||
Phase 3, and pull-from-Wayback at fetch time — all three implemented
|
||||
2026-06-10).
|
||||
|
||||
**Refinements (2026-05-22).** A code-review pass found and fixed several
|
||||
correctness and posture issues across the system:
|
||||
|
|
@ -103,7 +106,8 @@ Three smaller items remain documented and deferred:
|
|||
own design pass (archive pages aren't `match`ed Hakyll items in the
|
||||
normal way).
|
||||
- **`archive.py suggest`** (bibliography discovery — diff `.bib` URLs
|
||||
against the manifest) is documented but not implemented.
|
||||
against the manifest) is documented but not implemented. *(Implemented
|
||||
2026-06-10 — see the status note below.)*
|
||||
- **The controlled-host end-to-end link-rot test** (reserve
|
||||
`archive-test.levineuwirth.org`, run it through a 14-day-spanning fail
|
||||
streak, watch the flip happen) is inherently a multi-week real-world
|
||||
|
|
@ -237,6 +241,77 @@ failure-closed paths:
|
|||
The genuinely-open questions that remain are collected at the end — the list is
|
||||
short.
|
||||
|
||||
**`archive.py suggest` implemented (2026-06-10).** The bibliography-discovery
|
||||
subcommand from the design is built: `make archive-suggest` scans `data/*.bib`
|
||||
for `url` / `doi` fields (per entry the `url` field wins; a DOI-only entry
|
||||
resolves to `https://doi.org/{doi}`), diffs them against `manifest.yaml` and
|
||||
`removed.yaml` under the same `normalize_url` equivalence the link-annotation
|
||||
filter uses, and prints manifest-ready `- url:` lines, each preceded by
|
||||
`# cited by {bibfile}:{key}` comments (a work cited from several papers prints
|
||||
once, with every citer). Read-only and offline: it never edits the manifest
|
||||
and performs no network I/O. Known equivalence gap, surfaced by the first real
|
||||
run: a work archived under its landing URL but cited via its DOI (e.g.
|
||||
FIPS 203 — archived as the `nvlpubs.nist.gov` PDF, cited as
|
||||
`doi.org/10.6028/NIST.FIPS.203`) is re-suggested, because nothing offline can
|
||||
equate the two forms. That is the same gap that denies those bibliography
|
||||
links an archive affordance, so the suggestion is honest — the fix, if wanted,
|
||||
is alias-level (manifest or `url_aliases`), not suppression in `suggest`.
|
||||
|
||||
**Manifest `aliases:` field implemented (2026-06-10).** The alias-level fix
|
||||
above: an optional authored `aliases:` list per manifest entry for equivalent
|
||||
URLs that normalisation cannot derive (the DOI ↔ landing-URL case). Aliases
|
||||
are matching metadata, not identity — editing them rewrites the index on the
|
||||
next fetch, no `refresh` involved. Enforcement mirrors canonical URLs on both
|
||||
sides of the build: the `archive.py fetch` pre-scan and `Archive.hs`'s
|
||||
`validateManifestEntries` each reject an alias that collides with another
|
||||
entry (directly or via aliases) or that matches a `removed.yaml` takedown,
|
||||
and `ArchiveIndex.flatIndex` additionally drops alias keys matching recorded
|
||||
takedowns so a stale index cannot keep annotating a removed work. `suggest`
|
||||
counts aliases as covered. Applied: `nist-fips-203` now carries
|
||||
`https://doi.org/10.6028/NIST.FIPS.203`, so the simd paper's DOI citation
|
||||
resolves to the archived PDF and `suggest` no longer re-suggests it.
|
||||
|
||||
**Link-rot scan scheduled (2026-06-10).** The scan now runs unattended: a
|
||||
systemd user timer (`systemd/archive-check.{service,timer}`, symlink-installed
|
||||
on the build machine) fires `make archive-check` daily. Going unattended
|
||||
exposed a hysteresis hazard hand-run scans masked — a machine left offline
|
||||
would record a `fail` against every entry, and three such scans spanning two
|
||||
weeks would flip the whole archive to `rotted` — so `cmd_check` gained a
|
||||
canary guard: `https://levineuwirth.org/` unreachable → scan inconclusive,
|
||||
state untouched, exit 0. Details in the Phase 5 section.
|
||||
|
||||
**Fetch-time Wayback fallback implemented (2026-06-10).** The design's
|
||||
"original already dead at first fetch → pull the most recent existing Wayback
|
||||
capture" promise is now real. `fetch_pdf`/`fetch_html` classify their
|
||||
failures (`ok` / `dead` / `skip`), and only a *dead* original — not a
|
||||
`noarchive` refusal, cap skip, or tooling failure — falls through to
|
||||
`fetch_from_wayback`, which pulls the raw `id_` capture bytes through the
|
||||
normal pipeline, honours a preserved `X-Archive-Orig-X-Robots-Tag:
|
||||
noarchive`, re-detects the artifact type against the capture (the dead
|
||||
original's Content-Type probe degrades to the html default), and records
|
||||
`fetched-from` + `wayback` in `PROVENANCE.json`. `refresh` deliberately
|
||||
excludes the fallback: a dead original fails the refresh and restores the
|
||||
prior first-hand snapshot. Mechanics in Wayback Machine — non-blocking.
|
||||
|
||||
**Bibliography annotation implemented (2026-06-10).** The Phase 3 follow-up
|
||||
is done: bibliography entries now carry the same archive affordance (and
|
||||
rotted-link flip) as body links. The gating CSL-URL check passed —
|
||||
citeproc renders entry URLs as `Link` inlines — so `Filters.Archive` exports
|
||||
`annotateBlock` and `Citations.hs` applies it after `enhanceEntry` in both
|
||||
`renderBibDiv` (essay bibliographies) and `renderBibliographyHtml`
|
||||
(`/bibliography/` pages). Verified on the rendered SIMD essay, including the
|
||||
DOI-cited FIPS 203 entry resolving through its manifest alias. Details under
|
||||
Link annotation — Bibliography.
|
||||
|
||||
**Search-UI archive filter implemented (2026-06-10).** The last deferred
|
||||
refinement with a code-shaped fix: the search page's filter panel gains an
|
||||
"archive" mode (exclude / only) and a "link status" multi-select (live /
|
||||
moved / rotted / error), backed by a new `data/archive-meta.json` emitted by
|
||||
`Archive.hs` — the archive analogue of `epistemic-meta.json`. The collision
|
||||
with the epistemic `status` filter is resolved by scoping (own state fields,
|
||||
button classes, and labels), not renaming. Verified in headless Chrome.
|
||||
Details in the Phase 5 Met note.
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
|
@ -388,6 +463,8 @@ auto-derived.
|
|||
# slug: auto-derived → arxiv-2403-12345 (override only to disambiguate)
|
||||
# title: auto-derived from the artifact / popup-proxy metadata
|
||||
# type: auto-detected (pdf | html)
|
||||
aliases: # optional — equivalent URLs that URL
|
||||
- "https://doi.org/10.48550/arXiv.2403.12345" # normalisation cannot derive
|
||||
tags: [research/ml] # optional — same slash-hierarchy as content
|
||||
note: > # optional — why this is referenced
|
||||
Cited in the scaling-laws essay; section 4 is the load-bearing part.
|
||||
|
|
@ -407,6 +484,7 @@ auto-derived.
|
|||
| `slug` | no | Override the auto-derived slug. Must be unique. |
|
||||
| `title` | no | Override the auto-derived title. |
|
||||
| `type` | no | `pdf` \| `html`. Auto-detected from `Content-Type` / extension. |
|
||||
| `aliases` | no | Equivalent URLs of the same work that normalisation cannot derive — above all a DOI form vs. the landing URL it resolves to. Matching metadata, not identity: edit freely, no `refresh` needed. Each must be unique across the manifest and absent from `removed.yaml`; both `archive.py` and the direct-build validator enforce this. |
|
||||
| `tags` | no | Slash-hierarchy tags (`Tags.hs`). Place the work on tag indexes. |
|
||||
| `note` | no | Author's reason for archiving; shown on the archive page. |
|
||||
| `visibility` | no | `public` (default) or `private`. |
|
||||
|
|
@ -456,6 +534,12 @@ immediately-prior snapshot's hash, so the last prior snapshot is reachable
|
|||
the artifact**, not in a rolling global file, so the immutable claim is
|
||||
genuinely immutable in git history.
|
||||
|
||||
One optional field: `fetched-from`, present only when the original was
|
||||
already dead at first fetch and the snapshot's bytes came from a Wayback
|
||||
capture instead (see Wayback Machine — non-blocking). It holds the exact raw
|
||||
(`id_`) capture URL that was fetched; its absence means a first-hand fetch
|
||||
from the original.
|
||||
|
||||
### Mutable state — `data/archive-state.json`
|
||||
|
||||
Written **only** by `tools/archive.py check`. Holds the volatile link-rot
|
||||
|
|
@ -745,7 +829,33 @@ separately, POSTs the outstanding URLs to `https://web.archive.org/save/`
|
|||
returned timestamped URL into each `PROVENANCE.json`. This second, independent
|
||||
copy means a rotted entry whose local artifact is somehow lost still has a
|
||||
fallback. If the original is *already* dead at first fetch, `archive.py fetch`
|
||||
pulls the most recent existing Wayback capture instead.
|
||||
pulls the most recent existing Wayback capture instead (implemented
|
||||
2026-06-10):
|
||||
|
||||
- **Dead means dead** — the fallback fires only when the document itself
|
||||
could not be retrieved (DNS failure, refused connection, timeout, HTTP
|
||||
error). A `noarchive` refusal, the size cap, or a local/tooling failure
|
||||
never falls through to a third-party copy.
|
||||
- **Raw bytes via `id_`** — the capture is fetched in its
|
||||
`…/web/{timestamp}id_/{url}` form, which replays the original response
|
||||
bytes without the Wayback toolbar or link rewriting; the normal pipeline
|
||||
(size cap, CSP + noindex injection, quality classification) applies.
|
||||
- **Preserved directives are honoured** — Wayback replays the original's
|
||||
response headers as `X-Archive-Orig-*`; a capture whose
|
||||
`X-Archive-Orig-X-Robots-Tag` carries `noarchive` is refused, since the
|
||||
dead original can no longer be asked directly.
|
||||
- **Honest provenance** — `PROVENANCE.json` gains an optional
|
||||
`fetched-from` field holding the exact raw capture URL, so the record
|
||||
never implies a first-hand fetch that did not happen; `wayback` is set
|
||||
to the capture immediately (so `archive-wayback` correctly skips the
|
||||
entry — a dead URL cannot be re-submitted).
|
||||
- **Lookup only** — the fallback queries the availability API; it never
|
||||
creates third-party state. Note the API has blind spots (it reports no
|
||||
capture for some URLs that arguably have one, e.g. certain large PDFs);
|
||||
a "no capture" skip is retried on the next build like any other skip.
|
||||
- **Never during `refresh`** — a deliberate re-snapshot of a dead original
|
||||
fails and restores the prior first-hand snapshot rather than silently
|
||||
downgrading it to third-party bytes.
|
||||
|
||||
### Politeness & safety
|
||||
|
||||
|
|
@ -866,6 +976,15 @@ So `archive.py` computes the equivalent-URL set per entry and stores it as
|
|||
`Backlinks.hs` matches an incoming link against any alias before keying it to
|
||||
the archive URL.
|
||||
|
||||
Forms that no offline normalisation can derive — above all a **DOI vs. the
|
||||
landing URL it resolves to** — are *authored*: the manifest's `aliases:` field
|
||||
lists equivalent URLs per entry. Authored aliases join the entry's alias set
|
||||
in `archive-index.json` (each with its own generated expansions, so a DOI
|
||||
alias's `http://` form matches like the canonical's would), they count as
|
||||
"covered" in `archive.py suggest`, and they are validated like canonical URLs:
|
||||
unique across the manifest and absent from `removed.yaml`, enforced by both
|
||||
the `archive.py fetch` pre-scan and `Archive.hs`'s direct-build validator.
|
||||
|
||||
### Granular backlinks (Phase 4 refinement)
|
||||
|
||||
If a citation targets a fragment — `…/abs/2403.12345#section-4`, or a PDF page
|
||||
|
|
@ -967,13 +1086,25 @@ This does **not** put the broken popup layer on the critical path, as the
|
|||
draft feared. `Citations.hs` already performs AST surgery on each bibliography
|
||||
entry (`enhanceEntry` — it wraps `file:` PDF links and appends keyword strips),
|
||||
so the realistic annotation hook is `enhanceEntry`, reusing `Filters.Archive`'s
|
||||
index lookup — no popup dependency. That is **deferred to a Phase 3 follow-up**:
|
||||
it first needs a check that `chicago-notes.csl` renders a cited work's
|
||||
index lookup — no popup dependency. That was **deferred to a Phase 3
|
||||
follow-up** pending a check that `chicago-notes.csl` renders a cited work's
|
||||
`url`/`doi` as a `Link` node (a CSL style that omits URLs would leave nothing
|
||||
to match). Phase 3 ships prose-link annotation; bibliography annotation is
|
||||
documented as in-scope and hookable via `enhanceEntry`, pending that check. A
|
||||
future popup rewrite may *also* consult `archive-index.json`, but the archive
|
||||
system depends on neither the current nor a future popup implementation.
|
||||
to match). A future popup rewrite may *also* consult `archive-index.json`, but
|
||||
the archive system depends on neither the current nor a future popup
|
||||
implementation.
|
||||
|
||||
**Implemented (2026-06-10).** The CSL-URL check passed empirically: citeproc
|
||||
with `chicago-notes.csl` renders entry URLs as real `Link` inlines.
|
||||
`Filters.Archive` now exports `annotateBlock` — the same annotation pass
|
||||
(affordance when live, primary-link flip when `rotted`), minus the header
|
||||
protection bibliography entries don't need — and `Citations.hs` composes it
|
||||
after `enhanceEntry` at both rendering sites: essay bibliographies
|
||||
(`renderBibDiv`) and the synthetic `/bibliography/` pages
|
||||
(`renderBibliographyHtml`). It runs *after* `enhanceEntry` so the PDF-link
|
||||
title wrap sees the entry's original inline shape. Verified on the rendered
|
||||
SIMD essay: the FIPS 203 entry (cited via its DOI — resolved through the
|
||||
manifest `aliases:` field) and the `cr.yp.to/aes-speed.html` entry both carry
|
||||
the affordance, correctly relativized.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -982,6 +1113,15 @@ system depends on neither the current nor a future popup implementation.
|
|||
`tools/archive.py check` issues a `HEAD` (falling back to a ranged `GET`) to
|
||||
every original URL in the manifest and updates `data/archive-state.json`.
|
||||
|
||||
**Offline guard (canary).** Before probing any target, `check` probes
|
||||
`https://levineuwirth.org/` itself. If the canary is unreachable, the machine
|
||||
is offline (or DNS is down) and no probe result would be evidence about the
|
||||
*targets* — the scan is declared inconclusive, the state file is left
|
||||
untouched, and the run exits 0. Without this, an unattended scheduled scan on
|
||||
a machine left offline would record a `fail` against every entry, and three
|
||||
such scans spanning two weeks would flip the entire archive to `rotted` at
|
||||
once. An empty manifest skips the canary too: no network I/O at all.
|
||||
|
||||
**Hysteresis is asymmetric.** Rotting is slow; recovery is fast.
|
||||
|
||||
- *Rotting.* A failed probe increments `consecutive-failures` and sets
|
||||
|
|
@ -1011,10 +1151,30 @@ clicks through to a working local snapshot instead of a 404, with no manual
|
|||
intervention — and only after the rot is confirmed, not guessed.
|
||||
|
||||
`check` is a slow network job, not something every `make build` should pay for.
|
||||
It runs on its own cadence — a periodic local `make archive-check`, or a
|
||||
scheduled remote agent. It is decoupled from the main build: the build consumes
|
||||
It runs on its own cadence, decoupled from the main build: the build consumes
|
||||
whatever `archive-state.json` exists.
|
||||
|
||||
**Scheduling (installed 2026-06-10).** A systemd *user* timer runs the scan
|
||||
daily on the build machine — the state file is local and gitignored, so the
|
||||
scan must run where builds happen, not on the VPS. The units live in the repo
|
||||
at `systemd/archive-check.{service,timer}` and are installed as symlinks, so
|
||||
the repo stays the source of truth (edits apply after
|
||||
`systemctl --user daemon-reload`):
|
||||
|
||||
```
|
||||
systemctl --user link ~/Repos/levineuwirth.org/systemd/archive-check.service
|
||||
systemctl --user enable --now ~/Repos/levineuwirth.org/systemd/archive-check.timer
|
||||
```
|
||||
|
||||
Cadence: daily at 12:00 (±30 min jitter) gives the hysteresis its minimum
|
||||
detection latency — 14 days — at one polite `HEAD` per cited host per day.
|
||||
`Persistent=true` fires a missed scan at the next login; the canary guard
|
||||
makes an offline fire harmless. The timer only runs while a session is up
|
||||
(`Linger=no`); `loginctl enable-linger` would lift that, but with
|
||||
`Persistent=true` and a 14-day rot floor it is not needed. Rendering remains
|
||||
pull-based by design: a status flip reaches the live site at the next
|
||||
`make deploy` (always a clean build), and `/build/` shows the current tallies.
|
||||
|
||||
---
|
||||
|
||||
## Build-pipeline integration
|
||||
|
|
@ -1351,7 +1511,9 @@ field; the hook is `Citations.hs`'s `enhanceEntry`, pending a CSL-URL check —
|
|||
not popup-gated) and **pull-from-Wayback when the original is dead at fetch
|
||||
time** (it belongs with Phase 5 link-rot detection, where a dead URL is the
|
||||
central case and a Wayback-sourced artifact's provenance can be handled
|
||||
properly). The live `make archive-wayback` run is author-initiated — it submits
|
||||
properly). *(Both implemented 2026-06-10 — see the Bibliography note under
|
||||
Link annotation, and Wayback Machine — non-blocking.)* The live
|
||||
`make archive-wayback` run is author-initiated — it submits
|
||||
public captures to a third-party service.
|
||||
|
||||
### Phase 4 — Backlinks & similar-pages indexing
|
||||
|
|
@ -1470,17 +1632,25 @@ verification the author runs (or a CI cron); the hysteresis logic itself is
|
|||
unit-tested deterministically in `next_state`, and the rendering side is
|
||||
verified by the hand-crafted `rotted` state file.
|
||||
|
||||
**Search-UI filter (`search-filters.js`) — partial.** The data-side is in
|
||||
place: every archive page carries `data-pagefind-filter="type:archive,
|
||||
status:$status$"`, so Pagefind's filter index now distinguishes archive hits
|
||||
by rot status and (when @pagefind-ui@ is configured to show filters) lists
|
||||
them as a filterable facet. The remaining work — wiring a custom UI control
|
||||
into `search-filters.js` — is a deliberate refinement, not done in Phase 5:
|
||||
its existing `status` filter is reserved for *epistemic* status (working
|
||||
model / drafting / etc.) sourced from `data/epistemic-meta.json`, so adding an
|
||||
archive `status` dimension needs a name to avoid the collision plus new
|
||||
filter-panel buttons. Search-UX best iterated with the live page in front of
|
||||
the author.
|
||||
**Search-UI filter (`search-filters.js`) — implemented (2026-06-10).** The
|
||||
data-side was in place since Phase 5 (every archive page carries
|
||||
`data-pagefind-filter="type:archive, status:$status$"`); the deferred UI
|
||||
work is now done, with the naming collision resolved by *scoping, not
|
||||
renaming*: the epistemic `status` filter keeps its name and namespace, and
|
||||
the archive dimension lives in its own state fields (`archiveMode`,
|
||||
`archiveStatus`), button classes (`filter-archive-mode-btn`,
|
||||
`filter-archive-status-btn`), and panel rows ("archive", "link status").
|
||||
Mechanics follow the established epistemic pattern rather than the Pagefind
|
||||
filter API: `Archive.hs` emits `data/archive-meta.json` (routed page path →
|
||||
link-rot status, the analogue of `data/epistemic-meta.json`), which
|
||||
`search-filters.js` fetches lazily and applies to rendered results — both
|
||||
Pagefind and semantic. Semantics: `archiveMode` is a single-select toggle —
|
||||
`exclude` hides `/archive/` results, `only` shows nothing else; `link
|
||||
status` is a multi-select that further restricts *archive* results to the
|
||||
chosen statuses and never affects native content; both compose freely with
|
||||
the epistemic filters. Verified in headless Chrome against a harness with a
|
||||
simulated `rotted` entry: exclude / only / rotted-only / rotted+draft all
|
||||
behave per the table above.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
13
Makefile
13
Makefile
|
|
@ -1,4 +1,4 @@
|
|||
.PHONY: build deploy sign download-model download-pdfjs download-leaflet compress-assets convert-images pdf-thumbs pdfs watch clean dev audit-marks archive-gc archive-wayback archive-check
|
||||
.PHONY: build deploy sign download-model download-pdfjs download-leaflet compress-assets convert-images pdf-thumbs pdfs watch clean dev audit-marks archive-gc archive-wayback archive-check archive-suggest
|
||||
|
||||
# deploy's prerequisite order (clean -> build -> sign) is only correct
|
||||
# serially; under `make -j` they could interleave. This build has no
|
||||
|
|
@ -218,6 +218,17 @@ archive-wayback:
|
|||
python3 tools/archive.py wayback; \
|
||||
fi
|
||||
|
||||
# Print works cited in data/*.bib but not yet archived, as manifest-ready
|
||||
# lines the author copies by hand. Read-only — it never edits the manifest
|
||||
# (bibliography auto-seeding is rejected by design; see ARCHIVE.md).
|
||||
# Offline: scans local files only, no network.
|
||||
archive-suggest:
|
||||
@if [ -d .venv ]; then \
|
||||
uv run python tools/archive.py suggest; \
|
||||
else \
|
||||
python3 tools/archive.py suggest; \
|
||||
fi
|
||||
|
||||
# Probe every archived URL for link rot, updating data/archive-state.json.
|
||||
# A slow network job — opt-in, never run by `make build`. Asymmetric
|
||||
# hysteresis: `rotted` needs 3 consecutive failures over >=14 days; a
|
||||
|
|
|
|||
|
|
@ -11,6 +11,10 @@
|
|||
slug: nist-fips-203
|
||||
title: "FIPS 203 — Module-Lattice-Based Key-Encapsulation Mechanism Standard"
|
||||
type: pdf
|
||||
aliases:
|
||||
# The DOI form the simd paper cites; URL normalisation cannot equate
|
||||
# a DOI with the landing URL it resolves to, so it is authored here.
|
||||
- "https://doi.org/10.6028/NIST.FIPS.203"
|
||||
tags: [research]
|
||||
note: >
|
||||
The ML-KEM standard. Cited in the SIMD / post-quantum systems work;
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@
|
|||
module Archive (archiveRules, archiveBuildStats) where
|
||||
|
||||
import Control.Exception (SomeException, catch)
|
||||
import Control.Monad (filterM, forM, when)
|
||||
import Control.Monad (filterM, forM, forM_, when)
|
||||
import Data.Function (on)
|
||||
import Data.List (groupBy, intercalate, sort, sortBy)
|
||||
import qualified Data.Map.Strict as Map
|
||||
|
|
@ -39,6 +39,7 @@ import Data.Time (Day, diffDays, fromGregorian,
|
|||
getCurrentTime, utctDay)
|
||||
import qualified Data.Aeson as A
|
||||
import Data.Aeson ((.:), (.:?))
|
||||
import qualified Data.ByteString.Lazy.Char8 as LBS
|
||||
import qualified Data.Yaml as Y
|
||||
import System.Directory (doesDirectoryExist, doesFileExist,
|
||||
listDirectory)
|
||||
|
|
@ -63,6 +64,9 @@ import ArchiveIndex (ArchiveStatus (..), statusName,
|
|||
-- Phase 4) and need no Haskell-side binding.
|
||||
data ManifestEntry = ManifestEntry
|
||||
{ meUrl :: String
|
||||
, meAliases :: [String] -- ^ authored equivalent URLs of the
|
||||
-- same work (e.g. its DOI form);
|
||||
-- matching metadata, not identity
|
||||
, meNote :: Maybe String
|
||||
, mePaywalled :: Bool
|
||||
, meVisibility :: String -- ^ "public" (default) | "private"
|
||||
|
|
@ -71,6 +75,7 @@ data ManifestEntry = ManifestEntry
|
|||
instance A.FromJSON ManifestEntry where
|
||||
parseJSON = A.withObject "ManifestEntry" $ \o -> do
|
||||
url <- o .: "url"
|
||||
aliases <- fromMaybe [] <$> o .:? "aliases"
|
||||
note <- o .:? "note"
|
||||
paywalled <- fromMaybe False <$> o .:? "paywalled"
|
||||
visibility <- fromMaybe "public" <$> o .:? "visibility"
|
||||
|
|
@ -81,7 +86,7 @@ instance A.FromJSON ManifestEntry where
|
|||
"manifest entry " ++ url
|
||||
++ ": visibility must be \"public\" or \"private\", got "
|
||||
++ show visibility
|
||||
return (ManifestEntry url note paywalled visibility)
|
||||
return (ManifestEntry url aliases note paywalled visibility)
|
||||
|
||||
newtype RemovedEntry = RemovedEntry { reUrl :: String }
|
||||
|
||||
|
|
@ -197,21 +202,30 @@ validateManifestEntries manifest removed = go Map.empty manifest
|
|||
where
|
||||
go _ [] = return ()
|
||||
go seen (entry : rest) = do
|
||||
let url = meUrl entry
|
||||
norm = normalizeUrl (T.pack url)
|
||||
when (norm `Set.member` removed) $ do
|
||||
hPutStrLn stderr $
|
||||
"[archive] FATAL: manifest URL " ++ show url
|
||||
++ " is also recorded in removed.yaml; refusing to publish "
|
||||
++ "a deliberately removed work."
|
||||
exitFailure
|
||||
case Map.lookup norm seen of
|
||||
Just prior -> do
|
||||
-- The canonical URL and every authored alias must be unique
|
||||
-- across the manifest and absent from removed.yaml — the same
|
||||
-- equivalence the Python pre-scan enforces, so a direct Hakyll
|
||||
-- build fails just as closed. Within-entry duplicates (an alias
|
||||
-- normalising to its own entry's URL) are deduped, not errors.
|
||||
let url = meUrl entry
|
||||
norms = Set.toList . Set.fromList $
|
||||
map (normalizeUrl . T.pack) (url : meAliases entry)
|
||||
forM_ norms $ \norm -> do
|
||||
when (norm `Set.member` removed) $ do
|
||||
hPutStrLn stderr $
|
||||
"[archive] FATAL: manifest URLs " ++ show prior ++ " and "
|
||||
++ show url ++ " normalise to the same archive target."
|
||||
"[archive] FATAL: manifest entry " ++ show url
|
||||
++ " matches removed.yaml (directly or via `aliases:`); "
|
||||
++ "refusing to publish a deliberately removed work."
|
||||
exitFailure
|
||||
Nothing -> go (Map.insert norm url seen) rest
|
||||
case Map.lookup norm seen of
|
||||
Just prior -> do
|
||||
hPutStrLn stderr $
|
||||
"[archive] FATAL: manifest entries " ++ show prior
|
||||
++ " and " ++ show url ++ " normalise to the same "
|
||||
++ "archive target (directly or via `aliases:`)."
|
||||
exitFailure
|
||||
Nothing -> return ()
|
||||
go (foldr (\n m -> Map.insert n url m) seen norms) rest
|
||||
|
||||
-- | Scan @archive/<slug>/PROVENANCE.json@ into a @url -> (slug, Provenance)@
|
||||
-- map. The directory name is the slug; the join key is the URL.
|
||||
|
|
@ -349,6 +363,26 @@ archiveRules = do
|
|||
|
||||
mapM_ archiveEntryRule entries
|
||||
archiveIndexRule entries
|
||||
archiveMetaRule entries
|
||||
|
||||
-- | @data/archive-meta.json@ — routed page path -> link-rot status, the
|
||||
-- client-side manifest behind the search page's archive filter (same
|
||||
-- pattern as @data/epistemic-meta.json@). Keys use the routed
|
||||
-- @.../index.html@ form to match @search-filters.js@'s @normUrl@.
|
||||
-- Named @archive-meta@, and its field @status@ scoped under it, so the
|
||||
-- epistemic @status@ filter namespace is untouched.
|
||||
archiveMetaRule :: [ArchiveEntry] -> Rules ()
|
||||
archiveMetaRule entries =
|
||||
create ["data/archive-meta.json"] $ do
|
||||
route idRoute
|
||||
compile $ do
|
||||
_ <- loadAll "archive/*/PROVENANCE.json" :: Compiler [Item String]
|
||||
let metaMap = Map.fromList
|
||||
[ ( "/archive/" ++ pvSlug (aeProv e) ++ "/index.html"
|
||||
, Map.singleton ("status" :: String)
|
||||
(statusName (aeStatus e)) )
|
||||
| e <- entries ]
|
||||
makeItem (LBS.unpack (A.encode metaMap))
|
||||
|
||||
-- | One @/archive/<slug>/@ page.
|
||||
archiveEntryRule :: ArchiveEntry -> Rules ()
|
||||
|
|
|
|||
|
|
@ -124,6 +124,13 @@ readUrlSet path = do
|
|||
Left e -> ioError . userError $
|
||||
"[archive] FATAL: " ++ path ++ ": " ++ show e
|
||||
|
||||
-- | Normalised URLs recorded as deliberate takedowns in @removed.yaml@.
|
||||
-- Consulted independently of 'activeUrls': a takedown must also knock
|
||||
-- out any *alias* key a stale index still carries for another entry.
|
||||
{-# NOINLINE removedUrls #-}
|
||||
removedUrls :: Set Text
|
||||
removedUrls = unsafePerformIO (readUrlSet removedPath)
|
||||
|
||||
-- | Canonical URLs still permitted to participate in link annotation.
|
||||
-- Filtering the generated index at build time makes a direct Hakyll build
|
||||
-- respect authored manifest/removal state even when archive.py did not run.
|
||||
|
|
@ -131,8 +138,7 @@ readUrlSet path = do
|
|||
activeUrls :: Set Text
|
||||
activeUrls = unsafePerformIO $ do
|
||||
manifest <- readUrlSet manifestPath
|
||||
removed <- readUrlSet removedPath
|
||||
return (manifest `Set.difference` removed)
|
||||
return (manifest `Set.difference` removedUrls)
|
||||
|
||||
-- | @canonical-url -> entry@. Absent/malformed file -> empty; entries no
|
||||
-- longer permitted by the authored manifest/removal state are removed.
|
||||
|
|
@ -164,13 +170,18 @@ rawState = unsafePerformIO $ do
|
|||
-- @archive-index.json@, each fed through 'normalizeUrl'. Both keys and
|
||||
-- lookups are normalised, so a citation form the alias set cannot
|
||||
-- enumerate (e.g. an unbounded arXiv version, or any tracking-laden
|
||||
-- variant of a clean manifest URL) still resolves.
|
||||
-- variant of a clean manifest URL) still resolves. Alias keys that
|
||||
-- match a recorded takedown are dropped, so a stale index (archive.py
|
||||
-- not run since removed.yaml changed) cannot keep annotating a removed
|
||||
-- work through an alias of a still-active entry.
|
||||
{-# NOINLINE flatIndex #-}
|
||||
flatIndex :: Map Text String
|
||||
flatIndex = Map.fromList
|
||||
[ (normalizeUrl key, ieSlug e)
|
||||
[ (nkey, ieSlug e)
|
||||
| (canon, e) <- Map.toList rawIndex
|
||||
, key <- canon : ieAliases e
|
||||
, let nkey = normalizeUrl key
|
||||
, nkey `Set.notMember` removedUrls
|
||||
]
|
||||
|
||||
-- | @slug -> status@: each entry's status, looked up by its canonical URL
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ import Text.Pandoc.Citeproc (processCitations)
|
|||
import Text.Pandoc.Walk
|
||||
|
||||
import BibExtras (BibExtra (..), emptyBibExtra, parseBibExtras)
|
||||
import qualified Filters.Archive (annotateBlock)
|
||||
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
|
|
@ -96,7 +97,7 @@ renderBibliographyHtml bibPaths extras keys = do
|
|||
processed <- runIOorExplode $ processCitations doc
|
||||
let refsDivs = concatMap unwrapRefs (pandocBlocks processed)
|
||||
ordered = reorderByKeys keys refsDivs
|
||||
enhanced = map (enhanceEntry extras) ordered
|
||||
enhanced = map (annotateArchive . enhanceEntry extras) ordered
|
||||
return (renderEntries "csl-bib-body" enhanced)
|
||||
where
|
||||
pandocBlocks (Pandoc _ bs) = bs
|
||||
|
|
@ -263,7 +264,7 @@ extractBibliography extras citeOrder frKeys blocks =
|
|||
-- @\<div class="bib-keywords"\>@ appended when @keywords:@ is set.
|
||||
renderBibDiv :: Map String BibExtra -> [Text] -> [Text] -> Block -> (Text, Text)
|
||||
renderBibDiv extras citeOrder _frKeys (Div _ children) =
|
||||
let enhanced = map (enhanceEntry extras) children
|
||||
let enhanced = map (annotateArchive . enhanceEntry extras) children
|
||||
keyIndex = Map.fromList (zip citeOrder [0 :: Int ..])
|
||||
(citedEntries, furtherEntries) =
|
||||
partition (isCited keyIndex) enhanced
|
||||
|
|
@ -281,6 +282,16 @@ renderBibDiv _ _ _ _ = ("", "")
|
|||
-- Bib entry enhancement (Phase 6a)
|
||||
-- ---------------------------------------------------------------------------
|
||||
|
||||
-- | Bibliography-side archive annotation: the same affordance (and
|
||||
-- rotted-link flip) 'Filters.Archive' gives body links, applied to a
|
||||
-- rendered CSL entry. The bibliography never travels the body filter
|
||||
-- chain — it is extracted and rendered to an HTML string for the
|
||||
-- template's @$bibliography$@ field — so the entry Divs are annotated
|
||||
-- here, after 'enhanceEntry' (whose PDF-link title wrap must see the
|
||||
-- entry's original inline shape).
|
||||
annotateArchive :: Block -> Block
|
||||
annotateArchive = Filters.Archive.annotateBlock
|
||||
|
||||
-- | Augment a single @csl-entry@ Div with the custom fields we parsed
|
||||
-- from the .bib file. Other Blocks pass through unchanged.
|
||||
enhanceEntry :: Map String BibExtra -> Block -> Block
|
||||
|
|
|
|||
|
|
@ -22,7 +22,12 @@
|
|||
--
|
||||
-- No-op when @data/archive-index.json@ is absent. When no rot scan has
|
||||
-- run, every entry is 'Live' — no link is ever flipped.
|
||||
module Filters.Archive (apply) where
|
||||
--
|
||||
-- 'annotateBlock' exposes the same pass for rendered blocks that never
|
||||
-- travel the body filter chain — @Citations@ applies it to each
|
||||
-- CSL-rendered bibliography entry, so a bibliography URL gets the same
|
||||
-- affordance (and the same rotted-link flip) as a body link.
|
||||
module Filters.Archive (apply, annotateBlock) where
|
||||
|
||||
import qualified Data.Text as T
|
||||
import Text.Pandoc.Definition
|
||||
|
|
@ -42,6 +47,15 @@ apply doc
|
|||
| otherwise =
|
||||
walk unprotectLink . walk annotateInlines . walk protectHeader $ doc
|
||||
|
||||
-- | The annotation pass for a single already-rendered block, outside the
|
||||
-- body filter chain. No header protection — the callers' blocks
|
||||
-- (CSL bibliography entries) contain none. Identity when the index is
|
||||
-- absent.
|
||||
annotateBlock :: Block -> Block
|
||||
annotateBlock b
|
||||
| archiveIndexIsEmpty = b
|
||||
| otherwise = walk annotateInlines b
|
||||
|
||||
-- | Sentinel class marking a link the annotation walk must skip. It
|
||||
-- only exists between the protect and unprotect walks inside 'apply'.
|
||||
skipClass :: T.Text
|
||||
|
|
|
|||
|
|
@ -102,6 +102,22 @@ Filters<span class="filter-toggle-badge"></span>
|
|||
<button class="filter-btn filter-ordinal-btn" data-field="stability" data-index="4">established</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="filter-row">
|
||||
<span class="filter-label">archive</span>
|
||||
<div class="filter-options">
|
||||
<button class="filter-btn filter-archive-mode-btn" data-value="exclude" title="Hide archived-copy pages from results">exclude</button>
|
||||
<button class="filter-btn filter-archive-mode-btn" data-value="only" title="Show only archived-copy pages">only</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="filter-row">
|
||||
<span class="filter-label">link status</span>
|
||||
<div class="filter-options">
|
||||
<button class="filter-btn filter-archive-status-btn" data-value="live">live</button>
|
||||
<button class="filter-btn filter-archive-status-btn" data-value="moved">moved</button>
|
||||
<button class="filter-btn filter-archive-status-btn" data-value="rotted">rotted</button>
|
||||
<button class="filter-btn filter-archive-status-btn" data-value="error">error</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="filter-row filter-row-actions">
|
||||
<button class="filter-clear-btn">Clear all</button>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -1,9 +1,15 @@
|
|||
/* search-filters.js — Epistemic effort filters for the search page.
|
||||
/* search-filters.js — Epistemic effort + archive filters for the search page.
|
||||
*
|
||||
* Loads /data/epistemic-meta.json (a map of URL → epistemic fields)
|
||||
* Loads /data/epistemic-meta.json (a map of URL → epistemic fields) and
|
||||
* /data/archive-meta.json (a map of /archive/ page URL → link-rot status)
|
||||
* and hides search results whose source page doesn't match the active
|
||||
* filters. Works for both Pagefind keyword results and semantic results.
|
||||
*
|
||||
* Naming: the epistemic `status` filter (draft / working model / …) and
|
||||
* the archive link status (live / moved / rotted / error) are distinct
|
||||
* dimensions — the latter lives under `archiveMode` / `archiveStatus`
|
||||
* and its own button classes, never touching the epistemic namespace.
|
||||
*
|
||||
* Reuses the same CSS classes and filter-panel markup as library.html
|
||||
* so the two pages look and behave identically.
|
||||
*/
|
||||
|
|
@ -28,10 +34,13 @@
|
|||
scope: null,
|
||||
novelty: null,
|
||||
practicality: null,
|
||||
stability: null
|
||||
stability: null,
|
||||
archiveMode: null, /* null | 'exclude' | 'only' */
|
||||
archiveStatus: [] /* link-rot statuses; archive results only */
|
||||
};
|
||||
|
||||
var epistemicMeta = null; /* URL → {status, confidence, …} loaded lazily */
|
||||
var archiveMeta = null; /* /archive/ URL → {status: live|moved|…} */
|
||||
|
||||
/* ---- Persistence ---- */
|
||||
|
||||
|
|
@ -54,8 +63,9 @@
|
|||
/* ---- Metadata loading ---- */
|
||||
|
||||
var metaPromise = null;
|
||||
var archiveMetaPromise = null;
|
||||
|
||||
function loadMeta() {
|
||||
function loadEpistemicMeta() {
|
||||
if (epistemicMeta) return Promise.resolve(epistemicMeta);
|
||||
if (metaPromise) return metaPromise;
|
||||
metaPromise = fetch('/data/epistemic-meta.json')
|
||||
|
|
@ -65,6 +75,22 @@
|
|||
return metaPromise;
|
||||
}
|
||||
|
||||
function loadArchiveMeta() {
|
||||
if (archiveMeta) return Promise.resolve(archiveMeta);
|
||||
if (archiveMetaPromise) return archiveMetaPromise;
|
||||
archiveMetaPromise = fetch('/data/archive-meta.json')
|
||||
.then(function (r) { return r.ok ? r.json() : {}; })
|
||||
.catch(function () { return {}; })
|
||||
.then(function (data) { archiveMeta = data; return data; });
|
||||
return archiveMetaPromise;
|
||||
}
|
||||
|
||||
/* Both maps load together: epistemicMeta doubles as the
|
||||
"metadata is ready" marker in the apply/observer paths. */
|
||||
function loadMeta() {
|
||||
return Promise.all([loadEpistemicMeta(), loadArchiveMeta()]);
|
||||
}
|
||||
|
||||
/* ---- Filtering logic ---- */
|
||||
|
||||
function passes(meta) {
|
||||
|
|
@ -100,8 +126,29 @@
|
|||
return true;
|
||||
}
|
||||
|
||||
/* Archive dimension. `archiveMode` governs whether /archive/ pages
|
||||
appear at all (exclude) or alone (only); `archiveStatus` further
|
||||
restricts archive results to the selected link-rot statuses and
|
||||
never affects non-archive results. */
|
||||
function isArchiveUrl(p) {
|
||||
return !!p && p.indexOf('/archive/') === 0;
|
||||
}
|
||||
|
||||
function passesArchive(url) {
|
||||
var isArch = isArchiveUrl(url);
|
||||
if (state.archiveMode === 'exclude' && isArch) return false;
|
||||
if (state.archiveMode === 'only' && !isArch) return false;
|
||||
if (state.archiveStatus.length && isArch && archiveMeta) {
|
||||
var rec = archiveMeta[url];
|
||||
var s = rec && rec.status;
|
||||
if (!s || state.archiveStatus.indexOf(s) === -1) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function hasActiveFilters() {
|
||||
if (state.status.length) return true;
|
||||
if (state.archiveMode !== null || state.archiveStatus.length) return true;
|
||||
var fields = ['confidence', 'importance', 'evidence', 'score',
|
||||
'scope', 'novelty', 'practicality', 'stability'];
|
||||
for (var i = 0; i < fields.length; i++) {
|
||||
|
|
@ -144,7 +191,8 @@
|
|||
if (!link) return;
|
||||
var url = normUrl(link.getAttribute('href'));
|
||||
var meta = url ? epistemicMeta[url] : null;
|
||||
el.classList.toggle('search-filtered', !passes(meta));
|
||||
el.classList.toggle('search-filtered',
|
||||
!(passesArchive(url) && passes(meta)));
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -160,7 +208,8 @@
|
|||
if (!link) return;
|
||||
var url = normUrl(link.getAttribute('href'));
|
||||
var meta = url ? epistemicMeta[url] : null;
|
||||
el.classList.toggle('search-filtered', !passes(meta));
|
||||
el.classList.toggle('search-filtered',
|
||||
!(passesArchive(url) && passes(meta)));
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -176,6 +225,8 @@
|
|||
function activeCount() {
|
||||
var n = 0;
|
||||
if (state.status.length) n++;
|
||||
if (state.archiveMode !== null) n++;
|
||||
if (state.archiveStatus.length) n++;
|
||||
var fields = ['confidence', 'importance', 'evidence', 'score',
|
||||
'scope', 'novelty', 'practicality', 'stability'];
|
||||
for (var i = 0; i < fields.length; i++) {
|
||||
|
|
@ -193,6 +244,14 @@
|
|||
btn.classList.toggle('is-active', state.status.indexOf(btn.dataset.value) !== -1);
|
||||
});
|
||||
|
||||
document.querySelectorAll('.filter-archive-mode-btn').forEach(function (btn) {
|
||||
btn.classList.toggle('is-active', state.archiveMode === btn.dataset.value);
|
||||
});
|
||||
|
||||
document.querySelectorAll('.filter-archive-status-btn').forEach(function (btn) {
|
||||
btn.classList.toggle('is-active', state.archiveStatus.indexOf(btn.dataset.value) !== -1);
|
||||
});
|
||||
|
||||
var ci = document.getElementById('filter-confidence');
|
||||
if (ci) ci.value = state.confidence !== null ? state.confidence : '';
|
||||
var si = document.getElementById('filter-score');
|
||||
|
|
@ -247,6 +306,26 @@
|
|||
});
|
||||
});
|
||||
|
||||
/* Archive mode buttons (exclude / only) — single-select toggle */
|
||||
document.querySelectorAll('.filter-archive-mode-btn').forEach(function (btn) {
|
||||
btn.addEventListener('click', function () {
|
||||
var v = btn.dataset.value;
|
||||
state.archiveMode = (state.archiveMode === v) ? null : v;
|
||||
loadMeta().then(applyFilters);
|
||||
});
|
||||
});
|
||||
|
||||
/* Archive link-status buttons — multi-select, archive results only */
|
||||
document.querySelectorAll('.filter-archive-status-btn').forEach(function (btn) {
|
||||
btn.addEventListener('click', function () {
|
||||
var v = btn.dataset.value;
|
||||
var i = state.archiveStatus.indexOf(v);
|
||||
if (i === -1) state.archiveStatus.push(v);
|
||||
else state.archiveStatus.splice(i, 1);
|
||||
loadMeta().then(applyFilters);
|
||||
});
|
||||
});
|
||||
|
||||
/* Threshold buttons (importance, evidence) */
|
||||
document.querySelectorAll('.filter-threshold-btn').forEach(function (btn) {
|
||||
btn.addEventListener('click', function () {
|
||||
|
|
@ -296,6 +375,8 @@
|
|||
state.novelty = null;
|
||||
state.practicality = null;
|
||||
state.stability = null;
|
||||
state.archiveMode = null;
|
||||
state.archiveStatus = [];
|
||||
applyFilters();
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
# levineuwirth.org link-rot scan — runs `make archive-check` in the repo.
|
||||
#
|
||||
# Installed as a symlink into this repo (so edits here take effect after
|
||||
# `systemctl --user daemon-reload`):
|
||||
#
|
||||
# systemctl --user link ~/Repos/levineuwirth.org/systemd/archive-check.service
|
||||
# systemctl --user enable --now ~/Repos/levineuwirth.org/systemd/archive-check.timer
|
||||
#
|
||||
# The scan only updates the gitignored data/archive-state.json; the next
|
||||
# `make deploy` consumes it. See ARCHIVE.md — Link-rot detection.
|
||||
|
||||
[Unit]
|
||||
Description=levineuwirth.org link-rot scan (archive.py check)
|
||||
Documentation=file://%h/Repos/levineuwirth.org/ARCHIVE.md
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=%h/Repos/levineuwirth.org
|
||||
ExecStart=/usr/bin/make archive-check
|
||||
Nice=10
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
# Daily trigger for archive-check.service. See that unit's header for
|
||||
# the install commands.
|
||||
|
||||
[Unit]
|
||||
Description=Daily levineuwirth.org link-rot scan
|
||||
|
||||
[Timer]
|
||||
# Daily: rot needs 3 consecutive fails spanning >=14 days, so a daily
|
||||
# scan gives the design's minimum detection latency (14 days) at one
|
||||
# polite HEAD per cited host per day. Persistent=true fires a missed
|
||||
# scan at the next login; the canary guard in archive.py makes a fire
|
||||
# while offline harmless (scan skipped, state untouched).
|
||||
OnCalendar=*-*-* 12:00
|
||||
RandomizedDelaySec=30min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
325
tools/archive.py
325
tools/archive.py
|
|
@ -15,13 +15,17 @@ Two artifact types:
|
|||
text extracted with BeautifulSoup.
|
||||
|
||||
Subcommands:
|
||||
fetch download missing artifacts, (re)generate sidecars + index
|
||||
fetch download missing artifacts, (re)generate sidecars + index;
|
||||
an original already dead at first fetch falls back to its most
|
||||
recent existing Wayback capture (recorded as `fetched-from`)
|
||||
refresh deliberately re-snapshot a single entry, recording the prior
|
||||
SHA in the new PROVENANCE.json's `previous-sha256`
|
||||
wayback submit archived URLs to the Wayback Machine as a second,
|
||||
independent copy; backfill the capture URL into PROVENANCE.json
|
||||
check HEAD/GET-probe every manifest URL for link rot, updating
|
||||
data/archive-state.json with asymmetric hysteresis
|
||||
suggest print works cited in data/*.bib (url / doi fields) but not in
|
||||
the manifest, as manifest-ready lines; never edits the manifest
|
||||
gc delete archive/<slug>/ directories listed in archive/removed.yaml
|
||||
|
||||
Failure policy:
|
||||
|
|
@ -72,6 +76,12 @@ STATE_OUT = REPO_ROOT / "data" / "archive-state.json"
|
|||
ROT_FAILS = 3 # consecutive failed scans before `rotted` is considered
|
||||
ROT_DAYS = 14 # ... and the streak must also span at least this many days
|
||||
|
||||
# Probed before a link-rot scan; unreachable -> the scan is inconclusive
|
||||
# (offline machine, dead DNS) and state is left untouched. Guards the
|
||||
# unattended timer-run case: three offline scans spanning two weeks would
|
||||
# otherwise flip every entry to `rotted` at once.
|
||||
CHECK_CANARY = "https://levineuwirth.org/"
|
||||
|
||||
SIZE_CAP = 25 * 1024 * 1024 # 25 MB per-artifact cap
|
||||
TIMEOUT = 60 # seconds, per network request
|
||||
WAYBACK_TIMEOUT = 120 # seconds — Save Page Now is slow
|
||||
|
|
@ -173,6 +183,25 @@ def entry_slug(entry: dict) -> str:
|
|||
return slug if slug else derive_slug(entry["url"])
|
||||
|
||||
|
||||
def entry_aliases(entry: dict) -> list[str]:
|
||||
"""The authored `aliases:` list of a manifest entry, validated. An
|
||||
alias is an equivalent URL of the same work that no offline
|
||||
normalisation can derive — e.g. its DOI form vs. the landing URL the
|
||||
artifact was fetched from. Aliases are matching metadata, not
|
||||
identity: editing them never requires a refresh (the index is
|
||||
rewritten from the manifest on every fetch). Malformed values are
|
||||
fatal — a typo'd alias would otherwise silently never match, and the
|
||||
resulting affordance gap is invisible."""
|
||||
aliases = entry.get("aliases", [])
|
||||
if not isinstance(aliases, list) or not all(
|
||||
isinstance(a, str) and a.startswith(("http://", "https://"))
|
||||
for a in aliases):
|
||||
err(f"manifest entry {entry.get('url')!r}: `aliases:` must be a "
|
||||
f"list of http(s) URLs, got {aliases!r}")
|
||||
sys.exit(1)
|
||||
return aliases
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hashing / type detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -238,9 +267,13 @@ def detect_type(url: str, override) -> str | None:
|
|||
# PDF fetch + text extraction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def fetch_pdf(url: str, dest: Path) -> bool:
|
||||
"""Download `url` to `dest`, enforcing the size cap. Returns True on
|
||||
success. A partial / over-cap download leaves no file behind."""
|
||||
def fetch_pdf(url: str, dest: Path) -> str:
|
||||
"""Download `url` to `dest`, enforcing the size cap. Returns "ok" on
|
||||
success, "dead" when the document itself could not be retrieved (DNS
|
||||
failure, refused connection, timeout, HTTP error status — the cases
|
||||
where a Wayback fallback is legitimate), or "skip" for a policy or
|
||||
local failure (noarchive directive, size cap, disk) that a fallback
|
||||
must not circumvent. A partial / over-cap download leaves no file."""
|
||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
try:
|
||||
|
|
@ -250,7 +283,7 @@ def fetch_pdf(url: str, dest: Path) -> bool:
|
|||
robots = (resp.headers.get("X-Robots-Tag") or "").lower()
|
||||
if "noarchive" in robots:
|
||||
err(f"{url}: response carries X-Robots-Tag: noarchive — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
total = 0
|
||||
with tmp.open("wb") as fh:
|
||||
for chunk in iter(lambda: resp.read(1 << 16), b""):
|
||||
|
|
@ -260,14 +293,18 @@ def fetch_pdf(url: str, dest: Path) -> bool:
|
|||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: exceeds {SIZE_CAP // (1024*1024)} MB cap "
|
||||
f"— skipped (commit deliberately with `git add -f`)")
|
||||
return False
|
||||
return "skip"
|
||||
fh.write(chunk)
|
||||
tmp.replace(dest)
|
||||
return True
|
||||
return "ok"
|
||||
except (urllib.error.URLError, TimeoutError) as exc:
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: fetch failed — {exc}")
|
||||
return "dead" # HTTPError is a URLError subclass
|
||||
except Exception as exc: # noqa: BLE001 — report any failure
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: fetch failed — {exc}")
|
||||
return False
|
||||
return "skip"
|
||||
|
||||
|
||||
def extract_text_pdf(pdf: Path, txt: Path) -> None:
|
||||
|
|
@ -388,10 +425,13 @@ def inject_archive_metas(path: Path) -> None:
|
|||
path.write_text(str(soup), encoding="utf-8")
|
||||
|
||||
|
||||
def fetch_html(url: str, dest: Path) -> bool:
|
||||
def fetch_html(url: str, dest: Path) -> str:
|
||||
"""Snapshot an HTML page with monolith into a single self-contained
|
||||
file at `dest`, then inject the archive CSP. Returns True on success;
|
||||
every failure path is non-fatal (warn + skip)."""
|
||||
file at `dest`, then inject the archive CSP. Returns "ok" on success,
|
||||
"dead" when the source document itself could not be retrieved (the
|
||||
Wayback-fallback-eligible case), or "skip" for every other failure —
|
||||
noarchive directives, the size cap, monolith problems — none of which
|
||||
a fallback may circumvent. Every failure path is non-fatal."""
|
||||
# Honour directives returned by preliminary probes before performing
|
||||
# the document fetch. The full document response is inspected below
|
||||
# and is also the exact body passed to monolith; do not let monolith
|
||||
|
|
@ -400,14 +440,14 @@ def fetch_html(url: str, dest: Path) -> bool:
|
|||
for h in (probe_headers(url),
|
||||
probe_headers_get(url))):
|
||||
err(f"{url}: response carries X-Robots-Tag: noarchive — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
|
||||
mono = find_monolith()
|
||||
if mono is None:
|
||||
err(f"{url}: monolith not found — vendor the binary at "
|
||||
f"tools/bin/monolith (see tools/monolith-version.txt) or set "
|
||||
f"$MONOLITH_BIN; HTML snapshot skipped")
|
||||
return False
|
||||
return "skip"
|
||||
verify_monolith(mono)
|
||||
|
||||
source = dest.with_suffix(dest.suffix + ".source.part")
|
||||
|
|
@ -419,7 +459,7 @@ def fetch_html(url: str, dest: Path) -> bool:
|
|||
robots = (resp.headers.get("X-Robots-Tag") or "").lower()
|
||||
if "noarchive" in robots:
|
||||
err(f"{url}: response carries X-Robots-Tag: noarchive — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
effective_url = resp.geturl()
|
||||
total = 0
|
||||
with source.open("wb") as fh:
|
||||
|
|
@ -430,17 +470,21 @@ def fetch_html(url: str, dest: Path) -> bool:
|
|||
source.unlink(missing_ok=True)
|
||||
err(f"{url}: source HTML exceeds "
|
||||
f"{SIZE_CAP // (1024*1024)} MB cap — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
fh.write(chunk)
|
||||
except (urllib.error.URLError, TimeoutError) as exc:
|
||||
source.unlink(missing_ok=True)
|
||||
err(f"{url}: fetch failed — {exc}")
|
||||
return "dead" # HTTPError is a URLError subclass
|
||||
except Exception as exc: # noqa: BLE001
|
||||
source.unlink(missing_ok=True)
|
||||
err(f"{url}: fetch failed — {exc}")
|
||||
return False
|
||||
return "skip"
|
||||
|
||||
if body_noarchive(source):
|
||||
source.unlink(missing_ok=True)
|
||||
err(f"{url}: response declares <meta name=robots> noarchive — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
|
||||
cmd = [mono, "--no-js", "--ignore-errors", "--quiet",
|
||||
"--timeout", str(TIMEOUT), "--user-agent", USER_AGENT,
|
||||
|
|
@ -452,12 +496,12 @@ def fetch_html(url: str, dest: Path) -> bool:
|
|||
source.unlink(missing_ok=True)
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: monolith timed out — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
except Exception as exc: # noqa: BLE001
|
||||
source.unlink(missing_ok=True)
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: monolith failed to run — {exc}")
|
||||
return False
|
||||
return "skip"
|
||||
finally:
|
||||
source.unlink(missing_ok=True)
|
||||
|
||||
|
|
@ -467,21 +511,21 @@ def fetch_html(url: str, dest: Path) -> bool:
|
|||
tail = output.decode("utf-8", errors="replace").strip().splitlines()
|
||||
err(f"{url}: monolith exited {proc.returncode} "
|
||||
f"({tail[-1] if tail else 'no output'}) — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
if not tmp.exists() or tmp.stat().st_size == 0:
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: monolith produced no output — skipped")
|
||||
return False
|
||||
return "skip"
|
||||
if tmp.stat().st_size > SIZE_CAP:
|
||||
size_mb = tmp.stat().st_size // (1024 * 1024)
|
||||
tmp.unlink(missing_ok=True)
|
||||
err(f"{url}: snapshot is {size_mb} MB, over the "
|
||||
f"{SIZE_CAP // (1024*1024)} MB cap — skipped "
|
||||
f"(commit deliberately with `git add -f`)")
|
||||
return False
|
||||
return "skip"
|
||||
inject_archive_metas(tmp)
|
||||
tmp.replace(dest)
|
||||
return True
|
||||
return "ok"
|
||||
|
||||
|
||||
def extract_text_html(snapshot: Path, txt: Path) -> None:
|
||||
|
|
@ -667,7 +711,57 @@ def _is_tracked_and_clean(*paths: Path) -> bool:
|
|||
# fetch subcommand
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_fetch() -> int:
|
||||
_WAYBACK_CAPTURE_RE = re.compile(
|
||||
r"^(https?://web\.archive\.org/web/)(\d{4,14})(/.+)$")
|
||||
|
||||
|
||||
def wayback_raw_url(capture: str) -> str:
|
||||
"""The raw-bytes form of a Wayback capture URL: the `id_` flag after
|
||||
the timestamp serves the original response bytes, without the Wayback
|
||||
toolbar or link rewriting. An unexpected shape passes through."""
|
||||
m = _WAYBACK_CAPTURE_RE.match(capture)
|
||||
return f"{m.group(1)}{m.group(2)}id_{m.group(3)}" if m else capture
|
||||
|
||||
|
||||
def fetch_from_wayback(url: str, entry: dict, slug_dir: Path
|
||||
) -> tuple[str, Path, str, str] | None:
|
||||
"""Fetch-time fallback for an original that is already dead: pull the
|
||||
most recent *existing* Wayback capture (raw bytes via `id_`). Returns
|
||||
(atype, artifact_path, capture_url, raw_url), or None when no capture
|
||||
exists or the capture fetch itself fails. Lookup only — a dead URL
|
||||
cannot be newly captured, so the fallback never creates third-party
|
||||
state."""
|
||||
capture = wayback_lookup(url)
|
||||
if capture is None:
|
||||
err(f"{url}: original unreachable and the Wayback Machine has no "
|
||||
f"capture — skipped")
|
||||
return None
|
||||
raw = wayback_raw_url(capture)
|
||||
# Wayback replays the original response headers as X-Archive-Orig-*;
|
||||
# a preserved noarchive directive is still the publisher's word, and
|
||||
# the dead original can no longer be asked directly.
|
||||
preserved = (probe_headers(raw).get("x-archive-orig-x-robots-tag")
|
||||
or "").lower()
|
||||
if "noarchive" in preserved:
|
||||
err(f"{url}: Wayback capture preserves X-Robots-Tag: noarchive "
|
||||
f"— skipped")
|
||||
return None
|
||||
# Re-resolve the artifact type against the raw capture: the original
|
||||
# is dead, so its own Content-Type probe degraded to the html
|
||||
# default; the capture replays the stored Content-Type.
|
||||
atype = detect_type(raw, entry.get("type"))
|
||||
if atype is None:
|
||||
return None
|
||||
art = slug_dir / ARTIFACT[atype]
|
||||
log(f"{url}: original dead — fetching Wayback capture {capture}")
|
||||
result = fetch_pdf(raw, art) if atype == "pdf" else fetch_html(raw, art)
|
||||
if result != "ok":
|
||||
err(f"{url}: Wayback capture fetch failed — skipped")
|
||||
return None
|
||||
return (atype, art, capture, raw)
|
||||
|
||||
|
||||
def cmd_fetch(wayback_fallback: bool = True) -> int:
|
||||
manifest = load_yaml_list(MANIFEST)
|
||||
# Removed URLs are compared in normalised form so a tracking-laden
|
||||
# variant cannot bypass a takedown the author already recorded.
|
||||
|
|
@ -676,19 +770,39 @@ def cmd_fetch() -> int:
|
|||
|
||||
# Pre-scan validation: reject canonical-form duplicates *before* any
|
||||
# fetch I/O, so a first colliding entry never gets partially processed
|
||||
# while a second's duplicate check halts.
|
||||
# while a second's duplicate check halts. The canonical URL and every
|
||||
# authored alias must be unique across the whole manifest — a shared
|
||||
# form would route one citation under two slugs. (An alias that
|
||||
# normalises to its own entry's URL is merely redundant: deduped here,
|
||||
# never an error.)
|
||||
seen: dict[str, str] = {}
|
||||
for entry in manifest:
|
||||
url = entry.get("url")
|
||||
if not url:
|
||||
continue
|
||||
norm = normalize_url(url)
|
||||
if norm in seen:
|
||||
err(f"manifest: {url!r} and {seen[norm]!r} normalise to the "
|
||||
f"same canonical form ({norm!r}). Drop one or distinguish "
|
||||
f"them; the link archive cannot route both under one slug.")
|
||||
keys = {normalize_url(url)}
|
||||
keys |= {normalize_url(a) for a in entry_aliases(entry)}
|
||||
for norm in sorted(keys):
|
||||
if norm in seen:
|
||||
err(f"manifest: {url!r} and {seen[norm]!r} share the "
|
||||
f"canonical form {norm!r} (directly or via `aliases:`). "
|
||||
f"Drop one or distinguish them; the link archive "
|
||||
f"cannot route both under one slug.")
|
||||
sys.exit(1)
|
||||
seen[norm] = url
|
||||
# A takedown recorded in removed.yaml is enforced against aliases
|
||||
# too — re-listing a removed work as an alias of another entry
|
||||
# would republish it under that entry's slug. (The canonical URL
|
||||
# gets the same check, with fetch-specific wording, in the
|
||||
# per-entry loop below.)
|
||||
hit = keys & removed_norms
|
||||
if hit:
|
||||
err(f"manifest entry {url!r}: canonical form {sorted(hit)[0]!r} "
|
||||
f"(direct or via `aliases:`) is recorded in "
|
||||
f"archive/removed.yaml as a deliberate takedown. To "
|
||||
f"re-archive it, remove the corresponding line from "
|
||||
f"removed.yaml first.")
|
||||
sys.exit(1)
|
||||
seen[norm] = url
|
||||
|
||||
index: dict[str, dict] = {}
|
||||
skipped = 0
|
||||
|
|
@ -757,11 +871,25 @@ def cmd_fetch() -> int:
|
|||
sys.exit(1)
|
||||
|
||||
# --- fetch the artifact if it is not already present --------------
|
||||
wb_capture: str | None = None # Wayback capture used, if any
|
||||
wb_raw: str | None = None # ... and its raw id_ form
|
||||
if not art.exists():
|
||||
slug_dir.mkdir(parents=True, exist_ok=True)
|
||||
log(f"fetching {url} [{atype}]")
|
||||
ok = fetch_pdf(url, art) if atype == "pdf" else fetch_html(url, art)
|
||||
if not ok:
|
||||
result = (fetch_pdf(url, art) if atype == "pdf"
|
||||
else fetch_html(url, art))
|
||||
if result == "dead" and wayback_fallback:
|
||||
# The original is already gone at first fetch — pull the
|
||||
# most recent existing Wayback capture instead. Only for
|
||||
# *dead* originals: a noarchive refusal or an over-cap
|
||||
# skip must never be circumvented via a third-party copy.
|
||||
fb = fetch_from_wayback(url, entry, slug_dir)
|
||||
if fb is not None:
|
||||
atype, art, wb_capture, wb_raw = fb
|
||||
txt = slug_dir / TEXTFILE[atype]
|
||||
txt_stamp = slug_dir / (TEXTFILE[atype] + ".sha256")
|
||||
result = "ok"
|
||||
if result != "ok":
|
||||
skipped += 1
|
||||
continue
|
||||
else:
|
||||
|
|
@ -795,17 +923,35 @@ def cmd_fetch() -> int:
|
|||
"archived": datetime.date.today().isoformat(),
|
||||
"source-date": entry.get("source-date"),
|
||||
"snapshot-quality": quality,
|
||||
"wayback": None,
|
||||
# When the fallback fired, the capture is already known —
|
||||
# cmd_wayback (which targets `wayback: null`) skips it,
|
||||
# correctly: a dead original cannot be re-submitted.
|
||||
"wayback": wb_capture,
|
||||
}
|
||||
if wb_raw is not None:
|
||||
# The snapshot's bytes came from the Wayback capture, not
|
||||
# the (dead) original. Recorded so the provenance never
|
||||
# implies a first-hand fetch that did not happen.
|
||||
prov["fetched-from"] = wb_raw
|
||||
atomic_write_json(prov_path, prov)
|
||||
log(f"{slug}: archived [{atype}, {quality}] ({prov['bytes']} bytes)")
|
||||
origin = " via Wayback" if wb_raw else ""
|
||||
log(f"{slug}: archived{origin} [{atype}, {quality}] "
|
||||
f"({prov['bytes']} bytes)")
|
||||
|
||||
# --- contribute to the Hakyll index -------------------------------
|
||||
# Generated equivalents of the canonical URL, plus the authored
|
||||
# `aliases:` (each with its own generated equivalents) — so a DOI
|
||||
# alias's http:// form matches just like the canonical's would.
|
||||
alias_set = set(url_aliases(url))
|
||||
for authored in entry_aliases(entry):
|
||||
alias_set.add(authored)
|
||||
alias_set.update(url_aliases(authored))
|
||||
alias_set.discard(url)
|
||||
index[url] = {
|
||||
"slug": slug,
|
||||
"type": prov.get("type", atype),
|
||||
"title": prov.get("title", slug),
|
||||
"aliases": url_aliases(url),
|
||||
"aliases": sorted(alias_set),
|
||||
}
|
||||
|
||||
# archive-index.json is always rewritten to mirror the manifest exactly.
|
||||
|
|
@ -917,7 +1063,13 @@ def cmd_refresh(argv: list[str]) -> int:
|
|||
|
||||
succeeded = False
|
||||
try:
|
||||
rc = cmd_fetch()
|
||||
# No Wayback fallback during a refresh: the author asked for a
|
||||
# fresh first-hand snapshot. If the original turns out to be dead,
|
||||
# the right outcome is "refresh fails, prior snapshot restored" —
|
||||
# not a silent downgrade of a committed first-hand snapshot to
|
||||
# third-party bytes. Adopting a Wayback copy stays a deliberate
|
||||
# act (a new entry whose original is already dead).
|
||||
rc = cmd_fetch(wayback_fallback=False)
|
||||
|
||||
# Success requires a new PROVENANCE.json *and* its declared
|
||||
# artifact on disk. `cmd_fetch` returns 0 even when individual
|
||||
|
|
@ -1134,14 +1286,23 @@ def cmd_check() -> int:
|
|||
except Exception: # noqa: BLE001
|
||||
old = {}
|
||||
|
||||
to_probe = [e["url"] for e in manifest
|
||||
if e.get("url") and normalize_url(e["url"]) not in removed_norms]
|
||||
|
||||
# Offline guard: when the canary is unreachable, no probe result below
|
||||
# is evidence about the *targets* — record nothing rather than a
|
||||
# spurious `fail` against every entry. Exit 0: an offline laptop is
|
||||
# not a unit failure; the scan simply retries on the next timer fire.
|
||||
if to_probe and probe_url(CHECK_CANARY)[0] == "fail":
|
||||
log(f"check: canary {CHECK_CANARY} unreachable — offline or DNS "
|
||||
f"down; scan inconclusive, state left untouched")
|
||||
return 0
|
||||
|
||||
today = datetime.date.today()
|
||||
state: dict[str, dict] = {}
|
||||
tally = {"live": 0, "moved": 0, "error": 0, "rotted": 0}
|
||||
|
||||
for entry in manifest:
|
||||
url = entry.get("url")
|
||||
if not url or normalize_url(url) in removed_norms:
|
||||
continue
|
||||
for url in to_probe:
|
||||
result, new_url = probe_url(url)
|
||||
rec = next_state(old.get(url, {}), result, new_url, today)
|
||||
state[url] = rec
|
||||
|
|
@ -1198,6 +1359,84 @@ def cmd_gc(ignore_orphans: bool) -> int:
|
|||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# suggest subcommand
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_BIB_ENTRY_RE = re.compile(r"@(\w+)\s*\{\s*([^,\s{}]+)\s*,")
|
||||
_BIB_FIELD_RE = re.compile(
|
||||
r"""^\s*(url|doi)\s*=\s*[{"]\s*([^}"]+?)\s*["}]""",
|
||||
re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def bib_citations(text: str) -> list[tuple[str, str]]:
|
||||
"""(citation-key, URL) pairs from one .bib file's text. Per entry the
|
||||
`url` field wins; a DOI-only entry resolves to https://doi.org/{doi}.
|
||||
@comment/@string/@preamble blocks carry no citation and are skipped."""
|
||||
out: list[tuple[str, str]] = []
|
||||
entries = list(_BIB_ENTRY_RE.finditer(text))
|
||||
for i, m in enumerate(entries):
|
||||
kind, key = m.group(1).lower(), m.group(2)
|
||||
if kind in ("comment", "string", "preamble"):
|
||||
continue
|
||||
end = entries[i + 1].start() if i + 1 < len(entries) else len(text)
|
||||
fields = {f.group(1).lower(): f.group(2)
|
||||
for f in _BIB_FIELD_RE.finditer(text[m.end():end])}
|
||||
url = fields.get("url")
|
||||
if not url and fields.get("doi"):
|
||||
url = "https://doi.org/" + fields["doi"]
|
||||
if url and url.startswith(("http://", "https://")):
|
||||
out.append((key, url))
|
||||
return out
|
||||
|
||||
|
||||
def cmd_suggest() -> int:
|
||||
"""Print works cited in data/*.bib but absent from the manifest, as
|
||||
manifest-ready lines. Read-only by design: tools never write
|
||||
manifest.yaml — the author reviews and copies lines by hand, so the
|
||||
manifest stays the *identity* of the archive, not a .bib cache."""
|
||||
manifest = load_yaml_list(MANIFEST)
|
||||
removed = load_yaml_list(REMOVED)
|
||||
covered = {normalize_url(e["url"])
|
||||
for e in manifest + removed if e.get("url")}
|
||||
for e in manifest:
|
||||
covered.update(normalize_url(a) for a in entry_aliases(e))
|
||||
|
||||
bib_files = sorted((REPO_ROOT / "data").glob("*.bib"))
|
||||
if not bib_files:
|
||||
log("suggest: no data/*.bib files to scan")
|
||||
return 0
|
||||
|
||||
# normalized URL -> first-seen verbatim URL + every citing entry, so
|
||||
# one work cited from three papers prints once, with all three citers.
|
||||
suggestions: dict[str, dict] = {}
|
||||
for bib in bib_files:
|
||||
for key, url in bib_citations(bib.read_text(encoding="utf-8")):
|
||||
norm = normalize_url(url)
|
||||
if norm in covered:
|
||||
continue
|
||||
s = suggestions.setdefault(norm, {"url": url, "cited": []})
|
||||
s["cited"].append(f"{bib.name}:{key}")
|
||||
|
||||
scanned = ", ".join(b.name for b in bib_files)
|
||||
if not suggestions:
|
||||
log(f"suggest: nothing to add — every URL cited in {scanned} is "
|
||||
f"already archived or deliberately removed")
|
||||
return 0
|
||||
|
||||
log(f"suggest: {len(suggestions)} cited work(s) not in the manifest "
|
||||
f"(scanned {scanned})")
|
||||
print()
|
||||
print("# Cited but not archived. Review each one, then copy the")
|
||||
print("# entries you want into archive/manifest.yaml.")
|
||||
for s in suggestions.values():
|
||||
print()
|
||||
for citer in s["cited"]:
|
||||
print(f"# cited by {citer}")
|
||||
print(f'- url: "{s["url"]}"')
|
||||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -1212,10 +1451,12 @@ def main(argv: list[str]) -> int:
|
|||
return cmd_wayback()
|
||||
if cmd == "check":
|
||||
return cmd_check()
|
||||
if cmd == "suggest":
|
||||
return cmd_suggest()
|
||||
if cmd == "gc":
|
||||
return cmd_gc(ignore_orphans="--ignore-orphans" in argv[1:])
|
||||
err(f"unknown subcommand {cmd!r} "
|
||||
f"(expected: fetch | refresh | wayback | check | gc)")
|
||||
f"(expected: fetch | refresh | wayback | check | suggest | gc)")
|
||||
return 2
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue