Initialize an absent root through the production entry point
Scope 6.4 deliverable 1, startup state 1. StoreEngine::open now creates a root rather than refusing to, and every harness that measured a store production did not build is re-pointed at it. States 3 and 4 still refuse. The four startup states are distinguished by one read-only classification that creates nothing. A FORMAT entry means state 2 on the strength of the name alone, because a FORMAT that does not decode is still FORMAT and treating an unreadable marker as "no marker, therefore empty" would authorize building a fresh tree over a populated root. A root holding only LOCK is empty, since lock_root creates that file as a side effect of asking whether the root is busy. The signer check moved above every root access, because once an absent root is initialized rather than refused, a signerless configuration would otherwise create a tree, write FORMAT and fsync the parent before failing -- a configuration error must not leave a root behind. Three defects found in review, all fixed here rather than deferred. A deeply absent path was created with create_dir_all and only its immediate parent fenced, so open could report success over ancestors a power loss could take. Ancestors are now created one at a time -- so what this call created is exactly what it fences, and a racing creator surfaces as AlreadyExists rather than being absorbed -- and fenced deepest-first, since a directory entry lives in the parent that names it and the reverse order can leave a fenced parent naming an unfenced child. An interrupted initialization was unrecoverable: any tree residue classified the root as non-empty-without-FORMAT and it was refused forever, with a message about legacy layouts that had nothing to do with what happened. A root being built now carries an INITIALIZING marker, installed by rename as the first durable act and removed as the last, so a durable partial tree always has a durable marker beside it and every crash window is resumable. Recognizing it requires both a byte-exact marker this crate alone writes and every entry in the root drawn from a closed set of names this store invented, so a foreign layout cannot be mistaken for abandoned initialization and overwritten -- the direction that matters, since refusing a resumable root costs an operator time and overwriting a real one costs their data. Sibling staging with atomic installation was the alternative and is structurally blocked: LOCK lives inside the root, so the root must exist before any mutation can be serialized, and renaming a tree onto a directory containing LOCK fails ENOTEMPTY. The classifier then ignored INITIALIZING.tmp by name regardless of type or contents, and initialization opened that name with create plus truncate. An operator's file there was destroyed silently, and a symlink there truncated a file outside the root to 27 bytes and then removed the link -- destroying data the store never owned and erasing the evidence, while open returned Ok and reported a working store. The justification for ignoring the name was that only this path could have written it, which is circular: that is the claim the classifier runs in order to establish. Every entry is now judged by lstat type before anything opens it, the temporary marker is validated as an exact regular marker or refused, and installation is create-new rather than create-truncate. Contract review 2026-07-28-D records the two no-follow open primitives this added to the frozen sys.rs, and the four further symlink hazards in segment.rs that are recorded rather than fixed -- the first of which lets two processes believe they hold one root lock. A fifo at that name made the pre-fix open block forever: one mkfifo in a configured root was an unbounded startup hang, not only a data hazard. The engine and the drive seam are now asserted to recover one crash image identically, closing a gap that was true by construction and untested. Charter item 8 applied to the harness: the in-crate test helper no longer calls segment::initialize_root, so every writer test builds its root through open; the ROOT_SEEDED_BY_NON_PRODUCTION_PATH disclosure is retired; and the fixture's root_seeded_by becomes a stable token matched by exact equality, with the history moved to an adjacent reason field -- a substring match passes on a value that has drifted to mean something else. The d0 contract test asserting open returns NotImplemented for any valid configuration is obsoleted by this deliverable and replaced with the stronger property: a signerless configuration is refused and leaves no root behind. It moves off a fixed /tmp path, which under the old check ordering would have created a real store root on every gate run on every machine. scripts/check-phase1.sh GATE_EXIT=0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
e050b6dddd
commit
cd40e2c892
File diff suppressed because it is too large
Load Diff
|
|
@ -336,6 +336,88 @@ pub(crate) fn unlink(path: &Path) -> io::Result<()> {
|
|||
std::fs::remove_file(path)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Opening a name this process does not yet own
|
||||
// ---------------------------------------------------------------------------
|
||||
//
|
||||
// Both of these exist for startup state 1, where the store has to look at, and
|
||||
// then write into, a directory it has not established is its own. Plain
|
||||
// `File::open` and `File::options().create(true).truncate(true)` both traverse a
|
||||
// symlink at the final component, so either one at a name an operator can
|
||||
// occupy is a write to a path outside the root. `open`/`stat` are therefore not
|
||||
// available on that path at all; these are.
|
||||
|
||||
/// Open `path` read-only for *verification*, refusing to traverse a symlink at
|
||||
/// the final component and refusing anything that is not a regular file.
|
||||
///
|
||||
/// `Ok(None)` means "there is no regular file at exactly this path": absent, a
|
||||
/// symlink, a directory, a fifo, a socket, a device. Every caller is asking
|
||||
/// whether the bytes at a name are its own, and for all of those the answer is
|
||||
/// no — so they collapse into one variant rather than being distinguished by a
|
||||
/// caller that would treat them identically.
|
||||
///
|
||||
/// `O_NOFOLLOW` is what makes the symlink case an error rather than a read of
|
||||
/// somebody else's file, and the `fstat` is what makes it a *regular file*
|
||||
/// rather than a name that merely opened: `O_NOFOLLOW` says nothing about
|
||||
/// directories or fifos. The type is read from the descriptor already opened,
|
||||
/// not from a second path lookup, so the answer is about the object this call
|
||||
/// holds and cannot be changed underneath it.
|
||||
///
|
||||
/// `O_NONBLOCK` is load-bearing rather than incidental. Opening a fifo for
|
||||
/// reading blocks until a writer arrives, so without it a fifo left in a
|
||||
/// configured root would hang startup indefinitely — a denial of service
|
||||
/// reached by `mkfifo`, and one this function exists to be immune to because
|
||||
/// its whole job is to look at names it does not trust.
|
||||
pub(crate) fn open_regular_nofollow(path: &Path) -> io::Result<Option<File>> {
|
||||
use rustix::fs::{FileType, Mode, OFlags};
|
||||
let fd = match rustix::fs::open(
|
||||
path,
|
||||
OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC | OFlags::NONBLOCK,
|
||||
Mode::empty(),
|
||||
) {
|
||||
Ok(fd) => fd,
|
||||
Err(rustix::io::Errno::NOENT) | Err(rustix::io::Errno::NOTDIR) => return Ok(None),
|
||||
// `O_NOFOLLOW` on a symlink is `ELOOP` on Linux and `EMLINK` on some
|
||||
// BSDs. Both mean "the final component is a symlink", which is exactly
|
||||
// the answer this function is being asked for.
|
||||
Err(rustix::io::Errno::LOOP) | Err(rustix::io::Errno::MLINK) => return Ok(None),
|
||||
Err(e) => return Err(io::Error::from_raw_os_error(e.raw_os_error())),
|
||||
};
|
||||
let file = File::from(fd);
|
||||
let stat =
|
||||
rustix::fs::fstat(&file).map_err(|e| io::Error::from_raw_os_error(e.raw_os_error()))?;
|
||||
if FileType::from_raw_mode(stat.st_mode) == FileType::RegularFile {
|
||||
Ok(Some(file))
|
||||
} else {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
/// Create `path` as a **new** regular file, or report that the name is taken.
|
||||
///
|
||||
/// `Ok(None)` means the name already exists — as anything at all, including a
|
||||
/// symlink. `O_CREAT | O_EXCL` is specified to fail with `EEXIST` on a symlink
|
||||
/// whether or not the link resolves, so this can never create or truncate a
|
||||
/// file outside the directory it names; `O_NOFOLLOW` is passed as well so the
|
||||
/// intent survives someone later relaxing the `O_EXCL`.
|
||||
///
|
||||
/// This is the only way anything in this crate may bring a new name into a
|
||||
/// directory the store has not yet established is its own. `create(true)` plus
|
||||
/// `truncate(true)` is the operation it replaces, and the difference is that
|
||||
/// this one cannot destroy a byte it did not write.
|
||||
pub(crate) fn create_new_nofollow(path: &Path) -> io::Result<Option<File>> {
|
||||
use rustix::fs::{Mode, OFlags};
|
||||
match rustix::fs::open(
|
||||
path,
|
||||
OFlags::WRONLY | OFlags::CREATE | OFlags::EXCL | OFlags::NOFOLLOW | OFlags::CLOEXEC,
|
||||
Mode::from_raw_mode(0o644),
|
||||
) {
|
||||
Ok(fd) => Ok(Some(File::from(fd))),
|
||||
Err(rustix::io::Errno::EXIST) => Ok(None),
|
||||
Err(e) => Err(io::Error::from_raw_os_error(e.raw_os_error())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Truncate a file to `len`.
|
||||
///
|
||||
/// In the funnel because it changes what is durable. `journal.rs` was calling
|
||||
|
|
|
|||
|
|
@ -348,32 +348,74 @@ fn the_pending_set_is_empty_at_phase_1_exit() {
|
|||
);
|
||||
}
|
||||
|
||||
/// `StoreEngine::open` still refuses startup state 1, which is why the Wave B
|
||||
/// rows seed their root with `segment::initialize_root`.
|
||||
/// `StoreEngine::open` builds an absent root, which is why the Wave B rows no
|
||||
/// longer seed one of their own.
|
||||
///
|
||||
/// Asserted rather than assumed, because the moment it stops being true the
|
||||
/// seeding disclosure in the fixture becomes a false statement about the
|
||||
/// harness, and a stale disclosure is worse than none: it tells a reader that a
|
||||
/// weakening still exists when the honest answer is that it does not.
|
||||
/// # This test used to assert the opposite, and that is the record
|
||||
///
|
||||
/// While `open` refused startup state 1, the Wave B rows created their root
|
||||
/// with `segment::initialize_root` and this test asserted the refusal, so that
|
||||
/// the moment it stopped being true the seeding disclosure in the fixture would
|
||||
/// be caught as a false statement about the harness rather than left standing.
|
||||
/// B1 landed state 1; the tripwire fired; the seeding is re-pointed at
|
||||
/// `StoreEngine::open` and the disclosure is retired. What is asserted here now
|
||||
/// is the property that replaced it — charter item 8 in the affirmative, at the
|
||||
/// altitude a consumer calls: the store these rows exercise is built by
|
||||
/// production, and a regression that returned `open` to refusing would be
|
||||
/// caught here rather than by every row failing for an unrelated-looking
|
||||
/// reason.
|
||||
#[test]
|
||||
fn the_production_open_still_refuses_to_create_a_root_and_the_fixture_says_so() {
|
||||
fn the_production_open_builds_the_root_the_wave_b_rows_exercise() {
|
||||
let directory = tempfile::tempdir().expect("tempdir");
|
||||
let options = StoreOptions::new(directory.path().join("root"));
|
||||
match StoreEngine::open(options) {
|
||||
Err(StoreError::NotImplemented(reason)) => {
|
||||
assert!(
|
||||
reason.contains("startup states 1"),
|
||||
"the refusal must still be the unbuilt startup states: {reason}"
|
||||
);
|
||||
}
|
||||
Ok(_) => panic!(
|
||||
"StoreEngine::open now creates an absent root, so the Wave B rows must be \
|
||||
re-pointed at it and the fixture's root_seeded_by disclosure retired"
|
||||
let root = directory.path().join("root");
|
||||
let layout = levcs_store::segment::RootLayout::new(&root);
|
||||
assert!(!layout.format_path().exists(), "the root must start absent");
|
||||
|
||||
// The same options the rows use, because an option set that differs from
|
||||
// theirs would assert about a store nobody drives.
|
||||
let engine = match StoreEngine::open(engine_matrix::options(&root, 1)) {
|
||||
Ok(engine) => engine,
|
||||
Err(StoreError::NotImplemented(reason)) => panic!(
|
||||
"StoreEngine::open refuses to build an absent root again ({reason}). The Wave \
|
||||
B rows create their store through it and assert that they did; a harness \
|
||||
that went back to seeding its own root would be asserting the production \
|
||||
submit path over a store production never built."
|
||||
),
|
||||
Err(other) => {
|
||||
panic!("StoreEngine::open refused an absent root for an unexpected reason: {other:?}")
|
||||
}
|
||||
}
|
||||
Err(other) => panic!("StoreEngine::open refused an absent root: {other:?}"),
|
||||
};
|
||||
assert!(
|
||||
layout.format_path().exists(),
|
||||
"StoreEngine::open returned without writing FORMAT"
|
||||
);
|
||||
drop(engine);
|
||||
|
||||
// And the fixture must say so, because the disclosure a reader gets is the
|
||||
// fixture's, not this file's.
|
||||
//
|
||||
// **Exact equality, against a short stable token.** This was
|
||||
// `.contains("StoreEngine::open")` against a paragraph, which is too weak
|
||||
// for a tripwire: the substring survives almost any drift in what the
|
||||
// paragraph means, so the assertion would keep passing on a value that had
|
||||
// come to describe a different seeding path. A token has nothing to drift
|
||||
// into — it either is this value or the fixture and this file disagree, and
|
||||
// the disagreement is the point. The essay lives in the adjacent
|
||||
// `root_seeded_by_reason`, which the loader requires non-empty, so moving
|
||||
// the history out of the matched value did not make it droppable.
|
||||
let fixture = harness::load_fixture();
|
||||
assert_eq!(
|
||||
fixture.root_seeded_by, "store_engine_open_state_1",
|
||||
"the fixture must record, as an exact token, that the Wave B rows build their \
|
||||
root through StoreEngine::open on an absent path — startup state 1, the \
|
||||
production entry point"
|
||||
);
|
||||
assert!(
|
||||
fixture
|
||||
.root_seeded_by_reason
|
||||
.contains("segment::initialize_root"),
|
||||
"the reason field carries the history the token no longer does: that the rows \
|
||||
once seeded their own root because open refused state 1, and that the \
|
||||
weakening is closed rather than re-worded"
|
||||
);
|
||||
}
|
||||
|
||||
// ===========================================================================
|
||||
|
|
@ -1348,18 +1390,12 @@ fn assert_full_expectation(
|
|||
observation.prior_recovered
|
||||
);
|
||||
|
||||
// The root-lock release window documented on
|
||||
// `engine_matrix::reopen_after_close`. Reported rather than asserted: the
|
||||
// wait is compensating for a defect in a file B4 does not own, and a
|
||||
// harness that stayed silent about it would be hiding the very thing it is
|
||||
// working around.
|
||||
if observation.lock_release_attempts > 1 {
|
||||
eprintln!(
|
||||
"crash_matrix: {row}: the root lock was still held after StoreEngine::drop \
|
||||
returned; the reopen needed {} attempts over {:?}",
|
||||
observation.lock_release_attempts, observation.lock_release_wait
|
||||
);
|
||||
}
|
||||
// The root-lock release window that used to be reported here is gone with
|
||||
// the wait that produced it: `engine_matrix::reopen_after_close` now
|
||||
// asserts that the *first* `StoreEngine::open` after `drop` succeeds, so
|
||||
// every row in this matrix carries that assertion at production altitude
|
||||
// rather than a number describing how long it had to wait (contract review
|
||||
// 2026-07-28-B, commit e050b6d).
|
||||
|
||||
// Charter item 7: the class is checked against a syscall count, not a
|
||||
// comment. A class that says no bytes were written must show no fence, and
|
||||
|
|
|
|||
|
|
@ -372,18 +372,51 @@ fn open_refuses_an_invalid_configuration_before_anything_else() {
|
|||
}
|
||||
}
|
||||
|
||||
/// Configuration validation runs before the not-implemented path, so a valid
|
||||
/// configuration reaches the engine and reports honestly that it is unbuilt.
|
||||
/// A configuration error must not be able to leave a root behind.
|
||||
///
|
||||
/// This replaces a D0-era assertion that `open` returned `NotImplemented` for
|
||||
/// any valid configuration, which B1 deliverable 1 obsoleted by implementing
|
||||
/// startup state 1. The replacement is the stronger property, and it is the
|
||||
/// one that had to be asserted the moment an absent root began to be
|
||||
/// *initialized* rather than refused: a configuration that passes
|
||||
/// `StoreOptions::validate` but omits the signer must still be refused, and
|
||||
/// refused before anything is created.
|
||||
///
|
||||
/// The ordering it pins is load-bearing rather than tidy. While state 1
|
||||
/// refused, the signer check could sit after the `FORMAT` probe harmlessly;
|
||||
/// once state 1 initializes, that same placement would create the directory
|
||||
/// tree, write `FORMAT`, and fsync the parent before failing — and this test,
|
||||
/// written against a fixed `/tmp` path as its predecessor was, would have
|
||||
/// silently created a real store root on every gate run on every machine. It
|
||||
/// therefore asserts the absence of the root, not merely the error, and it
|
||||
/// uses a path inside a temporary directory so that a future regression
|
||||
/// pollutes nothing outside the test.
|
||||
#[test]
|
||||
fn open_reports_not_implemented_for_a_valid_configuration() {
|
||||
let options = StoreOptions::new("/tmp/levcs-store-d0");
|
||||
fn a_valid_but_signerless_configuration_is_refused_and_creates_no_root() {
|
||||
let directory = tempfile::tempdir().expect("tempdir");
|
||||
let root = directory.path().join("root");
|
||||
|
||||
// Valid in every respect `validate` can see; `StoreOptions::new` registers
|
||||
// no signer, and instance composition is what supplies one.
|
||||
let options = StoreOptions::new(&root);
|
||||
|
||||
match StoreEngine::open(options) {
|
||||
Err(StoreError::NotImplemented(what)) => {
|
||||
assert!(what.contains("B1"), "the stub must name its owner: {what}");
|
||||
Err(StoreError::InvalidConfiguration(msg)) => {
|
||||
assert!(
|
||||
msg.contains("CommitEvidenceSigner"),
|
||||
"the refusal must name what is missing, got: {msg}"
|
||||
);
|
||||
}
|
||||
Err(other) => panic!("expected NotImplemented, got {other:?}"),
|
||||
Ok(_) => panic!("D0 has no engine"),
|
||||
Err(other) => panic!("expected InvalidConfiguration, got {other:?}"),
|
||||
Ok(_) => panic!("a configuration with no signer must not open a store"),
|
||||
}
|
||||
|
||||
assert!(
|
||||
!root.exists(),
|
||||
"the refused configuration left {} behind; a startup that cannot \
|
||||
sequence a transaction must not have initialized a root first",
|
||||
root.display()
|
||||
);
|
||||
}
|
||||
|
||||
/// Shard assignment is frozen. Per-repository sequence ownership is only sound
|
||||
|
|
|
|||
|
|
@ -269,7 +269,8 @@
|
|||
"later_append_allowed_before_recovery"
|
||||
],
|
||||
"reason": "With engine.rs the four fields Wave A could not reach are observable through the API a consumer calls: submit returns or refuses, transaction_status performs plan 5.1's two-root read, a second submit to the same shard says whether a later append is admitted, and a close-and-reopen through StoreEngine::open answers recovery_outcome the same way the Wave A rows answer it. Every Wave B row asserts the complete FailpointExpectation, field by field, against oracle::append_publication_expectation.",
|
||||
"root_seeded_by": "segment::initialize_root. StoreEngine::open still refuses startup state 1 - initializing an absent or empty root is B1 deliverable 1 and is unimplemented - so the root these rows exercise cannot be created through the production open. This is a charter item 8 weakening and it is recorded here rather than only in a report: what is asserted below is the production submit, status, and reopen path over a root production did not build. When B1 lands state 1 this becomes StoreEngine::open on an absent path and the weakening disappears.",
|
||||
"root_seeded_by": "store_engine_open_state_1",
|
||||
"root_seeded_by_reason": "The token above is matched by exact equality in crash_matrix.rs, which is what makes it a tripwire: a substring test passes on a value that has drifted to mean something else, so the matched value is a short stable token and the history lives here. StoreEngine::open on an absent path is the production entry point, startup state 1. Until B1 landed that state the root these rows exercise was created by segment::initialize_root, because open refused to build one, and this field disclosed that charter item 8 weakening: what was asserted below was the production submit, status, and reopen path over a root production did not build. The weakening is closed rather than re-described - the rows now build their store through open and assert that they did, checking the root is absent before the call and that FORMAT exists after, so an open that quietly stopped initializing is caught here instead of returning the harness to seeding its own store.",
|
||||
"panic_ownership": "B1 implements phase-aware panic ownership: a panic in scope 6.3 steps 1-3 is definitively absent and leaves the writer thread alive; a panic in steps 4-8 poisons and kills the writer, so a later append is refused NotReady rather than accepted; a panic in steps 9-10 still delivers every receipt. The Panic rows confirm this from outside the engine, through submit and transaction_status, rather than by reading the catch sites."
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -47,54 +47,55 @@ use levcs_protocol::oracle::{ImmediateStatus, RecoveredTailFact};
|
|||
use levcs_protocol::v2::TransactionEvidenceV1;
|
||||
use levcs_store::failpoints::{Failpoint, FailpointAction};
|
||||
use levcs_store::options::StoreOptions;
|
||||
use levcs_store::segment::{initialize_root, RootLayout};
|
||||
use levcs_store::segment::RootLayout;
|
||||
use levcs_store::transaction::StagedObject;
|
||||
use levcs_store::types::{
|
||||
CommitEvidenceSigner, CommitReceipt, DurabilityCounters, NamespaceId, OperationId, SignerError,
|
||||
StoreError, TransactionStatus,
|
||||
CommitEvidenceSigner, CommitReceipt, NamespaceId, OperationId, SignerError, StoreError,
|
||||
TransactionStatus,
|
||||
};
|
||||
use levcs_store::{StoreEngine, ValidatedTransaction};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Seeding a root
|
||||
// Building the root through production
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// **Charter item 8 disclosure, stated where it is committed rather than in a
|
||||
/// report only.**
|
||||
/// Create the root **through the entry point a consumer calls**, and prove that
|
||||
/// is what happened.
|
||||
///
|
||||
/// `StoreEngine::open` refuses startup state 1 — initializing an absent or
|
||||
/// empty root — by name (`engine.rs`, B1 deliverable 1, unimplemented). So a
|
||||
/// store that is about to be exercised through the production `submit` cannot
|
||||
/// be *created* through the production `open`. It is created here by calling
|
||||
/// `segment::initialize_root`, which is the same function state 1 will call
|
||||
/// when B1 implements it, but reached directly rather than through the
|
||||
/// entry point a consumer uses.
|
||||
/// # What this replaces, and why the replacement is the point
|
||||
///
|
||||
/// This is a real weakening and it is named rather than buried: everything
|
||||
/// below asserts against the production path *after* the root exists, and
|
||||
/// nothing below asserts anything about how a root comes into existence. When
|
||||
/// B1 lands startup state 1, this function becomes one line — `StoreEngine::open`
|
||||
/// on an absent path — and the Wave B rows gain that half for free.
|
||||
pub const ROOT_SEEDED_BY_NON_PRODUCTION_PATH: &str =
|
||||
"segment::initialize_root under the store-internals-adjacent public module, because \
|
||||
StoreEngine::open still refuses startup state 1 (B1 deliverable 1)";
|
||||
/// Until B1 landed startup state 1, `StoreEngine::open` refused to initialize
|
||||
/// an absent root, so a store about to be exercised through the production
|
||||
/// `submit` could not be *created* through the production `open`. These rows
|
||||
/// seeded it with `segment::initialize_root` instead, and that weakening was
|
||||
/// disclosed here, in the fixture, and in three places in the B4 report:
|
||||
/// everything below asserted against the production path over a root
|
||||
/// production had not built. Charter item 8 is precisely about that shape.
|
||||
///
|
||||
/// State 1 exists now, so the weakening is closed rather than merely
|
||||
/// re-described. The assertions around the call are what make this a check and
|
||||
/// not a rename: the root is confirmed absent before, and the `FORMAT` marker
|
||||
/// is confirmed present after, so a future `open` that quietly stopped
|
||||
/// initializing would fail here instead of silently returning the harness to
|
||||
/// seeding its own store.
|
||||
pub fn open_absent_root(root: &Path, shard_count: u16, max_index_runs: u32) -> StoreEngine {
|
||||
let layout = RootLayout::new(root);
|
||||
assert!(
|
||||
!layout.format_path().exists(),
|
||||
"the row must build its store through StoreEngine::open, so the root must be \
|
||||
absent before it: {}",
|
||||
root.display()
|
||||
);
|
||||
|
||||
fn now_micros() -> i64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_micros() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
let engine = StoreEngine::open(options_with_index_runs(root, shard_count, max_index_runs))
|
||||
.expect("StoreEngine::open initializes an absent root (startup state 1)");
|
||||
|
||||
pub fn seed_root(root: &Path, shard_count: u16) {
|
||||
initialize_root(
|
||||
&RootLayout::new(root),
|
||||
shard_count,
|
||||
[0x4b; 16],
|
||||
now_micros(),
|
||||
&DurabilityCounters::default(),
|
||||
)
|
||||
.expect("initialize the store root");
|
||||
assert!(
|
||||
layout.format_path().exists(),
|
||||
"StoreEngine::open returned without writing FORMAT, so the store these rows \
|
||||
exercise was not built by the production entry point after all"
|
||||
);
|
||||
engine
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
@ -191,6 +192,13 @@ fn evidence() -> TransactionEvidenceV1 {
|
|||
}
|
||||
}
|
||||
|
||||
fn now_micros() -> i64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_micros() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
fn deadline() -> i64 {
|
||||
now_micros() + 600_000_000
|
||||
}
|
||||
|
|
@ -495,64 +503,41 @@ pub fn disarm(serial: &Serial) {
|
|||
// Reopening after a close
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// How long the harness will wait for `LOCK` after `StoreEngine` was dropped.
|
||||
pub const LOCK_RELEASE_BUDGET: Duration = Duration::from_secs(30);
|
||||
|
||||
/// What one close-and-reopen cost.
|
||||
pub struct Reopen {
|
||||
pub engine: StoreEngine,
|
||||
/// Attempts made. `1` means the lock was already free when `drop` returned.
|
||||
pub attempts: u32,
|
||||
pub waited: Duration,
|
||||
}
|
||||
|
||||
/// Reopen a root after its engine was dropped, waiting for the root lock.
|
||||
/// Reopen a root after its engine was dropped. **One attempt.**
|
||||
///
|
||||
/// # This wait is compensating for a defect, and it is not the harness's
|
||||
/// # This is an assertion, and it used to be a wait
|
||||
///
|
||||
/// `StoreEngine::drop` closes every shard channel and joins every writer
|
||||
/// thread, so on its own account the root lock is released by the time `drop`
|
||||
/// returns. Measured, it is not: reopening immediately after `drop` returns
|
||||
/// fails `AlreadyLocked` in roughly one run in six, and the lock then becomes
|
||||
/// free between a few hundred microseconds and about 150 milliseconds later.
|
||||
/// Something outlives the join and holds the `RecoverySession`.
|
||||
/// thread, so the root lock is released by the time `drop` returns. Measured
|
||||
/// against the code as it stood, it was not: reopening immediately failed
|
||||
/// `AlreadyLocked` in roughly one run in six, and this function waited up to
|
||||
/// thirty seconds for the lock while publishing the attempt count.
|
||||
///
|
||||
/// This matters outside the harness. Scope 3.1 says `AlreadyLocked` is a
|
||||
/// refusal and **never a wait**, so a consumer that closes a store and reopens
|
||||
/// it — a recovery drill, an in-place restart, the Phase 2 migrator — gets a
|
||||
/// spurious refusal with no defined retry. It is reported to the lead and to
|
||||
/// B1 as a finding rather than fixed here: `engine.rs` is not B4's file, and
|
||||
/// re-deriving its shutdown sequence in the harness is exactly the local
|
||||
/// restatement the ownership matrix exists to prevent.
|
||||
/// Commit `e050b6d` fixed the cause rather than the symptom. `flock` locks live
|
||||
/// on the *open file description*, so a concurrently forked child that
|
||||
/// inherited the descriptor kept the lock alive past the close;
|
||||
/// `segment::lock_root` now returns an RAII `RootLock` that issues an explicit
|
||||
/// `LOCK_UN` before dropping the descriptor. The wait was correct while the
|
||||
/// defect stood, and it must not outlive it: scope 3.1 says `AlreadyLocked` is
|
||||
/// a refusal and **never a wait**, so a harness that waits on it asserts
|
||||
/// something the store does not promise, and a wait that succeeds on the second
|
||||
/// try is exactly how a recurrence of this defect would stay invisible.
|
||||
///
|
||||
/// The wait is therefore **bounded, measured, and reported**, not silent. It is
|
||||
/// not a retry-until-green: a single reopen either succeeds within the budget
|
||||
/// or the run fails, and `attempts` is published so a regression that lengthens
|
||||
/// the window shows up as a number rather than as an intermittent failure.
|
||||
pub fn reopen_after_close(root: &Path, shard_count: u16, max_index_runs: u32) -> Reopen {
|
||||
let started = Instant::now();
|
||||
let mut attempts = 0u32;
|
||||
loop {
|
||||
attempts += 1;
|
||||
match StoreEngine::open(options_with_index_runs(root, shard_count, max_index_runs)) {
|
||||
Ok(engine) => {
|
||||
return Reopen {
|
||||
engine,
|
||||
attempts,
|
||||
waited: started.elapsed(),
|
||||
}
|
||||
}
|
||||
Err(StoreError::AlreadyLocked) if started.elapsed() < LOCK_RELEASE_BUDGET => {
|
||||
std::thread::sleep(Duration::from_micros(200));
|
||||
}
|
||||
Err(StoreError::AlreadyLocked) => panic!(
|
||||
"the root lock was still held {:?} after StoreEngine::drop returned, over \
|
||||
{attempts} attempts. The known window is under a second; this is longer, \
|
||||
which means the holder is not merely slow to be reclaimed.",
|
||||
started.elapsed()
|
||||
),
|
||||
Err(other) => panic!("reopen through production recovery failed: {other:?}"),
|
||||
}
|
||||
/// What this buys, stated because it is the point: the lead's own one-attempt
|
||||
/// assertion sits in `segment.rs` against `lock_root`, one level *below* the
|
||||
/// entry point a consumer calls. This one is at production altitude, through
|
||||
/// `StoreEngine::open` — charter item 8. Nothing else asserts it there.
|
||||
pub fn reopen_after_close(root: &Path, shard_count: u16, max_index_runs: u32) -> StoreEngine {
|
||||
match StoreEngine::open(options_with_index_runs(root, shard_count, max_index_runs)) {
|
||||
Ok(engine) => engine,
|
||||
Err(StoreError::AlreadyLocked) => panic!(
|
||||
"the root lock was still held on the first StoreEngine::open after \
|
||||
StoreEngine::drop returned. Commit e050b6d releases it explicitly in \
|
||||
RootLock::drop, and scope 3.1 makes AlreadyLocked a refusal rather than a \
|
||||
wait, so this is a recurrence of that defect and not a slow reclaim to \
|
||||
sleep through."
|
||||
),
|
||||
Err(other) => panic!("reopen through production recovery failed: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -595,11 +580,6 @@ pub struct RowObservation {
|
|||
/// against a comment about where the fence sits.
|
||||
pub fences_before: u64,
|
||||
pub fences_after: u64,
|
||||
/// How many `StoreEngine::open` attempts the reopen needed, and how long it
|
||||
/// waited. `1` and a near-zero wait is the expected reading; anything else
|
||||
/// is the root-lock release window documented on [`reopen_after_close`].
|
||||
pub lock_release_attempts: u32,
|
||||
pub lock_release_wait: Duration,
|
||||
}
|
||||
|
||||
impl RowObservation {
|
||||
|
|
@ -641,13 +621,13 @@ pub fn drive_submit_row(
|
|||
let row = format!("{} [{}]", point.name(), action_name(action));
|
||||
let directory = tempfile::tempdir().expect("tempdir");
|
||||
let root = directory.path().join("root");
|
||||
seed_root(&root, 1);
|
||||
|
||||
let namespace = namespace_on_shard(0, 1, 0x5643_5342);
|
||||
let prior_operation = OperationId([0x11; 16]);
|
||||
let victim_operation = OperationId([0x22; 16]);
|
||||
|
||||
let engine = StoreEngine::open(options(&root, 1)).expect("open the seeded root");
|
||||
// Built by production, not merely exercised through it.
|
||||
let engine = open_absent_root(&root, 1, DEFAULT_MAX_INDEX_RUNS);
|
||||
|
||||
let prior = submit(&engine, create_transaction(namespace, 0x11));
|
||||
assert!(
|
||||
|
|
@ -678,10 +658,7 @@ pub fn drive_submit_row(
|
|||
disarm(serial);
|
||||
drop(engine);
|
||||
|
||||
let reopen = reopen_after_close(&root, 1, DEFAULT_MAX_INDEX_RUNS);
|
||||
let lock_release_attempts = reopen.attempts;
|
||||
let lock_release_wait = reopen.waited;
|
||||
let reopened = reopen.engine;
|
||||
let reopened = reopen_after_close(&root, 1, DEFAULT_MAX_INDEX_RUNS);
|
||||
let recovered = reopened
|
||||
.transaction_status(namespace, victim_operation)
|
||||
.expect("status read");
|
||||
|
|
@ -700,8 +677,6 @@ pub fn drive_submit_row(
|
|||
prior_recovered,
|
||||
fences_before,
|
||||
fences_after,
|
||||
lock_release_attempts,
|
||||
lock_release_wait,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -818,18 +793,12 @@ pub fn drive_root_cas_retry_row(
|
|||
|
||||
let directory = tempfile::tempdir().expect("tempdir");
|
||||
let root = directory.path().join("root");
|
||||
seed_root(&root, CONTENTION_SHARDS);
|
||||
|
||||
let namespaces: Vec<NamespaceId> = (0..CONTENTION_SHARDS)
|
||||
.map(|shard| namespace_on_shard(shard, CONTENTION_SHARDS, 0x0CA5_0000 + shard as u64))
|
||||
.collect();
|
||||
|
||||
let engine = StoreEngine::open(options_with_index_runs(
|
||||
&root,
|
||||
CONTENTION_SHARDS,
|
||||
CONTENTION_MAX_INDEX_RUNS,
|
||||
))
|
||||
.expect("open the seeded root");
|
||||
let engine = open_absent_root(&root, CONTENTION_SHARDS, CONTENTION_MAX_INDEX_RUNS);
|
||||
for (index, namespace) in namespaces.iter().enumerate() {
|
||||
let created = submit_plain(&engine, create_transaction(*namespace, 0x40 + index as u8));
|
||||
assert!(
|
||||
|
|
@ -944,10 +913,7 @@ pub fn drive_root_cas_retry_row(
|
|||
disarm(serial);
|
||||
drop(engine);
|
||||
|
||||
let reopen = reopen_after_close(&root, CONTENTION_SHARDS, CONTENTION_MAX_INDEX_RUNS);
|
||||
let lock_release_attempts = reopen.attempts;
|
||||
let lock_release_wait = reopen.waited;
|
||||
let reopened = reopen.engine;
|
||||
let reopened = reopen_after_close(&root, CONTENTION_SHARDS, CONTENTION_MAX_INDEX_RUNS);
|
||||
let recovered = reopened
|
||||
.transaction_status(namespace, victim_operation)
|
||||
.expect("status read");
|
||||
|
|
@ -972,8 +938,6 @@ pub fn drive_root_cas_retry_row(
|
|||
prior_recovered,
|
||||
fences_before,
|
||||
fences_after,
|
||||
lock_release_attempts,
|
||||
lock_release_wait,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -74,6 +74,20 @@ pub struct Fixture {
|
|||
pub wave_a_asserted: Vec<String>,
|
||||
pub wave_a_unasserted: Vec<String>,
|
||||
pub wave_b_asserted: Vec<String>,
|
||||
/// By what path the root the Wave B rows exercise was created, as a short
|
||||
/// stable token.
|
||||
///
|
||||
/// A **tripwire**, matched by exact equality in `crash_matrix.rs`. It was a
|
||||
/// paragraph matched with `.contains("StoreEngine::open")`, and a substring
|
||||
/// test is too weak for a tripwire: it keeps passing on a value that has
|
||||
/// drifted to mean something else, because the substring survives the
|
||||
/// drift. The history — that the non-production seeding weakness existed
|
||||
/// and is now closed — lives in [`Fixture::root_seeded_by_reason`], which is
|
||||
/// where an essay belongs; it is not in the value being matched.
|
||||
pub root_seeded_by: String,
|
||||
/// Why the token above is what it is. Required non-empty: a tripwire whose
|
||||
/// justification can be deleted silently is a token nobody can audit.
|
||||
pub root_seeded_by_reason: String,
|
||||
pub wave_b_exit_conditions: Vec<String>,
|
||||
}
|
||||
|
||||
|
|
@ -129,16 +143,28 @@ pub fn load_fixture() -> Fixture {
|
|||
.get("wave_b_assertion_scope")
|
||||
.expect("the fixture must record which halves Wave B asserts");
|
||||
let wave_b_asserted = string_list(wave_b_scope.get("asserted").expect("asserted"));
|
||||
assert!(
|
||||
wave_b_scope
|
||||
.get("root_seeded_by")
|
||||
.and_then(|v| v.as_str())
|
||||
.is_some_and(|r| !r.is_empty()),
|
||||
"the Wave B rows are driven against a root that StoreEngine::open cannot yet \
|
||||
create, and the fixture must say by what path it was created instead. Scope 5 \
|
||||
charter item 8 is about exactly this: a property asserted against a store that \
|
||||
production never built is asserted against the wrong store."
|
||||
);
|
||||
let root_seeded_by = wave_b_scope
|
||||
.get("root_seeded_by")
|
||||
.and_then(|v| v.as_str())
|
||||
.filter(|r| !r.is_empty())
|
||||
.expect(
|
||||
"the fixture must say by what path the root the Wave B rows exercise was \
|
||||
created. Scope 5 charter item 8 is about exactly this: a property asserted \
|
||||
against a store that production never built is asserted against the wrong \
|
||||
store, and a reader of the fixture must be able to tell which it is \
|
||||
without reading the harness.",
|
||||
)
|
||||
.to_string();
|
||||
let root_seeded_by_reason = wave_b_scope
|
||||
.get("root_seeded_by_reason")
|
||||
.and_then(|v| v.as_str())
|
||||
.filter(|r| !r.is_empty())
|
||||
.expect(
|
||||
"root_seeded_by is a short token matched by exact equality, so the reason it \
|
||||
carries that value must live beside it. A tripwire whose justification can \
|
||||
be dropped without anything failing is a token nobody can audit.",
|
||||
)
|
||||
.to_string();
|
||||
|
||||
let wave_b_exit_conditions = string_list(
|
||||
document
|
||||
|
|
@ -181,6 +207,8 @@ pub fn load_fixture() -> Fixture {
|
|||
wave_a_asserted,
|
||||
wave_a_unasserted,
|
||||
wave_b_asserted,
|
||||
root_seeded_by,
|
||||
root_seeded_by_reason,
|
||||
wave_b_exit_conditions,
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1174,6 +1174,49 @@ interface request against B1 arises from this. Scope §6.6 records the consequen
|
|||
land: the bounded retry at `tests/support/engine_matrix.rs:516` is now compensating for a
|
||||
defect that no longer exists and must become a one-attempt assertion.
|
||||
|
||||
##### Contract review 2026-07-28-D
|
||||
|
||||
**Two no-follow `open(2)` primitives added to the frozen `sys.rs`.** Granted, and the grant is
|
||||
narrower than it looks: `sys.rs` gains `open_regular_nofollow` and `create_new_nofollow`, both
|
||||
pure syscall wrappers — one `open`, one `fstat` on the descriptor already held, and errno-to-
|
||||
variant mapping. Neither knows what a marker is, reads content, or decides a startup state.
|
||||
|
||||
The defect that forced it. `StoreEngine::open`'s classifier ignored `INITIALIZING.tmp` by name
|
||||
regardless of type or contents, and initialization then opened that name with `create` plus
|
||||
`truncate`. An operator's regular file at that name was destroyed silently; a **symlink** at
|
||||
that name caused a file *outside the root* to be truncated to the marker's 27 bytes and the
|
||||
link then removed, so the store destroyed data it never owned and erased the evidence. Measured
|
||||
on a reverted copy: the victim went from 4096 bytes to 27, the link was gone, and `open`
|
||||
returned `Ok` reporting a working store.
|
||||
|
||||
The justification for ignoring the name had been that only this code path could have written
|
||||
it. That reasoning was circular — it is the claim the classifier runs in order to establish,
|
||||
and before the store owns the root it has no standing to assume anything about the contents.
|
||||
|
||||
Why the primitives belong in `sys.rs` rather than in `engine.rs`. These are new `open(2)` sites
|
||||
whose **flags are the safety property**. `sys.rs` is the sole funnel precisely so that the next
|
||||
author looking for how this crate opens files finds every such site in one place; an
|
||||
`O_NOFOLLOW`/`O_EXCL` open living alone in `engine.rs` is one the next author does not find,
|
||||
and what they write instead is `create(true).truncate(true)` — the defect this closes. The
|
||||
weaker alternative, a raw syscall at the call site with a comment, was rejected for that reason.
|
||||
Neither wrapper increments a durability counter, because neither produces durable bytes; the
|
||||
existing write and fence primitives still do the counting.
|
||||
|
||||
`O_NONBLOCK` is load-bearing rather than defensive. A fifo at `INITIALIZING.tmp` made the
|
||||
pre-fix `O_WRONLY` open block forever: a single `mkfifo` in a configured root was an unbounded
|
||||
startup hang, not merely a data hazard. That was found while writing the tests, not predicted.
|
||||
|
||||
**Four further symlink hazards are recorded and not fixed**, because they are in `segment.rs`
|
||||
and are A1's surface. Ranked: `lock_root` follows a final symlink at `LOCK` and is reachable on
|
||||
the state-2 path, where the classifier never runs — through a live link the process `flock`s a
|
||||
foreign inode and believes it holds the root lock, so two processes can both own one root and
|
||||
the exclusion scope §3.1 depends on is defeated. `write_fenced` repeats the `create`+`truncate`
|
||||
defect for `FORMAT.tmp`, closed from `open` by the new classifier but open to `initialize_root`'s
|
||||
direct callers. `initialize_root`'s `create_dir_all` follows a symlinked directory and builds a
|
||||
shard tree outside the root. `read_format` follows a link at `FORMAT`, read-only and lowest
|
||||
priority. The first is a correctness defect in the locking discipline and should be scheduled on
|
||||
its own, not folded into a later pass.
|
||||
|
||||
### Phase 1 — storage engine spine
|
||||
|
||||
Lead first defines sealed transaction/frame/snapshot interfaces and file ownership. That deliverable (D0) landed on 2026-07-24 as `crates/levcs-store`: the frozen public API compiling against `StoreError::NotImplemented`, the file-ownership split, strict configuration validation, the single durability syscall funnel with its counters and fault hooks, the failpoint registry in enforced one-to-one correspondence with `oracle::AppendFailpoint`, and the journal-level drive seam that lets the crash harness run in Wave A. The enforced gate is `scripts/check-phase1.sh`, which runs `check-phase0.sh` first so the Phase 0 freeze stays enforced. That work is scoped in `doc/phase1-storage-spine-scope.md`, which realizes this section as a file-ownership matrix, a frozen `levcs-store` API, a physical format and durability/recovery specification, per-package deliverables and acceptance criteria, the Wave A adversarial review charter, and the capacity analysis for P2 on the frozen reference hardware. This plan remains authoritative; that document is the Phase 1 realization of it and lists the decisions that must be resolved before Wave A starts.
|
||||
|
|
|
|||
Loading…
Reference in New Issue