Make a passing P2 bundle reachable, and measure real windows
Two things stood between the harness and a bundle that could encode a pass, and neither was a hardware or duration problem. `run_conditions.checkpointing` had to be "exercised", and no code path could produce it: `CheckpointProbe` had two variants and the probe mapped `Ok(_)` to `EnabledNotReached` unconditionally. The scope doc's claim that "what remains is a run long enough to take a checkpoint, not a mechanism to build" was wrong -- the harness took no checkpoint inside the measured window at all. It does now, on a thread that lives exactly as long as the submitters, so the cost lands in the reported rate. That is the honest place for it: a number that excluded index maintenance would not describe a steady state either. `Exercised` is derived from checkpoints the run took, never from the probe -- the probe can only establish that checkpointing is possible, which is what `enabled_not_reached` already says. Checkpointing made the emitter refuse: sealing had run and left a partial backlog, and `index_maintenance` requires a drained one for `runs_sealed`. A steady-state run always has a partial backlog wherever the clock stops, so the strict check and the periodic checkpoints could not both hold. The run now takes one closing checkpoint *after* the window, which changes no reported rate and leaves the strict check intact. The alternative -- accepting a bounded backlog -- would assert a steady state while carrying a backlog the bundle has no field to report. `run` was a stub that still said submit "is B1 NamespaceTxn ... cannot produce a P2 result". It is implemented, sharing one emitter with `emit-skeleton` so their measurement blocks cannot drift, and differing only in the window arithmetic and the `skeleton` flag. Commits are counted per one-minute window at the commit site: a counter per minute is bounded by the run's length, while a timestamp per commit is bounded by its throughput -- 22 million of them at target. Whole windows only, and a measured run with no whole window is refused rather than falling back to the skeleton's synthesized one. Both halves of section 3 are checked: the percentage the schema records, and the floor no single window may fall below, which nothing downstream would otherwise notice. `scripts/close-phase1.sh` runs the gate, the recovery campaign, and the P2 repetitions, and reports one checklist. It is a reporter and never a promoter: it exits non-zero unless every criterion passes and never edits a bundle to make one pass, so it is safe to rehearse on non-reference hardware -- which is how the `run` stub was found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JGdH5V43XWnj1PdHqiPktQ
This commit is contained in:
parent
b10d5eee88
commit
b5db29354e
|
|
@ -933,11 +933,19 @@ pub enum CheckpointProbe {
|
|||
/// `StoreEngine::checkpoint` refused `NotImplemented`, so no checkpoint was
|
||||
/// taken and none could have been.
|
||||
RefusedNotImplemented,
|
||||
/// `StoreEngine::checkpoint` returned a lease. The measured window still
|
||||
/// contains no checkpoint — the harness takes none inside it — so the
|
||||
/// truthful value is `enabled_not_reached`, and reaching one is the
|
||||
/// emitter's next change rather than a relabelling of this one.
|
||||
/// `StoreEngine::checkpoint` returned a lease, but no checkpoint was taken
|
||||
/// inside the measured window. The value is honest and cannot pass: a run
|
||||
/// whose lookup fan-out grows for its whole duration and which never
|
||||
/// checkpoints is not measuring the steady state a P2 number is supposed to
|
||||
/// characterize.
|
||||
EnabledNotReached,
|
||||
/// At least one checkpoint completed **inside the measured interval**.
|
||||
///
|
||||
/// Counted from checkpoints the run actually took, never from the ability
|
||||
/// to take one -- that is what `EnabledNotReached` already records, and the
|
||||
/// distinction between "the store can checkpoint" and "this measurement
|
||||
/// contains one" is the whole reason scope §7 requires this field.
|
||||
Exercised,
|
||||
}
|
||||
|
||||
/// Whether the index reached a steady state, from what is on the device.
|
||||
|
|
@ -1235,6 +1243,7 @@ impl RunConditions {
|
|||
checkpointing: match observations.checkpoint {
|
||||
CheckpointProbe::RefusedNotImplemented => "unimplemented",
|
||||
CheckpointProbe::EnabledNotReached => "enabled_not_reached",
|
||||
CheckpointProbe::Exercised => "exercised",
|
||||
},
|
||||
index_maintenance,
|
||||
index_run_ceiling,
|
||||
|
|
@ -3318,6 +3327,17 @@ struct EngineRun {
|
|||
/// Group publications, counted as fences: `journal::append_group_and_fence`
|
||||
/// performs exactly one per group and A1's acceptance pins that.
|
||||
groups: u64,
|
||||
/// Checkpoints that completed inside the measured interval. Zero is a
|
||||
/// truthful answer for a run too short to reach the interval, and it is
|
||||
/// what keeps `checkpointing` at `enabled_not_reached` rather than
|
||||
/// promoting a run that never checkpointed.
|
||||
checkpoints_taken: u64,
|
||||
/// Commits in each whole one-minute window of the measured interval.
|
||||
///
|
||||
/// Whole windows only: a trailing partial minute is not a one-minute window
|
||||
/// and including it would report a rate over less than a minute as though
|
||||
/// it were one, which §3's rule is specifically about.
|
||||
window_commits: Vec<u64>,
|
||||
signing_micros_p50: f64,
|
||||
/// Signing samples taken inside the measured interval, on the same basis.
|
||||
signings: u64,
|
||||
|
|
@ -3344,6 +3364,7 @@ fn run_engine(
|
|||
seconds: u64,
|
||||
shard_count: u16,
|
||||
submitters_per_shard: usize,
|
||||
checkpoint_interval: Duration,
|
||||
) -> Result<EngineRun, String> {
|
||||
use levcs_store::segment::RootLayout;
|
||||
use levcs_store::transaction::StagedObject;
|
||||
|
|
@ -3517,7 +3538,64 @@ fn run_engine(
|
|||
let started = Instant::now();
|
||||
let deadline = started + Duration::from_secs(seconds.max(1));
|
||||
|
||||
// Checkpoints taken inside the measured interval, by a thread that lives
|
||||
// exactly as long as the submitters do.
|
||||
//
|
||||
// Inside the window on purpose. A checkpoint taken before or after it would
|
||||
// leave the measured seconds describing a store that never compacted its
|
||||
// index -- fan-out growing for the whole run -- which is the condition
|
||||
// scope §7's stop clause exists to exclude. Its cost is therefore *in* the
|
||||
// reported rate, which is the honest place for it: a P2 number that
|
||||
// excluded checkpoint cost would not describe a steady state either.
|
||||
// Commits per one-minute window, indexed by whole minutes since the window
|
||||
// opened. Counted here rather than reconstructed from per-transaction
|
||||
// timestamps: the §3 rule needs a rate per minute, and a counter per minute
|
||||
// is bounded by the run's length while a timestamp per commit is bounded by
|
||||
// its throughput -- 22 million of them at the P2 target.
|
||||
let window_commits: Mutex<Vec<u64>> = Mutex::new(Vec::new());
|
||||
let checkpoints_taken = AtomicU64::new(0);
|
||||
let checkpoint_failure: Mutex<Option<String>> = Mutex::new(None);
|
||||
|
||||
std::thread::scope(|scope| {
|
||||
{
|
||||
let engine = &engine;
|
||||
let stop = &stop;
|
||||
let checkpoints_taken = &checkpoints_taken;
|
||||
let checkpoint_failure = &checkpoint_failure;
|
||||
scope.spawn(move || {
|
||||
// Polled rather than slept in one block, so the thread leaves
|
||||
// promptly when the window closes instead of holding the scope
|
||||
// open for a whole interval past it.
|
||||
let mut next = Instant::now() + checkpoint_interval;
|
||||
while !stop.load(Ordering::Relaxed) && Instant::now() < deadline {
|
||||
if Instant::now() < next {
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
continue;
|
||||
}
|
||||
match engine.checkpoint() {
|
||||
Ok(_) => {
|
||||
checkpoints_taken.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
// Recorded and fatal to the run rather than retried. A
|
||||
// checkpoint that fails mid-measurement leaves the
|
||||
// store in a state the bundle would have to describe,
|
||||
// and continuing would report a rate for a run whose
|
||||
// index maintenance stopped working partway.
|
||||
Err(error) => {
|
||||
let mut slot =
|
||||
checkpoint_failure.lock().unwrap_or_else(|e| e.into_inner());
|
||||
if slot.is_none() {
|
||||
*slot = Some(format!("{error:?}"));
|
||||
}
|
||||
stop.store(true, Ordering::Relaxed);
|
||||
return;
|
||||
}
|
||||
}
|
||||
next = Instant::now() + checkpoint_interval;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
for shard in 0..shard_count {
|
||||
for submitter in 0..submitters_per_shard {
|
||||
let namespace = namespaces[shard as usize];
|
||||
|
|
@ -3530,6 +3608,7 @@ fn run_engine(
|
|||
let unaccounted = &unaccounted;
|
||||
let results = &results;
|
||||
let latencies = &latencies;
|
||||
let window_commits = &window_commits;
|
||||
let objects_new = &objects_new;
|
||||
let raw_bytes = &raw_bytes;
|
||||
scope.spawn(move || {
|
||||
|
|
@ -3691,6 +3770,19 @@ fn run_engine(
|
|||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner())
|
||||
.push(micros);
|
||||
{
|
||||
// The window this commit landed in, from the
|
||||
// same clock the deadline uses. A commit is
|
||||
// counted when it is acknowledged, which is
|
||||
// what the rate is a rate of.
|
||||
let minute = started.elapsed().as_secs() / 60;
|
||||
let mut windows =
|
||||
window_commits.lock().unwrap_or_else(|p| p.into_inner());
|
||||
if windows.len() <= minute as usize {
|
||||
windows.resize(minute as usize + 1, 0);
|
||||
}
|
||||
windows[minute as usize] += 1;
|
||||
}
|
||||
results.lock().unwrap_or_else(|p| p.into_inner()).push((
|
||||
namespace,
|
||||
operation,
|
||||
|
|
@ -3792,6 +3884,26 @@ fn run_engine(
|
|||
signing.sort_unstable();
|
||||
let signing_p50 = percentile(&signing, 0.50) as f64;
|
||||
|
||||
// One last checkpoint, **after** the measured window has closed.
|
||||
//
|
||||
// A steady-state run is publishing deltas continuously and sealing them
|
||||
// periodically, so at the instant the window closes there is essentially
|
||||
// always a partial backlog -- it is an artifact of where the clock stopped,
|
||||
// not a statement about whether maintenance kept pace. `index_maintenance`
|
||||
// requires a drained backlog to declare `runs_sealed`, and that check is
|
||||
// worth keeping strict: the alternative is asserting a steady state while
|
||||
// carrying a backlog the bundle has no field to report.
|
||||
//
|
||||
// Draining here reconciles the two without weakening either. It is outside
|
||||
// the measured interval, so it changes no reported rate; the checkpoints
|
||||
// that establish maintenance kept pace are the periodic ones *inside* it,
|
||||
// counted separately and reported as `checkpointing`.
|
||||
if checkpoints_taken.load(Ordering::Relaxed) > 0 {
|
||||
engine
|
||||
.checkpoint()
|
||||
.map_err(|error| format!("the closing checkpoint failed: {error:?}"))?;
|
||||
}
|
||||
|
||||
drop(ack);
|
||||
drop(engine);
|
||||
|
||||
|
|
@ -3847,7 +3959,15 @@ fn run_engine(
|
|||
// What the store answers about checkpoints, asked of the store rather than
|
||||
// asserted about it. Asked after the reconciliation so a checkpoint the day
|
||||
// this stops refusing cannot move what the reconciliation read.
|
||||
let checkpoint = probe_checkpointing(&reopened)?;
|
||||
// `Exercised` comes from the run, never from the probe: the probe can only
|
||||
// establish that a checkpoint is *possible*, and the field is about whether
|
||||
// this measurement contains one. A run that took none falls through to the
|
||||
// probe and reports `enabled_not_reached`, which cannot pass.
|
||||
let checkpoint = if checkpoints_taken.load(Ordering::Relaxed) > 0 {
|
||||
CheckpointProbe::Exercised
|
||||
} else {
|
||||
probe_checkpointing(&reopened)?
|
||||
};
|
||||
// Read from the live engine, after the checkpoint probe, so it describes
|
||||
// the state the bundle is about — and before the handle is dropped, because
|
||||
// the reading only exists while the root is open.
|
||||
|
|
@ -3883,7 +4003,29 @@ fn run_engine(
|
|||
// as index runs rather than counted as directory entries.
|
||||
let index_scan = scan_index_runs(root, shard_count);
|
||||
|
||||
// A checkpoint that failed mid-window invalidates the measurement rather
|
||||
// than reducing it: the reported rate would be for a run whose index
|
||||
// maintenance stopped partway, which is not a steady state either.
|
||||
if let Some(error) = checkpoint_failure.into_inner().unwrap_or(None) {
|
||||
return Err(format!(
|
||||
"a checkpoint failed inside the measured interval: {error}. The run is \
|
||||
discarded rather than reported, because its rate would describe a store \
|
||||
that stopped maintaining its index partway through."
|
||||
));
|
||||
}
|
||||
|
||||
// Whole windows only. The run stops at its deadline, so the last bucket is
|
||||
// almost always a fraction of a minute; keeping it would divide a partial
|
||||
// minute's commits by sixty seconds and report a rate no window achieved.
|
||||
let mut window_commits = window_commits
|
||||
.into_inner()
|
||||
.unwrap_or_else(|p| p.into_inner());
|
||||
let whole_windows = (elapsed.as_secs() / 60) as usize;
|
||||
window_commits.truncate(whole_windows);
|
||||
|
||||
Ok(EngineRun {
|
||||
window_commits,
|
||||
checkpoints_taken: checkpoints_taken.load(Ordering::Relaxed),
|
||||
facts: RunFacts {
|
||||
initialization,
|
||||
mutation: MutationPath::StoreEngineSubmit,
|
||||
|
|
@ -3991,6 +4133,14 @@ fn check_global_uniqueness(records: &[AckRecord]) -> Result<UniquenessCheck, Str
|
|||
struct MeasuredRun {
|
||||
latencies_micros: Vec<u64>,
|
||||
groups: u64,
|
||||
/// Checkpoints completed inside the measured interval. The journal seam has
|
||||
/// no engine and therefore no checkpoint, so it is always zero there --
|
||||
/// which is why the drive path declares `unimplemented` rather than this.
|
||||
checkpoints_taken: u64,
|
||||
/// Commits per whole one-minute window, empty for a run shorter than a
|
||||
/// minute. Empty is what makes the §3 rule unevaluable rather than
|
||||
/// vacuously satisfied.
|
||||
window_commits: Vec<u64>,
|
||||
transactions: u64,
|
||||
/// Counted from the objects the store actually staged on the submit path,
|
||||
/// and derived as `transactions * 3` on the drive path, where there are no
|
||||
|
|
@ -4028,6 +4178,9 @@ struct MeasuredRun {
|
|||
impl From<SkeletonRun> for MeasuredRun {
|
||||
fn from(run: SkeletonRun) -> Self {
|
||||
Self {
|
||||
// The journal seam has no engine to checkpoint.
|
||||
checkpoints_taken: 0,
|
||||
window_commits: Vec::new(),
|
||||
groups: run.groups,
|
||||
transactions: run.transactions,
|
||||
objects_new: run.transactions * OBJECTS_PER_COMMIT,
|
||||
|
|
@ -4061,6 +4214,8 @@ impl From<SkeletonRun> for MeasuredRun {
|
|||
impl From<EngineRun> for MeasuredRun {
|
||||
fn from(run: EngineRun) -> Self {
|
||||
Self {
|
||||
checkpoints_taken: run.checkpoints_taken,
|
||||
window_commits: run.window_commits.clone(),
|
||||
groups: run.groups,
|
||||
transactions: run.transactions,
|
||||
objects_new: run.objects_new,
|
||||
|
|
@ -4338,24 +4493,27 @@ fn dispatch(subcommand: &str, flags: &Flags) -> Result<ExitCode, String> {
|
|||
}
|
||||
}
|
||||
} else if subcommand == "emit-skeleton" {
|
||||
emit_skeleton(&repo_root, &workload, &profile, flags)
|
||||
emit_bundle(&repo_root, &workload, &profile, flags, true)
|
||||
} else if subcommand == "run" {
|
||||
Err(
|
||||
"the P2 run goes through StoreEngine::submit, which is B1 NamespaceTxn \
|
||||
(scope 6-B1). Wave A can produce a P1-micro number and a skeleton \
|
||||
bundle; it cannot produce a P2 result."
|
||||
.to_string(),
|
||||
)
|
||||
emit_bundle(&repo_root, &workload, &profile, flags, false)
|
||||
} else {
|
||||
Err(format!("unknown subcommand {subcommand:?}\n\n{USAGE}"))
|
||||
}
|
||||
}
|
||||
|
||||
fn emit_skeleton(
|
||||
/// Emit one bundle, either as a skeleton or as a measured P2 run.
|
||||
///
|
||||
/// One function for both because the two must not drift: a skeleton whose
|
||||
/// measurement block is assembled by different code from the run's would let a
|
||||
/// field be checked in rehearsal and unchecked in the campaign. What differs is
|
||||
/// stated in `skeleton` and nowhere else -- the window arithmetic below, and
|
||||
/// the `skeleton` flag the outcome rule reads.
|
||||
fn emit_bundle(
|
||||
repo_root: &Path,
|
||||
workload: &FrozenWorkload,
|
||||
profile: &FrozenProfile,
|
||||
flags: &Flags,
|
||||
skeleton: bool,
|
||||
) -> Result<ExitCode, String> {
|
||||
if flags.get("allow-unsigned").is_none() {
|
||||
return Err(
|
||||
|
|
@ -4370,8 +4528,24 @@ fn emit_skeleton(
|
|||
|
||||
let root = PathBuf::from(flags.required("root")?);
|
||||
let out = PathBuf::from(flags.required("out")?);
|
||||
let seconds = flags.number::<u64>("seconds", 2)?;
|
||||
// A skeleton is a rehearsal and defaults to seconds; a measured run defaults
|
||||
// to the frozen `p2_p3_measured_seconds`, so the campaign's length comes
|
||||
// from the workload rather than from whatever the caller typed.
|
||||
let seconds = flags.number::<u64>(
|
||||
"seconds",
|
||||
if skeleton {
|
||||
2
|
||||
} else {
|
||||
workload.measured_seconds
|
||||
},
|
||||
)?;
|
||||
let group_len = flags.number::<usize>("group-len", 16)?;
|
||||
// How often the measured window takes a checkpoint. The default is short
|
||||
// enough that even a brief run reaches one, because a run that silently
|
||||
// never checkpointed is exactly the state scope §7 forbids a P2 number from
|
||||
// having been obtained in.
|
||||
let checkpoint_interval =
|
||||
Duration::from_millis(flags.number::<u64>("checkpoint-interval-millis", 1_000)?);
|
||||
|
||||
// Precheck 1 runs against the skeleton's own budget, not P2's: the point
|
||||
// is to exercise the code path and the arithmetic, and demanding 280 GB
|
||||
|
|
@ -4402,7 +4576,14 @@ fn emit_skeleton(
|
|||
let shards = flags.number::<u16>("shards", 4)?;
|
||||
let submitters = flags.number::<usize>("submitters-per-shard", group_len.max(1))?;
|
||||
run_engine(
|
||||
&root, &ack_path, ack_fault, group_len, seconds, shards, submitters,
|
||||
&root,
|
||||
&ack_path,
|
||||
ack_fault,
|
||||
group_len,
|
||||
seconds,
|
||||
shards,
|
||||
submitters,
|
||||
checkpoint_interval,
|
||||
)?
|
||||
.into()
|
||||
} else if path == "drive" {
|
||||
|
|
@ -4466,6 +4647,52 @@ fn emit_skeleton(
|
|||
0.0
|
||||
};
|
||||
|
||||
// The §3 window rule, computed from whole one-minute windows the run
|
||||
// actually recorded.
|
||||
//
|
||||
// A skeleton has none -- it is shorter than a minute by design -- so it
|
||||
// reports the single whole-run rate and a trivially-satisfied percentage,
|
||||
// and its `skeleton` flag is what stops the outcome rule reading that as a
|
||||
// pass. A measured run with no whole window is refused outright rather than
|
||||
// falling back to the same synthesis: a P2 bundle whose window rule was
|
||||
// evaluated over a synthesized window would state the rule as met without
|
||||
// having tested it.
|
||||
let target_rate = flags.number::<u64>("target-rate", 75_000)? as f64;
|
||||
let (window_rates, windows_meeting_target) = if run.window_commits.is_empty() {
|
||||
if !skeleton {
|
||||
return Err(format!(
|
||||
"refusing to emit a measured bundle: the run recorded no whole one-minute \
|
||||
window (it ran for {seconds}s). Section 3's rule is about one-minute \
|
||||
windows, and a run too short to contain one cannot have met it. Pass \
|
||||
--seconds 60 or more, or emit-skeleton if a rehearsal was intended."
|
||||
));
|
||||
}
|
||||
(vec![rate], 100.0)
|
||||
} else {
|
||||
let rates: Vec<f64> = run
|
||||
.window_commits
|
||||
.iter()
|
||||
.map(|commits| *commits as f64 / 60.0)
|
||||
.collect();
|
||||
// Both halves of the rule. The percentage is what the schema records;
|
||||
// the floor is checked here because a single window below 90% of target
|
||||
// fails §3 outright however good the percentage is, and nothing
|
||||
// downstream would notice it.
|
||||
let meeting = rates.iter().filter(|r| **r >= target_rate).count();
|
||||
let percent = (meeting as f64 / rates.len() as f64) * 100.0;
|
||||
let floor = target_rate * 0.90;
|
||||
let percent = if rates.iter().any(|r| *r < floor) {
|
||||
// Reported as a failure of the rule rather than as a separate
|
||||
// field, because the schema has one number for it and a bundle that
|
||||
// passed the percentage while dipping below the floor must not read
|
||||
// as having met the rule.
|
||||
0.0
|
||||
} else {
|
||||
percent
|
||||
};
|
||||
(rates, percent)
|
||||
};
|
||||
|
||||
let mut histogram_input = String::new();
|
||||
for value in &sorted {
|
||||
let _ = write!(histogram_input, "{value},");
|
||||
|
|
@ -4508,14 +4735,8 @@ fn emit_skeleton(
|
|||
latency_p99: percentile(&sorted, 0.99),
|
||||
latency_max: sorted.last().copied().unwrap_or(0),
|
||||
histogram_digest: digest_hex(histogram_input.as_bytes()),
|
||||
// A run shorter than a minute has no one-minute windows. What is
|
||||
// reported is the single whole-run rate, and the percentage is
|
||||
// therefore trivially 100 — which is why the bundle's `outcome` is
|
||||
// `preliminary` and its own gate verdict `not-applicable`. A P2 run
|
||||
// computes real windows; nothing here should be read as having met the
|
||||
// section 3 window rule.
|
||||
one_minute_windows: vec![rate],
|
||||
windows_meeting_target_percent: 100.0,
|
||||
one_minute_windows: window_rates.clone(),
|
||||
windows_meeting_target_percent: windows_meeting_target,
|
||||
ack_journal_digest: run.ack_journal_digest.clone(),
|
||||
acknowledged_loss: run.acknowledged_loss,
|
||||
torn_transactions: run.torn_transactions,
|
||||
|
|
@ -5431,8 +5652,19 @@ sys.exit(1 if errors else 0)
|
|||
let directory = tempfile::tempdir().expect("tempdir");
|
||||
let root = directory.path().join("root");
|
||||
let ack = directory.path().join("ack-journal");
|
||||
let run = run_engine(&root, &ack, AckJournalFault::None, 4, 1, 2, 1)
|
||||
.expect("an engine-driven run");
|
||||
// A one-second run with a half-second interval, so the measured window
|
||||
// contains a checkpoint rather than depending on the default cadence.
|
||||
let run = run_engine(
|
||||
&root,
|
||||
&ack,
|
||||
AckJournalFault::None,
|
||||
4,
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
Duration::from_millis(500),
|
||||
)
|
||||
.expect("an engine-driven run");
|
||||
|
||||
// The exclusion is not vacuous: creating the repositories really did
|
||||
// fence and really did sign, so there is something to exclude. Without
|
||||
|
|
@ -5513,6 +5745,7 @@ sys.exit(1 if errors else 0)
|
|||
1,
|
||||
2,
|
||||
0,
|
||||
Duration::from_secs(3_600),
|
||||
)
|
||||
.err()
|
||||
.expect("a run with no submitter measures nothing");
|
||||
|
|
@ -5527,6 +5760,7 @@ sys.exit(1 if errors else 0)
|
|||
1,
|
||||
0,
|
||||
1,
|
||||
Duration::from_secs(3_600),
|
||||
)
|
||||
.err()
|
||||
.expect("no shard is no repository and no transaction");
|
||||
|
|
|
|||
|
|
@ -0,0 +1,228 @@
|
|||
#!/usr/bin/env bash
|
||||
# Run every piece of Phase 1 exit evidence and report it as a checklist.
|
||||
#
|
||||
# Scope §7 lists seven exit criteria and two stop-condition clauses. Most are
|
||||
# already asserted by tests that run in the ordinary gate; what this script adds
|
||||
# is the part that cannot run anywhere except on reference hardware -- the P2
|
||||
# campaign -- and a single place that says, in one screen, which criteria are
|
||||
# met and which are not.
|
||||
#
|
||||
# It is deliberately a *reporter*, not a promoter. It exits non-zero unless
|
||||
# every criterion passes, and it never edits a bundle to make one pass. A run on
|
||||
# a machine that does not match a frozen hardware profile will correctly report
|
||||
# the P2 rows as failed, which is what makes it safe to run here as a rehearsal.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/close-phase1.sh [--work DIR] [--seconds N] [--reps N] [--cycles N]
|
||||
# [--skip-recovery] [--skip-gate]
|
||||
set -euo pipefail
|
||||
|
||||
repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$repo_root"
|
||||
|
||||
work="$repo_root/target/phase1-closure"
|
||||
seconds=120
|
||||
reps=3
|
||||
cycles=100
|
||||
run_recovery=1
|
||||
run_gate=1
|
||||
|
||||
usage() {
|
||||
sed -n '2,17p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--work) work="$2"; shift 2 ;;
|
||||
--seconds) seconds="$2"; shift 2 ;;
|
||||
--reps) reps="$2"; shift 2 ;;
|
||||
--cycles) cycles="$2"; shift 2 ;;
|
||||
--skip-recovery) run_recovery=0; shift ;;
|
||||
--skip-gate) run_gate=0; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
evidence="$repo_root/evidence/phase1-closure-$stamp"
|
||||
|
||||
# Every row this script can report on. `record` appends one; nothing else
|
||||
# writes to the checklist, so a criterion with no row is a criterion nobody
|
||||
# measured rather than one that silently passed.
|
||||
checklist=()
|
||||
record() { checklist+=("$1|$2|$3"); }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Preflight
|
||||
# ---------------------------------------------------------------------------
|
||||
echo "== preflight ==" >&2
|
||||
|
||||
target_dir="$(cargo metadata --no-deps --format-version 1 \
|
||||
| python3 -c 'import json,sys; print(json.load(sys.stdin)["target_directory"])')"
|
||||
if [ -z "$target_dir" ]; then
|
||||
echo "could not resolve cargo's target directory from cargo metadata" >&2
|
||||
exit 70
|
||||
fi
|
||||
|
||||
if ! python3 -c 'import jsonschema' 2>/dev/null; then
|
||||
echo "jsonschema is not installed; bundles could not be validated" >&2
|
||||
exit 70
|
||||
fi
|
||||
|
||||
mkdir -p "$work" "$evidence"
|
||||
work="$(cd -- "$work" && pwd)"
|
||||
|
||||
# The frozen profile requires nodatacow on the journal and segment directories,
|
||||
# and store-bench refuses on mismatch rather than recording it. Setting the
|
||||
# attribute on an empty parent and creating each store root inside it is the
|
||||
# only way an unprivileged run can satisfy that.
|
||||
bench_parent="$work/bench"
|
||||
rm -rf "$bench_parent"
|
||||
mkdir -p "$bench_parent"
|
||||
chattr +C "$bench_parent" 2>/dev/null || true
|
||||
|
||||
fs_type="$(stat -f -c %T "$work" 2>/dev/null || echo unknown)"
|
||||
if [ "$fs_type" = "tmpfs" ]; then
|
||||
echo "the work directory is on tmpfs; no durability claim survives that." >&2
|
||||
echo "Pass --work with a persistent filesystem." >&2
|
||||
exit 70
|
||||
fi
|
||||
echo "work=$work fs=$fs_type evidence=$evidence" >&2
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The ordinary gate
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ "$run_gate" -eq 1 ]; then
|
||||
echo "== phase 1 gate ==" >&2
|
||||
if bash "$repo_root/scripts/check-phase1.sh" >"$evidence/check-phase1.log" 2>&1; then
|
||||
record "Gate (fmt, workspace tests, crash matrix, whitespace)" PASS "check-phase1.log"
|
||||
else
|
||||
record "Gate (fmt, workspace tests, crash matrix, whitespace)" FAIL "check-phase1.log"
|
||||
fi
|
||||
else
|
||||
record "Gate (fmt, workspace tests, crash matrix, whitespace)" SKIP "--skip-gate"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Acknowledged crash recovery
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ "$run_recovery" -eq 1 ]; then
|
||||
echo "== crash recovery campaign ($cycles cycles) ==" >&2
|
||||
if bash "$repo_root/scripts/verify-store-recovery.sh" \
|
||||
--cycles "$cycles" --work "$work/recovery" \
|
||||
>"$evidence/verify-store-recovery.log" 2>&1; then
|
||||
record "Acknowledged crash recovery ($cycles SIGKILL cycles)" PASS "verify-store-recovery.log"
|
||||
else
|
||||
record "Acknowledged crash recovery ($cycles SIGKILL cycles)" FAIL "verify-store-recovery.log"
|
||||
fi
|
||||
else
|
||||
record "Acknowledged crash recovery" SKIP "--skip-recovery"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The P2 campaign
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# Release build, because `environment_fidelity` refuses `reference_profile`
|
||||
# without one -- a debug bundle is a diagnostic bundle whatever the hardware.
|
||||
echo "== building store-bench (release) ==" >&2
|
||||
cargo build -q --release -p levcs-store \
|
||||
--features bench-harness,store-internals,store-privileged --bin store-bench >&2
|
||||
|
||||
p2_pass=1
|
||||
for rep in $(seq 1 "$reps"); do
|
||||
echo "== P2 repetition $rep of $reps (${seconds}s) ==" >&2
|
||||
root="$bench_parent/p2-$rep"
|
||||
out="$evidence/storage-primitive-$rep.json"
|
||||
rm -rf "$root"
|
||||
if ! "$target_dir/release/store-bench" run \
|
||||
--root "$root" --out "$out" --path submit \
|
||||
--seconds "$seconds" >"$evidence/p2-$rep.log" 2>&1; then
|
||||
record "P2 repetition $rep" FAIL "p2-$rep.log"
|
||||
p2_pass=0
|
||||
continue
|
||||
fi
|
||||
|
||||
verdict="$(python3 - "$repo_root/bench/result-schema.json" "$out" <<'PY'
|
||||
import json, sys
|
||||
import jsonschema
|
||||
|
||||
schema = json.load(open(sys.argv[1]))
|
||||
bundle = json.load(open(sys.argv[2]))
|
||||
|
||||
errors = sorted(
|
||||
jsonschema.Draft202012Validator(schema).iter_errors(bundle),
|
||||
key=lambda e: list(e.path),
|
||||
)
|
||||
if errors:
|
||||
print("INVALID " + "; ".join(f"{list(e.path)}: {e.message}" for e in errors[:3]))
|
||||
raise SystemExit(0)
|
||||
|
||||
rc = bundle.get("run_conditions", {})
|
||||
# The four conditions a passing storage_primitive bundle must declare, checked
|
||||
# by name so the reason a repetition did not close is the field rather than a
|
||||
# schema error twenty levels down.
|
||||
missing = [
|
||||
f"{k}={rc.get(k)!r}"
|
||||
for k, want in (
|
||||
("initialization_path", "store_engine_open"),
|
||||
("mutation_path", "store_engine_submit"),
|
||||
("checkpointing", "exercised"),
|
||||
("index_maintenance", "runs_sealed"),
|
||||
)
|
||||
if rc.get(k) != want
|
||||
]
|
||||
fidelity = rc.get("environment_fidelity")
|
||||
if fidelity != "reference_profile":
|
||||
missing.append(f"environment_fidelity={fidelity!r}")
|
||||
outcome = bundle.get("outcome")
|
||||
if outcome != "pass":
|
||||
missing.append(f"outcome={outcome!r}")
|
||||
|
||||
print("PASS" if not missing else "SHORT " + ", ".join(missing))
|
||||
PY
|
||||
)"
|
||||
case "$verdict" in
|
||||
PASS) record "P2 repetition $rep (schema-valid, conditions met)" PASS "$(basename "$out")" ;;
|
||||
*) record "P2 repetition $rep" FAIL "$(basename "$out"): $verdict"; p2_pass=0 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$p2_pass" -eq 1 ]; then
|
||||
record "P2 >=75k commits/s, p99 <=50ms, $reps repetitions" PASS "storage-primitive-*.json"
|
||||
else
|
||||
record "P2 >=75k commits/s, p99 <=50ms, $reps repetitions" FAIL "see repetitions above"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Checklist
|
||||
# ---------------------------------------------------------------------------
|
||||
echo >&2
|
||||
echo "===========================================================" >&2
|
||||
echo " Phase 1 exit checklist ($stamp)" >&2
|
||||
echo "===========================================================" >&2
|
||||
failed=0
|
||||
for row in "${checklist[@]}"; do
|
||||
IFS='|' read -r what verdict artifact <<<"$row"
|
||||
printf ' %-6s %-56s %s\n' "$verdict" "$what" "$artifact" >&2
|
||||
[ "$verdict" = "FAIL" ] && failed=1
|
||||
done
|
||||
echo "-----------------------------------------------------------" >&2
|
||||
echo " evidence archived in: $evidence" >&2
|
||||
|
||||
{
|
||||
echo "# Phase 1 closure run $stamp"
|
||||
echo
|
||||
printf '| verdict | criterion | artifact |\n|---|---|---|\n'
|
||||
for row in "${checklist[@]}"; do
|
||||
IFS='|' read -r what verdict artifact <<<"$row"
|
||||
printf '| %s | %s | `%s` |\n' "$verdict" "$what" "$artifact"
|
||||
done
|
||||
} >"$evidence/CHECKLIST.md"
|
||||
|
||||
if [ "$failed" -eq 1 ]; then
|
||||
echo "CLOSURE_EXIT=1 (at least one criterion did not pass)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "CLOSURE_EXIT=0" >&2
|
||||
Loading…
Reference in New Issue