LeVCS/crates/levcs-store/tests/fixtures/phase1-failpoints.json

{
  "schema_version": 1,
  "owner": "A3 StoreHarness",
  "scope": "doc/phase1-storage-spine-scope.md 4-A3 deliverable 2",
  "notes": [
    "One row per levcs_store::failpoints::Failpoint. The row set, its order, and each row's wave are asserted against Failpoint::ALL and Failpoint::wave(); the wave partition is pinned by name in failpoints.rs and is never restated as a count here.",
    "Each row carries two independent derivations of the same answer: physical_state_class -> required_outcome through the class table in tests/support/group_model.rs, and levcs_protocol::oracle::append_publication_expectation(point).recovery_outcome through the frozen Phase 0 oracle. crash_matrix.rs asserts the observed outcome equals both. Two derivations that must agree is the point; a single one would only restate itself.",
    "No catch-all match arm exists in crash_matrix.rs or in the class table. Contract review 2026-07-24-A shipped an unsound recovery classification because a catch-all left most outcomes unasserted.",
    "Wave B rows are driven in-process through StoreEngine::submit rather than through a child process. HardExit is the only action that requires a parent, and no Wave B row uses it; the four fields Wave B adds are statements about a live engine that a dead child cannot be asked about. recovery_outcome is still observed by closing the engine and reopening through StoreEngine::open, so both waves answer that half the same way."
  ],
  "wave_a_assertion_scope": {
    "asserted": [
      "physical_state_class",
      "recovery_outcome"
    ],
    "unasserted": [
      "shard_poisoned",
      "immediate_status",
      "acknowledgment_allowed",
      "later_append_allowed_before_recovery"
    ],
    "reason": "drive.rs has no engine, status root, sequencer, or signer, so there is no status to observe, no acknowledgment to permit or refuse, and no later-append admission control. These four fields are equally unassertable for every Wave A row, so they do not distinguish any row from another. Wave B re-asserts every row's complete FailpointExpectation through StoreEngine::submit."
  },
  "wave_b_exit_conditions": [
    "The pending set is empty: no row carries the wave value that names an undriven obligation. Met. The literal is deliberately absent from this file, because scripts/check-phase1.sh greps for it and a check that matches its own documentation is a check that fails on a green tree.",
    "Every row asserts its complete FailpointExpectation through submit, not only the two halves Wave A could reach. Met for the eight Wave B rows; the nine Wave A rows keep the two halves drive.rs can observe, since drive.rs has no status root to observe the others with and driving them twice would not add a fact.",
    "The Panic action is exercised on DuringCommittedRootBuild, BeforeRootCas, and DuringRootCasRetry, so the panic-in-publication coverage vacated by ruling WriterPanicAfterFence into Wave A is not lost. The failpoint enum names a location; the driver chooses the action; the two axes are independent."
  ],
  "rows": [
    {
      "failpoint": "BeforeAppend",
      "wave": "A",
      "physical_state_class": "NoBytes",
      "required_outcome": "AbsentRetriable",
      "victim_placement": "first",
      "drive": {
        "action": "hard-exit",
        "fault": "none"
      },
      "rationale": "The fault precedes the first byte of the group. Nothing is written, so the contiguous adopted prefix is pinned at p = 0."
    },
    {
      "failpoint": "AfterMarkedResolving",
      "wave": "B",
      "physical_state_class": "NoBytes",
      "required_outcome": "AbsentRetriable",
      "victim_placement": "first",
      "drive": null,
      "submit": {
        "actions": [
          "fail"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "Marking Resolving writes no journal byte, so the physical state is identical to BeforeAppend. The row is distinguished only by immediate_status and shard_poisoned, which is why it needed an engine: the status root is the only place the difference exists."
    },
    {
      "failpoint": "DuringFrameWriteTorn",
      "wave": "A",
      "physical_state_class": "PartialFrame",
      "required_outcome": "AbsentRetriable",
      "victim_placement": "last",
      "drive": {
        "action": "hard-exit",
        "fault": "short-write-tears-victim"
      },
      "rationale": "A short write is not an error; it leaves a durable prefix. The tear removes the 48-byte trailer of the victim, so the frame fails completeness condition 4 and can never be adopted. Frames before it are whole but unfenced, so p ranges over 0..=victim."
    },
    {
      "failpoint": "AfterFrameWrite",
      "wave": "A",
      "physical_state_class": "WholeFrameUnfenced",
      "required_outcome": "EitherWhole",
      "victim_placement": "group-wide",
      "drive": {
        "action": "hard-exit",
        "fault": "none"
      },
      "rationale": "Every frame's bytes are complete and no fence has returned. Whether any of them reached durable storage is exactly what recovery must decide by re-reading the device."
    },
    {
      "failpoint": "EvidenceHandoffFailure",
      "wave": "B",
      "physical_state_class": "NoBytes",
      "required_outcome": "AbsentRetriable",
      "victim_placement": "first",
      "drive": null,
      "submit": {
        "actions": [
          "fail"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "Fires at scope 6.3 step 2, before the group is marked Resolving and before any byte is written. Contract review 2026-07-26-A gave it the BeforeAppend shape: a routine SignerError::Unavailable is an availability event, and poisoning the shard for it would trade a real availability property for a safety property that was never at risk."
    },
    {
      "failpoint": "BeforeFence",
      "wave": "A",
      "physical_state_class": "WholeFrameUnfenced",
      "required_outcome": "EitherWhole",
      "victim_placement": "group-wide",
      "drive": {
        "action": "hard-exit",
        "fault": "none"
      },
      "rationale": "Identical physical state to AfterFrameWrite; the two rows differ only in where in the code the process died, and both must resolve through the device, never deterministically."
    },
    {
      "failpoint": "FenceFailed",
      "wave": "A",
      "physical_state_class": "WholeFrameUnfenced",
      "required_outcome": "EitherWhole",
      "victim_placement": "group-wide",
      "drive": {
        "action": "fail",
        "fault": "fence-eio"
      },
      "rationale": "On Linux an fdatasync error may be reported exactly once, after which the kernel has already dropped the dirty pages, so a retry can return success while the data is permanently gone. A failed fence therefore proves nothing about durability in either direction, which is why contract review 2026-07-24-A moved this row to EitherWhole. The shard poisons and never retries the fence."
    },
    {
      "failpoint": "FenceAmbiguous",
      "wave": "A",
      "physical_state_class": "WholeFrameUnfenced",
      "required_outcome": "EitherWhole",
      "victim_placement": "group-wide",
      "drive": {
        "action": "hard-exit",
        "fault": "none"
      },
      "rationale": "The process dies inside the fence call with no return value at all. Recovery must reopen with a fresh descriptor and re-read from the device before classifying; this row and FenceFailed are what scope 5 charter item 4 directs a reviewer to attack."
    },
    {
      "failpoint": "AfterSuccessfulFence",
      "wave": "A",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": {
        "action": "hard-exit",
        "fault": "none"
      },
      "rationale": "fdatasync returned success over the whole group, so p is pinned at group_len. Recovery must publish the complete durable prefix; a shorter prefix here is acknowledged loss, not a tolerable outcome."
    },
    {
      "failpoint": "DuringCommittedRootBuild",
      "wave": "B",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail",
          "panic"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "The fence returned before the subtree build begins, so the frame is durable and the transaction is committed on the device while the publication that would make it visible never happens. Recovery replays the fenced frame and publishes the receipt."
    },
    {
      "failpoint": "AllocationFailureBeforePublication",
      "wave": "B",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "Same physical state as DuringCommittedRootBuild and reached one step later: the group is fenced, the subtree is built, and the allocation that would carry it into the root fails. Poisoning, because it is inside steps 4-8."
    },
    {
      "failpoint": "BeforeRootCas",
      "wave": "B",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail",
          "panic"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "The last instant at which a fenced group is still unpublished. Everything before the compare-and-swap has succeeded, so the transaction is durable; the shard is poisoned because a group that fenced and did not publish leaves the status root claiming Resolving for a committed transaction."
    },
    {
      "failpoint": "DuringRootCasRetry",
      "wave": "B",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail",
          "panic"
        ],
        "requires_root_cas_contention": true
      },
      "rationale": "Reached only after a lost compare-and-swap: another shard published between this shard's load of the committed root and its swap. The re-merge is against a newer root and the subtree is unchanged, so the outcome is identical to BeforeRootCas - which is the point, because a retry that resolved differently from a first attempt would make publication order observable."
    },
    {
      "failpoint": "WriterPanicBeforeFence",
      "wave": "A",
      "physical_state_class": "WholeFrameUnfenced",
      "required_outcome": "EitherWhole",
      "victim_placement": "group-wide",
      "drive": {
        "action": "panic",
        "fault": "none"
      },
      "rationale": "Unwind rather than a clean error return, so destructors run. A Drop impl that truncates, rewinds the cursor, or flushes a buffer would show up here and nowhere else."
    },
    {
      "failpoint": "WriterPanicAfterFence",
      "wave": "A",
      "physical_state_class": "WholeFrameFenced",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": {
        "action": "panic",
        "fault": "none"
      },
      "rationale": "Ruled into Wave A: the claim it certifies, that a fenced frame survives a panic-unwind death of the writer, is a journal-layer claim and drive.rs contains the panic site. Its whole differential against AfterSuccessfulFence is unwind versus clean error return, and the driver's HardExit path deliberately cannot catch a destructor bug because _exit(3) runs no destructors."
    },
    {
      "failpoint": "AfterRootCasBeforeWaiterWake",
      "wave": "B",
      "physical_state_class": "WholeFrameFencedAndPublished",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "Post-publication. The fence succeeded and the root published, so the transaction is committed; a waiter that never wakes is a hung request, not an absent transaction, and the receipt stays queryable through transaction_status."
    },
    {
      "failpoint": "BeforeResponse",
      "wave": "B",
      "physical_state_class": "WholeFrameFencedAndPublished",
      "required_outcome": "Committed",
      "victim_placement": "group-wide",
      "drive": null,
      "submit": {
        "actions": [
          "fail"
        ],
        "requires_root_cas_contention": false
      },
      "rationale": "Post-publication and pre-response: the receipt is already durable and idempotently retrievable by status or retry, and no reappend is required."
    }
  ],
  "wave_b_assertion_scope": {
    "asserted": [
      "physical_state_class",
      "recovery_outcome",
      "shard_poisoned",
      "immediate_status",
      "acknowledgment_allowed",
      "later_append_allowed_before_recovery"
    ],
    "reason": "With engine.rs the four fields Wave A could not reach are observable through the API a consumer calls: submit returns or refuses, transaction_status performs plan 5.1's two-root read, a second submit to the same shard says whether a later append is admitted, and a close-and-reopen through StoreEngine::open answers recovery_outcome the same way the Wave A rows answer it. Every Wave B row asserts the complete FailpointExpectation, field by field, against oracle::append_publication_expectation.",
    "root_seeded_by": "store_engine_open_state_1",
    "root_seeded_by_reason": "The token above is matched by exact equality in crash_matrix.rs, which is what makes it a tripwire: a substring test passes on a value that has drifted to mean something else, so the matched value is a short stable token and the history lives here. StoreEngine::open on an absent path is the production entry point, startup state 1. Until B1 landed that state the root these rows exercise was created by segment::initialize_root, because open refused to build one, and this field disclosed that charter item 8 weakening: what was asserted below was the production submit, status, and reopen path over a root production did not build. The weakening is closed rather than re-described - the rows now build their store through open and assert that they did, checking the root is absent before the call and that FORMAT exists after, so an open that quietly stopped initializing is caught here instead of returning the harness to seeding its own store.",
    "panic_ownership": "B1 implements phase-aware panic ownership: a panic in scope 6.3 steps 1-3 is definitively absent and leaves the writer thread alive; a panic in steps 4-8 poisons and kills the writer, so a later append is refused NotReady rather than accepted; a panic in steps 9-10 still delivers every receipt. The Panic rows confirm this from outside the engine, through submit and transaction_status, rather than by reading the catch sites."
  }
}