lvol/blob: hold client IO across leadership promotion; demote candidates on writer conflict

[schmidt-scaled]

Problem

RCA of k8s_native_scale_break 2026-07-16 23:44 (graceful shutdown of an LVS primary under 64-pod load): a normal leadership failover turned into ~6,000 client-visible INTERNAL DEVICE ERROR (00/06) completions and ext4 journal aborts (filesystems remounted read-only) on 10 PVCs, within 3 seconds.

Root cause chain (validated against pod logs + this source):

1. Pre-confirmation pass-through. The accept-and-promote path on a non-leader (vbdev_hublvol_submit_request -> spdk_lvs_check_active_process) flips the blobstore leader flag immediately, but the confirming lvolstore update starts only after a 500 ms poller delay. Blob IO is queued solely on frozen_refcnt, so IO accepted in that window executes straight into the journal from a node whose leadership was never confirmed. The JC flags those premature appends as a writer conflict (observed at 23:44:22.960, before either survivor's update had started), which demotes the distribs (b_block_new_io=1) — blocking the very superblock write the confirmation update needs.
2. Conflict handler skips candidates. spdk_lvs_change_leader_state() gated its whole freeze -> block_port -> demote sequence on lvs->leader, so mid-promotion candidates were ignored: no port block (total blocked ports: 0 throughout the incident on both survivors).
3. Flush-on-failure. Each failed confirmation attempt flushed the frozen queues with EIO (spdk_blob_update_failed_cleanup per pending lvol) — 4 attempts over 2.5 s, thousands of errors through open ports. As a non-path NVMe status, initiator multipath does not retry these; ext4 aborts its journal on the failed writes.

Fix

Commit 1 — demote promotion candidates on writer conflict. spdk_lvs_change_leader_state() now matches update_in_progress stores in addition to established leaders, so a candidate gets the same freeze -> port-block -> demote treatment that makes established-leader conflicts invisible to clients. Retry state is reset in both conflict paths.

Commit 2 — promotion_hold: hold client IO across the promotion.

• New blobstore flag promotion_hold, engaged at the promotion trigger: user blob IO submitted while set is queued on the existing per-channel queued_io list (same mechanism as the per-blob freeze). No premature journal appends -> the conflict source of this incident is gone.
• Failed confirmation attempts now retry in place (500 ms, existing 4-attempt budget) with all IO still held — no more flush-EIO per attempt.
• The hold is released + flushed on exactly three events:
  • confirmed (spdk_lvs_set_leader(true)): held ops execute in order;
  • writer-conflict demotion: ops abort behind the ports the conflict path blocked;
  • retry budget exhausted: new spdk_lvs_promotion_give_up() blocks the LVS data port and rejects the hublvol port first (mirroring the conflict demotion sequence, incl. the 10 s rule-removal poller), then fails the queues behind the block — initiators see a path drop and retry elsewhere, never a device error.
• Hardening for conflict-during-promotion: late completions of a cancelled update no longer re-claim leadership or trip the timeout_trigger abort; a cancelled candidate's pending_update_lvols are drained via spdk_blob_update_failed_cleanup (releasing their blob_freeze_on_failover refs, preventing double TAILQ insert on the next promotion).

Expected behavior change

Failover of an LVS primary under load: clients see a ~1-2 s latency blip (IO held) instead of EIO. If promotion cannot complete (conflict, wedged JC), IO fails behind blocked ports -> multipath path-down semantics instead of non-retryable device errors.

Status / review notes

• UNTESTED — authored on a non-build host. Needs compile + CI, and a failover soak: graceful shutdown of an LVS primary with saturated tertiary redirect traffic (the 64-pod scale-break scenario) plus the standard single/dual outage suites.
• Reviewers should check: (a) thread context of spdk_lvs_promotion_give_up (called from the failover cpl on the md thread; block_port uses system() — same as the existing conflict path); (b) bs_flush_held_io_channel op-skip condition for still-frozen blobs; (c) interaction with spdk_lvs_nonleader_timeout's residual spdk_bs_set_leader(true) side effect (left in place; harmless under the hold, but could be removed for clarity).
• Base: R26.2-PRE (traced branch). If main-latest builds from a different branch, cherry-pick both commits there — the touched functions are identical modulo small line offsets.

Full incident analysis: sbcli/k8s_scale_break_rca_validation.md.

🤖 Generated with Claude Code