Conversation
reworked cleanup implementation for lvol migration
…siting to cleanup target
…lt values and API parity params change (#1198) * Add restart phase for mass create tests * Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change * Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change * Fixing reserved cpu set in openshift baremetal k8s and changing default values and API parity params change * Add K8s host-level core dump collection to e2e test framework Previously, check_core_dump() only detected core dumps inside SPDK pods at /etc/simplyblock/ but never copied them, and completely missed host-level core dumps at /var/lib/systemd/coredump/. When SPDK crashed in K8s, all crash evidence was silently lost after test runs. Changes: - k8s_utils.py: Add copy_core_dumps_from_spdk_pod() to kubectl cp core files from inside SPDK pods to the NFS log directory - k8s_utils.py: Add collect_host_core_dumps() which uses the running SPDK pod (privileged, accesses host via /proc/1/root/) as primary path, with platform-aware fallback when SPDK pod is down: - OpenShift: oc debug node/ + chroot /host - Vanilla K8s: privileged pod with nsenter - Talos: privileged pod with hostPath volume mount (no host binaries needed) - k8s_utils.py: Promote get_all_k8s_node_names() and detect_openshift() from continuous_k8s_native_failover.py to shared K8sUtils class - cluster_test_base.py: Enhance check_core_dump() to actually copy pod core dumps and collect host-level core dumps via _check_host_core_dumps_k8s() - continuous_k8s_native_failover.py: Refactor to delegate to K8sUtils methods * Separate mass stress test into restart and no-restart scenarios Existing mass create/delete tests had node outages (Phase 3b/7b) baked into every run, conflating capacity testing with restart resilience. This splits them into two distinct scenario types: - Existing tests: ENABLE_NODE_OUTAGE defaults to False, so all current tests now run at full entity count without node restarts, enabling proper max-capacity delete testing. - New restart variants: 6 new classes (Docker + K8s) with a 6000 total entity cap, node outages at Phase 3b/7b, and PERSISTENT_RETRY=True. Entity cap formula (MAX_ENTITY_COUNT // (1 + SNAPSHOTS_PER_LVOL)) dynamically reduces lvol count per snapshot ratio: 1 snap/lvol → 3000 lvols 6 snaps/lvol → 857 lvols 10 snaps/lvol → 545 lvols * Fix restore wait to fail test immediately on restore failure _wait_for_restore_task_done had two bugs that masked restore failures: 1. except Exception swallowed AssertionError raised when task status was "failed", turning it into a warning log and continuing to poll until timeout. Added explicit re-raise for AssertionError. 2. After timeout expired, the method logged a warning and returned, letting callers proceed with checksum verification on incomplete restores — producing misleading "checksum mismatch" failures. Now asserts immediately on timeout. Also adds missing non-restart mass stress test variants: - MassCreateDeletePersistent_300x10_6Snap_K8s - MassCreateDeletePersistent_300x10_10Snap_K8s - MassCreateDeletePersistent_300x10_10Snap_Docker * Capping clone creates * Capping clone creates * Fix K8s mass stress test: cleanup deleting infra PVCs, FIO leak, stall timeouts - Scope PVC cleanup to test prefixes only (mcd-pvc-*, clone-pvc-*). The cleanup was deleting ALL PVCs in the namespace including simplyblock infrastructure PVCs, destabilising the cluster. Remove the kubectl delete pvc --all fallback on timeout. - Delete Phase 6 FIO Jobs before Phase 7 clone PVC deletion. FIO pods left running block PVC finalizer removal, causing 50 PVCs to stall for 300s+ and fail the test. - Add DELETE_STALL_TIMEOUT (600s) separate from BOUND_STALL_TIMEOUT for delete verification. CSI provisioner processes PV deletions serially (~1.5s/vol), so 300s is too short at 850+ PVCs. - Fix backend verification to call sbcli_utils.list_lvols() and sbcli_utils.list_snapshots() instead of k8s_utils which lacks these methods, causing AttributeError and skipping verification. * Increases max entity to 9000 * Fix K8s mass create cleanup: respect deadline and preserve_resources_on_failure - Increase MAX_TEST_DURATION from 6h to 10h for K8s tests - Increase CLEANUP_TIMEOUT from 30min to 1h - All cleanup steps now check deadline before running - Replace unbounded sbcli_utils.delete_all_snapshots() with deadline-aware _cleanup_delete_backend_snapshots_with_deadline() - Increase volumesnapshot batch delete size from 50 to 200 - Skip internal cleanup when preserve_resources_on_failure is set and test has failed (soft failure or exception) * Fix upgrade test infrastructure: migration script, version detection, helm release - Docker R25→R26 migration script: add missing mini lvol/snapshot re-write steps that existed in K8s version and UPGRADE.md but were absent in Docker - Docker _is_r25_to_r26_upgrade(): fix detection to handle 'main' as target branch (trigger when base is R25 and target is NOT R25, instead of requiring target to start with 'r26') - K8s HELM_RELEASE_SBCLI: change default from empty string to 'sbcli' so the R25 sbcli chart is actually uninstalled during R25→R26 migration - K8s workflow: export HELM_RELEASE_SBCLI env var to test execution step with conditional value based on upgrade_type * Update K8s upgrade workflow: r25_base_config default to remove_snode_init_container The p2p-migration branch is outdated; remove_snode_init_container is the current R25 branch used for K8s R25→R26 upgrade bootstrapping. * Update K8s workflow image defaults: docker.io → ECR/Docker Hub shorthand Replace outdated docker.io/simplyblock/ registry prefixes across all K8s workflow files to match the current convention (topology-suite pattern): - simplyblock_repository: public.ecr.aws/simply-block/simplyblock - operator_repository: simplyblock/simplyblock-operator - csi_repository: simplyblock/spdkcsi - spdk_image: simplyblock/spdk:main-latest Affected workflows: k8s-native-e2e, k8s-native-stress, k8s-native-e2e-add-node, k8s-native-e2e-node-migration, monitoring-suite-k8s-native, k8s-native-upgrade. * Align upgrade workflow defaults with e2e/stress pipelines - Fix IP defaults: 192.168.10.211 → .210 (mgmt), 205-208 → 201-204 (storage) - Remove NR_HUGEPAGES input (hardcode '2048' in env, override via EXTRA_SN_ARGS) - Fix BOOTSTRAP_DATA_CHUNKS default: 2 → 1 (match e2e/stress) - Fix BOOTSTRAP_ENABLE_NODE_AFFINITY default: false → true - Add missing inputs: EXTRA_CLUSTER_ARGS, EXTRA_SN_ARGS, CLUSTER_SECURITY - Add "Write cluster security/backup config" step before bootstrap - Add EXTRA_CLUSTER_ARGS/EXTRA_SN_ARGS support to bootstrap step Applied to: upgrade-bootstrap.yml, upgrade-bootstrap-single.yml, upgrade-bootstrap-single-v2.yml. * Fix upgrade test: mandatory target images, unconditional DB migration, pip error handling - Make --target_spdk_image and --target_docker_image required in upgrade_e2e.py to prevent silent failures when images are empty - Rename _is_r25_to_r26_upgrade() to _needs_db_migration() and run migration for all cross-version upgrades (not just R25→R26) - Add raise_on_error=True to _pip_install_target so pip failures are caught - Add default TARGET_SPDK_IMAGE and TARGET_DOCKER_IMAGE env vars in all 3 upgrade workflow files * Fix Slack notification skip and add test names to pipeline run-name - Add send_slack_notification to e2e-bootstrap.yml workflow_dispatch inputs (was only in workflow_call, causing null != false to skip Slack step) - Add run-name with test class/case to e2e-bootstrap.yml and e2e-docker.yml * Fix api-parity-audit: pass send_slack_notification and lower max-subsys to 40 * Fix api-parity-audit: register TestAPIParityAudit in ALL_TESTS and add parity keyword - Add TestAPIParityAudit to ALL_TESTS list in e2e/__init__.py so the e2e runner can discover it - Import get_parity_tests and add "parity" keyword handler in e2e.py - Fix K8s backup merge detection in TestBackupPolicyVersionsOne: detect pruning by backup count decrease (K8s CRD) in addition to status field (Docker/sbcli mode) * Add missing sbcli_utils_v2.py required by TestAPIParityAudit The file was missed when test_api_parity_audit.py was originally committed (e2ee3ba). The import fails in CI because the module is not tracked. Force-added past the sbcli* gitignore pattern. * Fix api-parity-audit: register TestAPIParityAudit in ALL_TESTS and add parity keyword - Fix get_io_stats() missing cluster_id argument in cluster.iostats audit - Add detailed summary output to test log (findings by category) - Fail the test on error-level findings instead of silently passing - Register TestAPIParityAudit in ALL_TESTS so --testname discovery works - Disable lvol-level backup delete tests (SFAM-2792): TestBackupRetentionMergeAfterDelete, TestBackupDeleteAndRestore, TestBackupDeleteInProgress - Revert SBCLI_BRANCH default back to 'main' (bootstrap uses it) * Add parity report artifact upload to e2e pipeline Copy api_parity_report.html and api_parity_findings.json from NFS into sbcli/e2e/logs/parity_report/ so they are included in the existing log artifact upload step. * Disable backup delete operations in stress tests (SFAM-2792) - BackupStressMarathon: remove delete_and_backup from weighted operations and comment out _do_delete_and_backup method - BackupStressRetentionMergeCycles: disable entirely since its sole purpose is delete-merge-restore cycles * Add parity audit findings summary to e2e-bootstrap job summary When TEST_CLASS is TestAPIParityAudit, parse the api_parity_findings.json sidecar and append a severity breakdown table (errors/warnings/info) to the GitHub Actions job summary.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+45
to
+57
| uses: ./.github/workflows/e2e-bootstrap.yml | ||
| with: | ||
| TEST_CLASS: TestAPIParityAudit | ||
| RUN_LABEL: api-parity | ||
| STORAGE_PRIVATE_IPS: ${{ inputs.STORAGE_PRIVATE_IPS || '192.168.10.201 192.168.10.202 192.168.10.203 192.168.10.204' }} | ||
| API_INVOKE_URL: ${{ inputs.API_INVOKE_URL || 'http://192.168.10.210/' }} | ||
| BASTION_IP: ${{ inputs.BASTION_IP || '192.168.10.210' }} | ||
| MNODES: ${{ inputs.MNODES || '192.168.10.210' }} | ||
| SBCLI_BRANCH: ${{ inputs.sbcli_branch || 'main' }} | ||
| CUSTOM_IMAGES: ${{ inputs.CUSTOM_IMAGES || 'spdk="simplyblock/spdk:main-latest" docker="simplyblock/simplyblock:main"' }} | ||
| send_slack_notification: ${{ inputs.send_slack_notification || github.event_name == 'schedule' }} | ||
| BOOTSTRAP_MAX_SUBSYS: "40" | ||
| secrets: inherit |
More recent linter versions support more checks and enable them by default. Generally, we should try to adopt them to arrive at a more cherent codebase. Instead of pinning an old ruff version, this explicitly disables all checks that we fail at present. We should try and remove the violations and enable the checks successively. For those checks that we simply do not agree with, we can document explicit exceptions.
…l this process before it's completion
…d CVEs The Trivy job in .github/workflows/security.yml was failing daily. Two separate causes: - docker/Dockerfile's `COPY . /app` was shipping the repo's local .tox/.mypy_cache/.ruff_cache/.pytest_cache dirs into the scanned image (.tox alone added ~900MB across several stale python3.9/3.12/3.13 virtualenvs, each with its own old, genuinely vulnerable pip/wheel). None of these are excluded by .dockerignore, so every local tox/mypy/ ruff run before a build fattens the shipped image with real CVEs. - The remaining 3 findings (msgpack 1.1.2, setuptools 70.3.0) come from pip 26.2's own vendored bundle (pip/_vendor), not from anything we install; pip still ships those exact versions upstream, so there is no fix available from our side. Added .trivyignore.yaml to suppress just those 3 finding IDs, wired via the trivyignores input. Verified locally end-to-end (build docker/Dockerfile, scan with aquasec/trivy:latest replicating the workflow's flags) — image shrank from 1.96GB to 1.09GB and the scan now exits 0 with zero findings. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fix: lvol migration getting killed and cleanup not running
#1203) calculate_core_allocations()'s colocate_lvs branch put lvol_poller_core on app_thread_core's slot. Since jc_singleton_core is assigned in every branch, storage_node_ops.py's poller-group RPC unconditionally preferred jc_singleton_mask (added in e3e8fd0 to force the two onto the same core) � which silently clobbered the >=32 vCPU tier's deliberately dedicated lvol_poller core every time, defeating the point of giving it one. Colocate with jc_singleton_core's slot instead, so lvol_poller_mask becomes the single source of truth: equal to jc_singleton_mask when nothing dedicated was assigned, distinct when it was. Update add_node()/_restart_storage_node_impl() to use lvol_poller_mask directly (jc_singleton_mask only as a last-resort fallback if that reservation came up empty) and fix their now-stale comments to match. Add regression coverage for calculate_core_allocations' colocation behavior across all three size tiers � there was none before.
…9 incident) A healthy node was force-shut and restarted a second time: while its restart task was still running on the parallel pool, the dispatch mode flipped (fd_dead_recovery_allowed went false as the first domain peers came back ONLINE) and the main loop's inline path — which consulted neither _restart_inflight nor _node_inflight — re-entered the same task. Every guard in the second entry was blinded by two lost updates from full-object writes of stale in-memory copies: the defer path's task.write_to_db un-canceled the task and wiped its owner lease, and _persist_target_device_event reverted the node's committed in_restart->online flip back to in_restart. - tasks_runner_restart.main(): single dispatch path — every execution checks the inflight maps and registers its future; serialized mode submits identically and waits (fut.result()), so parallel<->inline mode flips are harmless in both directions. - tasks_runner_restart: new _task_finish/_task_update helpers write tasks via db.atomic_update (CAS on the fresh row, updated_at lease stamp) — a write can no longer resurrect a concurrently canceled/done task or clobber its owner lease. All task_runner_node writes converted; a lost CAS means another actor owns the outcome and the runner stops. Plus a fresh task re-read immediately before the destructive shutdown step. - distr_controller._persist_target_device_event: atomic_update whose mutator touches only the device entries — concurrent node status flips survive device-event fan-out during restart waves. - storage_node_ops.shutdown_storage_node: honour the result of the final OFFLINE set_node_status instead of reporting success over a half-committed shutdown; deliberately NOT whitelisting the RESTARTING->OFFLINE flip (it would strand a genuine concurrent restart whose final ONLINE CAS then gets refused). - shutdown_storage_node/check_node_shutdown_preconditions: thread current_restart_task_id (bare uuid) through so the restart runner's own cleanup shutdown is not reported as a competing restart task. - tests/unit/tasks: fake db mirrors the atomic_update contract. Follow-ups (not in this change): unconditional node-RESTARTING entry guard + lease-aware watchdog skip, atomic _add_task creation (duplicate task TOCTOU), gating the self-heal force-ONLINE branch, converting task_runner_device writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design rule: the control plane forces LVS leadership only where it is structural and race-free — lvstore creation, activation, and the restart flow's fenced demote->grant handoff (recreate_lvstore ### 5/### 7). Recovery paths repair the redirect topology and wait for IO-driven self-promotion; they never grant. Incident 2026-07-30 (sb_logs_20260730_195000_30m, LVS_9): the guarded last-resort grant fired from the lvol-migration runner 0.45s before the dead primary's restart task was even created (the "no handoff task active" guard raced task creation), seating the secondary as writer. When the primary's restart later ran its own fenced demote->grant handoff, writer_conflict events fired on jm_vuid=9 and the demoted secondary's JM write lock lingered for its full ~64s lease (jfi_r_wr_lock tms_delta=64103ms on both JMs), blocking the new leader's journal writes for that window. Earlier motivations stand too: a CP-forced grant outside the restart flow skips the primary's blob-md reload (2026-07-06 LVS_13 stale metadata) and the run-20260725 grant/demote flapping. - _recover_leaderless_lvs: keep the single-flight lock, hublvol repair and bounded self-promotion wait; drop the last-resort bdev_lvol_set_leader(leader=True) — still-leaderless now returns None and object operations keep failing fast (no_leader_cache) until IO promotes the primary or a restart re-places leadership. - Remove the grant-only guard helpers _leadership_moving_tasks_active and _taker_jm_quorum_ok. - tests: leaderless recovery must never call bdev_lvol_set_leader. Open data-plane item (separate): the demote path does not release the JM write lock, so every handoff — including IO-driven self-promotion — waits out the ~64s lease before the new writer can write the journal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion The namespaced-subsystem pick (get_next_available_subsystem_on_node) and the in_creation record write were separate reads/writes, so two concurrent creates/clones could both count the same shared subsystem as having one free namespace slot and both join it past capacity. DBController.claim_lvol_ns_slot now runs pick + record write in ONE FDB transaction: the record itself is the slot claim (occupancy is recounted from lvol records inside the transaction). A per-node allocator key gives concurrent claims a read conflict so the loser retries and recounts with the winner's record present, while the lvol-table read itself is a snapshot read (no conflict range over the whole table — unrelated lvol writes must not abort claims). Wired into create_lvol, snapshot clone, and the -32602 add-time fallback, which now also excludes the subsystem SPDK just rejected instead of being able to re-pick it forever. release_lvol_ns_slot is the rollback half: record + mini removed in one transaction, releasing the slot atomically. All create/clone rollback paths and the delete-path record removals go through it. Also: get_next_available_subsystem_on_node treats only all_lvols=None as "not provided" — an empty in-transaction snapshot result used to trigger a fresh full-table read outside the caller's transaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 5c4c2ff gated every migration target-lvol create on _ensure_lvstore_primary_leader, which queries bdev_lvol_get_lvstores — a method the migration-tier mock SPDK server never implemented. Every migration integration test has failed at start_migration since ("Lvstore lvs_tgt not found on <node>", 112 failures, red since 2026-07-27's first post-merge run). The mock node is by construction the sole primary/leader of its lvstore, so the handler reports lvs_primary + "lvs leadership" true and returns an empty list for unknown lvs names (which the guard maps to its not-found error). Registered in _METHOD_ERROR_CODES so failure-rate injection exercises the guard's retry path too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sts, replica-gate suspend Three independent breakages: 1. argcomplete 3.7.1 (released 2026-08-04, unpinned) uses PEP 604 unions at runtime and breaks py3.9 at import — the whole unit job died at collection. Pinned <3.7.1. 2. 13 mock-based migration tests drifted from production signatures: _get_target_secondary_node grew a required src_node_id (overlap-drain rule), _collect_snap_ancestry now returns the set (the protect helper was folded into set subtraction), _delete_bdev_blocking passes sync/special_delete and polls time.monotonic(), start_migration grew cluster preconditions, and snaps owned by a volume that migrated away are now cleanup candidates. Updated the tests to current behavior and added coverage for the new overlap-drain and migrated-away-owner rules. 3. Real bug caught by test_secondary_blocked_when_secondary_in_bad_state: a target secondary/tertiary in a transient state (in_restart, ...) returned an error triple from the phase handlers, which the dispatcher counts against the migration retry budget — 5 quick retries later the migration entered cleanup_target and FAILED because a replica was restarting. The six replica-state gates now suspend via error_message without charging retries, matching the target-offline and cluster-not-active gates (same design as 5e57b45 for source-offline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nvariant, no FD migration Role placement on FD clusters only affects availability (durability is covered independently by chunk anti-affinity and the JM per-domain caps), so the role-level contract is: every LVS keeps at least one cross-domain non-leader role, ensuring a full-domain outage never strands an LVS with zero paths while its data is still readable. - planner: fd_interleaved_host_order, rotation_layout (public, with the FTT+1 host floor), compute_fd_layout_violations, fd_balance_violation. Interleaved rotation makes every secondary cross-domain when balanced and confines the +1 case to exactly one degraded LVS (tertiary-covered). - preconditions: add/remove/current admission on per-HOST (mgmt_ip) domain counts — spread <= 1, and >= 2 hosts per domain once an HA layout exists. Pre-activation assembly stays unrestricted. - executor: FD-aware fresh planning — recover the actual rotation from the secondary-pointer chain (DB-order fallback repairs drift), pass the actual layout as current_layout, and choose the newcomer insertion point by trying cyclic shifts; refuse when no placement satisfies the invariant (e.g. FTT1 at odd populations). Multi-slot hosts are refused on FD expansion for now. - storage_node_ops: host failure-domain is immutable (re-add/slot-add with a different id is rejected — FD migration is not supported); expansion add and node removal run the FD admission checks; _pick_replica_relocation_node enforces cross-domain HARD when the relocated replica would be the primary's only cross-domain role. - cluster_ops: fresh activation of an FD cluster requires >= 2 domains, EQUAL host counts per domain and single-domain hosts, and assigns sec/tert via the interleaved rotation instead of greedy picks. Re-activation is deliberately not blocked (recovery path). tests: tests/unit/test_fd_topology_policy.py (38 cases); full unit tier green (912 passed), ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test takes a random node offline for 2s mid-migration and asserted STATUS_DONE unconditionally. When the random pick is the TARGET and a runner tick observes the outage, the runner's deliberate policy is to fail the migration into cleanup_target (rollback) — not suspend/resume as the docstring claimed — so the test failed ~25% of runs by construction (observed: green and red slow-job runs on the same sha). Accept both documented outcomes for the target case: DONE (outage window missed between ticks) or FAILED with a verified completed rollback (phase cleanup_target, volume still served from the source). Source/passive picks still require DONE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update tests.yml
This makes dependency and venv handling automatic.
This avoids version warnings about exectutions on the wrong node version.
In this case (method not found) we use RPC jc_get_jm_status
Three hard limits, enforced on every create path (lvol create, snapshot create, clone): - 6000 objects per lvstore (MAX_OBJECTS_PER_LVSTORE): lvols + clones + snapshots, counted against the owning node — each node owns exactly one lvstore, so this is the per-lvstore count and a host temporarily serving a second LVS (takeover) gives each active lvstore an independent budget. Replaces the per-core cap (cores x 2000); the hard limit no longer scales with the SPDK core mask and no longer exempts nodes with a missing/unparseable mask. check_node_object_limit -> check_lvstore_object_limit. - 50 namespaces per subsystem (MAX_NAMESPACES_PER_SUBSYSTEM): caller-supplied max_namespace_per_subsys above the cap is rejected at create; joins into legacy subsystems recorded with a larger max stop at the cap (get_next_available_subsystem_on_node bounds the recorded max — the FDB claim transaction recounts through the same function, so the join path is enforced transactionally); new subsystems seeded in _claim_lvol_ns_slot_tx clamp their recorded max to the cap. The default for namespaced creates stays LVO_MAX_NAMESPACES_PER_SUBSYS (32). - 75 subsystems per node (MAX_SUBSYSTEMS_PER_NODE): applied as a ceiling over the node's configured max_lvol (max_subsystems_for_node = min(max_lvol, 75), preserving smaller or zero configured values) at placement, the advisory pre-check, the clone path, and authoritatively in _claim_lvol_ns_slot_tx. Counting stays on the TTL-cached mini records (no full-table scan on the create path — run-20260721 regression). Concurrent creates within one cache window can overshoot the lvstore cap slightly; strict never-exceed would need a transactional per-lvstore counter like the namespace-slot claim. Tests: tests/unit/test_object_limit_per_lvstore.py (replaces the per-core suite), tests/unit/test_subsystem_limits.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run mass_create_delete_docker-20260730-134740: the async+sync delete
chain of a SINGLE lvol/snapshot took 4-7 minutes (a 7-object
clone->snapshot chain ~50 min), while every SPDK step completes in
milliseconds (e.g. SNAP_2379: API phase-1 15:35:22, the monitor's one
and only touch at 15:39:11 finished poll + 3 sync deletes + DB in
73 ms). The latency was entirely monitor-side:
1. Inline sleeps inside the SERIAL per-object loop: snapshot_monitor
slept 1s after every phase-1 (1,006 sleeps = ~17 min of one hour;
cycle time 4-5 min), lvol_monitor slept 3s. Removed - the
delete-status poll already handles a still-running async delete
(ret 1 = retry next cycle).
2. Leadership probe (1-3 bdev_lvol_get_lvstores RPCs) ran once PER
OBJECT per cycle. Now cached per cycle/pass; mid-cycle leadership
moves are still handled by the poll error codes (-35/4).
3. Chain hops cost a full extra cycle each:
- the clone-blocker check used the cycle-start mini-lvol snapshot,
so a snapshot stayed blocked a whole cycle after its clone
finished; re-read the single blocking record before deferring.
- the 30s idle interval bounds every hop; while any in-deletion
object exists both monitors now re-scan every 2s
(LVOL_MONITOR_DELETION_INTERVAL_SEC), idle clusters keep 30s.
Expected: single-object delete finishes within one short cycle
(seconds); an N-hop chain drains in ~N short cycles instead of
N x 4-7 min. Full test suite passes (923 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… to batch migration for consistency
…ce to a subsystem with active migration
…tion use ensuring 10 second cooldown between detch and attach
…d compression-thread path (#1217) The compression-thread JM CPU layout was gated behind JM_COMPRESSION_THREAD_ENABLED, permanently False on main with no way to enable it -- both the dedicated-compression-core allocation and the lvol_poller/jc_singleton colocation it gated were dead code. Removed the flag and the dead compression-core allocation entirely (bdev_jm_create's compression_thread/compression_cpu_mask params, the RPC call, and the model field are left in place but now always inert, since nothing references the deleted constant anymore). Behavior change: colocate_lvs (lvol_poller co-locating with jc_singleton's core below 32 vCPU, freeing a core) is now unconditional on vcpu count instead of also requiring the removed flag -- this part of the gated logic is kept and permanently enabled, only the compression-specific half is dropped. The return tuple's 9th (compression_core) slot is kept, always empty via the existing assigned.get(..., []) default, rather than removed -- several callers index into this tuple positionally (distribution[8]), and shrinking it would silently IndexError rather than fail loudly. tests/unit/test_lvol_poller_jc_colocation.py (previously untracked, written for the still-unmerged lvol_poller/jc_singleton colocation fix that landed without it) rewritten to drop the now-removed compression_enabled parameter and assert the sole remaining behavior across all three size tiers.
batch migration web api endpoints
2026-08-06 soak iter-50: a manual CLI `sn restart` and the restart task runner drove the SAME node's restart concurrently. The task runner calls restart_storage_node with force=True (bypassing every status/task guard), try_set_node_restarting's FDB tx only excluded PEERS (it explicitly skipped the target node), and the task lease could not discriminate the actors: both share the one NODE_RESTART task, and the lease owner id (hostname) collides for a CLI and a runner service on the same mgmt host. Result: dueling spdk_process_start calls replacing each other's SPDK container mid-restart (banners 3s apart), the refused CLI attempt's failure cleanup killing the runner's container, and the runner's retry shutting down over the CLI's next in-flight attempt at finalization. a61b00a's inflight maps could not catch this — they are process-local to the runner service. The claim is an (owner-token, timestamp) pair on the StorageNode row: - acquired atomically inside _try_set_node_restarting_tx: refuse when the target itself is RESTARTING/IN_SHUTDOWN under a FRESH claim held by anyone else — in BOTH tx modes (allow_concurrent_peers relaxes peer exclusion, never same-node exclusion). force does not bypass it. - a stale claim (older than RESTART_CLAIM_TTL_SEC) or an empty one is takeover-able: the transferable-ownership resume path for a driver that died mid-restart stays fully alive. - heartbeated by the restart_storage_node wrapper (owner-matched CAS, lands only once acquired), released on every exit path. - the wrapper's failure cleanup (_kill_spdk_until_dead + OFFLINE flip) now runs only when THIS call holds the claim — proven by token, not inferred from pre_status, which was also true for attempts refused before acquisition and made their cleanup destroy the rightful owner's in-flight SPDK container. - check_node_shutdown_preconditions refuses (not force-overridable, not task-id-exemptable — both actors share the task) to shut down a node mid-transition under a live foreign claim; a dead driver's claim expires within the TTL, which is the sanctioned takeover wait. - task_runner_node defers without consuming retry budget when a live foreign claim exists, mirroring the peer-exclusion pre-check. 27 new unit tests (tests/unit/test_restart_claim.py); unit tier 950 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.