Add stuck pod describe - #1208
Open
RaunakJalan wants to merge 74 commits into
Open
Conversation
When FIO pods are stuck in PodInitializing or time out during wait_fio, save kubectl describe output to stuck_pod_describes/ directory for post-mortem debugging of volume mount or CSI failures.
TestSingleNodeOutage: The checksum utility pod was deleted with --wait=false, causing a name collision when re-created 24s later. The stale pod returned empty checksums, failing the assertion. Fix: delete_pod() now accepts wait=True; _generate_checksums_dual uses it to block until the pod is fully removed. TestSingleNodeFailure: Stale VolumeSnapshots (snapshot-1, snapshot-2) from TestSingleNodeOutage persisted because cleanup used --wait=false and cleanup_k8s_leftovers only matched snap-* prefix. The next test's kubectl apply hit "persistentVolumeClaimName is immutable", silently reusing the stale snapshot that pointed to a deleted backend object. Fixes: - delete_pod/delete_volume_snapshot: add wait parameter - _generate_checksums_dual: wait for pod deletion - Teardown: wait for snapshot deletion + catch-all for untracked snapshots - cleanup_k8s_leftovers: match snapshot-* in addition to snap-* - create_volume_snapshot: detect and remove stale snapshot before apply
- Create audit pool in Phase 0 before read-only audits so pool.get, pool.iostats, volume.crud, and snapshot.crud are no longer skipped with "no pools available" - Change all not_tested and interface_error severity from WARNING/INFO to ERROR — if an interface fails, it should fail the test - Capture full API call details in every finding: the CLI command or HTTP method+path, HTTP status code, and response preview - _run_cli now returns a dict with data, stdout, stderr, and command so failures include the actual stderr output for debugging - Count mismatches now include sample IDs from each interface so it's clear which items are present vs missing - Report HTML updated with new columns: API Call, HTTP Status, Response preview, and sample IDs for count mismatches - pool.crud uses a separate pool name (parity_crud_pool) so it doesn't conflict with the audit pool lifecycle
Full end-to-end runbook covering Phase 1 (R25.x legacy Helm deployment), Phase 2 (pre-upgrade data setup with FIO/MD5/snapshots/clones), Phase 3 (10-step maintenance window migration), and Phase 4 (post-upgrade validation including old data verify, new provisioning, and outage tests).
- TC-BCK-018: wait for backup completion before PVC deletion (K8s operator re-resolves PVC during reconciliation, causing BackupSourceResolutionError when PVC is deleted mid-backup) - TC-BCK-172: pass restore_size="10G" for resized lvol restore (PVC size must match backup size) - TC-BCK-175: remove -d debug flag that caused false positive error assertion on stderr - Move topology backup tests (TestBackupAfterNodeAdd, TestBackupWithFioOnNewNode, TestBackupAfterNodeMigration, TestBackupDuringMigration) into separate get_backup_topology_tests() so they don't run in the regular backup pipeline without required NEW_NODE_IPS / migrate_to_worker params - Add "backup-topology" keyword to e2e.py test runner
Only TestSequentialNodeAdd needs 2 new nodes; all other add-node tests work with 1.
- csi_repository: add default simplyblock/spdkcsi (was empty) - csi_tag: add default latest (was empty) - ifc_names: br-ex:enp2s0f0 (was ens18:enp1s0) - cluster_environment: openshift-baremetal (was local)
- Add cluster_security input (none/backup) to add-node and migration workflows (both workflow_call and workflow_dispatch) - Deploy MinIO + backup-credentials secret when backup is enabled - Add BACKUP_SPEC to StorageCluster CR for backup-enabled runs - Auto-detect backup tests by testname containing "Backup" - Pass cluster_security through topology suite parent workflows
Remove cluster_security from workflow_dispatch inputs (exceeds GitHub's 25-input limit). Keep it in workflow_call for programmatic use. Backup is auto-enabled when testname contains "Backup" so no functionality lost.
Change `inputs.tls_enabled == 'true'` to `inputs.tls_enabled` in if conditions. The truthy check works for both boolean true (from workflow_call) and string "true" (from workflow_dispatch), avoiding type coercion issues when parent workflow passes boolean to child.
The DaemonSet schedules storage-node pods on any node with the simplyblock.io/role=mgmt-plane label. Pre-labeling new_worker_nodes caused pods to start on nodes not yet in the StorageNodeSet, resulting in Init:CrashLoopBackOff. The test itself handles labeling when it adds the node via StorageNodeSet CR update.
Labels persist across pipeline runs. During cleanup, remove simplyblock.io/role from new_worker_nodes and reset their hugepages so the DaemonSet doesn't schedule storage-node pods on them before the test adds them via StorageNodeSet CR.
…elete stuck pods Three fixes for K8s add-node and migration pipeline failures: 1. Pipeline cleanup: remove /etc/simplyblock from all worker nodes (both initial and new) during cleanup phase, preventing stale device config from causing init container CrashLoopBackOff on subsequent runs. 2. Add-node test: add new workers one at a time instead of all at once. Each node's StorageNode CR is created, stale pods are deleted, and the node is waited on to come online before proceeding to the next. 3. Both tests: after creating StorageNode/StorageNodeOps CRs, delete any existing simplyblock-storage-node-ds pods on the target worker so the DaemonSet recreates them with correct StorageNodeSet configuration.
When creating StorageNode CRs for add-node expansion, read driveSizeRange and pcieModel from the parent StorageNodeSet and include them in the overrides block. This ensures the init container can discover the correct SSD devices on the new worker node.
Root cause: after creating a StorageNode CR, the test immediately deleted the stale DaemonSet pod. The operator hadn't yet updated the per-node-config ConfigMap with the new worker's MAX_LVOL value, so the recreated pod started with MAX_LVOL=0 and crashed in s-node-api-config-generator init container. Fix: poll the per-node-config ConfigMap until it has an entry for the worker node before deleting any stale pods. This ensures the DaemonSet recreates the pod with the correct configuration.
Three issues fixed: 1. TestSequentialNodeAdd / TestAddNodeSnapshotCloneOnNewNode fail with "Only 4/5 snode-spdk pods" because they don't wait for the operator to populate the per-node-config ConfigMap before the DaemonSet pod starts. Added wait_for_per_node_config + delete_storage_node_pods_on_worker calls to both K8s code paths in test_add_node_edge_cases.py. 2. Topology suite Slack summaries show "?/? passed, ? failed" because the regex patterns only handle the k8s-native summary format (inline "**Total:** N") but not the e2e-bootstrap format (table with emojis). Updated all three parent workflows to handle both formats. 3. K8s child workflows always send individual Slack notifications even when send_slack_notification=false because the condition (inputs.send_slack_notification || 'true') == 'true' evaluates to true for both true and false inputs. Changed to != false.
RaunakJalan
force-pushed
the
add-stuck-pod-describe
branch
from
August 3, 2026 05:38
aac190b to
703a271
Compare
…tion The wait_for_per_node_config and delete_storage_node_pods_on_worker calls were placed AFTER the StorageNodeOps CR creation, which meant our test was deleting pods while the operator was actively managing the migration. This broke the operator's DNS/endpoint resolution, causing it to hang at "waiting for DNS to be published" indefinitely. Move these calls BEFORE the StorageNodeOps CR creation so the stale crashing pod (MAX_LVOL=0) is fixed first, and the operator finds a healthy pod when it starts the migration.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| self.container_nodes[ip] = containers | ||
|
|
||
| try: | ||
| cluster_details = self.sbcli_utils.wait_for_cluster_status( |
| self.storage_nodes.append(ip) | ||
|
|
||
| try: | ||
| cluster_details = self.sbcli_utils.wait_for_cluster_status( |
RaunakJalan
force-pushed
the
add-stuck-pod-describe
branch
from
August 5, 2026 20:41
4db4738 to
71339bf
Compare
…ine with UPGRADE.md - Add mc admin trace background logging to 8 workflow files for backup test runs: K8s (port-forward): k8s-native-e2e, k8s-native-e2e-node-migration, k8s-native-e2e-add-node Docker (direct): e2e-bootstrap, stress-run-bootstrap, monitoring-suite-docker, upgrade-bootstrap, upgrade-bootstrap-single - Trace logs are saved as artifacts for post-run debugging - Align k8s_major_upgrade.py with UPGRADE.md: add post-upgrade old data verification (FIO verify-only, fresh IO, new snapshots on old PVCs), node outage test, pre-upgrade state capture, and final checklist assertions - Update UPGRADE.md: add worker node labeling section, Pool CR name must match backend, StorageCluster CR name must match upgrade secret - Add worker node label step to k8s-native-upgrade.yaml for R25 storage plane discovery
Previous test runs leave io.simplyblock.storagenodeset labels on worker nodes. When a new run starts, the operator re-deploys the DaemonSet which immediately schedules pods on all labeled nodes — including migration targets and add-node spares that aren't in the StorageNodeSet workerNodes list. Those pods crash with MAX_LVOL=0 because the per-node-config ConfigMap has no entry for them. Add a cleanup step to all five K8s workflows that removes the storagenodeset label from all worker nodes before the new run begins.
The simplyblock image tag is 'main', not 'main-latest'. The incorrect default caused upgrade tests to fail when no custom image was specified.
The operator expects a Running storage-node pod on the migration target worker before it can process a StorageNodeOps CR. Previously, the pod only existed due to stale labels from earlier runs. On a clean cluster, no pod was scheduled and the operator hung at "waiting for storage-node pod on worker". Fix by adding the target worker to the StorageNodeSet (via StorageNode CR with expand=true) before creating the StorageNodeOps. This follows the same pattern as add-node: create CR -> wait for ConfigMap -> fix stale pods -> wait for snode-spdk pod -> then proceed with migration.
This reverts commit 439a0d0.
- Replace hardcoded minioadmin credentials with MINIO_ACCESS_KEY / MINIO_SECRET_KEY env vars from GitHub secrets in all 5 Docker workflows (e2e-bootstrap, monitoring-suite-docker, upgrade-bootstrap, upgrade-bootstrap-single, stress-run-bootstrap) - Add /tmp fallback for mc binary install when /usr/local/bin write fails (curl exit 23 on runners with permission/disk issues) - Fix TEST_CLASS defaults: use exact class names (TestMajorUpgrade, TestMajorUpgradeSingleNode) instead of substrings that match multiple test classes
The node-migration and add-node K8s pipelines were missing this input, so test teardown deleted lvols while FIO was still running, causing spurious err=121 (Remote I/O error). Default to true (matching k8s-e2e).
Move preserve_resources_on_failure to workflow_call only in the migration pipeline and topology suite (same pattern as cluster_security). Defaults to true when not provided.
Run the operator's cleanup-simplyblock.sh before cleanup_k8s.sh for more thorough cleanup of stale resources after failed migrations. Also fixes 25-input limit for node-migration workflow_dispatch by moving preserve_resources_on_failure to workflow_call only.
Previous test runs leave behind cluster-scoped StorageClass and VolumeSnapshotClass objects with provisioner=csi.simplyblock.io. These are not cleaned up by namespace deletion. Now deleted in the cleanup phase by checking the provisioner/driver field.
… set storagenode ndcs/npcs - R25 spdk-csi chart auto-creates StorageClass 'simplyblock-csi-sc' from logicalVolume config. Maintenance upgrade now uses it instead of creating its own (which would fail without an operator). - Changed logicalVolume.pool_name from 'testing1' to 'testpool' to match the pool created by the test via sbcli-dev. - Set storagenode.numDataChunks and numParityChunks from cluster params (were defaulting to 1).
Workflow: - logicalVolume.pool_name set to 'testing1' (matching R25 convention) - Removed logicalVolume.snapshot (not in R25 chart) - storagenode.numPartitions=0 (matching R25 default) - Removed storagenode.numDataChunks/numParityChunks (not valid R25 params) - Added --create-namespace to match actual R25 install command Test (_run_maintenance_upgrade): - Pool created as 'testing1' to match chart's logicalVolume.pool_name - Skips _create_storage_classes() entirely — uses the chart-created 'simplyblock-csi-sc' StorageClass from the logicalVolume config - Maps XFS SC to the same chart SC (R25 has no XFS variant)
- Add ::add-mask:: before writing CLUSTER_SECRET to GITHUB_ENV at all 3 locations - Wrap secret retrieval in set +x/set -x to prevent bash trace leaking secret - Change storagenode.numPartitions from 0 to 1 for R25 spdk-csi install
RaunakJalan
force-pushed
the
add-stuck-pod-describe
branch
from
August 5, 2026 20:45
71339bf to
dabad4c
Compare
Prevents stale NVMe-oF connections from causing nvme connect failures in subsequent test runs (Invalid argument on /dev/nvme-fabrics).
The upgrade test should not fail if pre-upgrade FIO doesn't complete. The goal is testing the upgrade path, not the old version's IO. Changes: - Reduce FIO runtime from 120s to 60s - Wait up to 5 mins for FIO, catch failures as warnings - Clean up FIO pods before taking snapshots - Create snapshots/clones without running FIO on clones
| self._validate_all_fio(fio_timeout) | ||
| self.logger.info("Pre-upgrade FIO completed and validated") | ||
| except Exception as fio_err: | ||
| pre_upgrade_fio_ok = False |
Pre-upgrade flow: 1. Create PVCs, run FIO (60s, non-fatal) 2. Clean up FIO pods 3. Create snapshots and clones (no FIO on clones initially) 4. Run FIO on clones (60s, non-fatal) 5. Capture MD5 checksums on all PVCs and clones Post-upgrade: - Verify MD5 checksums match pre-upgrade data - Ensures data integrity survived the maintenance window
cleanup_stale_fio_resources() deletes clone PVCs, snapshots, and test PVCs along with FIO jobs. Replace mid-test calls with a targeted _cleanup_fio_jobs_only() that only removes FIO jobs and configmaps, keeping PVCs available for utility pod mounting and md5sum.
| finally: | ||
| try: | ||
| self.k8s_utils.delete_pod(pod_name, wait=True) | ||
| except Exception: |
| finally: | ||
| try: | ||
| self.k8s_utils.delete_pod(pod_name, wait=True) | ||
| except Exception: |
Older versions require --force to shut down nodes that aren't in suspended state. The suspend call may silently fail, leaving nodes online and causing shutdown to error with "Node is not in suspended state". Using force=True bypasses this check.
Suspend fails with "Offline storage nodes found, cannot suspend node without --force" when any node is already offline (Step 6.1 scenario). Remove the suspend step entirely and just use shutdown --force which bypasses all state checks.
After Step 10 restarts all nodes one at a time, add an explicit wait for all SPDK pods to reach Ready state and all storage nodes to be online before ending the maintenance window. Prevents proceeding to post-upgrade steps while a node is still coming up.
After nodes register in sn list, poll until all report status=online before calling cluster activate. Prevents activating with nodes still starting up (SPDK pod NotReady / node offline).
R25 clusters don't have cert-manager since TLS wasn't supported. The target operator chart's validate-tls.yaml requires cert-manager CRDs when tls.enabled=true. Install cert-manager inside the test's _install_operator_chart if TLS is enabled and CRDs are missing.
| out, _ = self.k8s_utils._exec_kubectl( | ||
| "kubectl get crd certificates.cert-manager.io 2>/dev/null || true" | ||
| ) | ||
| if "certificates.cert-manager.io" in (out or ""): |
cert-manager _ensure_cert_manager now: - Uninstalls stale cert-manager release before install - Retries install up to 3 times, uninstalling between attempts Reverted helm install retry/uninstall logic - not needed there.
New test classes MassCreateRapidRestart_6k_3Snap_Docker and MassCreateRapidRestart_6k_3Snap_K8s: create 1500 lvols + 4500 snapshots (1:3 ratio), run 30 container stop/restart cycles without waiting for migration (60s cooldown), then delete lvols, create clones, and repeat 30 more restart cycles. Final summary prints per-iteration stop-to-online times for both phases (60 entries total). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| pass | ||
|
|
||
| # Fallback: parse table rows for UUID-like strings | ||
| import re |
| or entry.get("cluster_id") or "") | ||
| if cid and cid != self.cluster_id: | ||
| return cid | ||
| except (_json.JSONDecodeError, ValueError): |
| self.NUM_SUBSYSTEMS = effective_num_sub | ||
|
|
||
| total = self.NUM_SUBSYSTEMS * self.NS_PER_SUBSYSTEM | ||
| max_dur = getattr(self, 'MAX_TEST_DURATION', 24 * 3600) |
| f"{total} lvols, {self.SNAPSHOTS_PER_LVOL} snaps/lvol, " | ||
| f"{self.RAPID_RESTART_ITERATIONS} restart cycles per phase ===" | ||
| ) | ||
| test_start = time.time() |
| For each IP, finds the node UUID in Cluster-1, suspends it, | ||
| shuts it down, removes it, and runs deploy-cleaner on the host. | ||
| """ | ||
| mgmt_ip = self.mgmt_nodes[0] |
_bootstrap_second_cluster() now collects spare node IPs from both STORAGE_PRIVATE_IPS and NEW_NODE_IPS env vars. This aligns with the existing e2e-bootstrap.yml workflow which cleans NEW_NODE_IPS hosts without adding them to cluster 1 — making them ideal cluster 2 candidates. Also add TestBackupCrossClusterRestore to TOPOLOGY_MODIFYING_TESTS so inter-test cluster reset triggers when needed.
Add MassCreateRapidRestart_6k_3Snap_Docker and K8s to imports, ALL_TESTS, get_stress_tests(), and get_monitoring_tests(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The API enforces max_namespace_per_subsys=50. With NUM_SUBSYSTEMS=10, entity cap reduced 1500 lvols to 150/subsystem which was rejected. Changed to 30 subsystems x 50 ns/sub = 1500 lvols. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old spdk-csi helm chart sets helm.sh/resource-policy: keep on the simplyblock-snapshot-controller Deployment in kube-system. This causes the resource to survive helm uninstall, but it retains the stale meta.helm.sh/release-name: spdk-csi annotation. When the new simplyblock-operator chart tries to create the same resource, helm refuses with "invalid ownership metadata". Replace the incorrect re-annotation approach (_readopt_spdk_csi_resources) with explicit deletion of the orphaned resource after helm uninstall spdk-csi, in _uninstall_helm_releases(). The new operator chart then creates its own version cleanly.
The API rejects max_namespace_per_subsys > 50. Instead of fixing each test class individually, add enforcement in both orchestrator methods (_run_mass_create_delete_test and _run_mass_create_rapid_restart_test) that automatically redistributes lvols into more subsystems when NS_PER_SUBSYSTEM exceeds MAX_NS_PER_SUBSYSTEM (50). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After helm uninstall sbcli, the simplyblock-fdb-cluster-config ConfigMap is deleted despite resource-policy:keep annotations on other FDB resources. The new operator's admin-control pods mount this ConfigMap and get stuck in ContainerCreating: "configmap simplyblock-fdb-cluster- config not found". Three fixes: 1. Add ConfigMap to _FDB_KEEP_RESOURCES so it gets annotated with resource-policy:keep before helm uninstall. 2. Capture the FDB cluster file data BEFORE helm uninstall, and recreate the ConfigMap if it's missing afterward (fallback for cases where keep annotation doesn't work, e.g. resource owned by a different sub-chart). 3. Add explicit wait for admin-control pods to reach Ready state after operator chart install, with diagnostic event logging if pods remain in ContainerCreating.
- Step 1: Add ConfigMap simplyblock-fdb-cluster-config to FDB keep resources (8th resource). Admin pods mount this as fdb-cluster-file volume. - Step 2: Use shutdown --force instead of separate suspend+shutdown commands - Step 3.1: Delete orphaned simplyblock-snapshot-controller deployment in kube-system after helm uninstall spdk-csi (resource-policy: keep causes it to survive with stale ownership annotations) - Step 4.1: Verify FDB cluster-config ConfigMap survived helm uninstall sbcli, with recovery procedure to recreate from FDB pod if missing - Step 6: Add cert-manager prerequisite for TLS-enabled installs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.