Add e2e coverage for kubevirt-datamover VM backup/restore: multi-PVC, incremental sequence, and CBT restore - #2350
Conversation
|
Skipping CI for Draft Pull Request. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR adds a multi-PVC CirrOS VM fixture, Velero and KubeVirt inspection helpers, readiness checks, and CSI datamover tests for restore and incremental backup behavior. ChangesKubeVirt datamover validation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant BackupRestoreTest
participant KubernetesAPI
participant CSIDataMover
participant VirtLauncherPod
BackupRestoreTest->>KubernetesAPI: Wait for VM PVCs to become Bound
BackupRestoreTest->>CSIDataMover: Create full backup
CSIDataMover->>KubernetesAPI: Record DataUpload and checkpoint
BackupRestoreTest->>VirtLauncherPod: Run virsh and restart VM
BackupRestoreTest->>CSIDataMover: Create incremental backup
CSIDataMover->>KubernetesAPI: Record backup type and checkpoint
BackupRestoreTest->>CSIDataMover: Restore both VM PVCs
Possibly related PRs
Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (2 errors, 3 warnings)
✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/e2e/virt_backup_restore_suite_test.go`:
- Line 96: Update the Eventually assertion for allBound to include a failure
message containing the relevant namespace and PVC names, so timeout reports
identify which restore resources failed.
- Line 84: Update the PVC lookup in the suite polling flow to create a bounded
timeout context for each Kubernetes PersistentVolumeClaims.Get call instead of
using context.Background(). Pass that context to Get and ensure the context is
canceled after the request, while preserving the existing ten-minute polling
behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 5c2eeb1f-c30f-4c5e-8abd-dcc7735ac335
📒 Files selected for processing (2)
tests/e2e/sample-applications/virtual-machines/cirros-test/cirros-test-multipvc.yamltests/e2e/virt_backup_restore_suite_test.go
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
tests/e2e/virt_backup_restore_suite_test.go (1)
664-672: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe spec depends on a counter value produced by earlier specs.
runSequenceBackup("full")here is correct only whenbackupCountis 3.ginkgo.Orderedpreserves order, but it does not guarantee that the two earlier specs ran. A focused run, a label filter, or a--focuson this single spec leavesbackupCountat 0, and the assertion then checks the wrong expected type. Derive the number of preceding incremental backups explicitly, or set the annotation value frombackupCountso the expectation stays consistent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/virt_backup_restore_suite_test.go` around lines 664 - 672, Update the “hitting maxIncrementalBackups forces a full backup” spec so its expected backup type does not depend on earlier specs having run. Derive the required max-incremental-backups annotation or expected result from the current backupCount, ensuring focused or label-filtered execution still validates the intended full-backup behavior.tests/e2e/lib/virt_helpers.go (1)
1386-1394: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winHandle an unpopulated VMB status explicitly.
unstructured.NestedStringreturnsfound=falsewhenstatus.typeis absent. The code discards that bool, so a VirtualMachineBackup that is still initializing returns("", "", nil). The caller then fails withExpected: "full", Actual: "", which does not show that the status was not yet written. Return a distinct error when the field is missing, so the failure identifies the cause.♻️ Proposed change
- backupType, _, err = unstructured.NestedString(vmb.Object, "status", "type") - if err != nil { + backupType, found, err := unstructured.NestedString(vmb.Object, "status", "type") + if err != nil { return "", "", fmt.Errorf("failed to read status.type from VirtualMachineBackup %s/%s: %w", namespace, vmb.GetName(), err) } + if !found { + return "", "", fmt.Errorf("VirtualMachineBackup %s/%s has no status.type yet", namespace, vmb.GetName()) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/lib/virt_helpers.go` around lines 1386 - 1394, Update the status extraction in the VirtualMachineBackup helper to retain and validate the found flags returned by unstructured.NestedString for status.type and status.checkpointName. Return a distinct, descriptive error when either required field is absent, while preserving the existing wrapped errors for extraction failures and the successful return path when both fields are present.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/e2e/lib/backup.go`:
- Around line 149-160: Update lib.Scheme initialization to register
velerov2alpha1 via AddToScheme alongside the existing velerov1 registration, so
GetDataUploadForBackup can list DataUploadList without an unregistered-kind
error.
In `@tests/e2e/virt_backup_restore_suite_test.go`:
- Around line 622-651: Add meaningful failure messages to every Gomega assertion
in the BeforeAll and AfterAll hooks, including namespace deletion/creation,
application installation, VM status polling, VM readiness, VM removal, and
namespace cleanup. Each message should identify the failed action and include
the relevant incSeqNamespace and incSeqVMName values where applicable.
- Around line 344-349: Remove the temporary KubeVirt datamover plugin override
from the DPA setup in the test, including the UnsupportedOverrides
initialization and assignment for KubeVirtDatamoverPluginImageKey. If the plugin
fix is not yet available, replace the personal mutable image with a
project-owned digest-pinned image and retain explicit tracking for removing the
override.
- Around line 589-597: Prevent the outer AfterEach teardown from running between
specs in the “Kubevirt datamover incremental backup sequence” Describe. Scope or
condition tearDownBackupAndRestore so the ordered sequence remains deployed,
then add cleanup in this Describe’s AfterAll to remove the sequence resources
and restore the expected final state.
---
Nitpick comments:
In `@tests/e2e/lib/virt_helpers.go`:
- Around line 1386-1394: Update the status extraction in the
VirtualMachineBackup helper to retain and validate the found flags returned by
unstructured.NestedString for status.type and status.checkpointName. Return a
distinct, descriptive error when either required field is absent, while
preserving the existing wrapped errors for extraction failures and the
successful return path when both fields are present.
In `@tests/e2e/virt_backup_restore_suite_test.go`:
- Around line 664-672: Update the “hitting maxIncrementalBackups forces a full
backup” spec so its expected backup type does not depend on earlier specs having
run. Derive the required max-incremental-backups annotation or expected result
from the current backupCount, ensuring focused or label-filtered execution still
validates the intended full-backup behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: be76a5c0-d96a-49fb-a7cd-9b6173f7f69f
📒 Files selected for processing (4)
tests/e2e/lib/apps.gotests/e2e/lib/backup.gotests/e2e/lib/virt_helpers.gotests/e2e/virt_backup_restore_suite_test.go
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/e2e/lib/scheme.go (1)
31-31: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueHandle scheme registration errors consistently.
velerov2alpha1.AddToScheme(Scheme)currently returnsnilunconditionally, so this line cannot cause the statedDataUploadListfailure. Do not discard the error return; useutilruntime.Must(...)for all scheme registrations to fail fast if their behavior changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/lib/scheme.go` at line 31, Update the scheme initialization around velerov2alpha1.AddToScheme(Scheme) to pass its return value through utilruntime.Must instead of discarding it, and apply the same fail-fast wrapper consistently to every scheme registration in that initialization block.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/e2e/lib/scheme.go`:
- Line 31: Update the scheme initialization around
velerov2alpha1.AddToScheme(Scheme) to pass its return value through
utilruntime.Must instead of discarding it, and apply the same fail-fast wrapper
consistently to every scheme registration in that initialization block.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: afb294c1-b581-4b68-b0c5-c16a7bd1f4a9
📒 Files selected for processing (2)
tests/e2e/lib/scheme.gotests/e2e/virt_backup_restore_suite_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/e2e/virt_backup_restore_suite_test.go
|
Re review 4859513816 and the two remaining nitpicks from 4859450952:
Note Responses generated with Claude |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kaovilai The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
31998f5 to
5de0309
Compare
Adds a CirrOS VM template with two disks (existing rootdisk DataSource plus a blank data disk) and a corresponding backup/restore table entry using CSIDataMover, verifying both PVCs come back Bound after restore. Covers the "Multi-PVC VM" gap tracked in migtools/kubevirt-datamover-controller#99. Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Automates scenarios 1-3 from oadp-operator#2252 (a manual test writeup of kubevirt-datamover incremental-backup-sequence behavior): a full-then-incremental backup chain with no expected/actual type mismatch, a VM restart not invalidating the checkpoint chain, and hitting maxIncrementalBackups forcing a full backup via the per-VM override annotation. Scenario 4 (delete libvirt checkpoints with maxIncrementalBackups=0) hits an unfixed upstream bug (CNV-85377: virt-controller never falls back to full, VMB hangs Initializing forever) and is scaffolded as a real, compiling ginkgo.PIt rather than run, ready to flip to ginkgo.It once that bug is fixed. New lib helpers: VirtualMachineBackup GVR + GetVMBBackupType, GetDataUploadForBackup, GetVirtLauncherPod, RunVirshCommand, SetVMAnnotation, IsDeploymentRolledOut. Also pins the e2e DPA's kubevirt-datamover-plugin image to an unmerged PR build (quay.io/tkaovila/kubevirt-datamover-plugin:pr-41, see migtools/kubevirt-datamover-plugin#41) until that PR merges. Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
- Register velero v2alpha1 in the e2e suite scheme (GetDataUploadForBackup lists DataUploadList, which was otherwise unregistered -> runtime error). - Fix a real bug: the outer AfterEach undeploys CSI+datamover and deletes cirros-test after every spec it fires for, including this Describe's specs -- which would tear the VM down between the incremental-sequence scenarios. Collapse them into a single ordered It (with ginkgo.By steps) that does its own full teardown at the end, and register the case via updateLastBRcase/prepareBackupAndRestore so the shared AfterEach's now- redundant cleanup is a harmless no-op instead of acting on stale state. - Use a bounded context for the PVC Get in vmPvcsBound, instead of context.Background(). - Add failure messages naming the namespace/VM/backup to gomega assertions across the new setup/teardown/backup-sequence code. Not fixed: the personal-registry kubevirt-datamover-plugin image override is flagged as a pre-merge security concern, but it's an explicit, already- documented temporary pin (see prior commit) needed to exercise an unmerged upstream PR -- removing it would defeat this PR's own purpose until migtools/kubevirt-datamover-plugin#41 merges. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Matches the existing cmd/main.go convention (utilruntime.Must(X.AddToScheme(scheme))) instead of silently discarding registration errors via "_ =". Verified none of the 15 AddToScheme calls actually error (ran TestOADPE2E without a cluster; it got past package init/scheme registration and failed later on an unrelated connectivity assertion, not a panic). Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
GetVMBBackupType discarded the found bool from
unstructured.NestedString(status.type), so a VirtualMachineBackup
still initializing (no status.type yet) silently returned ("", "",
nil) instead of a clear error -- callers then failed with a
confusing "Expected: full, Actual: \"\"" instead of a message
identifying the real cause.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Adds a second temporary image override (KubeVirtDatamoverControllerImageKey -> quay.io/tkaovila/kdm-controller:issue73-phase3) alongside the existing plugin override, to exercise migtools/kubevirt-datamover-controller#124 (DataDownload controller for VM restore, issue #73 phase 3) pre-merge -- directly relevant to restore-from-CBT coverage. Both overrides are temporary and should be removed once their respective PRs merge. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Closes the #99 "Restore from KDM CBT backup" gap, now unblocked by the overridden kubevirt-datamover-controller image (migtools/kubevirt-datamover- controller#124, DataDownload controller, issue #73 phase 3). Per docs/design/kubevirt-datamover.md and the plugin's own README, restore just needs a normal Velero Restore -- the VirtualMachine RestoreItemAction plugin creates the DataDownload CR automatically from backup-recorded annotations, and separately discards the restored VMB/VMBT so restore doesn't re-trigger a backup. No manual CR driving needed, mirroring how backup already works. Verifies success via the existing generic lib.CreateRestoreFromBackup/IsRestoreDone/IsRestoreCompletedSuccessfully helpers plus a new lib.GetDataDownloadForRestore (mirrors GetDataUploadForBackup) confirming the DataDownload itself reached status.phase=Completed -- proving the datamover path actually engaged, not just that the restore completed some other way. Extracted runKubevirtDMBackup (shared create+wait+verify-backup helper) out of runSequenceBackup to avoid duplicating that logic a second time for the restore test. Also scaffolds two ginkgo.PIt (real, compiling, not run) for gaps explicitly still open per kubevirt-datamover-controller#73's phased plan: multi-disk restore (phase 4, not yet hardened -- new cirros-test-multipvc-cbt.yaml template ready for when it lands) and restore from an incremental (not full) backup (phase 5, not yet e2e-validated upstream even though the chain-rebase logic may already support it internally). Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
…e gate - IsDeploymentRolledOut: also require Status.Replicas == wantReplicas alongside UpdatedReplicas/AvailableReplicas -- without it, a rollout scaling down old replicas could report done while stale pods still lingered. - GetVMBBackupType: check the found bool for status.checkpointName too (same fix already applied to status.type), so an unpopulated field returns a clear error instead of silently returning "". - Gate both kubevirt-datamover pre-merge image overrides behind an explicit OADP_E2E_KDM_PREMERGE_IMAGES=true env var instead of always-on, so this suite defaults to the operator's normal images for anyone else running it and only pulls the mutable personal-registry builds when deliberately testing those two PRs. Scheduled cluster-run job updated to set this env var. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
The manager ClusterRole shipped with OADP only granted access to datauploads/datauploads.status -- there was no datadownloads, datadownloads/status, or events permission at all. Without this, the controller would hit RBAC-denied errors reconciling any DataDownload, regardless of image correctness, once a real velero restore actually tried to drive it (which migtools/kubevirt-datamover-plugin#41 now makes possible). Synced config/kubevirt-datamover-controller_rbac/role.yaml and the matching block in bundle/manifests/oadp-operator.clusterserviceversion.yaml (serviceAccountName: oadp-kubevirt-datamover-controller-manager) to byte-match config/rbac/role.yaml from migtools/kubevirt-datamover-controller PR #124 (issue #73 phase 3, commit 825d176), which added these rules on the source side but were never pulled into OADP's bundled copy -- normally done via `make update-kubevirt-datamover-manifests KUBEVIRT_DATAMOVER_PATH=...`, done here by hand since no local checkout of that repo is available in this environment. This is a real, pre-existing gap (not introduced by this branch's other changes) that this branch's e2e work would otherwise have hit blind, so fixing it here rather than filing it separately. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
… to plugin PR #44 VirtualMachineBackupTracker deletion can leave a VirtualMachineBackup's backup.kubevirt.io/vmbackup-protection finalizer stuck (kubevirt#18724), blocking namespace deletion forever in the incremental-sequence and restore specs. IsNamespaceDeletedClearingStuckVMBFinalizers works around it by clearing stale finalizers on every poll; safe to remove once that kubevirt issue is fixed. Also fixes a real race: VirtualMachineBackup is ephemeral and can be deleted by virt-controller before the overall backup finishes uploading, so checking its status after backup completion (as before) could find nothing. runKubevirtDMBackup now takes an onDataUploadFound callback invoked as soon as the DataUpload appears, letting the incremental- sequence spec check VMB status while it still exists. migtools/kubevirt-datamover-plugin#41 merged; the e2e image override now points at #44 instead, which fixes a still-open bug where restored PVCs come back with stale spec.volumeName/status, causing the datamover controller to reject them as "already bound". Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Resolving the CSV merge conflict against upstream's velero-SA RBAC restructuring (PR openshift#2349) by taking upstream's side lost a small generated permission block (events create/patch). make bundle regenerates it correctly. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
5de0309 to
c300f00
Compare
The kubevirt-datamover controller's flip-back logic (restoring a VM's run state once all sibling DataDownloads complete) needs to patch the VirtualMachine object, but the synced ClusterRole only granted get/list/watch. Sync from kubevirt-datamover-controller commit a69ee24 to add the missing verb. Also switch the e2e suite's pre-merge image overrides from mutable tags to digests, since both quay.io/tkaovila/* tags used here have already been rebuilt in place during development. Verified on a real cluster: "restore a VM from a full kubevirt-datamover CBT backup" now passes end-to-end (VM correctly halts at restore time, then flips back to Running once its DataDownload completes). Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
ClearStuckVMBFinalizers: clear only backup.kubevirt.io/vmbackup-protection instead of nulling the entire finalizer list, so any other finalizer a VMB might carry survives the workaround. GetVirtLauncherPod: skip pods that are terminating or not yet Running before matching on the domain annotation. During a VM restart (hit by the incremental-backup-sequence test), both the old and new virt-launcher pod can briefly share the same domain annotation; without this the old, terminating pod could be picked, breaking RunVirshCommand execs. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
migtools/kubevirt-datamover-plugin#44 iterated further (commit 4fb7ed9) after this suite's plugin digest was originally pinned, including a change to Progress()'s grace-period-anchor timing. Re-ran the restore-from-CBT-backup scenario against the new build to confirm that change doesn't affect this path: still passes end-to-end. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
* Add optional e2e-test-kubevirt-aws presubmit for oadp-dev oadp-1.5 and oadp-1.6 already run make TEST_VIRT=true test-e2e as a gating presubmit (e2e-test-kubevirt-aws); oadp-dev only had it as a daily periodic, with no way to get presubmit signal on a PR before merge. Adds the same job to the oadp-dev 5.0/5.1 variants, but as optional (manually triggered via /test, doesn't block merge) rather than gating like 1.5/1.6 -- oadp-dev is the active development branch and this KubeVirt path is newer/less proven there. This is needed for openshift/oadp-operator#2350 (kubevirt-datamover VM backup/restore e2e coverage) to actually be runnable against a PR before merge -- without this, those new specs only get exercised by the existing daily periodic, well after the PR that added them has merged. Related: openshift#83049 (kdm-controller/kdm-plugin presubmit e2e using this same TEST_VIRT=true path). Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Make e2e-test-kubevirt-aws gating for oadp-dev Matches oadp-1.5/oadp-1.6, where this same job is already gating rather than optional. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Drop 5.1 from this PR, keep only oadp-dev 5.0 Scope this to just the 5.0 variant for now. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> --------- Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
Note Responses generated with Claude CI red but not real fail — |
|
Note Responses generated with Claude /test all |
* Add optional e2e-test-kubevirt-aws presubmit for oadp-dev oadp-1.5 and oadp-1.6 already run make TEST_VIRT=true test-e2e as a gating presubmit (e2e-test-kubevirt-aws); oadp-dev only had it as a daily periodic, with no way to get presubmit signal on a PR before merge. Adds the same job to the oadp-dev 5.0/5.1 variants, but as optional (manually triggered via /test, doesn't block merge) rather than gating like 1.5/1.6 -- oadp-dev is the active development branch and this KubeVirt path is newer/less proven there. This is needed for openshift/oadp-operator#2350 (kubevirt-datamover VM backup/restore e2e coverage) to actually be runnable against a PR before merge -- without this, those new specs only get exercised by the existing daily periodic, well after the PR that added them has merged. Related: openshift#83049 (kdm-controller/kdm-plugin presubmit e2e using this same TEST_VIRT=true path). Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Make e2e-test-kubevirt-aws gating for oadp-dev Matches oadp-1.5/oadp-1.6, where this same job is already gating rather than optional. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> * Drop 5.1 from this PR, keep only oadp-dev 5.0 Scope this to just the 5.0 variant for now. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> --------- Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
Note Responses generated with Claude /test all |
|
Note Responses generated with Claude /test all -- retesting for migtools/kubevirt-datamover-controller#124 rebase (tip ff1c1b8) |
|
Note Responses generated with Claude /test all -- retesting for migtools/kubevirt-datamover-controller#124 real functional fix (tip 3b4b3d5): stale sibling DataDownload correlation bug that could permanently block VM run-state restore, not just a rebase |
|
Note Responses generated with Claude /test all -- retesting for migtools/kubevirt-datamover-controller#124 tip e66317e: 2 more real bugs fixed (missing APIReader fallback in isRestoreAlreadyProvisioned; qemu-img convert missing -n corrupting already-provisioned block-mode restore targets; operation-timeout could mark an already-successful upload Failed). Note: the previous run (tip 3b4b3d5) gave no real signal -- 6/9 e2e jobs failed purely on CI infra (AWS session-token expiry, Route53 zone conflicts/missing-zone, a step-registry script bug), none reached actual test code. |
Points the virt e2e suite's controller UnsupportedOverrides at the freshly built quay.io/tkaovila/kubevirt-datamover-controller image (commit e66317e), which fixes two real bugs found since the prior digest: a missing APIReader fallback in isRestoreAlreadyProvisioned, and a missing -n on qemu-img convert that could corrupt an already-provisioned block-mode restore target. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
Note Responses generated with Claude /test all -- new commit 307d369 updates the kubevirt-datamover-controller test image to quay.io/tkaovila/kubevirt-datamover-controller@sha256:6d3d80dc7f0096114f4b1da2379194bca100fc063f0fd26ee97e9c8d2e21856f (commit e66317e, now public and confirmed anonymously pullable) |
Both fixes are now merged upstream: migtools/kubevirt-datamover-plugin#41 and #44 (VM-eager-start race, restore RIAs), and migtools/kubevirt-datamover-controller#124 (issue #73 phase 3 DataDownload controller). The suite now exercises the default images per the TODO this override block left for itself. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
Note Responses generated with Claude /test all -- commit cf64841 removes the kubevirt-datamover plugin/controller image overrides now that both fixes are merged upstream (migtools/kubevirt-datamover-plugin#41, #44 and migtools/kubevirt-datamover-controller#124). This validates the suite against the default images instead of pinned personal-registry builds. |
|
/hold |
| // IsDeploymentReady, which only checks AvailableReplicas == Replicas and can false-positive | ||
| // against stale pre-rollout pods, this also requires the controller to have observed the | ||
| // latest generation and updated all replicas to it. | ||
| func IsDeploymentRolledOut(ocClient client.Client, namespace, dName string) wait.ConditionFunc { |
There was a problem hiding this comment.
Two exported helpers here have no callers in this PR: IsDeploymentRolledOut (this func) and ChecksumUploadPodQcow2Flattened in virt_helpers.go (its own comment says it's not called). tests/e2e/lib is shared code, so unused exports tend to rot. Can we drop both and add them back in the PR that actually calls them? If IsDeploymentRolledOut has a caller I missed, point me at it.
| - apiGroups: | ||
| - "" | ||
| resources: | ||
| - events |
There was a problem hiding this comment.
This events: create,patch rule isn't mentioned in the description, unlike the update on virtualmachines. Two things:
- Were role.yaml and the CSV regenerated via
make update-kubevirt-datamover-manifests, or hand-edited? This file syncs from the controller repo, so hand edits drift on the next sync and can diverge from the CSV. - Please add a line to the description explaining the events rule so the CSV grant is traceable.
| fullBackupPayloadSizeMiB = 32 | ||
| ) | ||
|
|
||
| ginkgo.It("restore a VM from a full kubevirt-datamover CBT backup", ginkgo.Label("virt", "kdm"), func() { |
There was a problem hiding this comment.
This It runs two full restore cycles plus the backup in one spec, with timeouts summing past an hour. I get why it can't split: the AfterEach recreates the DPA, which gets a new random BSL prefix, so a second It can't reuse this backup. Two small asks:
- Move that "can't split" note to a comment above the
It, so the next person sees it before trying to break it apart. It currently sits mid-spec near the rejected-restore block. - If the rejection-path half can ever run off its own small backup, that's the natural spec to break out later.
| } | ||
| return status | ||
| }, time.Minute, time.Second*10).ShouldNot(gomega.Equal("Running"), | ||
| "VM %s/%s unexpectedly reached Running status despite its DataDownload failing", alpineNamespace, alpineVMName) |
There was a problem hiding this comment.
A persistent GetVmStatus error returns "" here, which passes ShouldNot(Equal("Running")) for the full minute. So a broken API makes the "VM stayed halted" check pass exactly when we can't see the VM at all. The eager-wait polls swallow the same error harmlessly (they time out loudly), but this negative check needs to tell "confirmed not Running" apart from "couldn't read status". Fail on a run of consecutive errors, or require at least one valid non-Running read before passing.
| allBound := func() bool { | ||
| for _, name := range pvcNames { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) | ||
| pvc, err := kubernetesClientForSuiteRun.CoreV1().PersistentVolumeClaims(pvcNamespace).Get(ctx, name, metav1.GetOptions{}) |
There was a problem hiding this comment.
vmPvcsBound ignores its ocClient and namespace params and uses the kubernetesClientForSuiteRun global plus pvcNamespace instead. Works, but the unused params are misleading. Either use ocClient, or add a one-line note on why the global clientset is used.
| // RunVirshCommand execs `virsh <args...>` inside vmName's virt-launcher pod's compute | ||
| // container via ExecuteCommandInPodsSh. kubeConfig is the suite's *rest.Config (VirtOperator | ||
| // itself only carries a *kubernetes.Clientset, not a rest.Config, so it's passed in here). | ||
| func (v *VirtOperator) RunVirshCommand(kubeConfig *rest.Config, namespace, vmName string, args ...string) (string, error) { |
There was a problem hiding this comment.
On the CodeRabbit no-sensitive-data flag: fine to leave. The only caller is the CNV-85377 PIt, which doesn't run, and checkpoint output isn't sensitive. The live guest-exec path logs base64 dd-of-urandom, also harmless. Noting it so the thread closes with a reason.
|
thank you for your review kind sir. |
- Remove IsDeploymentRolledOut and ChecksumUploadPodQcow2Flattened:
both are unused exports in shared tests/e2e/lib code (confirmed no
callers). Can be re-added in the PR that actually calls them.
- Move the "why this spec can't be split into multiple Its" explanation
to a comment directly above the ginkgo.It declaration, instead of
mid-spec near the rejected-restore block; note that the rejection-path
half is the natural future split point if it can ever run off its own
backup.
- Fix a false-negative in the "VM stays halted" check: a persistent
GetVmStatus error returned "", which trivially satisfied
ShouldNot(Equal("Running")) for the whole Consistently window --
making an unreadable/broken API look identical to "confirmed halted".
Now fails loudly after 3 consecutive read errors instead of silently
passing.
- vmPvcsBound now uses its own ocClient/namespace parameters instead of
the kubernetesClientForSuiteRun global and a redundant pvcNamespace
parameter that always matched namespace at the only call site.
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Confirmed live: the test-oadp-operator image build failed with "curl: (6) Could not resolve host: github.com" fetching virtctl -- a transient DNS blip in the CI build environment, unrelated to any code change, that failed the whole image build over nothing. Adds --retry/--retry-delay to every curl-based network fetch in build/ci-Dockerfile (kubectl, virtctl, claude CLI install) and the equivalent tool-download curls in Makefile (operator-sdk, opm, codecov uploader). curl's default retry-on-error set already covers exit 6 (could not resolve host) along with connect/read/5xx errors, so no extra flags beyond --retry/--retry-delay are needed. Also wraps the openshift/velero clone in ci-Dockerfile in a 3-attempt retry loop (git has no built-in --retry), clearing the target directory before each attempt so a partial clone from a failed try doesn't block the next one with "destination path already exists". Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
/retest-required |
|
/retest |
| # rm -rf before each attempt: a clone that fails partway through can leave a | ||
| # non-empty target directory behind, which would make the next attempt fail | ||
| # with "destination path already exists" instead of actually retrying. | ||
| RUN for i in 1 2 3; do \ |
There was a problem hiding this comment.
This retry loop exits 0 even when all three clones fail. git clone ... && break || sleep 5: on failure the && break is skipped and || sleep 5 returns 0, so after three misses the loop's status is the last sleep, and the RUN succeeds with no velero checkout. The build then continues and only degrades failure-analysis later, instead of failing here.
Every other fetch in this file uses curl --retry, which does fail the build after exhausting retries. Worth making this one consistent, e.g. assert the result after the loop:
RUN for i in 1 2 3; do \
rm -rf /go/src/github.com/openshift/velero && \
git clone --depth 1 --branch oadp-dev \
https://github.com/openshift/velero.git \
/go/src/github.com/openshift/velero && break || sleep 5; \
done; \
test -d /go/src/github.com/openshift/velero/.git
| // lifecycle (creation, Running wait, teardown-before-return) exists in exactly | ||
| // one place. | ||
| func (v *VirtOperator) runInPVCBlockDeviceHelperPod(kubeConfig *rest.Config, namespace, pvcName, command string) (stdout string, err error) { | ||
| podName := "checksum-helper-" + pvcName |
There was a problem hiding this comment.
checksum-helper- + pvcName can exceed the 63-char DNS label limit for longer PVC names and fail pod creation with an opaque error. Fine for the current short names, but worth truncating or hashing the suffix so a future longer-named PVC scenario doesn't trip on it.
There was a problem hiding this comment.
This is our own e2e no customer input but will harden
|
@kaovilai: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Summary
Adds e2e coverage for three kubevirt-datamover VM backup/restore scenarios: multi-PVC VMs, an incremental backup sequence, and restoring a VM from a full CBT backup. Closes coverage gaps tracked in migtools/kubevirt-datamover-controller#99, openshift/oadp-operator#2252, and migtools/kubevirt-datamover-controller#73 (phases 0-3; phases 4/5 are out of scope, see Known limitations).
Design
Multi-PVC VM backup/restore
tests/e2e/sample-applications/virtual-machines/cirros-test/cirros-test-multipvc.yaml: a CirrOS VM with two disks — the existing rootdisk plus a second blank 100Mi data disk, so no extra image fetch is needed.ginkgo.Entry("no-application CSI datamover backup and restore, multi-PVC CirrOS VM", ...), following the existing "no-application" CirrOS pattern withBackupRestoreType: lib.CSIDataMover.vmPvcsBound()verifies both PVCs exist and areBoundafter restore.Incremental backup sequence
maxIncrementalBackupsforces a full backup (via the per-VMkubevirt-datamover.io/max-incremental-backupsannotation override, which takes effect immediately with no controller rollout to wait for).tests/e2e/libhelpers:VirtualMachineBackupGVR +GetVMBBackupType(reads.status.type/.status.checkpointName),GetDataUploadForBackup(reads the controller'sexpected-backup-typeannotation),SetVMAnnotation.maxIncrementalBackups=0) depends on an unfixed upstream bug (CNV-85377: virt-controller never falls back to full, the VMB hangsInitializingforever) and is scaffolded as a real, compilingginkgo.PIt, ready to flip toginkgo.Itonce that's fixed.Restore from a full kubevirt-datamover CBT backup
Restoring a VM from a kubevirt-datamover backup has to hold the VM inactive until its DataDownload(s) finish: otherwise virt-launcher spawns immediately on the restored VM object, and
WaitForFirstConsumerbinds the target PVC to a scratch volume before the DataDownload can rebind it. That race is closed by two companion PRs, both now merged:RestoreItemActionV2halts the VM's run strategy at restore time and stashes the original value in an annotation.Completed— driven by a watch on the phase transition itself rather than a single completion call site, so a controller-restart-mid-restore resume path can't skip it.updateonkubevirt.io/virtualmachinesto perform that flip; this PR syncs that verb intoconfig/kubevirt-datamover-controller_rbac/role.yamland the bundle CSV. That sync (viamake update-kubevirt-datamover-manifests) also picked up a pre-existingevents: create,patchrule already present in the controller's own generated RBAC — used byinternal/controller/kubevirt_datadownload_controller.go'sEventfcalls, unrelated to this PR's own changes but now correctly reflected here since the manifests are resynced as a whole.RunVirshCommand's virt-launcher pod lookup now skips pods that are terminating or not yetRunning: a VM restart (hit by the incremental-sequence test above) can briefly have both the old and new launcher pod sharing the same domain annotation.ClearStuckVMBFinalizers(workaround for kubevirt/kubevirt#18724: a deletedVirtualMachineBackupnever has itsvmbackup-protectionfinalizer released once its backing tracker is gone) now clears only that specific finalizer, rather than the VMB's entire finalizer list.GetDataDownloadForRestorealso surfaces thekubevirt-datamover.io/restore-block-modeannotation the plugin stamps on the DataDownload, and the test cross-checks it against the restored PVC's actualspec.volumeMode.Both companion PRs above are merged, so the suite exercises the default plugin/controller images — no image override needed.
Known limitations
ginkgo.PItand are explicitly out of scope here: multi-disk restore hardening (kubevirt-datamover-controller#73 phase 4) and incremental-chain restore validation (phase 5) — neither is implemented upstream yet.maxIncrementalBackups=0checkpoint-delete scenario stays pending on CNV-85377.Testing
All three in-scope scenarios pass end-to-end on a real OpenShift cluster (AWS, community HCO/KubeVirt) with the companion plugin and controller images deployed:
Always) once its DataDownload reachesCompleted; Block volumeMode confirmed via both the DataDownload annotation and the restored PVC itself.go build,go vet, andgolangci-lintall pass.Test plan
Note
Responses generated with Claude