Unable to Expunge Mirrored RBD Image from Cloudstack

[tverkade]

problem

I'm running into an issue in Cloudstack where when using ceph RBD as primary storage, and having snapshot-based mirroring in Ceph, I cannot delete and expunge the image without manually disabling mirroring:

Request failed. (530)
Failed to expunge the volume Volume {"id":1751,"instanceId":null,"name":"TV-REP-02","uuid":"6ad0914e-9172-45d3-87a0-1f34661b1d3f","volumeType":"DATADISK"} in Primary data store : com.cloud.utils.exception.CloudRuntimeException: com.ceph.rbd.RbdException: Failed to find out if snapshot .mirror.primary.625a7cf1-1946-48a3-8fc3-5a4de98bec29.b8365252-c64b-4862-850b-cb29e333ceb7 is protected - No such file or directory

versions

ACS: 4.22.0.0
KVM: Ubuntu 22.04
Ceph: version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)

The steps to reproduce the bug

1. Create a disk image in Cloudstack on the RBD storage pool
2. Enable mirroring on the image in ceph
3. Either define a schedule for snapshots at the pool level, define a snapshot schedule for the image itself, or manually initiate a snapshot of the image in ceph
4. Attempt to delete the image in Cloudstack with the expunge flag enabled (or without, it will just fail to delete it later)
5. Get the error
   Failed to find out if snapshot <Image and snapshot IDs> is protected - No such file or directory

What to do about it?

It would be excellent if when trying to delete an RBD image from Cloudstack if the command could be passed through first determine whether or not mirroring is enabled, and if so, disable it prior to attempting to delete it:

rbd info <pool>/<image ID>
rbd mirror image disable <pool>/<image ID>

And then run the actual delete command.

So for example, if the image ID is df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e':

root@ceph:/# rbd info cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e
rbd image 'df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e':
	size 25 GiB in 6400 objects
	order 22 (4 MiB objects)
	snapshot_count: 2
	id: 3ad0e7be094cd5
	block_name_prefix: rbd_data.3ad0e7be094cd5
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
	op_features:
	flags:
	create_timestamp: Fri Mar  6 20:14:52 2026
	access_timestamp: Fri Mar  6 20:14:52 2026
	modify_timestamp: Fri Mar  6 20:14:52 2026
	journal: 3ad0e7be094cd5
	mirroring state: enabled
	mirroring mode: snapshot
	mirroring global id: a29acb3e-0d54-4de9-ac62-c98e435462ca
	mirroring primary: true

root@ceph:/# rbd mirror image disable cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e
Mirroring disabled

It would also be excellent to include an option to enable mirroring on an image when it's created in Cloudstack, but that's a little bit outside of the scope of this issue.

[DaanHoogland]

@tverkade , I am right in understanding that you are basically asking for defensive code agains an out-of-bounds change in cheph, am I?
As you say/imply we could also natively support snapshot mirroring, which would be a good feature.

[tverkade]

@tverkade , I am right in understanding that you are basically asking for defensive code agains an out-of-bounds change in cheph, am I? As you say/imply we could also natively support snapshot mirroring, which would be a good feature.

@DaanHoogland yes, even if the scheduling is not managed by Cloudstack (it would be the default schedule applied to the pool), if there is a way that creating an RBD image in Cloudstack in a specific primary storage target and enable the snapshot mirroring feature for that RBD, that would be excellent.

So, for the example In my initial issue, creating an RBD image in Cloudstack that results in the image ID df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e with pool cloudstack would require snapshot mirroring enabled with:
image enable cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e snapshot

That would solve the biggest issue I'm dealing with right now, however that also only solves the issue of enabling mirroring. The other issue is that if you simply enable the snapshot mirroring, it will default to the pool's snapshot schedule. This is a problem as not staggering the schedule (which results in all VMs on this storage are taking snapshots and replicating changes at exactly the same time), would cause potentially noticeable performance issues. The other piece of this is that in order to maintain crash-consistency, we also need to make sure that all disks on a particular VM have a snapshot taken at the exact same time, or the changes on the remote side will be inconsistent with the data on the source.

I'm currently using a series of custom systemd units that trigger scripts to:

1. Query the Cloudstack API for all VMs and their disks that are configured the desired RBD pools
2. Query the images in ceph to see which already have a snapshot schedule and which do not
3. If a new disk is created on a VM that already has other disks with a snapshot schedule, the scripts will apply a schedule that matches the other disk(s).
4. If this is a new VM, it will generate a random offset between 1 and 55 minutes and use that for an hourly snapshot schedule
5. Query the Cloudstack API for any images marked for deletion that reside on RBD pools with mirroring, and disable mirroring so the disks can be expunged.

While this works, these scripts run every 5 minutes, and as I had previously mentioned it prevents expunging directly in the UI until after mirroring is disabled.

I hope this helps explain a bit more of what I'm running into