Bug 2354499 - Deadlock to recover from consistency group with images from multiple pools
Summary: Deadlock to recover from consistency group with images from multiple pools
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD-Mirror
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 8.1
Assignee: Prasanna Kumar Kalever
QA Contact: Chaitanya
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-03-24 10:23 UTC by aarsharm
Modified: 2025-06-26 12:29 UTC (History)
6 users (show)

Fixed In Version: ceph-19.2.1-74.el9cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-06-26 12:29:27 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-10946 0 None None None 2025-03-24 10:23:26 UTC
Red Hat Product Errata RHSA-2025:9775 0 None None None 2025-06-26 12:29:31 UTC

Description aarsharm 2025-03-24 10:23:03 UTC
Description of problem:
As consistency group mirroring is not supporting images from multiple pools as of now. We should keep a check during "mirror group enable" that group should not have images from multiple pools and report user with right error string to remove images belonging to other group and then proceed with mirroring.

Testing this negative scenario, current behavior is group mirroring state goes to 'enabling' and i am not able to recover back from that state.

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group image add  --image pool_2_image --group-pool pool_1 --group group_1 --image-pool pool_2

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1
2025-03-24T09:53:54.702+0000 7f7b02696200 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: mirroring is not enabled on a pool
2025-03-24T09:53:54.702+0000 7f7b02696200 -1 librbd::api::Mirror: group_enable: failed enabling image: pool_2_image: (22) Invalid argument

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror pool enable pool_2 image
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# 

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1
2025-03-24T09:54:34.542+0000 7f01ec4d9200 -1 librbd::api::Mirror: group_enable: enabling mirroring for group group_1 either in progress or was interrupted
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1
2025-03-24T09:54:43.872+0000 7fac07c61200 -1 librbd::api::Mirror: group_enable: enabling mirroring for group group_1 either in progress or was interrupted

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1
rbd group 'group_1':
        id: 398487e237b8
        mirroring state: enabling
        mirroring mode: snapshot
        mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0
        mirroring primary: false

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1
rbd: mirroring not enabled on the group
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]#

When i try to disable mirroring, it succeeds but mirroring state still shows enabling.

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group disable --group group_1 --pool pool_1
Mirroring disabled

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1
rbd group 'group_1':
        id: 398487e237b8
        mirroring state: enabling
        mirroring mode: snapshot
        mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0
        mirroring primary: false
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# 

using force with disabling group mirorring, still shows mirroring state enabling
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group disable --group group_1 --pool pool_1 --force
Mirroring disabled

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1
rbd group 'group_1':
        id: 398487e237b8
        mirroring state: enabling
        mirroring mode: snapshot
        mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0
        mirroring primary: false
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# 

Removing the other pool image from the group also gives below error as it consider mirroring to be enabled.

[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group image remove --image pool_2_image --group-pool pool_1 --group group_1 --image-pool pool_2
rbd: remove image error: 2025-03-24T09:56:51.520+0000 7fd25cb4e200 -1 librbd::api::Group: image_remove: cannot remove image from mirror enabled group
(22) Invalid argument
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# 


Version-Release number of selected component (if applicable):
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# ceph version
ceph version 19.2.1-57.el9cp (25ca432e5c2874ac833d0f13057a1b7d98913317) squid (stable)
[ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]#

Comment 6 errata-xmlrpc 2025-06-26 12:29:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775


Note You need to log in before you can comment on or make changes to this bug.