Bug 2354499
| Summary: | Deadlock to recover from consistency group with images from multiple pools | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | aarsharm |
| Component: | RBD-Mirror | Assignee: | Prasanna Kumar Kalever <prasanna.kalever> |
| Status: | CLOSED ERRATA | QA Contact: | Chaitanya <cdommeti> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.1 | CC: | cdommeti, ceph-eng-bugs, cephqe-warriors, idryomov, sangadi, tserlin |
| Target Milestone: | --- | ||
| Target Release: | 8.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-19.2.1-74.el9cp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-06-26 12:29:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775 |
Description of problem: As consistency group mirroring is not supporting images from multiple pools as of now. We should keep a check during "mirror group enable" that group should not have images from multiple pools and report user with right error string to remove images belonging to other group and then proceed with mirroring. Testing this negative scenario, current behavior is group mirroring state goes to 'enabling' and i am not able to recover back from that state. [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group image add --image pool_2_image --group-pool pool_1 --group group_1 --image-pool pool_2 [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1 2025-03-24T09:53:54.702+0000 7f7b02696200 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: mirroring is not enabled on a pool 2025-03-24T09:53:54.702+0000 7f7b02696200 -1 librbd::api::Mirror: group_enable: failed enabling image: pool_2_image: (22) Invalid argument [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror pool enable pool_2 image [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1 2025-03-24T09:54:34.542+0000 7f01ec4d9200 -1 librbd::api::Mirror: group_enable: enabling mirroring for group group_1 either in progress or was interrupted [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1 2025-03-24T09:54:43.872+0000 7fac07c61200 -1 librbd::api::Mirror: group_enable: enabling mirroring for group group_1 either in progress or was interrupted [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1 rbd group 'group_1': id: 398487e237b8 mirroring state: enabling mirroring mode: snapshot mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0 mirroring primary: false [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 rbd: mirroring not enabled on the group [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# When i try to disable mirroring, it succeeds but mirroring state still shows enabling. [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group disable --group group_1 --pool pool_1 Mirroring disabled [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1 rbd group 'group_1': id: 398487e237b8 mirroring state: enabling mirroring mode: snapshot mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0 mirroring primary: false [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# using force with disabling group mirorring, still shows mirroring state enabling [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd mirror group disable --group group_1 --pool pool_1 --force Mirroring disabled [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group info --group group_1 --pool pool_1 rbd group 'group_1': id: 398487e237b8 mirroring state: enabling mirroring mode: snapshot mirroring global id: 48ca1fcc-a534-44a2-ada4-6575e009eab0 mirroring primary: false [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# Removing the other pool image from the group also gives below error as it consider mirroring to be enabled. [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# rbd group image remove --image pool_2_image --group-pool pool_1 --group group_1 --image-pool pool_2 rbd: remove image error: 2025-03-24T09:56:51.520+0000 7fd25cb4e200 -1 librbd::api::Group: image_remove: cannot remove image from mirror enabled group (22) Invalid argument [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# Version-Release number of selected component (if applicable): [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]# ceph version ceph version 19.2.1-57.el9cp (25ca432e5c2874ac833d0f13057a1b7d98913317) squid (stable) [ceph: root@ceph-rbd1-cg-ds-tf74dn-node1-installer /]#