2361747 – [Scale] Mirror group operation failed with : librbd::mirror::GroupGetInfoRequest: 0x5560070f3690 handle_get_last_mirror_snapshot_state: failed to list group snapshots of group 'group_29': (22) Invalid argument

Bug 2361747 - [Scale] Mirror group operation failed with : librbd::mirror::GroupGetInfoRequest: 0x5560070f3690 handle_get_last_mirror_snapshot_state: failed to list group snapshots of group 'group_29': (22) Invalid argument

Summary: [Scale] Mirror group operation failed with : librbd::mirror::GroupGetInfoRequ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RBD-Mirror
Sub Component:
Version:	8.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	8.1
Assignee:	N Balachandran
QA Contact:	aarsharm
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2025-04-22 19:05 UTC by aarsharm
Modified:	2025-06-26 12:31 UTC (History)
CC List:	5 users (show)
Fixed In Version:	ceph-19.2.1-166.el9cp
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2025-06-26 12:30:56 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHCEPH-11233	0	None	None	None	2025-04-22 19:07:15 UTC
Red Hat Product Errata	RHSA-2025:9775	0	None	None	None	2025-06-26 12:30:59 UTC

Description aarsharm 2025-04-22 19:05:17 UTC

Description of problem:
In a scale setup, where 100 groups with 1 image each (size of image 22G)
Only one group i.e group_29 (out of 100 failed with below error)

[ceph: root@tala001 /]# rbd mirror group status --pool pool_2 --group group_29  --debug_rbd 0
rbd: failed to get mirror info for group: 2025-04-22T18:56:42.890+0000 7f39dd57b640 -1 librbd::mirror::GroupGetInfoRequest: 0x5619341d7680 handle_get_last_mirror_snapshot_state: failed to list group snapshots of group 'group_29': (22) Invalid argument
(22) Invalid argument
[ceph: root@tala001 /]#

[ceph: root@tala001 /]# rbd group snap list --pool pool_2 --group group_29  --debug_rbd 0
ID             NAME                                                                STATE     NAMESPACE
16e814b317b28  .mirror.primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.16e814b317b28  complete  mirror (primary peer_uuids:[])
18db94735015b  .mirror.primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.18db94735015b  complete  mirror (primary peer_uuids:[e2d30a1f-3c7d-45bc-b91e-ca81cc428230])
1b8dfe216f73d  .mirror.primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.1b8dfe216f73d  complete  mirror (demoted peer_uuids:[e2d30a1f-3c7d-45bc-b91e-ca81cc428230])
[ceph: root@tala001 /]#

site-b:
[ceph: root@tala005 /]# rbd mirror group status --pool pool_2 --group group_29  --debug_rbd 0
group_29:
  global_id:   6f803dc2-25d4-43de-985e-f1f3ea32e8bc
  state:       up+error
  description: bootstrap failed
  service:     tala005.jwqeia on tala005
  last_update: 2025-04-22 18:41:42
  images:
  peer_sites:
    name: site-a
    state: up+error
    description: bootstrap failed
    last_update: 2025-04-22 18:42:01
    images:
  snapshots:
    .mirror.non-primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.1b8dfe216f73d
    .mirror.primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.1330b1824f430
[ceph: root@tala005 /]# rbd group snap list --pool pool_2 --group group_29  --debug_rbd 0
ID             NAME                                                                    STATE     NAMESPACE                                                         
1b8dfe216f73d  .mirror.non-primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.1b8dfe216f73d  complete  mirror (demoted peer_uuids:[a3fa4ccf-91c5-4cf2-b717-470c30908b23] 359d7808-3dbb-4e59-8b66-ad1cfe162441:1b8dfe216f73d)
1330b1824f430  .mirror.primary.6f803dc2-25d4-43de-985e-f1f3ea32e8bc.1330b1824f430      complete  mirror (primary peer_uuids:[a3fa4ccf-91c5-4cf2-b717-470c30908b23])
[ceph: root@tala005 /]#


Version-Release number of selected component (if applicable):
[ceph: root@tala005 /]# ceph version
ceph version 19.2.1-151.el9cp (642684c3e6e5556e1132f363e49d5e26b605e8ef) squid (stable)
[ceph: root@tala005 /]#


Steps to Reproduce:
1. perform demote on all 100 groups on site-a
2. perform promote on all 100 groups on site-b
[ceph: root@tala005 /]# rbd mirror group promote --pool pool_2 --group group_1 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_2 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_3 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_4 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_5 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_6 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_7 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_8 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_9 --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_10  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_11  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_12  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_13  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_14  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_15  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_16  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_17  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_18  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_19  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_20  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_21  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_22  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_23  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_24  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_25  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_26  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_27  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_28  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_29  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_30  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_31  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_32  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_33  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_34  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_35  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_36  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_37  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_38  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_39  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_40  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_41  --debug_rbd 0
rbd mirror group promote --pool pool_2 --group group_100 --debug_rbd 0
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
rbd: error promoting group to primary
2025-04-22T18:08:07.262+0000 7f696547b380 -1 librbd::api::Mirror: group_promote: group group_5 is still primary within a remote cluster
rbd: error promoting group to primary
2025-04-22T18:08:07.300+0000 7fa0b411b380 -1 librbd::api::Mirror: group_promote: group group_6 is still primary within a remote cluster
rbd: error promoting group to primary
2025-04-22T18:08:07.338+0000 7fb601dfd380 -1 librbd::api::Mirror: group_promote: group group_7 is still primary within a remote cluster
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
rbd: error promoting group to primary
2025-04-22T18:08:50.183+0000 7f17de8e1380 -1 librbd::api::Mirror: group_promote: group group_32 is still primary within a remote cluster
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
rbd: error promoting group to primary
2025-04-22T18:09:26.233+0000 7f6e55252380 -1 librbd::api::Mirror: group_promote: group group_62 is still primary within a remote cluster
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
Group promoted to primary
[ceph: root@tala005 /]#

3. perform force promote on site-a for all groups
4. perform demote on all 100 groups on site-a
[ceph: root@tala001 /]# rbd mirror group demote --pool pool_2 --group group_1 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_2 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_3 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_4 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_5 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_6 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_7 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_8 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_9 --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_10  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_11  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_12  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_13  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_14  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_15  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_16  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_17  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_18  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_19  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_20  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_21  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_22  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_23  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_24  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_25  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_26  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_27  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_28  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_29  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_30  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_31  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_32  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_33  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_34  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_35  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_36  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_37  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_38  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_39  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_40  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_41  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_42  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_43  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_44  --debug_rbd 0
rbd mirror group demote --pool pool_2 --group group_100 --debug_rbd 0
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
2025-04-22T18:35:02.443+0000 7f249effd640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55fb88cbb5a0 handle_remove_peer_uuid: failed to remove group snapshot mirror peer: (17) File exists
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
2025-04-22T18:35:17.447+0000 7ff0ef7fe640 -1 librbd::mirror::GroupGetInfoRequest: 0x55f9f1258090 handle_get_last_mirror_snapshot_state: failed to list group snapshots of group 'group_29': (22) Invalid argument
rbd: failed to get mirror info for group: (22) Invalid argument
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
2025-04-22T18:35:32.395+0000 7fe426ffd640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55c89fd8ffc0 handle_remove_peer_uuid: failed to remove group snapshot mirror peer: (17) File exists
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
2025-04-22T18:36:02.419+0000 7f410affd640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55b774c685a0 handle_remove_peer_uuid: failed to remove group snapshot mirror peer: (17) File exists
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
Group demoted to non-primary
2025-04-22T18:36:32.427+0000 7f9899ffb640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55c6ea1f3390 handle_remove_peer_uuid: failed to remove group snapshot mirror peer: (17) File exists
Group demoted to non-primary
[ceph: root@tala001 /]#


Actual results: group_29 mirror group info is not displayed on site-a


Expected results: group_29 mirror group info should be displayed on site-a


Additional info: NA

Comment 10 errata-xmlrpc 2025-06-26 12:30:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775

Note You need to log in before you can comment on or make changes to this bug.