Description of problem: Seeing below error during promote on primary site. [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 2025-04-04T06:29:42.583+0000 7f3f49ffb640 -1 librbd::mirror::PromoteRequest: 0x7f3f30012ae0 handle_get_info: image is primary within a remote cluster or demotion is not propagated yet 2025-04-04T06:29:42.584+0000 7f3f4a7fc640 -1 librbd::mirror::PromoteRequest: 0x7f3f2800f260 handle_get_info: image is primary within a remote cluster or demotion is not propagated yet 2025-04-04T06:29:42.584+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed promoting image: image1: (16) Device or resource busy 2025-04-04T06:29:42.584+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed promoting image: image2: (16) Device or resource busy 2025-04-04T06:29:42.634+0000 7f3f49ffb640 -1 librbd::mirror::DemoteRequest: 0x7f3f3001bb80 handle_get_info: image is not primary 2025-04-04T06:29:42.634+0000 7f3f4a7fc640 -1 librbd::mirror::DemoteRequest: 0x7f3f3001bb80 handle_get_info: image is not primary 2025-04-04T06:29:42.634+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed demoting image: image1: (22) Invalid argument 2025-04-04T06:29:42.634+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed demoting image: image2: (22) Invalid argument rbd: error promoting group to primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f2db8049dc0 handle_create_snapshot: failed to create mirror snapshot: (30) Read-only file system 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::PromoteRequest: 0x7f2dc0004740 handle_create_promote_snapshot: failed to create promote snapshot: (30) Read-only file system 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::PromoteRequest: 0x559f17503110 handle_promote: failed to promote image: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f2dc0008480 handle_create_snapshot: failed to create mirror snapshot: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::PromoteRequest: 0x7f2dcc06f000 handle_create_promote_snapshot: failed to create promote snapshot: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::PromoteRequest: 0x7f2db8019140 handle_promote: failed to promote image: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed promoting image: image1: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed promoting image: image2: (30) Read-only file system 2025-04-04T06:30:00.882+0000 7f2dda7fc640 -1 librbd::mirror::DemoteRequest: 0x7f2db8016510 handle_get_info: image is not primary 2025-04-04T06:30:00.882+0000 7f2dda7fc640 -1 librbd::mirror::DemoteRequest: 0x7f2db8010df0 handle_get_info: image is not primary 2025-04-04T06:30:00.882+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed demoting image: image1: (22) Invalid argument 2025-04-04T06:30:00.882+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed demoting image: image2: (22) Invalid argument rbd: error promoting group to primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# Version-Release number of selected component (if applicable): [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# ceph version ceph version 19.2.1-57.el9cp (25ca432e5c2874ac833d0f13057a1b7d98913317) squid (stable) [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# Steps to Reproduce: 1. Create pool on both clusters 2. Enable image mirror mode on pool Primary site: [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror pool enable pool_1 image Secondary site: [ceph: root@ceph-rbd2-cg-new-np6kos-node1-installer /]# rbd mirror pool enable pool_1 image 3. Create 2 images on Primary site and add IO [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd create image1 --size 1G --pool pool_1 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd create image2 --size 1G --pool pool_1 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd bench --io-type write --io-threads 16 --io-total 1G --io-pattern rand --io-size 4096 pool_1/image1 bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern random SEC OPS OPS/SEC BYTES/SEC 1 15584 15615.5 61 MiB/s 2 31168 15591.9 61 MiB/s 3 45568 14809.5 58 MiB/s 4 59936 14991.6 59 MiB/s 5 75072 15017.5 59 MiB/s 6 90176 14879.6 58 MiB/s 7 103376 14415.5 56 MiB/s 8 117504 14618 57 MiB/s 9 133296 14486.4 57 MiB/s 10 147072 14379.7 56 MiB/s 11 161600 14247.6 56 MiB/s 12 176672 14670.8 57 MiB/s 13 191424 14559.6 57 MiB/s 14 204576 14440.7 56 MiB/s 15 217008 14001.1 55 MiB/s 16 230928 13937.9 54 MiB/s 17 245440 13544.9 53 MiB/s 18 259040 13715.1 54 MiB/s elapsed: 18 ops: 262144 ops/sec: 14297.3 bytes/sec: 56 MiB/s [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd bench --io-type write --io-threads 16 --io-total 1G --io-pattern rand --io-size 4096 pool_1/image2 bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern random SEC OPS OPS/SEC BYTES/SEC 1 16784 16231.7 63 MiB/s 2 30480 15255.5 60 MiB/s 3 44544 14808.8 58 MiB/s 4 59568 14892.1 58 MiB/s 5 73408 14539.3 57 MiB/s 6 88000 14346.4 56 MiB/s 7 102768 14420 56 MiB/s 8 115680 14170.4 55 MiB/s 9 128480 13779.5 54 MiB/s 10 141328 13698.9 54 MiB/s 11 152576 12889.3 50 MiB/s 12 165248 12480.9 49 MiB/s 13 178240 12582.3 49 MiB/s 14 192400 12791.6 50 MiB/s 15 204320 12613.4 49 MiB/s 16 219280 13367.4 52 MiB/s 17 231616 13324.1 52 MiB/s 18 244144 13186 52 MiB/s 19 255536 12627.1 49 MiB/s elapsed: 19 ops: 262144 ops/sec: 13147.8 bytes/sec: 51 MiB/s [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 4. Create group and add both images to the group [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd group create --pool pool_1 --group group_1 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd group image add --image image1 --group group_1 --group-pool pool_1 --image-pool pool_1 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd group image add --image image2 --group group_1 --group-pool pool_1 --image-pool pool_1 5. Enable mirroring on the group [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group enable --group group_1 --pool pool_1 Mirroring enabled [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 6. Wait for replay_state to change from 'syncing' to 'idle' for both images [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 group_1: global_id: 892cf4da-d334-40f3-9935-5a04a5bf70e4 state: up+stopped description: service: ceph-rbd1-cg-new-np6kos-node5.skrihw on ceph-rbd1-cg-new-np6kos-node5 last_update: 2025-04-04 06:19:44 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: local image is primary image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: local image is primary peer_sites: name: site-b state: up+replaying description: replaying last_update: 2025-04-04 06:19:39 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+replaying description: replaying, {"bytes_per_second":62326784.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1743747567,"replay_state":"syncing","seconds_until_synced":0,"syncing_percent":95,"syncing_snapshot_timestamp":1743747567} image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+replaying description: replaying, {"bytes_per_second":56492032.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1743747566,"replay_state":"syncing","seconds_until_synced":0,"syncing_percent":86,"syncing_snapshot_timestamp":1743747566} snapshots: .mirror.primary.892cf4da-d334-40f3-9935-5a04a5bf70e4.39994e6c7036 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 group_1: global_id: 892cf4da-d334-40f3-9935-5a04a5bf70e4 state: up+stopped description: service: ceph-rbd1-cg-new-np6kos-node5.skrihw on ceph-rbd1-cg-new-np6kos-node5 last_update: 2025-04-04 06:19:44 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: local image is primary image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: local image is primary peer_sites: name: site-b state: up+replaying description: replaying last_update: 2025-04-04 06:20:09 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+replaying description: replaying, {"bytes_per_second":1838421.33,"bytes_per_snapshot":678420480.0,"last_snapshot_bytes":678420480,"last_snapshot_sync_seconds":8,"local_snapshot_timestamp":1743747567,"remote_snapshot_timestamp":1743747567,"replay_state":"idle"} image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+replaying description: replaying, {"bytes_per_second":3787571.2,"bytes_per_snapshot":678547456.0,"last_snapshot_bytes":678547456,"last_snapshot_sync_seconds":8,"local_snapshot_timestamp":1743747566,"remote_snapshot_timestamp":1743747566,"replay_state":"idle"} snapshots: .mirror.primary.892cf4da-d334-40f3-9935-5a04a5bf70e4.39994e6c7036 [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 7. demote primary site [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group demote pool_1/group_1 Group demoted to non-primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 8. Promote secondary site [ceph: root@ceph-rbd2-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 Group promoted to primary [ceph: root@ceph-rbd2-cg-new-np6kos-node1-installer /]# 9. Verify group status, Site-b: up+stopped , Site-a: up+replaying [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 group_1: global_id: 892cf4da-d334-40f3-9935-5a04a5bf70e4 state: up+replaying description: replaying service: ceph-rbd1-cg-new-np6kos-node5.skrihw on ceph-rbd1-cg-new-np6kos-node5 last_update: 2025-04-04 06:24:14 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1743747734,"remote_snapshot_timestamp":1743747734,"replay_state":"idle"} image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+replaying description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1743747734,"remote_snapshot_timestamp":1743747734,"replay_state":"idle"} peer_sites: name: site-b state: up+stopped description: last_update: 2025-04-04 06:24:09 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: local image is primary image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: local image is primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 10. Do force promote on site-a + demote (split brain scenario) [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 --force Group promoted to primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group demote --group group_1 --pool pool_1 Group demoted to non-primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 11. Wait for split brain status [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 group_1: global_id: 892cf4da-d334-40f3-9935-5a04a5bf70e4 state: up+stopped description: service: ceph-rbd1-cg-new-np6kos-node5.skrihw on ceph-rbd1-cg-new-np6kos-node5 last_update: 2025-04-04 06:26:54 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: split-brain image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: split-brain peer_sites: name: site-b state: up+stopped description: last_update: 2025-04-04 06:26:39 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: local image is primary image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: local image is primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 12. Perform resync on site-a [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group resync --group group_1 --pool pool_1 Flagged group for resync from primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 13. Wait till site-a: status is up+replaying and site-b: up+stopped [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group status --group group_1 --pool pool_1 group_1: global_id: 892cf4da-d334-40f3-9935-5a04a5bf70e4 state: up+replaying description: replaying service: ceph-rbd1-cg-new-np6kos-node5.skrihw on ceph-rbd1-cg-new-np6kos-node5 last_update: 2025-04-04 06:28:44 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+replaying description: replaying, {"bytes_per_second":33921024.0,"bytes_per_snapshot":678420480.0,"last_snapshot_bytes":678420480,"last_snapshot_sync_seconds":7,"local_snapshot_timestamp":1743747734,"remote_snapshot_timestamp":1743747734,"replay_state":"idle"} image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+replaying description: replaying, {"bytes_per_second":33927372.8,"bytes_per_snapshot":678547456.0,"last_snapshot_bytes":678547456,"last_snapshot_sync_seconds":8,"local_snapshot_timestamp":1743747734,"remote_snapshot_timestamp":1743747734,"replay_state":"idle"} peer_sites: name: site-b state: up+stopped description: last_update: 2025-04-04 06:28:39 images: image: 3/c5e0d62c-3989-49b4-86fc-4857531e30d9 state: up+stopped description: local image is primary image: 3/cd23a6a7-4c27-46a4-9efb-3677c75b149f state: up+stopped description: local image is primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# 14. Perform demote on site-b: [ceph: root@ceph-rbd2-cg-new-np6kos-node1-installer /]# rbd mirror group demote pool_1/group_1 2025-04-04T06:29:41.716+0000 7f806bfff640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x562e347b9780 handle_remove_group_snapshot: failed to remove image snapshot metadata: (30) Read-only file system Group demoted to non-primary [ceph: root@ceph-rbd2-cg-new-np6kos-node1-installer /]# 15. Perform promote on site-a: [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 2025-04-04T06:29:42.583+0000 7f3f49ffb640 -1 librbd::mirror::PromoteRequest: 0x7f3f30012ae0 handle_get_info: image is primary within a remote cluster or demotion is not propagated yet 2025-04-04T06:29:42.584+0000 7f3f4a7fc640 -1 librbd::mirror::PromoteRequest: 0x7f3f2800f260 handle_get_info: image is primary within a remote cluster or demotion is not propagated yet 2025-04-04T06:29:42.584+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed promoting image: image1: (16) Device or resource busy 2025-04-04T06:29:42.584+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed promoting image: image2: (16) Device or resource busy 2025-04-04T06:29:42.634+0000 7f3f49ffb640 -1 librbd::mirror::DemoteRequest: 0x7f3f3001bb80 handle_get_info: image is not primary 2025-04-04T06:29:42.634+0000 7f3f4a7fc640 -1 librbd::mirror::DemoteRequest: 0x7f3f3001bb80 handle_get_info: image is not primary 2025-04-04T06:29:42.634+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed demoting image: image1: (22) Invalid argument 2025-04-04T06:29:42.634+0000 7f3f51b24200 -1 librbd::api::Mirror: group_promote: failed demoting image: image2: (22) Invalid argument rbd: error promoting group to primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group promote pool_1/group_1 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f2db8049dc0 handle_create_snapshot: failed to create mirror snapshot: (30) Read-only file system 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::PromoteRequest: 0x7f2dc0004740 handle_create_promote_snapshot: failed to create promote snapshot: (30) Read-only file system 2025-04-04T06:30:00.841+0000 7f2dda7fc640 -1 librbd::mirror::PromoteRequest: 0x559f17503110 handle_promote: failed to promote image: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f2dc0008480 handle_create_snapshot: failed to create mirror snapshot: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::snapshot::PromoteRequest: 0x7f2dcc06f000 handle_create_promote_snapshot: failed to create promote snapshot: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2dda7fc640 -1 librbd::mirror::PromoteRequest: 0x7f2db8019140 handle_promote: failed to promote image: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed promoting image: image1: (30) Read-only file system 2025-04-04T06:30:00.843+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed promoting image: image2: (30) Read-only file system 2025-04-04T06:30:00.882+0000 7f2dda7fc640 -1 librbd::mirror::DemoteRequest: 0x7f2db8016510 handle_get_info: image is not primary 2025-04-04T06:30:00.882+0000 7f2dda7fc640 -1 librbd::mirror::DemoteRequest: 0x7f2db8010df0 handle_get_info: image is not primary 2025-04-04T06:30:00.882+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed demoting image: image1: (22) Invalid argument 2025-04-04T06:30:00.882+0000 7f2de19a5200 -1 librbd::api::Mirror: group_promote: failed demoting image: image2: (22) Invalid argument rbd: error promoting group to primary [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# Note: Operation #15 was performed quickly after #14, which may have led us into promote failing. After this no matter, how many times i retry promote operation, this is stuck here in the same error. Also noticing other mirror group operations to fail due to this. for example: [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# rbd mirror group snapshot --pool pool_1 --group group_1 2025-04-04T06:55:50.769+0000 7f5a767fc640 -1 librbd::mirror::snapshot::GroupCreatePrimaryRequest: handle_get_last_mirror_snapshot_state: group group_1 is not primary 2025-04-04T06:55:50.769+0000 7f5a7dc1e200 -1 librbd::api::Mirror: group_snapshot_create: failed to create mirror snapshot for group 'group_1': (22) Invalid argument rbd: error creating snapshot: (22) Invalid argument [ceph: root@ceph-rbd1-cg-new-np6kos-node1-installer /]# Actual results: Promote failing Expected results: promote should succeed Additional info: NA
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775