Bug 2357179 - cross namespace mirror group enters into split-brain on a normal relocate operation
Summary: cross namespace mirror group enters into split-brain on a normal relocate ope...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD-Mirror
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 8.1
Assignee: Prasanna Kumar Kalever
QA Contact: Chaitanya
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-04-03 12:35 UTC by Chaitanya
Modified: 2025-06-26 12:22 UTC (History)
4 users (show)

Fixed In Version: ceph-19.2.1-109.el9cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-06-26 12:22:32 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-11050 0 None None None 2025-04-03 12:35:54 UTC
Red Hat Product Errata RHSA-2025:9775 0 None None None 2025-06-26 12:22:35 UTC

Description Chaitanya 2025-04-03 12:35:29 UTC
Description of problem:
seeing 'up+error' and 'split-brain detected' when I do 'demote' on primary and 'promote' on secondary. (initial state is up+stopped on primary and Up+ Replaying on secondary)

root@ceph-rbd1-cd-cg-x49tk0-node2 ~]# rbd mirror group demote p1/ns1/g1
2025-04-03T07:59:50.180+0000 7f5791140640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x559bac3c9ba0 handle_remove_group_snapshot: failed to remove image snapshot metadata: (30) Read-only file system
Group demoted to non-primary

[root@ceph-rbd2-cd-cg-x49tk0-node2 ~]# rbd mirror group promote p1/ns1/g1
Group promoted to primary

[root@ceph-rbd1-cd-cg-x49tk0-node2 ~]# rbd mirror group status p1/ns1/g1
g1:
 global_id:  6c3e1329-ac3a-46cf-8234-27c0a854d11a
 state:    up+error
 description: split-brain detected
 service:   ceph-rbd1-cd-cg-x49tk0-node5.qphwpz on ceph-rbd1-cd-cg-x49tk0-node5
 last_update: 2025-04-03 08:03:01
 images:
 peer_sites:
  name: ceph-rbd2
  state: up+stopped
  description: 
  last_update: 2025-04-03 08:03:05
  images:
   image:    3/49bec464-6559-49bd-a2b3-61982915cb07
   state:    up+stopped
   description: local image is primary

   image:    3/5904fee2-e6da-4e2f-ba1a-a80f9dfb2f9c
   state:    up+stopped
   description: local image is primary


This is happening only with the groups on namespaces.

As per dev, this is already WIP and is available in the next build.

Raising this BZ for tracking/reference purpose. 

Version-Release number of selected component (if applicable):
ceph version 19.2.1-57.el9cp

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
split brain seen

Expected results:
No split brain should be seen

Additional info:

Comment 6 errata-xmlrpc 2025-06-26 12:22:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775


Note You need to log in before you can comment on or make changes to this bug.