Description of problem: Snapshot based mirroring is not getting replicated on the secondary cluster. Journal based mirroring is working fine on the same clusters Version-Release number of selected component (if applicable): ceph version 14.2.11-183.el8cp (07313a094b455c7834e137d63da57830ef66b672) nautilus (stable) How reproducible: Create 2 clusters. create a pool (data) on both the clusters and initilize it for rbd enable one way rbd mirroring using https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/block_device_guide/mirroring-ceph-block-devices#configuring-one-way-mirroring-using-ansible_block . Create image and enable mirroring with snapshot mode Same if the image is enabled with journal mode mirroring is working fine As part of this i have created image1 and image3 with snapshot mode image2 with journal mode imgae2 is getting replicated but image1 and image3 are giving below status rbd: mirroring not enabled on the image Primary Cluster: [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror pool enable data image [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image enable data/image1 snapshot Mirroring enabled [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd create image2 --size 1024 --pool data --image-feature exclusive-lock [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2 rbd: mirroring not enabled on the image [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image enable data/image2 journal Mirroring enabled [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2 image2: global_id: 7633b077-67cf-4c63-b7bc-914c0599cced [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1 image1: global_id: dcf0226b-1235-4614-aebb-0076ce8a1fa0 snapshots: 4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd create image3 --size 1024 --pool data --image-feature exclusive-lock [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image3 rbd: mirroring not enabled on the image [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image enable data/image3 snapshot Mirroring enabled [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image3 image3: global_id: cf4f377e-ebc0-4441-a1c0-54b31bfbdffd snapshots: 7 .mirror.primary.cf4f377e-ebc0-4441-a1c0-54b31bfbdffd.3f2f355a-50dd-4803-bf9f-587eb756a83a (peer_uuids:[c4bd7d98-b5fa-4049-a80a-8fd9500078b7]) [root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2 image2: global_id: 7633b077-67cf-4c63-b7bc-914c0599cced Sceondary Cluster: [cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ sudo -i rbd mirror image status data/image3 rbd: mirroring not enabled on the image [cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ sudo -i rbd mirror image status data/image2 image2: global_id: 7633b077-67cf-4c63-b7bc-914c0599cced state: up+replaying description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} service: ceph-rbd2-rbd-mirror-up-1625303521156-node2-client on ceph-rbd2-rbd-mirror-up-1625303521156-node2-client last_update: 2021-07-05 03:16:25 [cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ Please find the commands executed on both clusters https://docs.google.com/document/d/1Qg-_lEDJh5n0pHSRJq9T8n9qsUbP47pxEtFlDuead8E/edit# Primary Node Ip :10.0.210.65 Secondary Node IP : 10.0.211.44 Steps to Reproduce: 1. 2. 3. Actual results: should enable mirroring Expected results: Additional info:
Hi, Could you give me access to the doc you shared. Thanks, Sunny
Hi Sunny, I have shared the document with you. Is anything else I need to provide. Thanks, Amarnath
Hi Sunny, Can you please look in to this issue as we are blocked with upgrade from 4.2 to 4.3 due to snapshot based one-way rbd-mirror.
Hi Gopi, Looks like its fixed upstream. https://tracker.ceph.com/issues/52675. I will confirm this once and let you know. Thanks, Sunny
Hi Sunny, The tracker which is there in comment#5 back-ported to only octopus and pacific. I tested with latest nautilus build and facing the same issue. [root@ceph-rbd1-gpatta-rbd-mirror-v8aiiu-node1-installer ~]# ceph -v ceph version 14.2.11-208.el8cp (6738ba96f296a41c24357c12e8d594fbde457abc) nautilus (stable) [root@ceph-rbd2-gpatta-rbd-mirror-v8aiiu-node1-installer ~]# rbd mirror image status rbd-pool/image1 rbd: mirroring not enabled on the image
Hi Ilya, Any update on this as we are blocking with 4.3 one-way mirror. The tracker which is there in comment#5 back-ported to only octopus and pacific not to Nautilus.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 4.3 Security and Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1716