Bug 1979186 - Snapshot based mirroring not working
Summary: Snapshot based mirroring not working
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD-Mirror
Version: 4.2
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: 4.3
Assignee: Ilya Dryomov
QA Contact: Vasishta
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-05 07:29 UTC by Amarnath
Modified: 2022-05-05 07:53 UTC (History)
7 users (show)

Fixed In Version: ceph-14.2.22-47.el8cp, ceph-14.2.22-47.el7cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-05 07:53:24 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:1716 0 None None None 2022-05-05 07:53:50 UTC

Description Amarnath 2021-07-05 07:29:38 UTC
Description of problem:
Snapshot based mirroring is not getting replicated on the secondary cluster.
Journal based mirroring is working fine on the same clusters


Version-Release number of selected component (if applicable):
ceph version 14.2.11-183.el8cp (07313a094b455c7834e137d63da57830ef66b672) nautilus (stable)


How reproducible:
Create 2 clusters.
create a pool (data) on both the clusters and initilize it for rbd

enable one way rbd mirroring using 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/block_device_guide/mirroring-ceph-block-devices#configuring-one-way-mirroring-using-ansible_block .

Create image and enable mirroring with snapshot mode 
Same if the image is enabled with journal  mode mirroring is working fine

As part of this i have created image1 and image3 with snapshot mode
image2 with journal mode
imgae2 is getting replicated but image1 and image3 are giving below status
rbd: mirroring not enabled on the image


Primary Cluster:
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror pool enable data image
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd  mirror image enable data/image1 snapshot
Mirroring enabled
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd create image2 --size 1024 --pool data --image-feature exclusive-lock
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2
rbd: mirroring not enabled on the image
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd  mirror image enable data/image2 journal
Mirroring enabled
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2
image2:
  global_id:   7633b077-67cf-4c63-b7bc-914c0599cced
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image1
image1:
  global_id:   dcf0226b-1235-4614-aebb-0076ce8a1fa0
  snapshots:
    4 .mirror.primary.dcf0226b-1235-4614-aebb-0076ce8a1fa0.586afc86-f65e-41ea-87c9-4dccb22a6022 (peer_uuids:[])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd create image3 --size 1024 --pool data --image-feature exclusive-lock
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image3
rbd: mirroring not enabled on the image
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd  mirror image enable data/image3 snapshot
Mirroring enabled
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image3
image3:
  global_id:   cf4f377e-ebc0-4441-a1c0-54b31bfbdffd
  snapshots:
    7 .mirror.primary.cf4f377e-ebc0-4441-a1c0-54b31bfbdffd.3f2f355a-50dd-4803-bf9f-587eb756a83a (peer_uuids:[c4bd7d98-b5fa-4049-a80a-8fd9500078b7])
[root@ceph-rbd1-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ceph]# rbd mirror image status data/image2
image2:
  global_id:   7633b077-67cf-4c63-b7bc-914c0599cced


Sceondary Cluster:
[cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ sudo -i rbd mirror image status data/image3
rbd: mirroring not enabled on the image
[cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ sudo -i rbd mirror image status data/image2
image2:
  global_id:   7633b077-67cf-4c63-b7bc-914c0599cced
  state:       up+replaying
  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
  service:     ceph-rbd2-rbd-mirror-up-1625303521156-node2-client on ceph-rbd2-rbd-mirror-up-1625303521156-node2-client
  last_update: 2021-07-05 03:16:25
[cephuser@ceph-rbd2-rbd-mirror-up-1625303521156-node1-mon-mgr-installer ~]$ 


Please find the commands executed on both clusters
https://docs.google.com/document/d/1Qg-_lEDJh5n0pHSRJq9T8n9qsUbP47pxEtFlDuead8E/edit#

Primary Node  Ip :10.0.210.65
Secondary Node IP : 10.0.211.44



Steps to Reproduce:
1.
2.
3.

Actual results:
should enable mirroring


Expected results:


Additional info:

Comment 2 Sunny Kumar 2021-11-11 22:48:35 UTC
Hi,

Could you give me access to the doc you shared.

Thanks,
Sunny

Comment 3 Amarnath 2021-11-18 18:03:25 UTC
Hi Sunny,

I have shared the document with you.
Is anything else I need to provide.

Thanks,
Amarnath

Comment 4 Gopi 2021-11-23 10:03:59 UTC
Hi Sunny,

Can you please look in to this issue as we are blocked with upgrade from 4.2 to 4.3 due to snapshot based one-way rbd-mirror.

Comment 5 Sunny Kumar 2021-11-23 11:29:05 UTC
Hi Gopi,

Looks like its fixed upstream.

https://tracker.ceph.com/issues/52675.

I will confirm this once and let you know.

Thanks,
Sunny

Comment 11 Gopi 2021-12-20 11:39:23 UTC
Hi Sunny,

The tracker which is there in comment#5 back-ported to only octopus and pacific. I tested with latest nautilus build and facing the same issue.


[root@ceph-rbd1-gpatta-rbd-mirror-v8aiiu-node1-installer ~]# ceph -v
ceph version 14.2.11-208.el8cp (6738ba96f296a41c24357c12e8d594fbde457abc) nautilus (stable)

[root@ceph-rbd2-gpatta-rbd-mirror-v8aiiu-node1-installer ~]# rbd mirror image status rbd-pool/image1
rbd: mirroring not enabled on the image

Comment 12 Gopi 2022-01-11 05:31:26 UTC
Hi Ilya,

Any update on this as we are blocking with 4.3 one-way mirror.

The tracker which is there in comment#5 back-ported to only octopus and pacific not to Nautilus.

Comment 23 errata-xmlrpc 2022-05-05 07:53:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 4.3 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1716


Note You need to log in before you can comment on or make changes to this bug.