Bug 2277144

+++ This bug was initially created as a clone of Bug #2248176 +++

Description of problem:
-----------------------

The secondary cluster is configured with 3 mons, when a peer connection is established, the peer list/snapshot mirror status always displays only one mon host instead of all the mon details.

[root@ceph1-hk-m-4p31kb-node7 ~]# ceph fs snapshot mirror peer_list cephfs
{"c4f15c31-8d2b-445e-8fa7-3137a1b638ca": {"client_name": "client.mirror_remote", "site_name": "remote_site", "fs_name": "cephfs", "mon_host": "[v2:10.0.97.224:3300,v1:10.0.97.224:6789]"}}
[root@ceph1-hk-m-4p31kb-node7 ~]#

The secondary cluster is configured with 3 mon hosts : 
[root@ceph2-hk-m-4p31kb-node6 ~]# ceph mon dump
epoch 3
fsid b9052a36-7c88-11ee-aee7-fa163e9f50e7
last_changed 2023-11-06T09:45:25.866973+0000
created 2023-11-06T09:42:13.026736+0000
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:10.0.97.224:3300/0,v1:10.0.97.224:6789/0] mon.ceph2-hk-m-4p31kb-node1-installer
1: [v2:10.0.97.14:3300/0,v1:10.0.97.14:6789/0] mon.ceph2-hk-m-4p31kb-node3
2: [v2:10.0.99.147:3300/0,v1:10.0.99.147:6789/0] mon.ceph2-hk-m-4p31kb-node2
dumped monmap epoch 3
[root@ceph2-hk-m-4p31kb-node6 ~]#


Version-Release number of selected component (if applicable):
---------
ceph version 18.2.0-113.el9cp

How reproducible:
----------------
Always


Actual results:
--------------
ceph fs snapshot mirror peer-list is displaying only 1 host instead of all mon hosts

Expected results:
----------------
ceph fs snapshot mirror peer-list must display details of all mon hosts of the secondary cluster.

--- Additional comment from Hemanth Kumar on 2023-11-07 05:30:55 UTC ---



--- Additional comment from Venky Shankar on 2023-11-07 05:41:44 UTC ---

Hi Hemanth,

(In reply to Hemanth Kumar from comment #0)
> Description of problem:
> -----------------------
> 
> The secondary cluster is configured with 3 mons, when a peer connection is
> established, the peer list/snapshot mirror status always displays only one
> mon host instead of all the mon details.
> 
> [root@ceph1-hk-m-4p31kb-node7 ~]# ceph fs snapshot mirror peer_list cephfs
> {"c4f15c31-8d2b-445e-8fa7-3137a1b638ca": {"client_name":
> "client.mirror_remote", "site_name": "remote_site", "fs_name": "cephfs",
> "mon_host": "[v2:10.0.97.224:3300,v1:10.0.97.224:6789]"}}
> [root@ceph1-hk-m-4p31kb-node7 ~]#
> 
> The secondary cluster is configured with 3 mon hosts : 
> [root@ceph2-hk-m-4p31kb-node6 ~]# ceph mon dump
> epoch 3
> fsid b9052a36-7c88-11ee-aee7-fa163e9f50e7
> last_changed 2023-11-06T09:45:25.866973+0000
> created 2023-11-06T09:42:13.026736+0000
> min_mon_release 18 (reef)
> election_strategy: 1
> 0: [v2:10.0.97.224:3300/0,v1:10.0.97.224:6789/0]
> mon.ceph2-hk-m-4p31kb-node1-installer
> 1: [v2:10.0.97.14:3300/0,v1:10.0.97.14:6789/0] mon.ceph2-hk-m-4p31kb-node3
> 2: [v2:10.0.99.147:3300/0,v1:10.0.99.147:6789/0] mon.ceph2-hk-m-4p31kb-node2
> dumped monmap epoch 3
> [root@ceph2-hk-m-4p31kb-node6 ~]#
> 
> 
> Version-Release number of selected component (if applicable):
> ---------
> ceph version 18.2.0-113.el9cp
> 
> How reproducible:
> ----------------
> Always
> 
> 
> Actual results:
> --------------
> ceph fs snapshot mirror peer-list is displaying only 1 host instead of all
> mon hosts

Could you share the cli command you used to add the peer?

--- Additional comment from Hemanth Kumar on 2023-11-07 06:44:34 UTC ---

(In reply to Venky Shankar from comment #2)
> Hi Hemanth,
> 
> (In reply to Hemanth Kumar from comment #0)
> > Description of problem:
> > -----------------------
> > 
> > The secondary cluster is configured with 3 mons, when a peer connection is
> > established, the peer list/snapshot mirror status always displays only one
> > mon host instead of all the mon details.
> > 
> > [root@ceph1-hk-m-4p31kb-node7 ~]# ceph fs snapshot mirror peer_list cephfs
> > {"c4f15c31-8d2b-445e-8fa7-3137a1b638ca": {"client_name":
> > "client.mirror_remote", "site_name": "remote_site", "fs_name": "cephfs",
> > "mon_host": "[v2:10.0.97.224:3300,v1:10.0.97.224:6789]"}}
> > [root@ceph1-hk-m-4p31kb-node7 ~]#
> > 
> > The secondary cluster is configured with 3 mon hosts : 
> > [root@ceph2-hk-m-4p31kb-node6 ~]# ceph mon dump
> > epoch 3
> > fsid b9052a36-7c88-11ee-aee7-fa163e9f50e7
> > last_changed 2023-11-06T09:45:25.866973+0000
> > created 2023-11-06T09:42:13.026736+0000
> > min_mon_release 18 (reef)
> > election_strategy: 1
> > 0: [v2:10.0.97.224:3300/0,v1:10.0.97.224:6789/0]
> > mon.ceph2-hk-m-4p31kb-node1-installer
> > 1: [v2:10.0.97.14:3300/0,v1:10.0.97.14:6789/0] mon.ceph2-hk-m-4p31kb-node3
> > 2: [v2:10.0.99.147:3300/0,v1:10.0.99.147:6789/0] mon.ceph2-hk-m-4p31kb-node2
> > dumped monmap epoch 3
> > [root@ceph2-hk-m-4p31kb-node6 ~]#
> > 
> > 
> > Version-Release number of selected component (if applicable):
> > ---------
> > ceph version 18.2.0-113.el9cp
> > 
> > How reproducible:
> > ----------------
> > Always
> > 
> > 
> > Actual results:
> > --------------
> > ceph fs snapshot mirror peer-list is displaying only 1 host instead of all
> > mon hosts
> 
> Could you share the cli command you used to add the peer?

" ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote remote_site "

--- Additional comment from Hemanth Kumar on 2023-11-07 06:45:35 UTC ---

And to import : 
ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImI5MDUyYTM2LTdjODgtMTFlZS1hZWU3LWZhMTYzZTlmNTBlNyIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJyZW1vdGVfc2l0ZSIsICJrZXkiOiAiQVFERnQwaGxGVlBvR1JBQURUTUN0Wk5VS0NPbTFvKy9MNlpMZWc9PSIsICJtb25faG9zdCI6ICJbdjI6MTAuMC45Ny4yMjQ6MzMwMCx2MToxMC4wLjk3LjIyNDo2Nzg5XSJ9

--- Additional comment from Venky Shankar on 2023-11-07 08:00:16 UTC ---

(In reply to Hemanth Kumar from comment #4)
> And to import : 
> ceph fs snapshot mirror peer_bootstrap import cephfs
> eyJmc2lkIjogImI5MDUyYTM2LTdjODgtMTFlZS1hZWU3LWZhMTYzZTlmNTBlNyIsICJmaWxlc3lzd
> GVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZS
> I6ICJyZW1vdGVfc2l0ZSIsICJrZXkiOiAiQVFERnQwaGxGVlBvR1JBQURUTUN0Wk5VS0NPbTFvKy9
> MNlpMZWc9PSIsICJtb25faG9zdCI6ICJbdjI6MTAuMC45Ny4yMjQ6MzMwMCx2MToxMC4wLjk3LjIy
> NDo2Nzg5XSJ9

That gets decoded to:

> b'{"fsid": "b9052a36-7c88-11ee-aee7-fa163e9f50e7", "filesystem": "cephfs", "user": "client.mirror_remote", "site_name": "remote_site", "key": "AQDFt0hlFVPoGRAADTMCtZNUKCOm1o+/L6ZLeg==", "mon_host": "[v2:10.0.97.224:3300,v1:10.0.97.224:6789]"}'

and the mon_host comes from what has been set. What is mon_host set to in the global section?

--- Additional comment from Amarnath on 2023-11-09 06:23:45 UTC ---

Hi Venky,


Source cluster : 
[root@ceph1-amk-fs-tc-nmj42e-node7 ~]# ceph fs snapshot mirror peer_list cephfs
{"88d9f42f-a22d-4e4f-bb01-e2ae11c248a0": {"client_name": "client.mirror_remote_caps", "site_name": "remote_site_caps", "fs_name": "cephfs", "mon_host": "[v2:10.0.209.6:3300,v1:10.0.209.6:6789]"}}


Target cluster:
[root@ceph2-amk-fs-tc-nmj42e-node6 ~]# ceph-conf --show-config mon | grep mon_host
mon_host = [v2:10.0.209.6:3300/0,v1:10.0.209.6:6789/0] [v2:10.0.210.190:3300/0,v1:10.0.210.190:6789/0] [v2:10.0.206.59:3300/0,v1:10.0.206.59:6789/0]

In global conf also it has 3 mon ips listed. Is there any other place I need to check


Regards,
Amarnath

--- Additional comment from Venky Shankar on 2023-11-13 05:06:40 UTC ---

Hi Amarnath,

(In reply to Amarnath from comment #6)
> Hi Venky,
> 
> 
> Source cluster : 
> [root@ceph1-amk-fs-tc-nmj42e-node7 ~]# ceph fs snapshot mirror peer_list
> cephfs
> {"88d9f42f-a22d-4e4f-bb01-e2ae11c248a0": {"client_name":
> "client.mirror_remote_caps", "site_name": "remote_site_caps", "fs_name":
> "cephfs", "mon_host": "[v2:10.0.209.6:3300,v1:10.0.209.6:6789]"}}

This addr is what rados hands out when the mirroring module requires the mon_addr - https://github.com/ceph/ceph/blob/main/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L668

BTW, this shouldn't be an issue since (any) ceph client requires a single mon addr to bootstrap, so there should not be any functionality issue.

--- Additional comment from Hemanth Kumar on 2023-11-15 05:01:45 UTC ---

(In reply to Venky Shankar from comment #7)
> Hi Amarnath,
> 
> (In reply to Amarnath from comment #6)
> > Hi Venky,
> > 
> > 
> > Source cluster : 
> > [root@ceph1-amk-fs-tc-nmj42e-node7 ~]# ceph fs snapshot mirror peer_list
> > cephfs
> > {"88d9f42f-a22d-4e4f-bb01-e2ae11c248a0": {"client_name":
> > "client.mirror_remote_caps", "site_name": "remote_site_caps", "fs_name":
> > "cephfs", "mon_host": "[v2:10.0.209.6:3300,v1:10.0.209.6:6789]"}}
> 
> This addr is what rados hands out when the mirroring module requires the
> mon_addr -
> https://github.com/ceph/ceph/blob/main/src/pybind/mgr/mirroring/fs/
> snapshot_mirror.py#L668
> 
> BTW, this shouldn't be an issue since (any) ceph client requires a single
> mon addr to bootstrap, so there should not be any functionality issue.

Hi Venky,

What would be the behavior when the only Mon IP(displayed in the mirror status) is down on the target cluster?

--- Additional comment from Venky Shankar on 2023-11-15 09:38:34 UTC ---

(In reply to Hemanth Kumar from comment #8)
> (In reply to Venky Shankar from comment #7)
> > Hi Amarnath,
> > 
> > (In reply to Amarnath from comment #6)
> > > Hi Venky,
> > > 
> > > 
> > > Source cluster : 
> > > [root@ceph1-amk-fs-tc-nmj42e-node7 ~]# ceph fs snapshot mirror peer_list
> > > cephfs
> > > {"88d9f42f-a22d-4e4f-bb01-e2ae11c248a0": {"client_name":
> > > "client.mirror_remote_caps", "site_name": "remote_site_caps", "fs_name":
> > > "cephfs", "mon_host": "[v2:10.0.209.6:3300,v1:10.0.209.6:6789]"}}
> > 
> > This addr is what rados hands out when the mirroring module requires the
> > mon_addr -
> > https://github.com/ceph/ceph/blob/main/src/pybind/mgr/mirroring/fs/
> > snapshot_mirror.py#L668
> > 
> > BTW, this shouldn't be an issue since (any) ceph client requires a single
> > mon addr to bootstrap, so there should not be any functionality issue.
> 
> Hi Venky,
> 
> What would be the behavior when the only Mon IP(displayed in the mirror
> status) is down on the target cluster?

Hmm.. the client would fail to connect to the cluster if that mon isn't available. It isn't too much worrisome, but I accept now that its an irritation.

I'd need to dig more into what's required to have the whole set of mon addrs to fallback. BTW, you could also check if rbd-mirror too has this same construct.

Also, I'm lowering the prio of this BZ since its not too critical.

--- Additional comment from Hemanth Kumar on 2023-11-20 08:39:32 UTC ---

I'll check the status of rbd-mirror and update the BZ, clearing the needed info for now.

--- Additional comment from Hemanth Kumar on 2023-11-20 09:14:27 UTC ---

In RBD - only peer_uuid, name and clients are displayed. Mon Details are not part of any status command.

[root@tala007 ~]# rbd mirror pool info odf_test --debug-rbd 0
Mode: image
Site Name: site-b

Peer Sites: 

UUID: 1e138dbb-345a-4438-b0f3-53a9f7c741a5
Name: site-a
Mirror UUID: 972a80c5-fa23-47e3-beee-6f651269f3ec
Direction: rx-tx
Client: client.rbd-mirror-peer

--- Additional comment from Venky Shankar on 2023-11-23 04:40:24 UTC ---

(In reply to Hemanth Kumar from comment #11)
> In RBD - only peer_uuid, name and clients are displayed. Mon Details are not
> part of any status command.
> 
> [root@tala007 ~]# rbd mirror pool info odf_test --debug-rbd 0
> Mode: image
> Site Name: site-b
> 
> Peer Sites: 
> 
> UUID: 1e138dbb-345a-4438-b0f3-53a9f7c741a5
> Name: site-a
> Mirror UUID: 972a80c5-fa23-47e3-beee-6f651269f3ec
> Direction: rx-tx
> Client: client.rbd-mirror-peer

IMO it makes sense to keep it that way for cephfs-mirror too. The additional bit of information isn't particularly useful to the user.

--- Additional comment from Venky Shankar on 2023-11-23 05:02:17 UTC ---

Jos, please create an upstream tracker for the same.

--- Additional comment from errata-xmlrpc on 2024-04-04 20:17:05 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:129008-01
https://errata.engineering.redhat.com/advisory/129008

--- Additional comment from errata-xmlrpc on 2024-04-04 20:17:13 UTC ---

This bug has been added to advisory RHBA-2024:129008 by Thomas Serlin (tserlin)

--- Additional comment from Hemanth Kumar on 2024-04-05 11:30:34 UTC ---

Verified on Build: ceph version 18.2.0-180.el9cp.

The Target MON Host details are removed from the Peer List and mirror daemon status.

-----
[root@ceph1-hk-m-goihr7-node1-installer ~]# ceph fs snapshot mirror peer_list cephfs
{"f942439a-743b-405c-ad56-c3f92193bec4": {"client_name": "client.mirror_remote", "site_name": "remote_site", "fs_name": "cephfs"}}


[root@ceph1-hk-m-goihr7-node1-installer ~]# ceph fs snapshot mirror daemon status
[{"daemon_id": 14622, "filesystems": [{"filesystem_id": 1, "name": "cephfs", "directory_count": 2, "peers": [{"uuid": "f942439a-743b-405c-ad56-c3f92193bec4", "remote": {"client_name": "client.mirror_remote", "cluster_name": "remote_site", "fs_name": "cephfs"}, "stats": {"failure_count": 0, "recovery_count": 0}}]}]}]
-----

--- Additional comment from Amarnath on 2024-04-16 05:48:56 UTC ---

Hi Venky,

Do we require Doc_text for this BZ

Regards,
Amarnath

--- Additional comment from Rivka Pollack on 2024-04-18 13:09:06 UTC ---

Jos,

Please provide full bug fix release notes text information.
we need to know what this comes to fix (what the problem was previously that we needed to fix it, if possible).
thanks

--- Additional comment from Hemanth Kumar on 2024-04-25 12:20:09 UTC ---

Jos, Venky - 
Can I create a backport BZ to include this fix in 6.x, 7.1 also?

--- Additional comment from Jos Collin on 2024-04-25 12:29:39 UTC ---

(In reply to Hemanth Kumar from comment #19)
> Jos, Venky - 
> Can I create a backport BZ to include this fix in 6.x, 7.1 also?

Yes, please go ahead.