Bug 1971996

Description of problem: Create single primary and multiple secondary clusters 


Version-Release number of selected component (if applicable):
ceph version 16.2.0-72.el8cp


Steps to Reproduce:
1. Set up three clusters, one primary and two secondary
2. Create bootstrap token on primary and copy to secondary
on site-a(primary) - 
3. import the bootstrap on the secondary1
on site b(secondary1) - rbd mirror pool peer bootstrap import --direction rx-only --site-name site-b data bootstrap.site-a
successful
4. import the bootstrap on the secondary2
on site c(secondary2) - rbd mirror pool peer bootstrap import --direction rx-only --site-name site-c data bootstrap.site-a
FAILS



Actual results:
[ceph: root@ceph-mirror-1623713519646-node1-mon-mgr-installer-node-exporter ceph]# rbd mirror pool peer bootstrap import --site-name site-c data bootstrap.site-a 
rbd: error opening bootstrap.site-a
[ceph: root@ceph-mirror-1623713519646-node1-mon-mgr-installer-node-exporter ceph]# cd
[ceph: root@ceph-mirror-1623713519646-node1-mon-mgr-installer-node-exporter ~]# rbd mirror pool peer bootstrap import --site-name site-c data bootstrap.site-a 
2021-06-15T00:24:54.404+0000 7f11b59a52c0 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-06-15T00:24:54.405+0000 7f11b59a52c0 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-06-15T00:24:54.405+0000 7f11b59a52c0 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-06-15T00:24:54.430+0000 7f11b59a52c0 -1 librbd::api::Mirror: create_bootstrap_peer: multiple peers are not currently supported
rbd: failed to import peer bootstrap token


Expected results: bootstrap should be successful on secondary site-c


Additional Details:
1. Deploy clusters using cephadm(We need min two clusters for rbd-mirror)
2. Create pools and configure them for mirroring(on both clusters with same name)
3. Go to ceph shell and perform following steps:
 ceph osd pool create <pool-name>
 rbd pool init <pool-name>
 rbd mirror pool enable <pool-name> pool

4. On primary cluster(ex: site-a), do the following steps:
 ceph orch apply rbd-mirror --placement=nodename(This step is needed for  two way mirroring)
Create image: rbd create <image-name>  --size 2G --pool <pool-name> --image-feature exclusive-lock,journaling
 Bootstrap the storage peers: rbd mirror pool peer bootstrap create --site-name site-a <pool-name>
iii. Copy the output of the above command (you can also save this output into some file to import in later steps)

5. On secondary cluster(ex: site-b), do the following steps:
 ceph orch apply rbd-mirror --placement=nodename(where you want to configure mirror )
Ii. Create a file and paste output of the bootstrap command which you saved before.(make sure to create this file under root and inside the ceph shell)
iii. Import bootstrap file: rbd mirror pool peer bootstrap import --direction rx-only --site-name site-b <pool-name> filename

6. repeat step 5 on secondary cluster site-c