Bug 1910280

Created attachment 1741508 [details]
rnd-mirror-site-b daemon logs

From the logs:

Dec 23 04:30:12 ceph-rbd2-2ymane-1608709673158-node2-client rbd-mirror[22318]: 2020-12-23 04:30:12.265 7f0d0880d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
Dec 23 04:30:12 ceph-rbd2-2ymane-1608709673158-node2-client rbd-mirror[22318]: 2020-12-23 04:30:12.265 7f0d17ddc1c0 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication

Your container is not deployed with any CephX keys but the MONs are configured to required CephX.

(In reply to Jason Dillaman from comment #2)
> Your container is not deployed with any CephX keys but the MONs are
> configured to required CephX.

Actually, this looks like a documentation issue. Section 5.6.5 incorrectly states to start the daemon using the bootstrap token user, which is not correct. Each daemon should have its own client id, the key needs to be available on the daemon's node, and the service should be enabled using that client id.

Hi Jason,

I have started daemons as you mentioned in comment#3 but it's not working as expected. I observed one thing that, Section 5.6.4 when we do bootstrap, it should create two ceph users but i can see only one user in "ceph auth ls". So i am suspecting that might be the issue. 

As per doc:
-----------
[root@rbd-client-site-a ~]# rbd mirror pool peer bootstrap create --site-name site-a data > /root/bootstrap_token_site-a
NOTE
This example bootstrap command creates the client.rbd-mirror.site-a and the client.rbd-mirror-peer Ceph users.


Actual output from cluster:
------------------------------
ceph auth ls
installed auth entries:
osd.0
	key: AQCtZelfkRY7DxAAHmq1To33TPslySXB1nfnYg==
	caps: [mgr] allow profile osd
	caps: [mon] allow profile osd
	caps: [osd] allow *
osd.1
	key: AQCtZelfTsVsEhAA7fnwsfY/qhjy0HKDslWBBQ==
	caps: [mgr] allow profile osd
	caps: [mon] allow profile osd
	caps: [osd] allow *
.
.
.

client.bootstrap-rbd-mirror
	key: AQBpZelfR0P4IhAAnveJzh3J9f7be+yvnJRMoA==
	caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgw
	key: AQBpZelflVz4IhAAN2YoeALXfPlezqVr9KA45A==
	caps: [mon] allow profile bootstrap-rgw
client.crash
	key: AQDwZelfAAAAABAA3sTKfpj+6GVyfaTZxr04nA==
	caps: [mgr] allow profile crash
	caps: [mon] allow profile crash
client.rbd-mirror-peer
	key: AQCZdOlfiWYfCBAAi7sqfjYznx4QGydVXOSo2A==
	caps: [mon] profile rbd-mirror-peer
	caps: [osd] profile rbd
mgr.ceph-rbd2-2ymane-1609128602921-node1-monmgrinstaller
	key: AQCEZelfAAAAABAADm9wXXtobom2oxgl7+7kOQ==
	caps: [mds] allow *
	caps: [mon] allow profile mgr
	caps: [osd] allow *


Another thing is, when we deploy rbd-mirror using ansible first(which is successful) and deploy through cli, it's working, but this might be because of ansible deployment, daemon is active.

[root@site-a ubuntu]# ceph osd pool create mirror_pool 10
pool 'mirror_pool' created
[root@site-a ubuntu]# ceph osd pool application enable mirror_pool rbd
enabled application 'rbd' on pool 'mirror_pool'
[root@site-a ubuntu]# rbd pool init -p mirror_pool
[root@site-a ubuntu]# dnf install rbd-mirror
Updating Subscription Management repositories.
Last metadata expiration check: 12:43:29 ago on Sun 27 Dec 2020 09:53:43 PM UTC.
Package rbd-mirror-2:14.2.11-95.el8cp.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@site-a ubuntu]# rbd create image1 --size 1024 --pool mirror_pool --image-feature exclusive-lock,journaling
[root@site-a ubuntu]# rbd mirror pool enable mirror_pool  pool
[root@site-a ubuntu]# rbd mirror pool info  mirror_pool
Mode: pool
Site Name: site-a

Peer Sites: none
[root@site-a ~]# rbd mirror pool peer bootstrap create --site-name site-a mirror_pool > /root/bootstrap_token_site-a
[root@site-a ~]# scp bootstrap_token_site-a root@site-b:/root
root@site-b's password: 
bootstrap_token_site-a                                                                                                                     100%  293   499.8KB/s   00:00    
[root@site-a ~]# systemctl enable ceph-rbd-mirror.target
[root@site-a ~]# systemctl enable ceph-rbd-mirror
[root@site-a ~]# systemctl start ceph-rbd-mirror
[root@site-a ~]# rbd mirror image status mirror_pool/image1
image1:
  global_id:   87d9c516-a922-4f34-96b1-16aa1418856a
  state:       up+stopped
  description: local image is primary
  service:     site-a on site-a
  last_update: 2020-12-28 10:41:33
  peer_sites:
    name: site-b
    state: up+replaying
    description: replaying, {"bytes_per_second":17.5,"entries_behind_primary":0,"entries_per_second":0.2,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
    last_update: 2020-12-28 10:41:38

(In reply to Gopi from comment #4)

You need to make the BZ as needinfo if you are expecting a reply, otherwise I likely won't see it.

> I have started daemons as you mentioned in comment#3 but it's not working as
> expected. I observed one thing that, Section 5.6.4 when we do bootstrap, it
> should create two ceph users but i can see only one user in "ceph auth ls".
> So i am suspecting that might be the issue. 

No, bootstrap only creates "client.rbd-mirror-peer". Bootstrap does not create the per-daemon user. I am not sure where "client.rbd-mirror.site-a" came from since it's not in the bootstrap code. Perhaps you had an unclean environment.

$ ceph auth ls 2>&1 | grep client.rbd
$ rbd mirror pool peer bootstrap create
eyJmc2lkIjoiYmMwZDY2N2EtMGMzOS00MjZlLTllNjYtM2RiNDlkYTVmY2JlIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFEZC93Wmd2aC81SlJBQUZOK09iM0RlSE81NmU4RS9iSjluTFE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjQwODc5LHYxOjE5Mi4xNjguMS4zOjQwODgwXSJ9
$ ceph auth ls 2>&1 | grep client.rbd
client.rbd-mirror-peer
$ $ echo 'eyJmc2lkIjoiYmMwZDY2N2EtMGMzOS00MjZlLTllNjYtM2RiNDlkYTVmY2JlIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFEZC93Wmd2aC81SlJBQUZOK09iM0RlSE81NmU4RS9iSjluTFE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjQwODc5LHYxOjE5Mi4xNjguMS4zOjQwODgwXSJ9' | base64 -d | json_reformat 
{
    "fsid": "bc0d667a-0c39-426e-9e66-3db49da5fcbe",
    "client_id": "rbd-mirror-peer",
    "key": "AQDd/wZgvh/5JRAAFN+Ob3DeHO56e8E/bJ9nLQ==",
    "mon_host": "[v2:192.168.1.3:40879,v1:192.168.1.3:40880]"
}


> Another thing is, when we deploy rbd-mirror using ansible first(which is
> successful) and deploy through cli, it's working, but this might be because
> of ansible deployment, daemon is active.

Ansible creates the per-daemon users whereas the documentation for the manual deployment is missing that step.

Actually we need to add few steps before bootstrap command and give proper naming convention to keyring file to work rbd-mirror using cli. I have made the steps to work rbd-mirror and pasted the same below:

on primary:
-----------
Create user with proper name:
[root@ceph-rbd1 ~]# ceph auth get-or-create client.rbd-mirror.abc mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.abc.keyring

copy keyring to secondary cluster:
[root@ceph-rbd1 ~]# scp /etc/ceph/ceph.client.rbd-mirror.abc.keyring root@ceph-rbd2:/etc/ceph/

bootstrap:
[root@ceph-rbd1 ~]# rbd mirror pool peer bootstrap create --site-name rbd-mirror.abc rbd_miiror_pool > /root/bootstrap_token_rbd-mirror.abc

copy bootstrap to secondary cluster:
[root@ceph-rbd1 ~]# /root/bootstrap_token_rbd-mirror.abc root@ceph-rbd2:/root/


On secondary:
-------------
Create user:
[root@ceph-rbd2 ~]# ceph auth get-or-create client.rbd-mirror.xyz mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.xyz.keyring

copy keyring to primary cluster:
[root@ceph-rbd2 ~]# scp /etc/ceph/ceph.client.rbd-mirror.xyz.keyring root@ceph-rbd1:/etc/ceph/

import boostrap:
[root@ceph-rbd2 ~]# rbd mirror pool peer bootstrap import --site-name rbd-mirror.xyz --direction rx-tx rbd_miiror_pool /root/bootstrap_token_rbd-mirror.abc

@rmandyam please make changes to document and let me know if you get any confusion with steps or need any info from me.

@rmandyam 

Changing the target release to 4.2z2 as these steps (comment#7) are missed in 4.x doc and 5.0 follows cephadm procedure to configure rbd-mirror.

@Gopi can you please check the steps and if any step is missing can we include it here. Thanks!

@Ranjini can we have this pushed to the docs earliest. Thanks!

5.5. CONFIGURING ONE-WAY MIRRORING USING THE COMMAND-LINE INTERFACE
  
observed: ceph auth get-or-create client.rbd-mirrror.site-a mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirrror.site-a.keyring
expected: ceph auth get-or-create client.rbd-mirror.site-a mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-a.keyring

observed: rbd mirror pool peer bootstrap create --site-name site-a data > /root/bootstrap_token_site-a
expected: rbd mirror pool peer bootstrap create --site-name rbd-mirror.site-a data > /root/bootstrap_token_rbd-mirror.site-a

observed: ceph auth get-or-create client.rbd-mirrror.site-b mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirrror.site-b.keyring
expected: ceph auth get-or-create client.rbd-mirror.site-b mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-b.keyring

observed: rbd mirror pool peer bootstrap import --site-name site-b --direction rx-only data /root/bootstrap_token_site-a
expected: rbd mirror pool peer bootstrap import --site-name rbd-mirror.site-b --direction rx-only data /root/bootstrap_token_rbd-mirror.site-a

Please make the required changes corresponding to this.

5.6. CONFIGURING TWO-WAY MIRRORING USING THE COMMAND-LINE INTERFACE

observed: [root@rbd-client-site-b ~]# rbd mirror pool peer bootstrap import --site-name rbd-mirror.site-b --direction rx-only data /root/bootstrap_token_rbd-mirror.site-a

expected: [root@rbd-client-site-b ~]# rbd mirror pool peer bootstrap import --site-name rbd-mirror.site-b --direction rx-tx data /root/bootstrap_token_rbd-mirror.site-a



observed:
[root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror

expected:
[root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror


observed:
[root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror

Expected
[root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror

We need to enable both the users a and b on both site-a and site-b

on primary cluster site-a
[root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror.target
[root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror
[root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror
on seconadry clsuter site-b
[root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror.target
[root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror
[root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror
[root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror

Looks good. Moving to Verified.