Description of problem: On a RGW multiste setup ,tried to configure a new realm,zone with multiple daemons which failed to apply. Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1929155 Version-Release number of selected component (if applicable): ceph version 16.1.0-486.el8cp (f9701a56b7b8182352532afba8db2bf394c8585a) pacific (rc) Steps to Reproduce: 1. Configure 2 clusters ,and create a realm,zone on primary cluster . 2. Pull the realm and configure a working multisite relationship. 3. Try to configure a second realm,zone on secondary cluster, it fails. secondary cluster : /]# ceph -s cluster: id: 18d6b412-7be1-11eb-88c5-fa163ec46c5e health: HEALTH_OK services: mon: 1 daemons, quorum ceph-tj-site2-1614686272665-node1-mon-mgr-installer-node-export (age 7h) mgr: ceph-tj-site2-1614686272665-node1-mon-mgr-installer-node-export.nmbgef(active, since 7h) osd: 12 osds: 12 up (since 7h), 12 in (since 7h) rgw: 2 daemons active (india1.north.ceph-tj-site2-1614686272665-node2-mon-node-exporter-alertmanage.jsgdhb, india1.north.ceph-tj-site2-1614686272665-node6-mon-rgw-node-exporter.wuyiiv) /]# cat rgw1.yaml service_type: rgw service_id: rgw_bz placement: hosts: - ceph-tj-site2-1614686272665-node4-osd-node-exporter - ceph-tj-site2-1614686272665-node5-osd-node-exporter - ceph-tj-site2-1614686272665-node3-osd-node-exporter rgw_realm: test1 rgw_zone: tzone rgw_frontend_port: 8080 /]# ceph orch apply -i rgw1.yaml Scheduled rgw.rgw_bz update... --- service_type: rgw service_id: rgw_bz service_name: rgw.rgw_bz placement: hosts: - ceph-tj-site2-1614686272665-node4-osd-node-exporter - ceph-tj-site2-1614686272665-node5-osd-node-exporter - ceph-tj-site2-1614686272665-node3-osd-node-exporter spec: rgw_frontend_port: 8080 rgw_realm: rgw_bz rgw_zone: tzone status: running: 0 size: 3 events: - 2021-03-03T13:10:51.703577Z service:rgw.rgw_bz [INFO] "service was created" - '2021-03-03T13:10:55.821707Z service:rgw.rgw_bz [ERROR] "Failed to apply: failed to update RGW period: b"2021-03-03T13:10:55.596+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.597+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.597+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.605+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.606+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.606+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.608+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.609+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.609+0000 7f34118c1200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.rgw.rgw_bz.ceph-tj-site2-1614686272665-node3-osd-node-exporter.uigume.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-03-03T13:10:55.807+0000 7f34118c1200 0 failed reading obj info from .rgw.root:zone_info.afa33ecf-e2cc-41a6-9289-d3ec2f5e4de2: (2) No such file or directory\n2021-03-03T13:10:55.807+0000 7f34118c1200 0 WARNING: could not read zone params for zone id=afa33ecf-e2cc-41a6-9289-d3ec2f5e4de2 name=south\n2021-03-03T13:10:55.813+0000 7f34118c1200 -1 Cannot find zone id=a75f21bf-a15b-43dc-aab3-66e68b31111e (name=default)\n2021-03-03T13:10:55.813+0000 7f34118c1200 0 ERROR: failed to start notify service ((22) Invalid argument\n2021-03-03T13:10:55.813+0000 7f34118c1200 0 ERROR: failed to init services (ret=(22) Invalid argument)\ncouldn''t init storage provider\n""'
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
upstream fix PR: https://github.com/ceph/ceph/pull/39877/files * The ``ceph orch apply rgw`` syntax and behavior have changed. RGW services can now be arbitrarily named (it is no longer forced to be `realm.zone`). The ``--rgw-realm=...`` and ``--rgw-zone=...`` arguments are now optional, which means that if they are omitted, a vanilla single-cluster RGW will be deployed. When the realm and zone are provided, the user is now responsible for setting up the multisite configuration beforehand--cephadm no longer attempts to create missing realms or zones. multisite example setup commands: 2 clusters with osds needed Cluster 1: radosgw-admin realm create --default --rgw-realm=gold radosgw-admin zonegroup create --rgw-zonegroup=us --master --default --endpoints=http://<ip vm-00>:80 radosgw-admin zone create --rgw-zone=us-east --master --rgw-zonegroup=us --endpoints=http://<ip vm-00>:80 --access-key=1234567 --secret=098765 --default radosgw-admin period update --rgw-realm=gold --commit radosgw-admin user create --uid=repuser --display-name="Replication_user" --access-key=1234567 --secret=098765 --system ceph orch apply rgw rgwserviceid1 gold us-east --placement=vm-00 Cluster 2: radosgw-admin realm pull --rgw-realm=gold --url=http://<ip vm-00>:80 --access-key=1234567 --secret=098765 --default radosgw-admin period pull --url=http://<ip vm-00>:80 --access-key=1234567 --secret=098765 radosgw-admin zone create --rgw-zone=us-west --rgw-zonegroup=us --endpoints=http://<ip vm-03>:80 --access-key=1234567 --secret=098765 radosgw-admin period update --rgw-realm=gold --commit ceph orch apply rgw rgwserviceid1 gold us-west --placement=vm-03
Sage backported PR 39877 to pacific in https://github.com/ceph/ceph/pull/40135. This will be in the next weekly rebase I build downstream (March 22nd).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3294