Bug 2278873

I see it looking in /var/lib/ceph/ instead of /etc/ceph.

This can happen if you do not pass this parameter

  CephConfigPath: "/etc/ceph"

As described in 5.1 step 9

  https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html-single/framework_for_upgrades_16.2_to_17.1/index

Can you try again with that parameter passed?

Also, on the controller node can you see the missing key on the controller node in /etc/ceph?

That is configured in:

(undercloud) [stack@openstack ~]$ cat osp17.1_upgrade/ceph-params.yaml
±±±
parameter_defaults:

  CephSpecFqdn: false

  CephConfigPath: "/etc/ceph"

  CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms"

  DeployedCeph: true
±±±

(In reply to Kenny Tordeurs from comment #2)
> That is configured in:
> 
> (undercloud) [stack@openstack ~]$ cat osp17.1_upgrade/ceph-params.yaml
> ±±±
> parameter_defaults:
> 
>   CephSpecFqdn: false
> 
>   CephConfigPath: "/etc/ceph"
> 
>   CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms"
> 
>   DeployedCeph: true
> ±±±

Please share the `opentack overcloud ` ... `-e osp17.1_upgrade/ceph-params.yaml` you are using.


Also, as I asked previously, on the controller node can you see the missing key on the controller node in /etc/ceph?

> Please share the `opentack overcloud ` ... `-e osp17.1_upgrade/ceph-params.yaml` you are using.

I ask for the above because, if you have something like

-e osp17.1_upgrade/ceph-params.yaml -e /usr/share/...

Then the default in /usr/share will override the override in ceph-params.yaml

If we confirm the above and see that the keys are actually in /etc/ceph on the controller, then changing the argument order should be sufficient to address the issue.

These are the contents of the /etc/ceph/ directory but as you can see the naming of the keyrings do not match

$ ll /etc/ceph/
~~~
total 44
-rwxr-xr-x. 1 ceph ceph   0 May  2 10:35 cephadm.priv
-rwxr-xr-x. 1 ceph ceph   0 May  2 10:35 cephadm.pub
-rw-------. 1 ceph ceph 151 May  2 10:35 ceph.client.admin.keyring
-rw-------. 1 ceph ceph 131 May  2 10:35 ceph.client.crash.keyring
-rw-------. 1 ceph ceph 153 May  2 10:35 ceph.client.manila.keyring
-rw-r--r--. 1 ceph ceph 231 May  2 10:35 ceph.client.openstack.keyring
-rw-r--r--. 1 ceph ceph 138 May  2 10:35 ceph.client.radosgw.keyring
-rw-
~~~

~~~
7f565a11f200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.openstack/keyring: (2) No such file or directory
7f565a11f200 -1 AuthRegistry(0x5616d41c4ee0) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.openstack/keyring, disabling cephx
7f565a11f200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.openstack/keyring: (2) No such file or directory
7f565a11f200 -1 AuthRegistry(0x7ffe02ad13a0) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.openstack/keyring, disabling cephx
7f5647fff640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
7f564ca83640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
7f565a11f200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
~~~

Sharing the full command:

openstack overcloud upgrade prepare --yes \
--timeout 460 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack $CLUSTER \
-r ~/templates/roles_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-mds.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml \
-e ~/common_templates/aide.yaml \
-e ~/common_templates/ceph-config.yaml \
-e ~/common_templates/extraconfig-environment.yaml \
-e ~/templates/extraconfig-environment.yaml \
-e ~/common_templates/inject-trust-anchor-hiera.yaml \
-e ~/common_templates/keystone_domain_specific_ldap_backend.yaml \
-e ~/common_templates/network-environment-overrides.yaml \
-e ~/templates/network-environment-overrides.yaml \
-e ~/common_templates/nova-environment-overrides.yaml \
-e ~/templates/nova-environment-overrides.yaml \
-e ~/common_templates/octavia-environment.yaml \
-e ~/common_templates/pci_passthru_compute.yaml \
-e ~/common_templates/pci_passthru_controller.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
-e ~/common_templates/configure-barbican.yaml \
-e ~/common_templates/security.yaml \
-e ~/templates/cloudname.yaml \
-e ~/templates/enable-tls.yaml \
-e ~/templates/fencing.yaml \
-e ~/templates/rhsm.yaml \
-e ~/templates/storage-config.yaml \
-e ~/templates/upgrades-environment.yaml \
-e ~/osp17.1_upgrade/ceph-params.yaml \
-e ~/overcloud-deploy/$CLUSTER/$CLUSTER-network-environment.yaml \
-e ~/overcloud_adopt/baremetal-deployment.yaml \
-e ~/overcloud_adopt/generated-networks-deployed.yaml \
-e ~/overcloud_adopt/generated-vip-deployed.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \
-e ~/osp17.1_upgrade/system_upgrade.yaml \
-e ~/templates/containers-prepare-parameter.yaml \
--debug
2>&1 | tee $LOGFILE

*** This bug has been marked as a duplicate of bug 2278832 ***