openstack-glance: OSP12 OC deployment with external ceph: unable to create image (status: queued) Environment: instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch openstack-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch puppet-glance-11.2.0-0.20170718235959.302415b.el7ost.noarch python-glanceclient-2.7.0-0.20170622095732.d67b33b.el7ost.noarch python-glance-store-0.20.1-0.20170719150719.fafc8fd.el7ost.noarch python-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch Steps to reproduce: 1. Deploy overcloud with: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/external-ceph.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml \ --log-file overcloud_deployment_0.log 2. Try to create glance image Result: Image doesn't get created on the external ceph and stays with status "queued": (overcloud) [stack@undercloud-0 ~]$ openstack image list +--------------------------------------+--------+--------+ | ID | Name | Status | +--------------------------------------+--------+--------+ | 5646420d-efcb-478f-968a-375fea079290 | cirros | queued | +--------------------------------------+--------+--------+ Expected result: The image should be created on external ceph and become active. More details: (overcloud) [stack@undercloud-0 ~]$ cat /home/stack/external-ceph.yaml # ******************************************************************* # This file was created automatically by the sample environment # generator. Developers should use `tox -e genconfig` to update it. # Users are recommended to make changes to a copy of the file instead # of the original, if any customizations are needed. # ******************************************************************* # title: Deploy Using an External Ceph Cluster # description: | # A Heat environment file which can be used to enable the # use of an externally managed Ceph cluster. parameter_defaults: # The Ceph admin client key. Can be created with ceph-authtool --gen-print-key. # Type: string CephAdminKey: '' # The Ceph client key. Can be created with ceph-authtool --gen-print-key. Currently only used for external Ceph deployments to create the openstack user keyring. # Mandatory. This parameter must be set by the user. # Type: string CephClientKey: '<KEY>' # # Type: string CephClientUserName: openstack # The Ceph cluster FSID. Must be a UUID. # Mandatory. This parameter must be set by the user. # Type: string CephClusterFSID: '<FSID>' # List of externally managed Ceph Mon Host IPs. Only used for external Ceph deployments. # Type: string CephExternalMonHost: '<IPs>' # Whether to enable or not the Iscsi backend for Cinder # Type: boolean CinderEnableIscsiBackend: False # Whether to enable or not the Rbd backend for Cinder # Type: boolean CinderEnableRbdBackend: True # # Type: string CinderRbdPoolName: volumes # The short name of the Glance backend to use. Should be one of swift, rbd, or file # Type: string GlanceBackend: rbd # # Type: string GlanceRbdPoolName: images # The short name of the Gnocchi backend to use. Should be one of swift, rbd, or file # Type: string GnocchiBackend: rbd # # Type: string GnocchiRbdPoolName: metrics # Whether to enable or not the Rbd backend for Nova # Type: boolean NovaEnableRbdBackend: True # # Type: string NovaRbdPoolName: vms # The default features enabled when creating a block device image. Only applies to format 2 images. Set to '1' for Jewel clients using older Ceph servers. # Type: string RbdDefaultFeatures: '1' resource_registry: OS::TripleO::Services::CephClient: OS::Heat::None OS::TripleO::Services::CephExternal: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-external.yaml OS::TripleO::Services::CephMon: OS::Heat::None OS::TripleO::Services::CephOSD: OS::Heat::None ######################################################################## Checking logs on all controllers, one controllers has these errors; [root@overcloud-controller-2 ~]# grep -i error /var/log/containers/glance/api.log 2017-07-31 16:19:15.602 22 ERROR glance.api.v2.image_data [req-42f3deb0-8250-4911-8a77-61006543791f 5862a0cf8d014744a828425d5991a799 d7cb49479ee84e219feed7ca7514985c - default default] Failed to upload image data due to internal error: Error: error calling conf_read_file: errno EINVAL 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi [req-42f3deb0-8250-4911-8a77-61006543791f 5862a0cf8d014744a828425d5991a799 d7cb49479ee84e219feed7ca7514985c - default default] Caught error: error calling conf_read_file: errno EINVAL: Error: error calling conf_read_file: errno EINVAL 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi Traceback (most recent call last): 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1155, in __call__ 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi request, **action_args) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1194, in dispatch 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi return method(*args, **kwargs) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 269, in upload 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self._restore(image_repo, image) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.force_reraise() 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 134, in upload 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi image.set_data(data, size) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.base.set_data(data, size) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 480, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.force_reraise() 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 427, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.repo.set_data(data, size) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 192, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.image.set_data(data, size=size) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/location.py", line 439, in set_data 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi verifier=verifier) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 453, in add_to_backend 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi verifier) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 426, in store_add_to_backend 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi verifier=verifier) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 454, in add 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi rados_id=self.user) as conn: 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi return self.gen.next() 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 268, in get_connection 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi client = rados.Rados(conffile=conffile, rados_id=rados_id) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/rados.py", line 253, in __init__ 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi self.conf_read_file(conffile) 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/rados.py", line 302, in conf_read_file 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi raise make_ex(ret, "error calling conf_read_file") 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi Error: error calling conf_read_file: errno EINVAL 2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi
gfidente or johfulto might help here.
Ceph authentication is not set up [1]. Does your undercloud include this change? https://review.openstack.org/#/c/482500/ [1] as per the sosreport from controller-0.redhat.local-20170731174937 2017-07-31 17:50:12.481195 7f29891c9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2017-07-31 17:50:12.481204 7f29891c9700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2017-07-31 17:50:12.481205 7f29891c9700 0 librados: client.admin initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound [jfulton@skagra ceph]$ cat ceph_health_detail 2017-07-31 17:50:11.902068 7f05347f2700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2017-07-31 17:50:11.902088 7f05347f2700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication 2017-07-31 17:50:11.902090 7f05347f2700 0 librados: client.admin initialization error (2) No such file or directory Error connecting to cluster: ObjectNotFound
Verified on openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462