Bug 1476880

Summary: openstack-glance: OSP12 OC deployment with external ceph: unable to create image (status: queued)
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: John Fulton <johfulto>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: cyril, dprince, eglynn, fpercoco, gfidente, jjoyce, johfulto, mburns, pdeore, pgrist, rhel-osp-director-maint, sasha, scohen, srevivo
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170805163048 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:45:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2017-07-31 17:16:18 UTC 
openstack-glance: OSP12 OC deployment with external ceph: unable to create image (status: queued)

Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch

openstack-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch
puppet-glance-11.2.0-0.20170718235959.302415b.el7ost.noarch
python-glanceclient-2.7.0-0.20170622095732.d67b33b.el7ost.noarch
python-glance-store-0.20.1-0.20170719150719.fafc8fd.el7ost.noarch
python-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /home/stack/external-ceph.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yaml \
--log-file overcloud_deployment_0.log


2.  Try to create glance image



Result:
Image doesn't get created on the external ceph and stays with status "queued":
(overcloud) [stack@undercloud-0 ~]$ openstack image list 
+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| 5646420d-efcb-478f-968a-375fea079290 | cirros | queued |
+--------------------------------------+--------+--------+


Expected result:
The image should be created on external ceph and become active.



More details:

(overcloud) [stack@undercloud-0 ~]$ cat /home/stack/external-ceph.yaml
# *******************************************************************
# This file was created automatically by the sample environment
# generator. Developers should use `tox -e genconfig` to update it.
# Users are recommended to make changes to a copy of the file instead
# of the original, if any customizations are needed.
# *******************************************************************
# title: Deploy Using an External Ceph Cluster
# description: |
#   A Heat environment file which can be used to enable the
#   use of an externally managed Ceph cluster.
parameter_defaults:
  # The Ceph admin client key. Can be created with ceph-authtool --gen-print-key.
  # Type: string
  CephAdminKey: ''

  # The Ceph client key. Can be created with ceph-authtool --gen-print-key. Currently only used for external Ceph deployments to create the openstack user keyring.
  # Mandatory. This parameter must be set by the user.
  # Type: string
  CephClientKey: '<KEY>'

  # 
  # Type: string
  CephClientUserName: openstack

  # The Ceph cluster FSID. Must be a UUID.
  # Mandatory. This parameter must be set by the user.
  # Type: string
  CephClusterFSID: '<FSID>'

  # List of externally managed Ceph Mon Host IPs. Only used for external Ceph deployments.
  # Type: string
  CephExternalMonHost: '<IPs>'

  # Whether to enable or not the Iscsi backend for Cinder
  # Type: boolean
  CinderEnableIscsiBackend: False

  # Whether to enable or not the Rbd backend for Cinder
  # Type: boolean
  CinderEnableRbdBackend: True

  # 
  # Type: string
  CinderRbdPoolName: volumes

  # The short name of the Glance backend to use. Should be one of swift, rbd, or file
  # Type: string
  GlanceBackend: rbd

  # 
  # Type: string
  GlanceRbdPoolName: images

  # The short name of the Gnocchi backend to use. Should be one of swift, rbd, or file
  # Type: string
  GnocchiBackend: rbd

  # 
  # Type: string
  GnocchiRbdPoolName: metrics

  # Whether to enable or not the Rbd backend for Nova
  # Type: boolean
  NovaEnableRbdBackend: True

  # 
  # Type: string
  NovaRbdPoolName: vms

  # The default features enabled when creating a block device image. Only applies to format 2 images. Set to '1' for Jewel clients using older Ceph servers.
  # Type: string
  RbdDefaultFeatures: '1'

resource_registry:
  OS::TripleO::Services::CephClient: OS::Heat::None
  OS::TripleO::Services::CephExternal: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-external.yaml
  OS::TripleO::Services::CephMon: OS::Heat::None
  OS::TripleO::Services::CephOSD: OS::Heat::None




########################################################################


Checking logs on all controllers, one controllers has these errors;
[root@overcloud-controller-2 ~]# grep -i error /var/log/containers/glance/api.log
2017-07-31 16:19:15.602 22 ERROR glance.api.v2.image_data [req-42f3deb0-8250-4911-8a77-61006543791f 5862a0cf8d014744a828425d5991a799 d7cb49479ee84e219feed7ca7514985c - default default] Failed to upload image data due to internal error: Error: error calling conf_read_file: errno EINVAL
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi [req-42f3deb0-8250-4911-8a77-61006543791f 5862a0cf8d014744a828425d5991a799 d7cb49479ee84e219feed7ca7514985c - default default] Caught error: error calling conf_read_file: errno EINVAL: Error: error calling conf_read_file: errno EINVAL
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi Traceback (most recent call last):
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1155, in __call__
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     request, **action_args)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1194, in dispatch
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     return method(*args, **kwargs)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 269, in upload
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self._restore(image_repo, image)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.force_reraise()
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 134, in upload
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     image.set_data(data, size)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.base.set_data(data, size)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 480, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.force_reraise()
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 427, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.repo.set_data(data, size)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 192, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.image.set_data(data, size=size)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/location.py", line 439, in set_data
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     verifier=verifier)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 453, in add_to_backend
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     verifier)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 426, in store_add_to_backend
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     verifier=verifier)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 454, in add
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     rados_id=self.user) as conn:
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     return self.gen.next()
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 268, in get_connection
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     client = rados.Rados(conffile=conffile, rados_id=rados_id)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/rados.py", line 253, in __init__
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     self.conf_read_file(conffile)
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/rados.py", line 302, in conf_read_file
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi     raise make_ex(ret, "error calling conf_read_file")
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi Error: error calling conf_read_file: errno EINVAL
2017-07-31 16:19:15.638 22 ERROR glance.common.wsgi
Comment 1 Dan Prince 2017-07-31 18:04:09 UTC 
gfidente or johfulto might help here.
Comment 5 John Fulton 2017-08-02 19:38:57 UTC 
Ceph authentication is not set up [1]. Does your undercloud include this change? https://review.openstack.org/#/c/482500/ 

[1] as per the sosreport from controller-0.redhat.local-20170731174937

2017-07-31 17:50:12.481195 7f29891c9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2017-07-31 17:50:12.481204 7f29891c9700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2017-07-31 17:50:12.481205 7f29891c9700  0 librados: client.admin initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound
[jfulton@skagra ceph]$ cat ceph_health_detail
2017-07-31 17:50:11.902068 7f05347f2700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2017-07-31 17:50:11.902088 7f05347f2700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2017-07-31 17:50:11.902090 7f05347f2700  0 librados: client.admin initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound
Comment 11 Yogev Rabl 2017-11-16 19:07:02 UTC 
Verified on openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch
Comment 14 errata-xmlrpc 2017-12-13 21:45:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462