Bug 1481787
| Summary: | openstack-nova: check that the image data is not None | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | |
| Component: | openstack-nova | Assignee: | Stephen Finucane <stephenfin> | |
| Status: | CLOSED ERRATA | QA Contact: | Joe H. Rahme <jhakimra> | |
| Severity: | low | Docs Contact: | ||
| Priority: | low | |||
| Version: | 12.0 (Pike) | CC: | berrange, dasmith, eglynn, fpercoco, kchamart, lmarsh, lyarwood, mbooth, owalsh, sbauza, sclewis, sgordon, srevivo, stephenfin, vromanso | |
| Target Milestone: | z3 | Keywords: | TestOnly, Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-nova-16.0.4-5.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
Cause:
glanceclient can return 'None' from 'glanceclient.v2.images.Controller.data'. This is a valid value but nova was not handling it.
Consequence:
An unhandled error would be raised when booting an instance, resulting in a HTTP 500 error from nova-api.
Fix:
nova will now handle these values and raise an exception that can be handled by nova-api.
Result:
Users will now see a meaningful error message highlighting issues with the images returned by glance.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1481838 1623968 (view as bug list) | Environment: | ||
| Last Closed: | 2018-11-13 22:13:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1476448, 1486995 | |||
| Bug Blocks: | 1481838, 1623968 | |||
Running grep -i error /var/log/containers/glance/api.log on controller: ERROR glance.location [req-05617a17-ab4e-4550-95dc-a14a93045b54 bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Glance tried all active locations to get data for image a530e706-6cdf-42d0-8136-63bbe3dd9e32 but all have failed.: NotFound: Image /var/lib/glance/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32 not found (overcloud) [stack@undercloud-0 ~]$ openstack image show a530e706-6cdf-42d0-8136-63bbe3dd9e32 +------------------+---------------------------------------------------------------------------------+ | Field | Value | +------------------+---------------------------------------------------------------------------------+ | checksum | ee1eca47dc88f4879d8a229cc70a07c6 | | container_format | bare | | created_at | 2017-08-15T17:32:37Z | | disk_format | qcow2 | | file | /v2/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32/file | | id | a530e706-6cdf-42d0-8136-63bbe3dd9e32 | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | aaed41f2e25f494c9fadd01c340f25c8 | | properties | direct_url='file:///var/lib/glance/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32' | | protected | False | | schema | /v2/schemas/image | | size | 13287936 | | status | active | | tags | | | updated_at | 2017-08-15T17:32:38Z | | virtual_size | None | | visibility | shared | +------------------+---------------------------------------------------------------------------------+ [root@overcloud-controller-1 ~]# hostname
overcloud-controller-1
[root@overcloud-controller-1 ~]# df /var/lib/glance/images
df: ‘/var/lib/glance/images’: No such file or directory
[root@overcloud-controller-1 ~]# df /var/lib/glance
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda2 41930732 10585396 31345336 26% /
[root@overcloud-controller-1 ~]#
[root@overcloud-controller-1 ~]# hostname
overcloud-controller-1
[root@overcloud-controller-1 ~]# df /var/lib/glance/images
df: ‘/var/lib/glance/images’: No such file or directory
[root@overcloud-controller-1 ~]# df /var/lib/glance
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda2 41930732 10585396 31345336 26% /
[root@overcloud-controller-1 ~]#
[root@overcloud-controller-1 ~]# docker ps|grep glance
1ae94e8e4d60 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-08-14.2 "kolla_start" 3 hours ago Up 3 hours (unhealthy) glance_api
[root@overcloud-controller-1 ~]# docker inspect glance_api |jq ".[].Mounts"|grep -e Destination -e Source
"Destination": "/etc/hosts",
"Source": "/etc/hosts"
"Destination": "/etc/pki/tls/cert.pem",
"Source": "/etc/pki/tls/cert.pem"
"Destination": "/dev/log",
"Source": "/dev/log"
"Destination": "/etc/puppet",
"Source": "/etc/puppet"
"Destination": "/etc/pki/ca-trust/extracted",
"Source": "/etc/pki/ca-trust/extracted"
"Destination": "/var/lib/kolla/config_files/src",
"Source": "/var/lib/config-data/puppet-generated/glance_api"
"Destination": "/var/log/glance",
"Source": "/var/log/containers/glance"
"Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt",
"Source": "/etc/pki/tls/certs/ca-bundle.trust.crt"
"Destination": "/etc/ssh/ssh_known_hosts",
"Source": "/etc/ssh/ssh_known_hosts"
"Destination": "/etc/pki/tls/certs/ca-bundle.crt",
"Source": "/etc/pki/tls/certs/ca-bundle.crt"
"Destination": "/var/lib/kolla/config_files/src-ceph",
"Source": "/etc/ceph"
"Destination": "/etc/localtime",
"Source": "/etc/localtime"
"Destination": "/var/lib/kolla/config_files/config.json",
"Source": "/var/lib/kolla/config_files/glance_api.json"
[root@overcloud-controller-1 ~]# docker inspect glance_api |jq ".[].Mounts"|grep -e Destination -e Source|grep "/var/lib/glance"
So seems like the mount to NFS share didn't occur. The deployment command and the NFS yaml are below: (undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/nfs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml \ --log-file overcloud_deployment_0.log (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/nfs.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'retry=1' CinderNfsServers: 192.168.24.1:/export/cinder GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: '192.168.24.1:/export/glance' *** Bug 1481838 has been marked as a duplicate of this bug. *** Duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1476448, however this shows a minor issue in nova where it isn't checking if the image data is None. None is documented as a possible return value: https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data According to our records, this should be resolved by openstack-nova-17.0.5-3.d7864fbgit.el7ost. This build is available now. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3602 |
openstack-nova: failed to launch instance on OC: ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable Environment: libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64 openstack-nova-placement-api-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 openstack-nova-compute-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-conductor-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch python-novaclient-9.1.0-0.20170804194758.0a53d19.el7ost.noarch libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64 puppet-nova-11.3.0-0.20170805105252.30a205c.el7ost.noarch libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64 libvirt-daemon-kvm-3.2.0-14.el7.x86_64 openstack-nova-common-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-api-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-3.2.0-14.el7.x86_64 libvirt-libs-3.2.0-14.el7.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64 libvirt-daemon-config-network-3.2.0-14.el7.x86_64 libvirt-client-3.2.0-14.el7.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64 python-nova-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-novncproxy-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-console-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64 openstack-nova-scheduler-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64 openstack-nova-migration-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-network-3.2.0-14.el7.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64 instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch Steps to reproduce: Deploy overcloud with NFS as backend storage. Try to launch an instance Result: The instance ends up in error state: | fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 553, in build_instances | Looking for errors in /var/log/containers/nova-compute.log on compute: 2017-08-15 17:34:01.677 1 ERROR nova.image.glance [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Error writing to /var/lib/nova/instances/_base/cae3a4306eeb5643cb6caffbe1e3050645f8aee2.part: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Instance failed to spawn: TypeError: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Traceback (most recent call last): 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] yield resources 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] block_device_info=block_device_info) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] block_device_info=block_device_info) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fallback_from_host) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] instance, size, fallback_from_host) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] size=size) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 595, in create_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] prepare_template(target=base, *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] return f(*args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 237, in fetch_func_sync 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fetch_func(target=target, *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 446, in fetch_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] images.fetch_to_raw(context, image_id, target) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 134, in fetch_to_raw 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fetch(context, image_href, path_tmp) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 125, in fetch 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] IMAGE_API.download(context, image_href, dest_path=path) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 184, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] dst_path=dest_path) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 369, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] {'path': dst_path, 'exception': ex}) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] self.force_reraise() 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] six.reraise(self.type_, self.value, self.tb) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 353, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] for chunk in image_chunks: 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]