openstack-nova: failed to launch instance on OC: ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable Environment: libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64 openstack-nova-placement-api-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 openstack-nova-compute-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-conductor-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch python-novaclient-9.1.0-0.20170804194758.0a53d19.el7ost.noarch libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64 puppet-nova-11.3.0-0.20170805105252.30a205c.el7ost.noarch libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64 libvirt-daemon-kvm-3.2.0-14.el7.x86_64 openstack-nova-common-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-api-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-3.2.0-14.el7.x86_64 libvirt-libs-3.2.0-14.el7.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64 libvirt-daemon-config-network-3.2.0-14.el7.x86_64 libvirt-client-3.2.0-14.el7.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64 python-nova-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-novncproxy-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch openstack-nova-console-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64 openstack-nova-scheduler-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64 openstack-nova-migration-16.0.0-0.20170808024016.6d4fc21.el7ost.noarch libvirt-daemon-driver-network-3.2.0-14.el7.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64 instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch Steps to reproduce: Deploy overcloud with NFS as backend storage. Try to launch an instance Result: The instance ends up in error state: | fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 553, in build_instances | Looking for errors in /var/log/containers/nova-compute.log on compute: 2017-08-15 17:34:01.677 1 ERROR nova.image.glance [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Error writing to /var/lib/nova/instances/_base/cae3a4306eeb5643cb6caffbe1e3050645f8aee2.part: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Instance failed to spawn: TypeError: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Traceback (most recent call last): 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] yield resources 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] block_device_info=block_device_info) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] block_device_info=block_device_info) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fallback_from_host) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] instance, size, fallback_from_host) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] size=size) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 595, in create_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] prepare_template(target=base, *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] return f(*args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 237, in fetch_func_sync 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fetch_func(target=target, *args, **kwargs) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 446, in fetch_image 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] images.fetch_to_raw(context, image_id, target) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 134, in fetch_to_raw 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] fetch(context, image_href, path_tmp) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 125, in fetch 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] IMAGE_API.download(context, image_href, dest_path=path) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 184, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] dst_path=dest_path) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 369, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] {'path': dst_path, 'exception': ex}) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] self.force_reraise() 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] six.reraise(self.type_, self.value, self.tb) 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 353, in download 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] for chunk in image_chunks: 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable 2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]
Running grep -i error /var/log/containers/glance/api.log on controller: ERROR glance.location [req-05617a17-ab4e-4550-95dc-a14a93045b54 bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Glance tried all active locations to get data for image a530e706-6cdf-42d0-8136-63bbe3dd9e32 but all have failed.: NotFound: Image /var/lib/glance/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32 not found
(overcloud) [stack@undercloud-0 ~]$ openstack image show a530e706-6cdf-42d0-8136-63bbe3dd9e32 +------------------+---------------------------------------------------------------------------------+ | Field | Value | +------------------+---------------------------------------------------------------------------------+ | checksum | ee1eca47dc88f4879d8a229cc70a07c6 | | container_format | bare | | created_at | 2017-08-15T17:32:37Z | | disk_format | qcow2 | | file | /v2/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32/file | | id | a530e706-6cdf-42d0-8136-63bbe3dd9e32 | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | aaed41f2e25f494c9fadd01c340f25c8 | | properties | direct_url='file:///var/lib/glance/images/a530e706-6cdf-42d0-8136-63bbe3dd9e32' | | protected | False | | schema | /v2/schemas/image | | size | 13287936 | | status | active | | tags | | | updated_at | 2017-08-15T17:32:38Z | | virtual_size | None | | visibility | shared | +------------------+---------------------------------------------------------------------------------+
[root@overcloud-controller-1 ~]# hostname overcloud-controller-1 [root@overcloud-controller-1 ~]# df /var/lib/glance/images df: ‘/var/lib/glance/images’: No such file or directory [root@overcloud-controller-1 ~]# df /var/lib/glance Filesystem 1K-blocks Used Available Use% Mounted on /dev/vda2 41930732 10585396 31345336 26% / [root@overcloud-controller-1 ~]# [root@overcloud-controller-1 ~]# hostname overcloud-controller-1 [root@overcloud-controller-1 ~]# df /var/lib/glance/images df: ‘/var/lib/glance/images’: No such file or directory [root@overcloud-controller-1 ~]# df /var/lib/glance Filesystem 1K-blocks Used Available Use% Mounted on /dev/vda2 41930732 10585396 31345336 26% / [root@overcloud-controller-1 ~]# [root@overcloud-controller-1 ~]# docker ps|grep glance 1ae94e8e4d60 192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-08-14.2 "kolla_start" 3 hours ago Up 3 hours (unhealthy) glance_api [root@overcloud-controller-1 ~]# docker inspect glance_api |jq ".[].Mounts"|grep -e Destination -e Source "Destination": "/etc/hosts", "Source": "/etc/hosts" "Destination": "/etc/pki/tls/cert.pem", "Source": "/etc/pki/tls/cert.pem" "Destination": "/dev/log", "Source": "/dev/log" "Destination": "/etc/puppet", "Source": "/etc/puppet" "Destination": "/etc/pki/ca-trust/extracted", "Source": "/etc/pki/ca-trust/extracted" "Destination": "/var/lib/kolla/config_files/src", "Source": "/var/lib/config-data/puppet-generated/glance_api" "Destination": "/var/log/glance", "Source": "/var/log/containers/glance" "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt", "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt" "Destination": "/etc/ssh/ssh_known_hosts", "Source": "/etc/ssh/ssh_known_hosts" "Destination": "/etc/pki/tls/certs/ca-bundle.crt", "Source": "/etc/pki/tls/certs/ca-bundle.crt" "Destination": "/var/lib/kolla/config_files/src-ceph", "Source": "/etc/ceph" "Destination": "/etc/localtime", "Source": "/etc/localtime" "Destination": "/var/lib/kolla/config_files/config.json", "Source": "/var/lib/kolla/config_files/glance_api.json" [root@overcloud-controller-1 ~]# docker inspect glance_api |jq ".[].Mounts"|grep -e Destination -e Source|grep "/var/lib/glance"
So seems like the mount to NFS share didn't occur. The deployment command and the NFS yaml are below: (undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/nfs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml \ --log-file overcloud_deployment_0.log (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/nfs.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'retry=1' CinderNfsServers: 192.168.24.1:/export/cinder GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: '192.168.24.1:/export/glance'
*** Bug 1481838 has been marked as a duplicate of this bug. ***
Duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1476448, however this shows a minor issue in nova where it isn't checking if the image data is None. None is documented as a possible return value: https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data
According to our records, this should be resolved by openstack-nova-17.0.5-3.d7864fbgit.el7ost. This build is available now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3602