Hide Forgot
Description of problem: Fail to launch vm when using vmware hypervisor in OSP9. The same customer is able to get this working in OSP7 and OSP8. ##nova-compute.log trace snippet 2016-11-28 20:14:09.711 27169 ERROR nova.compute.manager [instance: 2971c638-8d28-47dd-aaa6-ede1ba8f4fd9] raise task_ex 2016-11-28 20:14:09.711 27169 ERROR nova.compute.manager [instance: 2971c638-8d28-47dd-aaa6-ede1ba8f4fd9] VimFaultException: An error occurred during host configuration. 2016-11-28 20:14:09.711 27169 ERROR nova.compute.manager [instance: 2971c638-8d28-47dd-aaa6-ede1ba8f4fd9] Faults: ['PlatformConfigFault'] 2016-11-28 20:14:09.711 27169 ERROR nova.compute.manager [instance: 2971c638-8d28-47dd-aaa6-ede1ba8f4fd9] 2016-11-28 20:14:09.712 27169 INFO nova.compute.manager [req-41fccaac-ce22-4cd6-ba4d-1da4f40dbf93 f143aab7cb15446fa9c4c54b39371dab 4e3d42a0846c4f6dbda4ab170311bd1f - - -] [instance: 2971c638-8d28-47dd-aaa6-ede1ba8f4fd9] Terminating instance ###I also noticed the same error from the vmware logs 2016-11-28T20:14:09.240Z info vpxd[7F348FCF9700] [Originator@6876 sub=Default opID=44f9f0c5-84] [VpxLRO] -- ERROR task-internal-2975229 -- -- VmprovWorkflow: vim.fault.PlatformConfigFault: --> Result: --> (vim.fault.PlatformConfigFault) { --> faultCause = (vmodl.MethodFault) null, --> text = "IPC disabled", --> msg = "An error occurred during host configuration." --> } --> Args: --> 2016-11-28T20:14:09.251Z info vpxd[7F348FCF9700] [Originator@6876 sub=vpxLro opID=44f9f0c5] [VpxLRO] -- FINISH task-795 2016-11-28T20:14:09.251Z info vpxd[7F348FCF9700] [Originator@6876 sub=Default opID=44f9f0c5] [VpxLRO] -- ERROR task-795 -- group-v243 -- vim.Folder.createVm: vim.fault.PlatformConfigFault: --> Result: --> (vim.fault.PlatformConfigFault) { --> faultCause = (vmodl.MethodFault) null, --> text = "IPC disabled", --> msg = "An error occurred during host configuration." --> } --> Args: --> --> Arg config: --> (vim.vm.ConfigSpec) { --> changeVersion = <unset>, --> name = "2971c638-8d28-47dd-aaa6-ede1ba8f4fd9", --> version = <unset>, --> uuid = "421feacf-6392-7667-694b-0fb32bea7414", --> instanceUuid = "2971c638-8d28-47dd-aaa6-ede1ba8f4fd9", Version-Release number of selected component (if applicable): openstack-nova-compute-13.1.1-7.el7ost.noarch Mon Oct 31 20:35:06 2016 How reproducible: 100% Steps to Reproduce: 1. set up vmware hypervisor in osp9 2.attempt to launch vm 3. Actual results: fails to launch Expected results: launchs vm successfully Additional info: [stack@undercloud-telus ~]$ glance image-list +--------------------------------------+--------------------+-------------+------------------+-------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+--------------------+-------------+------------------+-------------+--------+ | 0f364dfe-f940-415d-965d-fe8d723791c9 | telus-win-image | vmdk | bare | 23239262208 | active | | 3f232833-3933-4f68-b043-9b9bed20d1a0 | windows2012-sparse | vmdk | bare | 15012724736 | active | +--------------------------------------+--------------------+-------------+------------------+-------------+--------+ [stack@undercloud-telus ~]$ glance image-show windows2012-sparse +-------------------------------+--------------------------------------+ | Property | Value | +-------------------------------+--------------------------------------+ | Property 'vmware_adaptertype' | ide | | Property 'vmware_disktype' | sparse | | checksum | 5f9c9e9bf79298ff9dcc54becde5cf5d | | container_format | bare | | created_at | 2016-11-28T17:12:08.000000 | | deleted | False | | disk_format | vmdk | | id | 3f232833-3933-4f68-b043-9b9bed20d1a0 | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | windows2012-sparse | | owner | 4e3d42a0846c4f6dbda4ab170311bd1f | | protected | False | | size | 15012724736 | | status | active | | updated_at | 2016-11-28T17:14:03.000000 | +-------------------------------+--------------------------------------+ [stack@undercloud-telus ~]$ nova boot --flavor m1.medium --image windows2012-sparse testvm1 +--------------------------------------+-----------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | telus-test-compute-0.localdomain | | OS-EXT-SRV-ATTR:hostname | testvm1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | domain-c7.0302ea13-8681-4b48-9609-b4aea788ca63 | | OS-EXT-SRV-ATTR:instance_name | instance-00000007 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-wsi3uofb | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | 5nmHbroffa4F | | config_drive | | | created | 2016-11-29T20:25:31Z | | description | - | | flavor | m1.medium (3) | | hostId | 20da690b9907d2f7df0febc31ed00a39451879439ec1ce385335645f | | host_status | UP | | id | 1b2cbd38-72ca-4def-9f58-5554a8871528 | | image | windows2012-sparse (3f232833-3933-4f68-b043-9b9bed20d1a0) | | key_name | - | | locked | False | | metadata | {} | | name | testvm1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 4e3d42a0846c4f6dbda4ab170311bd1f | | updated | 2016-11-29T20:25:32Z | | user_id | f143aab7cb15446fa9c4c54b39371dab | +--------------------------------------+-----------------------------------------------------------+ [stack@undercloud-telus ~]$ nova show testvm1 +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | testvm1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000007 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-wsi3uofb | | OS-EXT-SRV-ATTR:root_device_name | /dev/sda | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2016-11-29T20:25:31Z | | description | - | | fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 392, in build_instances | | | context, request_spec, filter_properties) | | | File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 436, in _schedule_instances | | | hosts = self.scheduler_client.select_destinations(context, spec_obj) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 372, in wrapped | | | return func(*args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 51, in select_destinations | | | return self.queryclient.select_destinations(context, spec_obj) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method | | | return getattr(self.instance, __name)(*args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 32, in select_destinations | | | return self.scheduler_rpcapi.select_destinations(context, spec_obj) | | | File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 121, in select_destinations | | | return cctxt.call(ctxt, 'select_destinations', **msg_args) | | | File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 158, in call | | | retry=self.retry) | | | File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send | | | timeout=timeout, retry=retry) | | | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 470, in send | | | retry=retry) | | | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 461, in _send | | | raise result | | | ", "created": "2016-11-29T20:25:35Z"} | | flavor | m1.medium (3) | | hostId | | | host_status | | | id | 1b2cbd38-72ca-4def-9f58-5554a8871528 | | image | windows2012-sparse (3f232833-3933-4f68-b043-9b9bed20d1a0) | | key_name | - | | locked | False | | metadata | {} | | name | testvm1 | | os-extended-volumes:volumes_attached | [] | | status | ERROR | | tenant_id | 4e3d42a0846c4f6dbda4ab170311bd1f | | updated | 2016-11-29T20:25:35Z | | user_id | f143aab7cb15446fa9c4c54b39371dab | +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Created attachment 1226069 [details] seems to be new files added in OSP9, that osp7,8 do not seem to add
Created attachment 1226456 [details] vpxd.log from the time of failure
I couldn't find out anything about the specific error, so I bisected the issue. The fault is introduced by commit 600e86c456b51a57fa4e1ecba0bac84af3b624e2: https://review.openstack.org/#/c/288552/ This suggests that it's going to be related to network config somehow, but I'm still looking into exactly what this change means to the vmware driver.
I've had a look at the generated config spec both with and without the above change. This looks like the relevant diff (succeeds -> fails): backing = - (VirtualEthernetCardNetworkBackingInfo){ - deviceName = "br-int" - useAutoDetect = None - network = - (ManagedObjectReference){ - value = None - _type = "" - } - inPassthroughMode = None + (VirtualEthernetCardOpaqueNetworkBackingInfo){ + opaqueNetworkId = "br-int" + opaqueNetworkType = "opaque" }
So, the change I mention above fixed an inconsistency between the network api class which was loaded and the value of is_neutron passed to the driver. Previously is_neutron was calculated by looking at the class specified by network_api_class and checking if it was a subclass of the neutron api base class. network_api_class defaults to nova network and we don't set it, so is_neutron was always passed as False. However, we do set use_neutron=True, which causes network_api_class to be ignored and we load neutron anyway. Therefore we were loading neutron, but telling the driver we were using nova network. The fix with introduces the 'regression' fixes this. Because we're actually using neutron, the driver is now correctly told about this. The vmware driver uses is_neutron to pick an appropriate backend. It now picks VirtualEthernetCardOpaqueNetworkBackingInfo with a network id pointing at the value of vmware.integration-bridge. I strongly suspect that vsphere is not correctly configured for neutron networking, which is causing the failure.
WONTFIX/NOTABUG therefore QE Won't automate