Description of problem: We are unable to start nova-compute service on compute node. nova-compute.log on compute shows: 2018-05-12 16:42:47.323 305978 INFO os_vif [req-0a72cdea-843a-4932-b8a0-bc24c2f21d9f - - - - -] Successfully plugged vif VIFBridge(active=True,address=fa:16:3e:41:a9:2c,bridge_name='qbr8d027ff4-23',has_traffic_filtering=True,id=8d027ff4-2328-47df-9f9a-2c1a9914a83b,network=Network(9a98b244-b1d2-46b3-ab0e-be8456e3a984),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8d027ff4-23') 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service [req-0a72cdea-843a-4932-b8a0-bc24c2f21d9f - - - - -] Error starting thread. 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service Traceback (most recent call last): 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service service.start() 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 117, in start 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.manager.init_host() 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1154, in init_host 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self._init_instance(context, instance) 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 957, in _init_instance 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info) 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 703, in plug_vifs 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service self.vif_driver.plug(instance, vif) 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 771, in plug 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service vif_obj = os_vif_util.nova_to_osvif_vif(vif) 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 408, in nova_to_osvif_vif 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service {'type': vif['type'], 'func': funcname}) 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound' 2018-05-12 16:42:47.369 305978 ERROR oslo_service.service [stack@director:~]$ neutron port-list | grep fa:16:3e:41:a9:2c | 8d027ff4-2328-47df-9f9a-2c1a9914a83b | | fa:16:3e:41:a9:2c | {"subnet_id": "1f5ed9bc-aa7d-49bd-ac48-23b430fc0eb4", "ip_address": "172.19.9.17"} | [stack@director:~]$ neutron port-show 8d027ff4-2328-47df-9f9a-2c1a9914a83b +-----------------------+------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | overcloud-compute-7.localdomain | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-10-31T12:31:45Z | | description | | | device_id | b4ef4d0b-9e39-4741-a2dd-7fd7c066d13b | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "1f5ed9bc-aa7d-49bd-ac48-23b430fc0eb4", "ip_address": "172.19.9.17"} | | id | 8d027ff4-2328-47df-9f9a-2c1a9914a83b | | mac_address | fa:16:3e:41:a9:2c | | name | | | network_id | 9a98b244-b1d2-46b3-ab0e-be8456e3a984 | | port_security_enabled | True | | project_id | 3b2049626c954cdc9147beee2d34b441 | | qos_policy_id | | | revision_number | 184 | | security_groups | 97aa0764-c0b5-47d1-88b2-285673d46a31 | | | c7addc13-5a77-4322-953a-9d89d42468e6 | | | cecdad42-7c78-45e7-9ec2-fef1086dbb7e | | | de0a6da8-c44e-475f-90fd-1fb625840c52 | | status | ACTIVE | | tenant_id | 3b2049626c954cdc9147beee2d34b441 | | updated_at | 2018-05-12T15:37:46Z | +-----------------------+------------------------------------------------------------------------------------+
Would it be possible to reproduce this with debug enabled and attach DEBUG-level nova-compute logs to this BZ? I definitely see the ERROR in the sosreports, but there are some debug statements in the code that might help shed light on this. Cheers!
In addition to Artom's request, can you provide logs of Neutron server (from controllers)
The vif_type=unbound only appears when you create a port that is not assigned to any VM. In our case, the ports are well attached to VMs and we can see that, Neutron is showing correct information. My thinking is that the network info cache of Nova is not up to date. In nova/compute/manager.py there is a method called '_init_instance()' from where we call the driver to plug the VIFs. My thinking is to refresh the network info cache. diff --git a/nova/compute/manager.py b/nova/compute/manager.py index d7b8b71383..1ee4fe41df 100644 --- a/nova/compute/manager.py +++ b/nova/compute/manager.py @@ -952,7 +952,8 @@ class ComputeManager(manager.Manager): LOG.exception(msg, instance=instance) return - net_info = compute_utils.get_nw_info_for_instance(instance) + # Refreshed network info + net_info = self.network_api.get_instance_nw_info(context, instance) try: self.driver.plug_vifs(instance, net_info) except NotImplementedError as e: Can you apply that patch and let me know if that resolve the issue?
This bug has been open in NEEDINFO for over a month, so I'm closing this to keep the review queue sane. Please reopen the bug if this is still an issue.
Hi guys, We're hitting this exact same issue so we'll try testing the patch you suggested earlier and we'll update this back then. Thank you very much, David Hill
Hi, just had the same issue. Used the above patch and my nova-compute was able to handle the issue. So I would say the above patch solves/workarounds the issue. Thanks, Fabian
This is still an issue on master and I have posted a fix for master that should resolve this. Note that that fix takes a different approach to the one suggested by Sahid earlier in this BZ. The earlier solution would solve this specific issue here but there are other corner cases it wouldn't cover [1]. It also requires a lot of expensive calls to the neutron API on startup. As such, a more general approach is to simply fail the particular instance and allow the nova-compute service to continue starting. That's what we've done. [1] https://review.openstack.org/#/c/587498/1/nova/compute/manager.py@956
*** Bug 1655056 has been marked as a duplicate of this bug. ***
As per #c21, the compute agent can be restarted (systemctl restart openstack-nova-compute) successfully despite the presence of the unbound port.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0923
*** Bug 1760762 has been marked as a duplicate of this bug. ***