Bug 2221944
| Summary: | [17.1][ML2-OVS][OFFLOAD] vm is not able to start after hypervisor reboot | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Miguel Angel Nieto <mnietoji> |
| Component: | openstack-nova | Assignee: | OSP DFG:Compute <osp-dfg-compute> |
| Status: | CLOSED NOTABUG | QA Contact: | OSP DFG:Compute <osp-dfg-compute> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 17.1 (Wallaby) | CC: | dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, vromanso |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-17 16:03:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I am reproducing again with the log, each time i tried to start the instance, i get this error
2023-07-11 14:02:10.667 2 DEBUG oslo_concurrency.lockutils [req-b57778f6-3fa2-4a42-9476-492abe6223f2 f4ad469020f543edaf00514a09c0912d 5e533057d91741bcb54359c9a77c8a4d - default default] Lock "compute_resources"
released by "nova.compute.resource_tracker.ResourceTracker.update_usage" :: held 0.042s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:367
2023-07-11 14:02:10.667 2 INFO nova.compute.manager [req-b57778f6-3fa2-4a42-9476-492abe6223f2 f4ad469020f543edaf00514a09c0912d 5e533057d91741bcb54359c9a77c8a4d - default default] [instance: 7c882b21-d9f2-4d66-8a
78-af0509d0f1af] Successfully reverted task state from powering-on on failure for instance.
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server [req-b57778f6-3fa2-4a42-9476-492abe6223f2 f4ad469020f543edaf00514a09c0912d 5e533057d91741bcb54359c9a77c8a4d - default default] Exception during message h
andling: KeyError: 'pci_slot'
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server raise self.value [14/29104]
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 183, in decorated_function
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server raise self.value
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 154, in decorated_function
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 211, in decorated_function
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server raise self.value
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 200, in decorated_function
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 3174, in start_instance
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self._power_on(context, instance)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 3142, in _power_on
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.driver.power_on(context, instance,
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4016, in power_on
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info,
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 3840, in _hard_reboot
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.destroy(context, instance, network_info, destroy_disks=False,
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1499, in destroy
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.cleanup(context, instance, network_info, block_device_info,
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1569, in cleanup
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server return self._cleanup(
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1602, in _cleanup
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self._unplug_vifs(instance, network_info, True)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1372, in _unplug_vifs
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server self.vif_driver.unplug(instance, vif)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/vif.py", line 841, in unplug
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server vif_obj = os_vif_util.nova_to_osvif_vif(vif)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 546, in nova_to_osvif_vif
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server vif_obj = _nova_to_osvif_vif_ovs(vif)
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 339, in _nova_to_osvif_vif_ovs
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server port_profile=_get_ovs_representor_port_profile(vif),
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 313, in _get_ovs_representor_port_profile
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server representor_address=vif["profile"]['pci_slot'])
2023-07-11 14:02:10.671 2 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
I will update sos reports
I have seen that the vm has 2 ports, both of them are the same, vfs, but only one has configured binding_profile. Before reboot +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | testpmd-hwoffload-dut | ACTIVE | hwoffload_net_nic0_148=10.10.148.179; hwoffload_net_nic1_149=10.10.149.114; management_net_management=10.10.144.104, 10.46.228.42 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 272f2720-4980-407a-9b9d-c419d321ea36 | trex | ACTIVE | hwoffload_net_nic0_148=10.10.148.181; hwoffload_net_nic1_149=10.10.149.130; management_net_management=10.10.144.181, 10.46.228.36 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ openstack port show 63220377-6bad-4e3f-b5fd-1293f8907730 +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:02.3', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-17T14:08:49Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.149.114', subnet_id='114a2f53-b834-4f30-8de4-e38d5f77b098' | | id | 63220377-6bad-4e3f-b5fd-1293f8907730 | | ip_allocation | None | | mac_address | fa:16:3e:5f:15:e1 | | name | hwoffload_net_nic1_149_dut_normal_port-3 | | network_id | afa4542c-df7f-4dfd-bfe2-da229b7e6d54 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:11:53Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ openstack port show f38fde66-5b2a-437e-b8b7-e233d194c166 +-------------------------+------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2023-07-17T14:08:50Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.144.104', subnet_id='367849bb-c7b6-4546-a8e2-d9cbf8eaf8bc' | | id | f38fde66-5b2a-437e-b8b7-e233d194c166 | | ip_allocation | None | | mac_address | fa:16:3e:d2:7b:4a | | name | testpmd_dut_management_port_management | | network_id | ed86ab2f-af0a-4e0f-9036-15a8b6d57bfa | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 6 | | security_group_ids | 0055eb62-f432-49d8-ab56-d41774519d7a | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:14:15Z | +-------------------------+------------------------------------------------------------------------------+ openstack server stop 54954af3-7b7c-42fe-934d-ac923ba22fe7 openstack server list --long +--------------------------------------+-----------------------+---------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------+---------------------------+-------------------+-----------------------------------+------------+ | ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Flavor | Availability Zone | Host | Properties | +--------------------------------------+-----------------------+---------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------+---------------------------+-------------------+-----------------------------------+------------+ | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | testpmd-hwoffload-dut | SHUTOFF | None | Shutdown | hwoffload_net_nic0_148=10.10.148.179; hwoffload_net_nic1_149=10.10.149.114; management_net_management=10.10.144.104, 10.46.228.42 | trex_testpmd_mellanox | 3c01dc02-5743-437a-a6d8-24ebdf400b98 | perf_numa_0_hwoffload_dut | nova | computehwoffload-r740.localdomain | | | 272f2720-4980-407a-9b9d-c419d321ea36 | trex | ACTIVE | None | Running | hwoffload_net_nic0_148=10.10.148.181; hwoffload_net_nic1_149=10.10.149.130; management_net_management=10.10.144.181, 10.46.228.36 | trex_testpmd_mellanox | 3c01dc02-5743-437a-a6d8-24ebdf400b98 | perf_numa_0_trex | nova | computehwoffload-r730.localdomain | | +--------------------------------------+-----------------------+---------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------+---------------------------+-------------------+-----------------------------------+------------+ Reboot compute openstack port show 63220377-6bad-4e3f-b5fd-1293f8907730 +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:02.3', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-17T14:08:49Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.149.114', subnet_id='114a2f53-b834-4f30-8de4-e38d5f77b098' | | id | 63220377-6bad-4e3f-b5fd-1293f8907730 | | ip_allocation | None | | mac_address | fa:16:3e:5f:15:e1 | | name | hwoffload_net_nic1_149_dut_normal_port-3 | | network_id | afa4542c-df7f-4dfd-bfe2-da229b7e6d54 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:11:53Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ openstack port show f38fde66-5b2a-437e-b8b7-e233d194c166 +-------------------------+------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2023-07-17T14:08:50Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.144.104', subnet_id='367849bb-c7b6-4546-a8e2-d9cbf8eaf8bc' | | id | f38fde66-5b2a-437e-b8b7-e233d194c166 | | ip_allocation | None | | mac_address | fa:16:3e:d2:7b:4a | | name | testpmd_dut_management_port_management | | network_id | ed86ab2f-af0a-4e0f-9036-15a8b6d57bfa | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 7 | | security_group_ids | 0055eb62-f432-49d8-ab56-d41774519d7a | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:22:36Z | +-------------------------+------------------------------------------------------------------------------+ My previous comment is wrong, i chose wrong port. Both vfs seems to be configured after reboot (overcloud) [stack@undercloud-0 ~]$ openstack port show 63220377-6bad-4e3f-b5fd-1293f8907730 [34/684] +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:02.3', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-17T14:08:49Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.149.114', subnet_id='114a2f53-b834-4f30-8de4-e38d5f77b098' | | id | 63220377-6bad-4e3f-b5fd-1293f8907730 | | ip_allocation | None | | mac_address | fa:16:3e:5f:15:e1 | | name | hwoffload_net_nic1_149_dut_normal_port-3 | | network_id | afa4542c-df7f-4dfd-bfe2-da229b7e6d54 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:11:53Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show b4cfd029-96e9-4362-8662-eb9efaa18138 +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:02.5', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | connectivity='l2', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-17T14:08:48Z | | data_plane_status | None | | description | | | device_id | 54954af3-7b7c-42fe-934d-ac923ba22fe7 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.148.179', subnet_id='63a5c087-70f4-433e-b78c-68d13132d25f' | | id | b4cfd029-96e9-4362-8662-eb9efaa18138 | | ip_allocation | None | | mac_address | fa:16:3e:8d:0a:41 | | name | hwoffload_net_nic0_148_dut_normal_port-2 | | network_id | 9ce68cd1-d671-4a41-8ac7-fb23b554cefa | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 8dfa941bda934493a36f8ed4045188d0 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-17T14:11:53Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ I closed as a NOTABUG I realized that /tmp folder was full, when rebooting the compute there were some issues starting that containers as a result vm was not able to start Tested with RHOS-17.1-RHEL-9-20230712.n.1 |
Description of problem: I have the following vms: (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | testpmd-hwoffload-dut | ACTIVE | hwoffload_net_nic0_148=10.10.148.124; hwoffload_net_nic1_149=10.10.149.189; management_net_management=10.10.144.131, 10.46.228.38 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 893bf524-88be-41b6-b2e1-3dfaa00110c8 | trex | ACTIVE | management_net_management=10.10.144.178, 10.46.228.36; sriov_net_148=10.10.148.179; sriov_net_2_149=10.10.149.170 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port list | egrep "124|189" [36/920] | 05aa5634-77ff-412d-bd75-8d4bd2567699 | hwoffload_net_nic1_149_dut_normal_port-3 | fa:16:3e:71:c8:ee | ip_address='10.10.149.189', subnet_id='435ebee4-3dec-4b94-969b-955a980a4c5d' | ACTIVE | | 6ef54872-72b2-43ba-9ae0-70b534b99604 | hwoffload_net_nic0_148_dut_normal_port-2 | fa:16:3e:4c:dd:35 | ip_address='10.10.148.124', subnet_id='9a9cb526-4790-470f-80f4-3365c0e96b68' | ACTIVE | (overcloud) [stack@undercloud-0 ~]$ openstack port show 05aa5634-77ff-412d-bd75-8d4bd2567699 +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:01.4', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-10T14:05:18Z | | data_plane_status | None | | description | | | device_id | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.149.189', subnet_id='435ebee4-3dec-4b94-969b-955a980a4c5d' | | id | 05aa5634-77ff-412d-bd75-8d4bd2567699 | | ip_allocation | None | | mac_address | fa:16:3e:71:c8:ee | | name | hwoffload_net_nic1_149_dut_normal_port-3 | | network_id | 3233b8da-5c43-4c26-9a50-23d2293d6704 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 5e533057d91741bcb54359c9a77c8a4d | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-10T14:08:06Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show 6ef54872-72b2-43ba-9ae0-70b534b99604 +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-r740.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:18:02.0', pci_vendor_info='15b3:1018', physical_network='mx-network' | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-10T14:05:18Z | | data_plane_status | None | | description | | | device_id | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.10.148.124', subnet_id='9a9cb526-4790-470f-80f4-3365c0e96b68' | | id | 6ef54872-72b2-43ba-9ae0-70b534b99604 | | ip_allocation | None | | mac_address | fa:16:3e:4c:dd:35 | | name | hwoffload_net_nic0_148_dut_normal_port-2 | | network_id | 26abbd6a-84cd-4597-a591-23a3fb286aaa | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 5e533057d91741bcb54359c9a77c8a4d | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-10T14:08:05Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------+ I can stop/start vms: (overcloud) [stack@undercloud-0 ~]$ openstack server stop testpmd-hwoffload-dut (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | testpmd-hwoffload-dut | SHUTOFF | hwoffload_net_nic0_148=10.10.148.124; hwoffload_net_nic1_149=10.10.149.189; management_net_management=10.10.144.131, 10.46.228.38 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 893bf524-88be-41b6-b2e1-3dfaa00110c8 | trex | ACTIVE | management_net_management=10.10.144.178, 10.46.228.36; sriov_net_148=10.10.148.179; sriov_net_2_149=10.10.149.170 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack server start testpmd-hwoffload-dut (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | testpmd-hwoffload-dut | ACTIVE | hwoffload_net_nic0_148=10.10.148.124; hwoffload_net_nic1_149=10.10.149.189; management_net_management=10.10.144.131, 10.46.228.38 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 893bf524-88be-41b6-b2e1-3dfaa00110c8 | trex | ACTIVE | management_net_management=10.10.144.178, 10.46.228.36; sriov_net_148=10.10.148.179; sriov_net_2_149=10.10.149.170 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ But I stop the vm and reboot the compute (overcloud) [stack@undercloud-0 ~]$ openstack server stop testpmd-hwoffload-dut (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | testpmd-hwoffload-dut | SHUTOFF | hwoffload_net_nic0_148=10.10.148.124; hwoffload_net_nic1_149=10.10.149.189; management_net_management=10.10.144.131, 10.46.228.38 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 893bf524-88be-41b6-b2e1-3dfaa00110c8 | trex | ACTIVE | management_net_management=10.10.144.178, 10.46.228.36; sriov_net_148=10.10.148.179; sriov_net_2_149=10.10.149.170 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ (overcloud) [stack@undercloud-0 ~]$ ssh computehwoffload-r740.ctlplane "sudo reboot" Warning: Permanently added 'computehwoffload-r740.ctlplane' (ED25519) to the list of known hosts. After the compute is up, i start the vm (overcloud) [stack@undercloud-0 ~]$ openstack server start testpmd-hwoffload-dut But it is not able to start (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ | 7c882b21-d9f2-4d66-8a78-af0509d0f1af | testpmd-hwoffload-dut | SHUTOFF | hwoffload_net_nic0_148=10.10.148.124; hwoffload_net_nic1_149=10.10.149.189; management_net_management=10.10.144.131, 10.46.228.38 | trex_testpmd_mellanox | perf_numa_0_hwoffload_dut | | 893bf524-88be-41b6-b2e1-3dfaa00110c8 | trex | ACTIVE | management_net_management=10.10.144.178, 10.46.228.36; sriov_net_148=10.10.148.179; sriov_net_2_149=10.10.149.170 | trex_testpmd_mellanox | perf_numa_0_trex | +--------------------------------------+-----------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------------------------+ First time i reproduced this error, i saw this error in nova logs, but now i am not seeing any error in the logs 2023-07-10 13:05:39.426 2 DEBUG nova.network.os_vif_util [req-ea1cd0f4-217b-49b6-b18a-20ec3208741f - - - - -] Converting VIF {"id": "07ffa905-8c5e-4175-9200-26d8c9b74678", "address": "fa:16:3e:65:a8:74", "network": {"id": "8b51eb6d-3ce2-41f7-99fa-3828281206bc", "bridge": "br-int", "label": "hwoffload_net_nic0_148", "subnets": [{"cidr": "10.10.148.0/24", "dns": [], "gateway": {"address": "10.10.148.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "10.10.148.101", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {}}], "meta": {"injected": false, "tenant_id": "5e533057d91741bcb54359c9a77c8a4d", "mtu": 9000, "physical_network": "mx-network", "tunneled": false}}, "type": "ovs", "details": {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, "devname": "tap07ffa905-8c", "ovs_interfaceid": "07ffa905-8c5e-4175-9200-26d8c9b74678", "qbh_params": null, "qbg_params": null, "active": true, "vnic_type": "direct", "profile": {"capabilities": ["switchdev"]}, "preserve_on_delete": true, "delegate_create": true, "meta": {}} nova_to_osvif_vif /usr/lib/python3.9/site-packages/nova/network/os_vif_util.py:523 2023-07-10 13:05:39.444 2 ERROR oslo_service.service [req-ea1cd0f4-217b-49b6-b18a-20ec3208741f - - - - -] Error starting thread.: KeyError: 'pci_slot' 2023-07-10 13:05:39.444 2 ERROR oslo_service.service Traceback (most recent call last): 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/oslo_service/service.py", line 807, in run_service 2023-07-10 13:05:39.444 2 ERROR oslo_service.service service.start() 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/service.py", line 159, in start 2023-07-10 13:05:39.444 2 ERROR oslo_service.service self.manager.init_host() 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1448, in init_host 2023-07-10 13:05:39.444 2 ERROR oslo_service.service self._init_instance(context, instance) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1142, in _init_instance 2023-07-10 13:05:39.444 2 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1366, in plug_vifs 2023-07-10 13:05:39.444 2 ERROR oslo_service.service self.vif_driver.plug(instance, vif) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/vif.py", line 723, in plug 2023-07-10 13:05:39.444 2 ERROR oslo_service.service vif_obj = os_vif_util.nova_to_osvif_vif(vif) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 546, in nova_to_osvif_vif 2023-07-10 13:05:39.444 2 ERROR oslo_service.service vif_obj = _nova_to_osvif_vif_ovs(vif) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 339, in _nova_to_osvif_vif_ovs 2023-07-10 13:05:39.444 2 ERROR oslo_service.service port_profile=_get_ovs_representor_port_profile(vif), 2023-07-10 13:05:39.444 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/network/os_vif_util.py", line 313, in _get_ovs_representor_port_profile 2023-07-10 13:05:39.444 2 ERROR oslo_service.service representor_address=vif["profile"]['pci_slot']) 2023-07-10 13:05:39.444 2 ERROR oslo_service.service KeyError: 'pci_slot' Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230628.n.2 How reproducible: See description Actual results: Unable to start a vm that was stopped after we reboot hypervisor Expected results: We should be able to start vm Additional info: I will attach sos reports