1882757 – [OSP16.1][FFU]After the ffu in a hybrid env with sriov compute are in ERROR status

Bug 1882757 - [OSP16.1][FFU]After the ffu in a hybrid env with sriov compute are in ERROR status

Summary: [OSP16.1][FFU]After the ffu in a hybrid env with sriov compute are in ERROR s...

Keywords:
Status:	CLOSED DUPLICATE of bug 1850929
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Lukas Bezdicka
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-25 15:24 UTC by Candido Campos
Modified:	2023-03-21 19:35 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-10-21 10:58:56 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Comment 7 Stephen Finucane 2020-10-14 14:57:56 UTC

Apologies for the delay. I got to look at this in more detail today. I missed the fact that we were talking about the undercloud services and that the servers reporting in ERROR status were still functioning. Now that I've grasped that, it seems clear that this is a misconfiguration issue with ironic, presumably caused by use of a feature that is no longer supported in OSP 16. I see a couple of entries like the following in the nova-compute logs on the undercloud:

  2020-10-12 19:54:33.178 7 DEBUG ironicclient.common.http [req-1f01f33d-8b30-4173-852c-d1bfb6799fef - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 751d6e75-39b0-46c9-b716-790a23a7e94d is already attached to Ironic Port 8b6bf19a-39a2-4359-8b7a-5a32d1012261 (HTTP 409). Attempt 1 of 2 wrapper /usr/lib/python3.6/site-packages/ironicclient/common/http.py:300
  2020-10-12 19:54:34.780 7 ERROR ironicclient.common.http [req-1f01f33d-8b30-4173-852c-d1bfb6799fef - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 751d6e75-39b0-46c9-b716-790a23a7e94d is already attached to Ironic Port 8b6bf19a-39a2-4359-8b7a-5a32d1012261 (HTTP 409). Attempt 2 of 2: ironicclient.common.apiclient.exceptions.Conflict: Unable to attach VIF because VIF 751d6e75-39b0-46c9-b716-790a23a7e94d is already attached to Ironic Port 8b6bf19a-39a2-4359-8b7a-5a32d1012261 (HTTP 409)

I don't know how significant this issue is, but the next error is clearly something to worry about.

  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [req-1f01f33d-8b30-4173-852c-d1bfb6799fef - - - - -] [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] Virtual interface plugging failed for instance. The port binding:host_id may need to be manually updated.: nova.exception.VirtualInterfacePlugException: Cannot attach VIF 479d238a-ae64-4eee-8a12-11ff51ed212a to the node d3550858-0ca7-40fd-869a-27b3cfe8e09d due to error: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool for conductor group "". (HTTP 400)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] Traceback (most recent call last):
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1612, in _plug_vif
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     port_id, retry_on_conflict=False)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/client_wrapper.py", line 180, in call
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     return self._multi_getattr(client, method)(*args, **kwargs)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/v1/node.py", line 422, in vif_attach
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     self.update(path, data, http_method="POST")
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/v1/node.py", line 367, in update
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     params=params)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/common/base.py", line 249, in _update
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     resp, body = self.api.json_request(method, url, **kwargs)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 464, in json_request
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     resp = self._http_request(url, method, **kwargs)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 289, in wrapper
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     return func(self, url, method, **kwargs)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/ironicclient/common/http.py", line 446, in _http_request
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     error_json.get('debuginfo'), method, url)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] ironicclient.common.apiclient.exceptions.BadRequest: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool for conductor group "". (HTTP 400)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] During handling of the above exception, another exception occurred:
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] Traceback (most recent call last):
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1088, in _init_instance
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     self.driver.plug_vifs(instance, net_info)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1683, in plug_vifs
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     self._plug_vifs(node, instance, network_info)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1653, in _plug_vifs
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     self._plug_vif(node, port_id)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]   File "/usr/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1635, in _plug_vif
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]     raise exception.VirtualInterfacePlugException(msg)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8] nova.exception.VirtualInterfacePlugException: Cannot attach VIF 479d238a-ae64-4eee-8a12-11ff51ed212a to the node d3550858-0ca7-40fd-869a-27b3cfe8e09d due to error: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool for conductor group "". (HTTP 400)
  2020-10-12 19:55:46.158 7 ERROR nova.compute.manager [instance: 917219a7-4caf-4793-9654-8efac10ea7a8]

From [1], it seems you need to update your configuration to use 'ipxe' instead of 'pxe_imitool', since the latter was removed in Rocky.

  The deprecated classic drivers pxe_ipmitool and agent_ipmitool have been removed. Please use the ipmi hardware type instead.

Finally, as this is an Ironic issue, I'm moving this to the hardware provisioning DFG who should be able to advise further.

PS: For some reason I wasn't able to download the logs at [2]. I guess the webserver there is doing something funky due to the file extension?

[1] https://docs.openstack.org/releasenotes/ironic/rocky.html#relnotes-11-0-0-stable-rocky-upgrade-notes
[2] http://cougar11.scl.lab.tlv.redhat.com/DFG-network-neutron-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp-ipv4-vlan-ml2ovs-sriov/56/undercloud-0.tar.gz?undercloud-0/var/log/containers/nova/nova-compute.log.3.gz

Comment 9 Lukas Bezdicka 2020-10-21 10:58:56 UTC


*** This bug has been marked as a duplicate of bug 1850929 ***

Note You need to log in before you can comment on or make changes to this bug.