Bug 1976125 - [BM][IPI] redfish inspect fails on nodes with nics where mac="": Expected a MAC address but received .
Summary: [BM][IPI] redfish inspect fails on nodes with nics where mac="": Expected a M...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Derek Higgins
QA Contact: Lubov
URL:
Whiteboard:
Depends On:
Blocks: 1991979
TreeView+ depends on / blocked
 
Reported: 2021-06-25 09:20 UTC by Yurii Prokulevych
Modified: 2021-10-18 17:37 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1991979 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:36:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ironic-image pull 200 0 None None None 2021-08-09 19:52:42 UTC
OpenStack gerrit 798844 0 None NEW Skip port create if MAC is blank 2021-06-30 12:54:15 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:37:08 UTC

Description Yurii Prokulevych 2021-06-25 09:20:49 UTC
Version:
--------
4.9.0-0.nightly-2021-06-24-082405

$ openshift-install version
openshift-baremetal-install version
openshift-baremetal-install 4.9.0-0.nightly-2021-06-24-082405
built from commit a049187192ea1c590b286e098ae223a04eff28b7
release image registry.ci.openshift.org/ocp/release@sha256:0a85bd21409ec017ce088c086eec8518ff164e8d8c479f6df88129e1baa1e711

Platform:
---------
baremetal

Please specify:
* IPI (automated install with `openshift-install`.

What happened?
--------------
2 out of 4 nodes failed introspection with error:

 Expected a MAC address but received .


What did you expect to happen?
------------------------------

Nodes are successfully introspected and deployed

How to reproduce it (as minimally and precisely as possible)?
-------------------------------------------------------------
Deploy OCP cluster with 4 worker nodes

Comment 3 Derek Higgins 2021-06-30 10:12:18 UTC
The problem here appears to be in redfish inspection,

taking a look logs for openshift-worker-2, here
2021-06-25T08:32:12.795333564Z 2021-06-25 08:32:12.789 1 WARNING ironic.drivers.modules.inspect_utils [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Port already exists for MAC address BC:97:E1:D9:9A:C0 for node 2eaa2c7f-9b17-4eea-99dc-c8dedb6ff2bd: ironic.common.exception.MACAlreadyExists: A port with MAC address bc:97:e1:d9:9a:c0 already exists.^[[00m
2021-06-25T08:32:12.799011134Z 2021-06-25 08:32:12.798 1 INFO ironic.drivers.modules.inspect_utils [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Port created for MAC address BC:97:E1:D9:9A:C1 for node 2eaa2c7f-9b17-4eea-99dc-c8dedb6ff2bd^[[00m
2021-06-25T08:32:12.799236491Z 2021-06-25 08:32:12.799 1 DEBUG ironic.common.states [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Exiting old state 'inspecting' in response to event 'fail' on_exit /usr/lib/python3.6/site-packages/ironic/common/states.py:307^[[00m
2021-06-25T08:32:12.799357757Z 2021-06-25 08:32:12.799 1 DEBUG ironic.common.states [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Entering new state 'inspect failed' in response to event 'fail' on_enter /usr/lib/python3.6/site-packages/ironic/common/states.py:313^[[00m
2021-06-25T08:32:12.810215423Z 2021-06-25 08:32:12.810 1 ERROR ironic.conductor.task_manager [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Node 2eaa2c7f-9b17-4eea-99dc-c8dedb6ff2bd moved to provision state "inspect failed" from state "inspecting"; target provision state is "manageable": ironic.common.exception.InvalidMAC: Expected a MAC address but received .^[[00m

The exception is getting thrown when the code tries to create a ports here[1], the code is looping through
the enabled nics found over redfish and creating ports for them, the first one already existed (BMO created this one),
it created a port for the 2nd one, and the 3rd failed, further up the logs show the redish representation for this device

2021-06-25T08:32:12.552318459Z 2021-06-25 08:32:12.551 1 DEBUG sushy.resources.base [req-3a18f6e8-0c30-458b-8b9d-d599d32c2539 ironic-user - - - -] Received representation of EthernetInterface /redfish/v1/Systems/System.Embedded.1/EthernetInterfaces/NIC.Slot.7-2-1: {'_oem_vendors': None, 'description': 'NIC in Slot 7 Port 2 Partition 1', 'identity': 'NIC.Slot.7-2-1', 'links': {'oem_vendors': None}, 'mac_address': '', 'name': 'System Ethernet Interface', 'permanent_mac_address': '', 'speed_mbps': 1000, 'status': {'health': 'ok', 'health_rollup': None, 'state': 'enabled'}} refresh /usr/lib/python3.6/site-packages/sushy/resources/base.py:634^[[00m

It is marked as enabled ('state': 'enabled') but isn't reporting a MAC ('mac_address': '')

I'll push a patch upto ironic to ignore nics with no MAC reported, in the meantime as a workaround are you able to disable the nic?

1 - https://github.com/openstack/ironic/blob/77be4c6c690cfa8f52a10ada2b9c1ecabccd423f/ironic/drivers/modules/inspect_utils.py#L46

Comment 4 Marius Cornea 2021-07-07 15:39:12 UTC
Slot 7(NIC in Slot 7) was specifically disabled from BIOS to prevent the NIC ports from showing up in the OS as they're not being in use. As a workaround I re-enabled the slot in BIOS and introspection passed.

Comment 7 Bob Fournier 2021-08-09 19:52:43 UTC
This has been picked up in ironic-image change https://github.com/openshift/ironic-image/pull/200, moving to ON_QA.

Comment 11 errata-xmlrpc 2021-10-18 17:36:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.