Bug 1915828 - Latest Dell firmware (04.40.00.00) fails to install IPI on BM using idrac-virtualmedia protocol
Summary: Latest Dell firmware (04.40.00.00) fails to install IPI on BM using idrac-vir...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Iury Gregory Melo Ferreira
QA Contact: Lubov
URL:
Whiteboard:
Depends On:
Blocks: dit
TreeView+ depends on / blocked
 
Reported: 2021-01-13 14:00 UTC by rlopez
Modified: 2021-02-24 15:53 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: Upgrade Dell Firmware to 4.40.00.00 will use eHTML5 as Virtual Console Plugin. Consequence: Nodes won't be deployed and will show error in their status. Workaround (if any): Change the Virtual Console Plugin to HTML5 and run the deployment again. Result: Nodes should be successfully deployed.
Clone Of:
Environment:
Last Closed: 2021-02-24 15:52:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Ironic Conductor log (4.31 MB, text/plain)
2021-01-13 14:00 UTC, rlopez
no flags Details
Ironic Conductor log 2021-01-13 (725.42 KB, text/plain)
2021-01-14 14:47 UTC, rlopez
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift ironic-image pull 142 0 None closed Bug 1915828: Latest Dell firmware (04.40.00.00) fails to install IPI on BM using idrac-virtualmedia protocol 2021-02-16 08:07:31 UTC
OpenStack gerrit 772089 0 None MERGED Fix ExtendedInfo error handling for non-list item 2021-02-16 08:07:31 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:53:00 UTC

Description rlopez 2021-01-13 14:00:33 UTC
Created attachment 1747075 [details]
Ironic Conductor log

The latest Dell iDRAC firmware (04.40.00.00) fails to power up node when attempting an install of IPI on BM.

Error seen:

2021-01-07 22:13:29.424 1 DEBUG sushy.exceptions [req-678b18e8-64cf-4d09-a2ac-b7d6ac2293b5 bootstrap-user - - - -] HTTP response for GET https://r640-c12u09-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService: status code: 401, error: Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid., extended: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}] __init__ /usr/lib/python3.6/site-packages/sushy/exceptions.py:121^[[00m
Version:

Nightly version="4.7.0-0.nightly-2020-12-21-131655"

Attaching Ironic conductor log.

Comment 1 Dmitry Tantsur 2021-01-13 14:05:17 UTC
Note that the authentication error is a red herring (fixed in https://opendev.org/openstack/sushy/commit/e730987a186be68de9555dfef66661f68b10c57a), unless followed by another error.

Comment 2 Dmitry Tantsur 2021-01-13 14:43:25 UTC
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector [req-57135e8a-0e48-4c72-93a6-c1e929271aaa - - - - -] Unable to start managed inspection for node ad9003ec-ebbe-4902-b2a7-eeaca8380426: 'idrac': KeyError: 'idrac'
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last):
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/inspector.py", line 204, in _start_managed_inspection
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector     task.driver.boot.prepare_ramdisk(task, ramdisk_params=params)
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 447, in prepare_ramdisk
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector     mode, d_info)
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 479, in prepare_deploy_iso
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector     return prepare_iso_image()
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 318, in _prepare_iso_image
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector     img_handler = ImageHandler(task.node.driver)
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 69, in __init__
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector     self._container = self._SWIFT_MAP[driver].get("container")
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector KeyError: 'idrac'

You seem to be using old images, could you check if there are newer ones? Both problems have been fixed already.

Comment 3 rlopez 2021-01-14 13:50:48 UTC
Hey Dmitry, Let me re-run the test and give you the latest log. It's likely I accidently uploaded ironic log from one of the other tests.

Comment 4 rlopez 2021-01-14 14:47:52 UTC
Created attachment 1747445 [details]
Ironic Conductor log 2021-01-13

Comment 5 rlopez 2021-01-14 14:52:15 UTC
2021-01-14 14:34:21.998 1 ERROR sushy.connector [req-243e82b5-9a6d-490a-afea-81169c0caec4 bootstrap-user - - - -] Authentication error detected. Cannot proceed: HTTP GET https://r640-c12u07-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService returned code 401. Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid. Extended information: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}]: sushy.exceptions.AccessError: HTTP GET https://r640-c12u07-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService returned code 401. Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid. Extended information: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}]^[[00m
2021-01-14 14:34:21.998 1 DEBUG sushy.resources.



from openshift install log


time="2021-01-14T14:41:12Z" level=error msg="Error: could not inspect: could not inspect node, node is currently 'inspect failed', last error was 'Inspection clean up failed: unable to clean up ramdisk boot: 'str' object has no attribute 'get''"
time="2021-01-14T14:41:12Z" level=error
time="2021-01-14T14:41:12Z" level=error msg="  on ../../tmp/openshift-install-812982727/masters/main.tf line 1, in resource \"ironic_node_v1\" \"openshift-master-host\":"
time="2021-01-14T14:41:12Z" level=error msg="   1: resource \"ironic_node_v1\" \"openshift-master-host\" {"
time="2021-01-14T14:41:12Z" level=error
time="2021-01-14T14:41:12Z" level=error
time="2021-01-14T14:41:12Z" level=error

Comment 6 rlopez 2021-01-14 14:57:00 UTC
I want to add that this latest build allow the systems to power up which I didn't experience with the 4.7.0-0.nightly-2020-12-21-131655

Comment 7 Dmitry Tantsur 2021-01-19 15:46:52 UTC
The failure:

2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector [req-a7a4d983-bbd1-4b21-922d-9230e10412d8 - - - - -] Unable to clean up ramdisk boot for node bfaf3069-bc79-495e-a8a2-8d7d6ffab4a2: AttributeError: 'str' object has no attribute 'get'
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last):
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/inspector.py", line 112, in _tear_down_managed_boot
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     task.driver.boot.clean_up_ramdisk(task)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 477, in clean_up_ramdisk
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     _eject_vmedia(task, sushy.VIRTUAL_MEDIA_CD)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 206, in _eject_vmedia
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     v_media.eject_media()
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/resources/manager/virtual_media.py", line 105, in eject_media
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     self._conn.post(target_uri)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/connector.py", line 212, in post
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     **extra_session_req_kwargs)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/connector.py", line 118, in _op
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     exceptions.raise_for_response(method, url, response)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 170, in raise_for_response
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     raise HTTPError(method, url, response)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 113, in __init__
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     index = self._get_most_severe_msg_index(ext_info)
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 129, in _get_most_severe_msg_index
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector     if m.get('Severity') == sev:
2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector AttributeError: 'str' object has no attribute 'get'

We need to make sure raising exceptions does not raise itself. Unfortunately, this hides the real cause.

Comment 9 Brian J. Beaudoin 2021-01-19 20:28:43 UTC
Seen after a failed 4.6 install when using `redfish-virtualmedia`. If the installation is not cleaned up properly, one needs to access the iDRAC under Configuration->Virtual Media and press "Disconnect" under "Remote File Share". Once disconnected, the Ironic inspector is able to set and connect the remote media properly and boot the system.

Comment 15 Tomas Sedovic 2021-01-20 09:30:54 UTC
Setting target release to 4.7. We are still investigating the root cause and that will determine whether we'll be able to resolve it in that time frame.

The current workaround is to downgrade the firmware from 4.40.00.00 to an earlier version. Roger Lopez has seen the following ones to work: 4.32.10.00, 4.22.00.53, 4.22.00.00.

Comment 16 Iury Gregory Melo Ferreira 2021-01-20 14:18:19 UTC
Roger, Pablo

I've pushed a patch to sushy that would prevent the exception from occurring and would give some logs that would help us understand the problem.
Can you use the latest FW and apply the sushy patch in the deployment so we can get the results?
Thank you!

Comment 20 Lubov 2021-02-03 08:36:45 UTC
Verified on 4.7.0-0.nightly-2021-02-01-180932

Comment 25 errata-xmlrpc 2021-02-24 15:52:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.