Created attachment 1747075 [details] Ironic Conductor log The latest Dell iDRAC firmware (04.40.00.00) fails to power up node when attempting an install of IPI on BM. Error seen: 2021-01-07 22:13:29.424 1 DEBUG sushy.exceptions [req-678b18e8-64cf-4d09-a2ac-b7d6ac2293b5 bootstrap-user - - - -] HTTP response for GET https://r640-c12u09-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService: status code: 401, error: Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid., extended: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}] __init__ /usr/lib/python3.6/site-packages/sushy/exceptions.py:121^[[00m Version: Nightly version="4.7.0-0.nightly-2020-12-21-131655" Attaching Ironic conductor log.
Note that the authentication error is a red herring (fixed in https://opendev.org/openstack/sushy/commit/e730987a186be68de9555dfef66661f68b10c57a), unless followed by another error.
2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector [req-57135e8a-0e48-4c72-93a6-c1e929271aaa - - - - -] Unable to start managed inspection for node ad9003ec-ebbe-4902-b2a7-eeaca8380426: 'idrac': KeyError: 'idrac' 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last): 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/inspector.py", line 204, in _start_managed_inspection 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector task.driver.boot.prepare_ramdisk(task, ramdisk_params=params) 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 447, in prepare_ramdisk 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector mode, d_info) 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 479, in prepare_deploy_iso 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector return prepare_iso_image() 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 318, in _prepare_iso_image 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector img_handler = ImageHandler(task.node.driver) 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_utils.py", line 69, in __init__ 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector self._container = self._SWIFT_MAP[driver].get("container") 2021-01-07 22:48:45.630 1 ERROR ironic.drivers.modules.inspector KeyError: 'idrac' You seem to be using old images, could you check if there are newer ones? Both problems have been fixed already.
Hey Dmitry, Let me re-run the test and give you the latest log. It's likely I accidently uploaded ironic log from one of the other tests.
Created attachment 1747445 [details] Ironic Conductor log 2021-01-13
2021-01-14 14:34:21.998 1 ERROR sushy.connector [req-243e82b5-9a6d-490a-afea-81169c0caec4 bootstrap-user - - - -] Authentication error detected. Cannot proceed: HTTP GET https://r640-c12u07-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService returned code 401. Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid. Extended information: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}]: sushy.exceptions.AccessError: HTTP GET https://r640-c12u07-drac.cloud.lab.eng.bos.redhat.com/redfish/v1/SessionService returned code 401. Base.1.0.GeneralError: The authentication credentials included with this request are missing or invalid. Extended information: [{'@odata.type': '#Message.v1_0_0.Message', 'MessageId': 'Base.1.2.AccessDenied', 'Message': 'The authentication credentials included with this request are missing or invalid.', 'MessageArgs': [], 'MessageArgs': 0, 'RelatedProperties': [], 'RelatedProperties': 0, 'Severity': 'Critical', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.'}]^[[00m 2021-01-14 14:34:21.998 1 DEBUG sushy.resources. from openshift install log time="2021-01-14T14:41:12Z" level=error msg="Error: could not inspect: could not inspect node, node is currently 'inspect failed', last error was 'Inspection clean up failed: unable to clean up ramdisk boot: 'str' object has no attribute 'get''" time="2021-01-14T14:41:12Z" level=error time="2021-01-14T14:41:12Z" level=error msg=" on ../../tmp/openshift-install-812982727/masters/main.tf line 1, in resource \"ironic_node_v1\" \"openshift-master-host\":" time="2021-01-14T14:41:12Z" level=error msg=" 1: resource \"ironic_node_v1\" \"openshift-master-host\" {" time="2021-01-14T14:41:12Z" level=error time="2021-01-14T14:41:12Z" level=error time="2021-01-14T14:41:12Z" level=error
I want to add that this latest build allow the systems to power up which I didn't experience with the 4.7.0-0.nightly-2020-12-21-131655
The failure: 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector [req-a7a4d983-bbd1-4b21-922d-9230e10412d8 - - - - -] Unable to clean up ramdisk boot for node bfaf3069-bc79-495e-a8a2-8d7d6ffab4a2: AttributeError: 'str' object has no attribute 'get' 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last): 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/inspector.py", line 112, in _tear_down_managed_boot 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector task.driver.boot.clean_up_ramdisk(task) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 477, in clean_up_ramdisk 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector _eject_vmedia(task, sushy.VIRTUAL_MEDIA_CD) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/redfish/boot.py", line 206, in _eject_vmedia 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector v_media.eject_media() 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/resources/manager/virtual_media.py", line 105, in eject_media 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector self._conn.post(target_uri) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/connector.py", line 212, in post 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector **extra_session_req_kwargs) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/connector.py", line 118, in _op 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector exceptions.raise_for_response(method, url, response) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 170, in raise_for_response 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector raise HTTPError(method, url, response) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 113, in __init__ 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector index = self._get_most_severe_msg_index(ext_info) 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/sushy/exceptions.py", line 129, in _get_most_severe_msg_index 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector if m.get('Severity') == sev: 2021-01-14 14:41:09.602 1 ERROR ironic.drivers.modules.inspector AttributeError: 'str' object has no attribute 'get' We need to make sure raising exceptions does not raise itself. Unfortunately, this hides the real cause.
Seen after a failed 4.6 install when using `redfish-virtualmedia`. If the installation is not cleaned up properly, one needs to access the iDRAC under Configuration->Virtual Media and press "Disconnect" under "Remote File Share". Once disconnected, the Ironic inspector is able to set and connect the remote media properly and boot the system.
Setting target release to 4.7. We are still investigating the root cause and that will determine whether we'll be able to resolve it in that time frame. The current workaround is to downgrade the firmware from 4.40.00.00 to an earlier version. Roger Lopez has seen the following ones to work: 4.32.10.00, 4.22.00.53, 4.22.00.00.
Roger, Pablo I've pushed a patch to sushy that would prevent the exception from occurring and would give some logs that would help us understand the problem. Can you use the latest FW and apply the sushy patch in the deployment so we can get the results? Thank you!
Verified on 4.7.0-0.nightly-2021-02-01-180932
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633