Bug 1928250

Summary: Ironic unable to set boot mode on HPE Gen9 on OSP16.1
Product: Red Hat OpenStack Reporter: alink
Component: openstack-ironicAssignee: Steve Baker <sbaker>
Status: CLOSED ERRATA QA Contact: Paras Babbar <pbabbar>
Severity: medium Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: msecaur, pbabbar, pweeks, rhos-maint, sbaker, slinaber, xili
Target Milestone: z6Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-ironic-13.0.7-1.20210407082057.3d77e61.el8ost python-sushy-2.0.3-1.20210406233537.0241cd9.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1944409 (view as bug list) Environment:
Last Closed: 2021-05-26 13:51:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 6 Steve Baker 2021-03-03 19:31:34 UTC
It would also be useful to enable debug logging on the ironic-conductor log. This will show the actual request and response being made in the PATCH call.

Debug logging is enabled with /etc/ironic/ironic.conf [DEFAULT] debug = True

Comment 13 Steve Baker 2021-03-04 20:39:22 UTC
This is the change which removed the Content-Type header[1] is unrelated and didn't explain why it was deleted. It is recommended to always set a Content-Type on requests, I'll propose adding it back upstream and I'll start the process of backporting it to 16.1 and 16.2, but if you need this change in a short time frame you'll likely need to ask for a hotfix.

[1] https://review.opendev.org/c/openstack/sushy/+/607809

Comment 14 Matthew Secaur 2021-03-04 20:46:37 UTC
Steve,

Is there a way that we can confirm that this will fix the issue? After I posted the output, I realized that it starts with the word "error". Is this what we expect?

{"error":{"@Message.ExtendedInfo":[{"MessageId":"Base.0.10.Success"}],"code":"iLO.0.10.ExtendedInfo","message":"See @Message.ExtendedInfo for more information."}}
  ^^^^^
  |||||

Is this normal?

Comment 15 Steve Baker 2021-03-04 21:26:19 UTC
Its unusual to package a success message inside an error attribute, but I've traced the code calls from ironic to sushy and nothing actually parses the content of the response, it is assumed to succeed if it returns a 200 status code.

Comment 16 Matthew Secaur 2021-03-04 21:32:47 UTC
Steve,

This has all been incredibly helpful. Thanks!

Comment 17 Matthew Secaur 2021-03-04 21:59:08 UTC
Steve,

Weren't we already getting a 200 status code in comment #8? The additional header changed the output, but the status code was the same. I suppose what we really need to know is if the ILO board actually *processed* the command, not really what status code it gives.

Am I way off base here?

Comment 18 Steve Baker 2021-03-04 22:47:02 UTC
comment #8 included the Content-Type and the response was a success. The ironic packaged in 16.1 is not setting the Content-Type header in any calls, and it looks like this is why Gen9 servers are returning an error.

It would still be useful to get the ironic-conductor debug logs to clearly show the exact request and response.

Comment 23 Steve Baker 2021-03-09 01:16:46 UTC
OK, I think I see what is happening now.

In 13.x (queens) Ironic had no concept of setting the boot mode (uefi vs legacy bios). In 16.x this is now supported and the redfish driver will send a request to set the boot mode regardless of whether the redfish BMC supports that.

It looks like HPE Gen9 doesn't support setting the boot mode through redfish. There is a mechanism in Ironic to silently ignore a failure to set the boot mode when the Ironic *driver* doesn't support it, but this isn't triggered when the BMC lacks support.

Let me talk to my team about the best way to handle this.

Comment 26 Steve Baker 2021-03-21 19:50:44 UTC
The upstream fix is now merged, I can now proceed with backports to 16.1

Comment 29 Steve Baker 2021-04-07 00:25:50 UTC
The python-sushy changes are now in built package python-sushy-2.0.3-1.20210406233537.0241cd9.el8

Comment 32 Steve Baker 2021-04-07 20:54:33 UTC
*** Bug 1893927 has been marked as a duplicate of this bug. ***

Comment 44 errata-xmlrpc 2021-05-26 13:51:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097

Comment 45 Red Hat Bugzilla 2023-09-15 01:32:50 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days