Bug 1986238

This is caused by mismatch in sushy (which is trying to set Inserted property while attaching vMedia) and SuperMicro BMC (which treats Inserted and WriteProtected attributes as read only). Looking into this.

Fix candidate is under review (see external tracker link).

I believe we will need a minor Ironic change to match as well, I will look into this as well.

Adding reference to proposed Ironic change.

It looks like we may need to split this into backportable and non-backportable components. Changes are up, added extra links as well as the upstream story requested in reviews.

I tested https://review.opendev.org/c/openstack/sushy/+/802690/5 on Dell R640, HP e910 and SuperMicro X11 and all tests passed. 

Waiting for upstream reviews.

There has been a report that https://review.opendev.org/c/openstack/sushy/+/802690/5 breaks virtual media on Lenovo (model number SD530 I think).

I uploaded https://review.opendev.org/c/openstack/sushy/+/802690/6 with a fix that should enable sushy to support both. Re-tested successfully on Dell/HP/Supermicro, requested a fellow upstream contributor who has access to Lenovo SD530 to test.

https://review.opendev.org/c/openstack/sushy/+/802690/6 has merged into master. I will start looking into backports now.

Backport is in CI and up for reviews https://review.opendev.org/c/openstack/sushy/+/803197/

Both master and stable/wallaby changes have merged. Will post an update when ironic-image including the fix is available.

https://github.com/openshift/ironic-image/pull/204 has now merged.

Hi, we don't have supermicro machines. Can this bz be verified on your site, please?

I'm happy to test if I can be provided a method to do so in (preferred) OpenShift 4.8.

Unfortunately I cannot verify this bz: have no suprmicro machine
Closing as OtherQA

I have a SuperMicro system that I plan to add to my testing pipeline today to try to verify this.

(Not sure if I should change bug status for this...?)

(In reply to Jeff Uphoff from comment #24)
> I have a SuperMicro system that I plan to add to my testing pipeline today
> to try to verify this.

I'd appreciate it :)

> 
> (Not sure if I should change bug status for this...?)

Feel free to return it to on-qa

I didn't get this verified before I had to turn my hardware over to someone else for some testing work. I'll pick this back up once I have access to the hardware again.

https://bugzilla.redhat.com/show_bug.cgi?id=2003035 is the backport request, but we cannot proceed until this patch is verified. Could someone please do it?

Hello Jeff,

Are you able to verified this BZ while leveraging Dave's HW?

Thanks,

Bertrand

Perhaps? I'd need info on where it is, IPs, BMC info, etc. to try adding it to our Jenkins pipeline.

In https://bugzilla.redhat.com/show_bug.cgi?id=2003035 it was confirmed that 4.9 works (while 4.8 does not), so I'm marking this as verified.

Now that this BZ has been verified, are we clear to have it backported?

Where are we tracking the OCP 4.8 Backport? 

Do we need a BZ tracking this Backport in OCP 4.8?

@Bertand we are tracking 4.8 in https://bugzilla.redhat.com/show_bug.cgi?id=2003035

Dmitry marked the bug as verified, but I got an answer from Dave via e-mail today and we need to discuss if the validation was sufficient.

Thanks Lury. 

I was expecting the same Summary between 2003035 (OCP 4.9)  and 1986238 (OCP 4.8). 

All good.

(In reply to Bertrand from comment #64)
> Thanks Lury. 
> 
> I was expecting the same Summary between 2003035 (OCP 4.9)  and 1986238 (OCP
> 4.8). 
> 
> All good.

Self correcting for the record: 2003035 (OCP 4.8.z)  and 1986238 (OCP 4.9)

Unfortunately, this is not working for the Supermicro X12 using this backported fix.  :(

I see a different error message this time:

  Normal  ProvisioningError    15s   metal3-baremetal-controller  Image provisioning failed: Failed to deploy. Exception: HTTP POST https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The value http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso for the property Image is of a different format than the property can accept. Extended information: [{'MessageId': 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': 'Correct the value for the property in the request body and resubmit the request if the operation failed.', 'Message': 'The value http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso for the property Image is of a different format than the property can accept.', 'MessageArgs': ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties': ['[`~#$%&*()=+{}| \t;"\',<>?]']}]

I have tried two different BMC firmware versions here.  Perhaps it doesn't like the '?'?

It is worth mentioning that the ISO does mount to a Supermicro X11 system.

(In reply to Dave Cain from comment #66)
> Unfortunately, this is not working for the Supermicro X12 using this
> backported fix.  :(
> 
> I see a different error message this time:
> 
>   Normal  ProvisioningError    15s   metal3-baremetal-controller  Image
> provisioning failed: Failed to deploy. Exception: HTTP POST
> https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/
> VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The
> value
> http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> than the property can accept. Extended information: [{'MessageId':
> 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution':
> 'Correct the value for the property in the request body and resubmit the
> request if the operation failed.', 'Message': 'The value
> http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> than the property can accept.', 'MessageArgs':
> ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties':
> ['[`~#$%&*()=+{}| \t;"\',<>?]']}]
> 
> I have tried two different BMC firmware versions here.  Perhaps it doesn't
> like the '?'?
> 
> It is worth mentioning that the ISO does mount to a Supermicro X11 system.

May be separate bug on this error, WDYT? 

Assigning the bz qa on you

(In reply to Lubov from comment #67)
> (In reply to Dave Cain from comment #66)
> > Unfortunately, this is not working for the Supermicro X12 using this
> > backported fix.  :(
> > 
> > I see a different error message this time:
> > 
> >   Normal  ProvisioningError    15s   metal3-baremetal-controller  Image
> > provisioning failed: Failed to deploy. Exception: HTTP POST
> > https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/
> > VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The
> > value
> > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> > than the property can accept. Extended information: [{'MessageId':
> > 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution':
> > 'Correct the value for the property in the request body and resubmit the
> > request if the operation failed.', 'Message': 'The value
> > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> > than the property can accept.', 'MessageArgs':
> > ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties':
> > ['[`~#$%&*()=+{}| \t;"\',<>?]']}]
> > 
> > I have tried two different BMC firmware versions here.  Perhaps it doesn't
> > like the '?'?
> > 
> > It is worth mentioning that the ISO does mount to a Supermicro X11 system.
> 
> May be separate bug on this error, WDYT? 
> 
> Assigning the bz qa on you

I discussed the problem described in the error message with Dave in real time and we've done some additional investigation on the X12 machine. Based on the outcomes of this investigation I am fairly confident that:
1) the original issue is resolved (otherwise we wouldn't move on to the next issue which this error describes)
2) the error message in the comment I'm replying to is a new issue.

Dave can you please open a new BZ to cover the overly restrictive validation of the virtual media URL in SuperMicro X12?

(In reply to Jacob Anders from comment #68)
> (In reply to Lubov from comment #67)
> > (In reply to Dave Cain from comment #66)
> > > Unfortunately, this is not working for the Supermicro X12 using this
> > > backported fix.  :(
> > > 
> > > I see a different error message this time:
> > > 
> > >   Normal  ProvisioningError    15s   metal3-baremetal-controller  Image
> > > provisioning failed: Failed to deploy. Exception: HTTP POST
> > > https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/
> > > VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The
> > > value
> > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> > > than the property can accept. Extended information: [{'MessageId':
> > > 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution':
> > > 'Correct the value for the property in the request body and resubmit the
> > > request if the operation failed.', 'Message': 'The value
> > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format
> > > than the property can accept.', 'MessageArgs':
> > > ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.
> > > iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties':
> > > ['[`~#$%&*()=+{}| \t;"\',<>?]']}]
> > > 
> > > I have tried two different BMC firmware versions here.  Perhaps it doesn't
> > > like the '?'?
> > > 
> > > It is worth mentioning that the ISO does mount to a Supermicro X11 system.
> > 
> > May be separate bug on this error, WDYT? 
> > 
> > Assigning the bz qa on you
> 
> I discussed the problem described in the error message with Dave in real
> time and we've done some additional investigation on the X12 machine. Based
> on the outcomes of this investigation I am fairly confident that:
> 1) the original issue is resolved (otherwise we wouldn't move on to the next
> issue which this error describes)
> 2) the error message in the comment I'm replying to is a new issue.
> 
> Dave can you please open a new BZ to cover the overly restrictive validation
> of the virtual media URL in SuperMicro X12?

Dave, do you agree with my assessment as per the previous comment? I wanted to make sure we're all on the same page so that we can work towards closing this bug. Adding a needinfo.

+1 to open a new BZ with the new information.

Yes, agree with you Jacob.  

The original issue surfaced in this BZ appears to be addressed (RedFish VirtualMedia.InsertMedia request).  I will open a new BZ for the URL problems exhibited in https://bugzilla.redhat.com/show_bug.cgi?id=1986238#c66.

Thanks much!

Thank you Dave!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759