Description of problem: The OpenShift Infrastructure Operator (assisted installer) currently cannot provision Supermicro X12 based systems. It does power up the system, but fails during the discovery ISO attach stage. Relevant component information and versioning: Supermicro Chassis SYS-210P-FRDN6T Supermicro Board X12SPM-LN6TF Supermicro Firmware Version 1.00.03 Supermicro Firmware Build Time 04/21/2021 Supermicro Redfish Version 1.8.0 Supermicro BIOS Firmware Version 1.1 Supermicro BIOS Build Time 04/29/2021 Supermicro SFT-DCMS-SINGLE license applied Red Hat OpenShift version 4.8.2 RHCOS 48.84.202107202156-0 Kernel 4.18.0-305.10.2.el8_4.x86_64 assisted-service-operator.v99.0.0-unreleased hive-operator.v1.1.9 openshift-gitops-operator.v1.1.2 How reproducible: Every time Steps to Reproduce: 1. Install Assisted Installer Operator, Hive, onto a 4.8 cluster as the hub or management cluster 2. Create an InfraEnv, AgentClusterInstall, ClusterDeployment, and BMH host files contingent for an OCP 4.8.2 deployment spoke cluster including Supermicro X12 system 3. Observe failure message for X12 system after kicking off install process Actual results: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Registered 111s metal3-baremetal-controller Registered new host Normal InspectionSkipped 100s metal3-baremetal-controller disabled by annotation Normal ProfileSet 100s metal3-baremetal-controller Hardware profile set: unknown Normal BMCAccessValidated 100s metal3-baremetal-controller Verified access to BMC Normal PowerOn 99s metal3-baremetal-controller Host powered on Normal ProvisioningStarted 45s metal3-baremetal-controller Image provisioning started for https://assisted-service-assisted-installer.apps.volt.cars.lab/api/assisted-install/v1/clusters/07245160-7133-4f5d-ae93-6d0d51a4c84b/downloads/image?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbHVzdGVyX2lkIjoiMDcyNDUxNjAtNzEzMy00ZjVkLWFlOTMtNmQwZDUxYTRjODRiIn0.zsbNAS89TsmXRhcggyfieepPNHVV7BaDIN0SLjiJPVeMecOiO6SjNqF0fD_2uR0xmnhkqBV0CS6Pw3S7WRN3Fw Normal ProvisioningError 25s metal3-baremetal-controller Image provisioning failed: Failed to deploy. Exception: HTTP POST https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The property Inserted is not in the list of valid properties for the resource. Extended information: [{'MessageId': 'Base.1.4.PropertyUnknown', 'Severity': 'Warning', 'Resolution': 'Remove the unknown property from the request body and resubmit the request if the operation failed.', 'Message': 'The property Inserted is not in the list of valid properties for the resource.', 'MessageArgs': ['Inserted'], 'RelatedProperties': ['Inserted']}, {'MessageId': 'Base.1.4.PropertyUnknown', 'Severity': 'Warning', 'Resolution': 'Remove the unknown property from the request body and resubmit the request if the operation failed.', 'Message': 'The property WriteProtected is not in the list of valid properties for the resource.', 'MessageArgs': ['WriteProtected'], 'RelatedProperties': ['WriteProtected']}, {'MessageId': 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': 'Correct the value for the property in the request body and resubmit the request if the operation failed.', 'Message': 'The value http://10.40.0.123:6180/redfish/boot-6cdaa3ba-2d1b-4b66-933f-c3a719bee062.iso?filename=tmptt81btxv.iso for the property Image is of a different format than the property can accept.', 'MessageArgs': ['http://10.40.0.123:6180/redfish/boot-6cdaa3ba-2d1b-4b66-933f-c3a719bee062.iso?filename=tmptt81btxv.iso', 'Image'], 'RelatedProperties': ['[`~#$%&*()=+{}| \t;"\',<>?]']}] $ cat 04-bmh-tesla-cars-lab.yaml --- apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: du1-ldc1-tesla-cars-lab namespace: assisted-installer labels: infraenvs.agent-install.openshift.io: "infraenv-ran-skylark-cars-lab" annotations: inspect.metal3.io: disabled bmac.agent-install.openshift.io/role: worker spec: online: true bootMACAddress: 3c:ec:ef:30:52:64 automatedCleaningMode: disabled bmc: address: redfish-virtualmedia://172.28.11.42/redfish/v1/Systems/1/ credentialsName: bmc-du1-ldc1-tesla-cars-lab disableCertificateVerification: True Expected results: Successful provisioning with Redfish, like with other hardware OEM platforms: https://docs.openshift.com/container-platform/4.8/installing/installing_bare_metal_ipi/ipi-install-installation-workflow.html#ipi-install-configuration-files Additional info: Let me know if you need anything specific, happy to provide.
This is caused by mismatch in sushy (which is trying to set Inserted property while attaching vMedia) and SuperMicro BMC (which treats Inserted and WriteProtected attributes as read only). Looking into this.
Fix candidate is under review (see external tracker link). I believe we will need a minor Ironic change to match as well, I will look into this as well.
Adding reference to proposed Ironic change.
It looks like we may need to split this into backportable and non-backportable components. Changes are up, added extra links as well as the upstream story requested in reviews.
I tested https://review.opendev.org/c/openstack/sushy/+/802690/5 on Dell R640, HP e910 and SuperMicro X11 and all tests passed. Waiting for upstream reviews.
There has been a report that https://review.opendev.org/c/openstack/sushy/+/802690/5 breaks virtual media on Lenovo (model number SD530 I think). I uploaded https://review.opendev.org/c/openstack/sushy/+/802690/6 with a fix that should enable sushy to support both. Re-tested successfully on Dell/HP/Supermicro, requested a fellow upstream contributor who has access to Lenovo SD530 to test.
https://review.opendev.org/c/openstack/sushy/+/802690/6 has merged into master. I will start looking into backports now.
Backport is in CI and up for reviews https://review.opendev.org/c/openstack/sushy/+/803197/
Both master and stable/wallaby changes have merged. Will post an update when ironic-image including the fix is available.
https://github.com/openshift/ironic-image/pull/204 has now merged.
Hi, we don't have supermicro machines. Can this bz be verified on your site, please?
I'm happy to test if I can be provided a method to do so in (preferred) OpenShift 4.8.
Unfortunately I cannot verify this bz: have no suprmicro machine Closing as OtherQA
I have a SuperMicro system that I plan to add to my testing pipeline today to try to verify this. (Not sure if I should change bug status for this...?)
(In reply to Jeff Uphoff from comment #24) > I have a SuperMicro system that I plan to add to my testing pipeline today > to try to verify this. I'd appreciate it :) > > (Not sure if I should change bug status for this...?) Feel free to return it to on-qa
I didn't get this verified before I had to turn my hardware over to someone else for some testing work. I'll pick this back up once I have access to the hardware again.
https://bugzilla.redhat.com/show_bug.cgi?id=2003035 is the backport request, but we cannot proceed until this patch is verified. Could someone please do it?
Hello Jeff, Are you able to verified this BZ while leveraging Dave's HW? Thanks, Bertrand
Perhaps? I'd need info on where it is, IPs, BMC info, etc. to try adding it to our Jenkins pipeline.
In https://bugzilla.redhat.com/show_bug.cgi?id=2003035 it was confirmed that 4.9 works (while 4.8 does not), so I'm marking this as verified.
Now that this BZ has been verified, are we clear to have it backported? Where are we tracking the OCP 4.8 Backport? Do we need a BZ tracking this Backport in OCP 4.8?
@Bertand we are tracking 4.8 in https://bugzilla.redhat.com/show_bug.cgi?id=2003035 Dmitry marked the bug as verified, but I got an answer from Dave via e-mail today and we need to discuss if the validation was sufficient.
Thanks Lury. I was expecting the same Summary between 2003035 (OCP 4.9) and 1986238 (OCP 4.8). All good.
(In reply to Bertrand from comment #64) > Thanks Lury. > > I was expecting the same Summary between 2003035 (OCP 4.9) and 1986238 (OCP > 4.8). > > All good. Self correcting for the record: 2003035 (OCP 4.8.z) and 1986238 (OCP 4.9)
Unfortunately, this is not working for the Supermicro X12 using this backported fix. :( I see a different error message this time: Normal ProvisioningError 15s metal3-baremetal-controller Image provisioning failed: Failed to deploy. Exception: HTTP POST https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The value http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso for the property Image is of a different format than the property can accept. Extended information: [{'MessageId': 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': 'Correct the value for the property in the request body and resubmit the request if the operation failed.', 'Message': 'The value http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso for the property Image is of a different format than the property can accept.', 'MessageArgs': ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d.iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties': ['[`~#$%&*()=+{}| \t;"\',<>?]']}] I have tried two different BMC firmware versions here. Perhaps it doesn't like the '?'? It is worth mentioning that the ISO does mount to a Supermicro X11 system.
(In reply to Dave Cain from comment #66) > Unfortunately, this is not working for the Supermicro X12 using this > backported fix. :( > > I see a different error message this time: > > Normal ProvisioningError 15s metal3-baremetal-controller Image > provisioning failed: Failed to deploy. Exception: HTTP POST > https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/ > VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The > value > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > than the property can accept. Extended information: [{'MessageId': > 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': > 'Correct the value for the property in the request body and resubmit the > request if the operation failed.', 'Message': 'The value > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > than the property can accept.', 'MessageArgs': > ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties': > ['[`~#$%&*()=+{}| \t;"\',<>?]']}] > > I have tried two different BMC firmware versions here. Perhaps it doesn't > like the '?'? > > It is worth mentioning that the ISO does mount to a Supermicro X11 system. May be separate bug on this error, WDYT? Assigning the bz qa on you
(In reply to Lubov from comment #67) > (In reply to Dave Cain from comment #66) > > Unfortunately, this is not working for the Supermicro X12 using this > > backported fix. :( > > > > I see a different error message this time: > > > > Normal ProvisioningError 15s metal3-baremetal-controller Image > > provisioning failed: Failed to deploy. Exception: HTTP POST > > https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/ > > VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The > > value > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > > than the property can accept. Extended information: [{'MessageId': > > 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': > > 'Correct the value for the property in the request body and resubmit the > > request if the operation failed.', 'Message': 'The value > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > > than the property can accept.', 'MessageArgs': > > ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties': > > ['[`~#$%&*()=+{}| \t;"\',<>?]']}] > > > > I have tried two different BMC firmware versions here. Perhaps it doesn't > > like the '?'? > > > > It is worth mentioning that the ISO does mount to a Supermicro X11 system. > > May be separate bug on this error, WDYT? > > Assigning the bz qa on you I discussed the problem described in the error message with Dave in real time and we've done some additional investigation on the X12 machine. Based on the outcomes of this investigation I am fairly confident that: 1) the original issue is resolved (otherwise we wouldn't move on to the next issue which this error describes) 2) the error message in the comment I'm replying to is a new issue. Dave can you please open a new BZ to cover the overly restrictive validation of the virtual media URL in SuperMicro X12?
(In reply to Jacob Anders from comment #68) > (In reply to Lubov from comment #67) > > (In reply to Dave Cain from comment #66) > > > Unfortunately, this is not working for the Supermicro X12 using this > > > backported fix. :( > > > > > > I see a different error message this time: > > > > > > Normal ProvisioningError 15s metal3-baremetal-controller Image > > > provisioning failed: Failed to deploy. Exception: HTTP POST > > > https://172.28.11.42/redfish/v1/Managers/1/VirtualMedia/CD1/Actions/ > > > VirtualMedia.InsertMedia returned code 400. Base.v1_4_0.GeneralError: The > > > value > > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > > > than the property can accept. Extended information: [{'MessageId': > > > 'Base.1.4.PropertyValueFormatError', 'Severity': 'Warning', 'Resolution': > > > 'Correct the value for the property in the request body and resubmit the > > > request if the operation failed.', 'Message': 'The value > > > http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > > iso?filename=tmp1p2l6hew.iso for the property Image is of a different format > > > than the property can accept.', 'MessageArgs': > > > ['http://10.40.0.122:6180/redfish/boot-dc5f752f-314b-4cea-a93c-85f008f5ce4d. > > > iso?filename=tmp1p2l6hew.iso', 'Image'], 'RelatedProperties': > > > ['[`~#$%&*()=+{}| \t;"\',<>?]']}] > > > > > > I have tried two different BMC firmware versions here. Perhaps it doesn't > > > like the '?'? > > > > > > It is worth mentioning that the ISO does mount to a Supermicro X11 system. > > > > May be separate bug on this error, WDYT? > > > > Assigning the bz qa on you > > I discussed the problem described in the error message with Dave in real > time and we've done some additional investigation on the X12 machine. Based > on the outcomes of this investigation I am fairly confident that: > 1) the original issue is resolved (otherwise we wouldn't move on to the next > issue which this error describes) > 2) the error message in the comment I'm replying to is a new issue. > > Dave can you please open a new BZ to cover the overly restrictive validation > of the virtual media URL in SuperMicro X12? Dave, do you agree with my assessment as per the previous comment? I wanted to make sure we're all on the same page so that we can work towards closing this bug. Adding a needinfo.
+1 to open a new BZ with the new information.
Yes, agree with you Jacob. The original issue surfaced in this BZ appears to be addressed (RedFish VirtualMedia.InsertMedia request). I will open a new BZ for the URL problems exhibited in https://bugzilla.redhat.com/show_bug.cgi?id=1986238#c66. Thanks much!
Thank you Dave!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759