Bug 2104275

Summary: Supermicro server FirmwareSchema CR does not contain allowable_values, attribute_type and read_only flag
Product: OpenShift Container Platform Reporter: tali <tali>
Component: Bare Metal Hardware ProvisioningAssignee: Dmitry Tantsur <dtantsur>
Bare Metal Hardware Provisioning sub component: ironic QA Contact: Jad Haj Yahya <jhajyahy>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: janders, lshilin
Version: 4.11Keywords: Triaged
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-17 19:51:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description tali@redhat.com 2022-07-05 20:37:07 UTC
Description of problem:

The FirmwareSchema CR generated for a Supermicro server
does not contain allowable_values, attribute_type and read_only flag. Hence BMO could not properly validate the user supplied attribute settings.

The schema Spec should contain the following fields for each attribute. 
spec:
  schema:
    AcpiHpet:
      allowable_values:
      - Enabled
      - Disabled
      attribute_type: Enumeration
      read_only: false

These fields are not populated for a Supermicro server. 
apiVersion: metal3.io/v1alpha1
kind: FirmwareSchema
metadata:
  creationTimestamp: "2022-07-05T18:27:13Z"
  generation: 1
  name: schema-2ab2e563
  namespace: cnfde11
  ownerReferences:
  - apiVersion: metal3.io/v1alpha1
    kind: HostFirmwareSettings
    name: cnfde11.ptp.lab.eng.bos.redhat.com
    uid: b4c319db-9434-4711-a1e3-52d9755dd0d9
  resourceVersion: "5107971"
  uid: 1ae853ae-5515-4733-a762-78d7445804e0
spec:
  schema:
    2xRefresh: {}
    ACSControl: {}
    AES-NI: {}
    ARISupport: {}
    ATS: {}
    Above4GDecoding: {}
    AdjacentCachePrefetch: {}
    AggressiveLinkPowerManagement: {}
    AggressiveLinkPowerManagement$2: {}
    AutonomousCoreC-State: {}
    Bitspersecond: {}
    Bitspersecond$2: {}
    Bitspersecond$3: {}
    BootOption#1$3: {}
    BootOption#1$4: {}
    BootOption#1$5: {}
    BootOption#1$6: {}
    BootOption#1$7: {}
    BootOption#1$8: {}
    BootOption#2$3: {}
    BootOption#2$4: {}
    BootOption#3$3: {}
    BootOption#3$4: {}
    BootOption#4$3: {}
    BootOption#4$4: {}
    BootOption#5$3: {}
    BootOption#5$4: {}
    BootOption#6$3: {}
    BootOption#6$4: {}
    BootOption#7$3: {}
    BootOption#7$4: {}
    BootOption#8$3: {}
    BootOption#8$4: {}
    BootOption#9$2: {}
    BootOption#10: {}
    BootOption#11: {}
    BootOption#12: {}
    BootOption#13: {}
    BootOption#14: {}
    BootOption#15: {}
    BootOption#16: {}
    BootOption#17: {}
    Bootmodeselect: {}
    BootupNumLockState: {}
    BusMasterEnable: {}
    CPUC6report: {}
    ChangeSettings: {}
    ChangeSettings$2: {}
    CoherencySupport(Non-Isoch): {}
    ConfigTDP: {}
    ConfigureSATAas: {}
    ConfiguresSATAas: {}
    ConsoleRedirection: {}
    ConsoleRedirection$2: {}
    ConsoleRedirection$3: {}
    CoresEnabled: {}
    CorrectableErrorThreshold: {}
    DCUIPPrefetcher: {}
    DCUStreamerPrefetcher: {}
    DataBits: {}
    DataBits$2: {}
    DataScramblingforDDR4: {}
    DegradePrecedence: {}
    DisableTPH: {}
    EISTPSDFunction: {}
    ENERGY_PERF_BIAS_CFGmode: {}
    EVDFXFeatures: {}
    EndorsementHierarchy: {}
    EnforcePOR: {}
    EnhancedHaltState(C1E): {}
    EnhancedPPR: {}
    EraseEventLog: {}
    ExecuteDisableBit: {}
    ExtendedAPIC: {}
    FlowControl: {}
    FlowControl$2: {}
    FlowControl$3: {}
    HTTPBootOneTime: {}
    HardwareP-States: {}
    HardwarePrefetcher: {}
    HighPrecisionEventTimer: {}
    HotPlug: {}
    HotPlug$2: {}
    HotPlug$3: {}
    HotPlug$4: {}
    HotPlug$5: {}
    HotPlug$6: {}
    HotPlug$7: {}
    HotPlug$8: {}
    HotPlug$9: {}
    HotPlug$10: {}
    HotPlug$11: {}
    HotPlug$12: {}
    HotPlug$13: {}
    HotPlug$14: {}
    Hyper-Threading[ALL]: {}
    IMCInterleaving: {}
    INT19TrapResponse: {}
    IODirectoryCache(IODC): {}
    IOU0(IIOPCIeBr1): {}
    IOU1(IIOPCIeBr2): {}
    IOU2(IIOPCIeBr3): {}
    IPv4HTTPSupport: {}
    IPv4PXESupport: {}
    IPv6HTTPSupport: {}
    IPv6PXESupport: {}
    InstallWindows7USBSupport: {}
    Intel®VMDforVolumeManagementDeviceforPStack0: {}
    Intel®VMDforVolumeManagementDeviceforPStack1: {}
    Intel®VMDforVolumeManagementDeviceforPStack2: {}
    Intel®VTforDirectedI/O(VT-d): {}
    IntelRunSure: {}
    IntelVirtualizationTechnology: {}
    InterruptRemapping: {}
    IsocMode: {}
    KTIPrefetch: {}
    LEGACYtoEFIsupport: {}
    LLCDeadLineAlloc: {}
    LLCPrefetch: {}
    LegacyOSRedirectionResolution: {}
    LegacyOSRedirectionResolution$2: {}
    LegacySerialRedirectionPort: {}
    LegacyUSBSupport: {}
    LinkL0pEnable: {}
    LinkL1Enable: {}
    LinkSpeed: {}
    LinkSpeed$2: {}
    Local/RemoteThreshold: {}
    LogSystemBootEvent: {}
    M.2PCI-E3.0X4OPROM: {}
    MECI: {}
    METW: {}
    MMCFGBase: {}
    MMIOHighBase: {}
    MMIOHighGranularitySize: {}
    MaximumReadRequest: {}
    Mediadetectcount: {}
    MemoryFrequency: {}
    MemoryRankSparing: {}
    Mirrormode: {}
    Monitor/Mwait: {}
    NVMeFirmwareSource: {}
    NetworkStack: {}
    OnboardLAN1OptionROM: {}
    OnboardLANDevice: {}
    OnboardVideoOptionROM: {}
    OperationMode: {}
    OptionROMMessages: {}
    Out-of-BandMgmtPort: {}
    PCI-ECompletionTimeoutDisable: {}
    PCI-EPortMaxPayloadSize: {}
    PCI-EPortMaxPayloadSize$2: {}
    PCIePLLSSC: {}
    PHRandomization: {}
    PPINControl: {}
    PPRType: {}
    PXEbootwaittime: {}
    PackageCState: {}
    PagePolicy: {}
    Parity: {}
    Parity$2: {}
    PassThroughDMA: {}
    PatrolScrub: {}
    PatrolScrubInterval: {}
    PendingOperation: {}
    PlatformHierarchy: {}
    Port60/64Emulation: {}
    Port61hBit-4Emulation: {}
    PostedInterrupt: {}
    PostedInterruptThrottle: {}
    PowerButtonFunction: {}
    PowerPerformanceTuning: {}
    PowerTechnology: {}
    PrioritizeTPH: {}
    PuttyKeyPad: {}
    PuttyKeyPad$2: {}
    QuietBoot: {}
    RSC-R1UW-2E16SLOT1PCI-EX16OPROM: {}
    RSC-R1UW-2E16SLOT2PCI-EX16OPROM: {}
    Re-tryBoot: {}
    RecorderMode: {}
    RecorderMode$2: {}
    RedirectionAfterBIOSPOST: {}
    RedirectionAfterBIOSPOST$2: {}
    RelaxedOrdering: {}
    ResetKeysType: {}
    Resolution100x31: {}
    Resolution100x31$2: {}
    RestoreonACPowerLoss: {}
    SATAController: {}
    SATADeviceType: {}
    SATADeviceType$2: {}
    SATADeviceType$3: {}
    SATADeviceType$4: {}
    SATADeviceType$5: {}
    SATADeviceType$6: {}
    SATADeviceType$7: {}
    SATADeviceType$8: {}
    SATAHDDUnlock: {}
    SATAHDDUnlock$2: {}
    SDDCPlusOne: {}
    SHA256PCRBank: {}
    SMBIOSEventLog: {}
    SMCIBIOS-BasedTPMProvisionSupport: {}
    SNC: {}
    SR-IOVSupport: {}
    SecureBoot: {}
    SecureBootMode: {}
    SecurityDeviceSupport: {}
    SerialPort1: {}
    SerialPort2: {}
    SerialPort2Attribute: {}
    SoftwareControlledT-States: {}
    SpeedStep(P-States): {}
    SpinUpDevice: {}
    SpinUpDevice$2: {}
    SpinUpDevice$3: {}
    SpinUpDevice$4: {}
    SpinUpDevice$5: {}
    SpinUpDevice$6: {}
    SpinUpDevice$7: {}
    SpinUpDevice$8: {}
    SpinUpDevice$9: {}
    SpinUpDevice$10: {}
    SpinUpDevice$11: {}
    SpinUpDevice$12: {}
    SpinUpDevice$13: {}
    SpinUpDevice$14: {}
    StaleAtoS: {}
    StaticVirtualLockstepMode: {}
    StopBits: {}
    StopBits$2: {}
    StorageHierarchy: {}
    TXTSupport: {}
    TerminalType: {}
    TerminalType$2: {}
    TerminalType$3: {}
    TurboMode: {}
    VGAPriority: {}
    VT-UTF8ComboKeySupport: {}
    VT-UTF8ComboKeySupport$2: {}
    WHEASupport: {}
    WaitFor"F1"IfError: {}
    WatchDogFunction: {}
    WhenLogisFull: {}
    XHCIHand-off: {}
    XPTPrefetch: {}
    sSATAController: {}
    sSATADeviceType: {}
    sSATADeviceType$2: {}
    sSATADeviceType$3: {}
    sSATADeviceType$4: {}
    sSATADeviceType$5: {}
    sSATADeviceType$6: {}
    tCCD_LRelaxation: {}
    tRFCOptimizationfor16GbBasedDIMM: {}

The following invalid setting was not detected.
apiVersion: metal3.io/v1alpha1
kind: HostFirmwareSettings
metadata:
    name: "cnfde11.ptp.lab.eng.bos.redhat.com"
    namespace: "cnfde11"
spec:
  settings:
    PowerButtonFunction: "test string"

Version-Release number of selected component (if applicable):
- Latest upstream assisted-service-operator
- OCP 4.11 on hub (4.11.0-fc.3)
- 4.10 spoke


How reproducible:
100%

Steps to Reproduce:
1. Deploy OCP 4.11 hub with upstream assisted-service-operator
2. Try to deploy spoke using manually created CRs including a HostFirmwareSettings CR with an invalid setting

Actual results:
HostFirmwareSettings validation passed.

Expected results:
"Invalid BIOS setting" event should be generated.

Additional info:

Comment 3 tali@redhat.com 2022-07-05 20:48:31 UTC
The must-gather is available: https://drive.google.com/file/d/1ssNvPHkQZX8_eLwUdfQNy4mNgCxAs4Gl/view?usp=sharing

Comment 4 Dmitry Tantsur 2022-07-26 12:08:14 UTC
From what I can see, Ironic cannot find the BIOS attribute registry, thus the schema only contains the currently available fields. Could you please cURL your BMC to verify:

 curl -k https://<BMC IP>/redfish/v1/

If the JSON output contains a Registry link, could you follow it further?

Comment 5 tali@redhat.com 2022-07-26 13:39:41 UTC
Here are the info you requested:

curl -ksu ADMIN:ADMIN https://10.16.231.98/redfish/v1/ | jq .
{
  "@odata.type": "#ServiceRoot.v1_5_2.ServiceRoot",
  "@odata.id": "/redfish/v1/",
  "Id": "RootService",
  "Name": "Root Service",
  "RedfishVersion": "1.8.0",
  "UUID": "00000000-0000-0000-0000-3CECEF59834C",
  "Systems": {
    "@odata.id": "/redfish/v1/Systems"
  },
  "Chassis": {
    "@odata.id": "/redfish/v1/Chassis"
  },
  "Managers": {
    "@odata.id": "/redfish/v1/Managers"
  },
  "Tasks": {
    "@odata.id": "/redfish/v1/TaskService"
  },
  "SessionService": {
    "@odata.id": "/redfish/v1/SessionService"
  },
  "AccountService": {
    "@odata.id": "/redfish/v1/AccountService"
  },
  "EventService": {
    "@odata.id": "/redfish/v1/EventService"
  },
  "UpdateService": {
    "@odata.id": "/redfish/v1/UpdateService"
  },
  "CertificateService": {
    "@odata.id": "/redfish/v1/CertificateService"
  },
  "Registries": {
    "@odata.id": "/redfish/v1/Registries"
  },
  "JsonSchemas": {
    "@odata.id": "/redfish/v1/JsonSchemas"
  },
  "Links": {
    "Sessions": {
      "@odata.id": "/redfish/v1/SessionService/Sessions"
    }
  },
  "Oem": {
    "Supermicro": {}
  }
}

curl -ksu ADMIN:ADMIN https://10.16.231.98/redfish/v1/Registries | jq
{
  "@odata.type": "#MessageRegistryFileCollection.MessageRegistryFileCollection",
  "@odata.id": "/redfish/v1/Registries",
  "Name": "Registry File Collection",
  "Description": "Registry Repository",
  "Members": [
    {
      "@odata.id": "/redfish/v1/Registries/BiosAttributeRegistry.v1_0_0"
    },
    {
      "@odata.id": "/redfish/v1/Registries/Base.v1_4_0"
    },
    {
      "@odata.id": "/redfish/v1/Registries/Event.v1_0_0"
    },
    {
      "@odata.id": "/redfish/v1/Registries/SMC.v1_0_0"
    }
  ],
  "Members": 4
}


curl -ksu ADMIN:ADMIN https://10.16.231.98/redfish/v1/Registries/BiosAttributeRegistry.v1_0_0 | jq .
{
  "@odata.type": "#MessageRegistryFile.v1_1_3.MessageRegistryFile",
  "@odata.id": "/redfish/v1/Registries/BiosAttributeRegistry.v1_0_0",
  "Id": "BiosAttributeRegistry.v1_0_0",
  "@Redfish.Copyright": "Copyright 2014-2019 DMTF. All rights reserved.",
  "Name": "BIOS Attribute Registry File",
  "Description": "BIOS Attribute Registry File locations",
  "Languages": [
    "en"
  ],
  "Registry": "BiosAttributeRegistry.1.0.0",
  "Location": [
    {
      "Language": "en",
      "Uri": "/registries/BiosAttributeRegistry.1.0.0.json"
    }
  ],
  "Oem": {}
}

Comment 6 Dmitry Tantsur 2022-07-26 14:21:32 UTC
Thank you, appreciated? Could go one step further and fetch (and attach) /registries/BiosAttributeRegistry.1.0.0.json please?

Comment 7 tali@redhat.com 2022-07-26 15:10:07 UTC
It looks like the uri is incorrect. SuperMicro support page says "it is a isolated issue on the 1.73.10 release". This serve is on Firmware Revision: 01.73.12. Let's see if we can upgrade it to a newer version.

wget --no-check-certificate  --user ADMIN --password ADMIN https://10.16.231.98/redfish/v1/Registries/BiosAttributeRegistry.1.0.0.json
--2022-07-26 11:00:14--  https://10.16.231.98/redfish/v1/Registries/BiosAttributeRegistry.1.0.0.json
Connecting to 10.16.231.98:443... connected.
WARNING: cannot verify 10.16.231.98's certificate, issued by ‘CN=IPMI,OU=Software,O=Super Micro Computer,L=San Jose,ST=California,C=US’:
  Self-signed certificate encountered.
    WARNING: certificate common name ‘IPMI’ doesn't match requested host name ‘10.16.231.98’.
HTTP request sent, awaiting response... 404 Not Found
2022-07-26 11:00:15 ERROR 404: Not Found.


curl -ksu ADMIN:ADMIN https://10.16.231.98/redfish/v1/Registries/BiosAttributeRegistry.1.0.0.json
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
 </body>
</html>

Comment 8 Dmitry Tantsur 2022-07-28 11:33:49 UTC
Hmm, I checked your machine, and apparently this URL is correct: https://10.16.231.98/registries/BiosAttributeRegistry.1.0.0.json (note lower-case "registries" and no /redfish/v1 prefix). I will now check if we build the absolute URL correctly.

Comment 9 Dmitry Tantsur 2022-07-28 11:45:29 UTC
I think see the issue. The Bios resource references registry BiosAttributeRegistry.v1_0_0, which exists, but its identity is BiosAttributeRegistry.1_0_0 (note the missing "v"). This confuses our code. This is probably something we can work around, although I wonder what their logic behind it was..

Comment 10 tali@redhat.com 2022-07-28 12:17:33 UTC
Right, the location uri is correct. Good catch!:-)

Comment 12 Dmitry Tantsur 2022-08-24 13:10:06 UTC
Would you be able to verify this bug on 4.12? I'm afraid our QE may not have the same hardware.

Comment 13 tali@redhat.com 2022-08-24 13:14:17 UTC
Yes, I will verify the fix on 4.12.

Comment 15 Dmitry Tantsur 2022-08-26 08:44:01 UTC
Okay, we have another issue then: https://review.opendev.org/c/openstack/ironic/+/854760

Comment 17 Dmitry Tantsur 2022-08-31 10:07:16 UTC
Hi! The second fix will be available in the next accepted build, could you please test it again?

Comment 18 tali@redhat.com 2022-09-06 14:48:28 UTC
I was informed that there were some issues with OLM in OCP 4.12. 
I will wait for the OLM team to merge the fix prior to installing 4.12 latest.

Comment 19 Dmitry Tantsur 2022-09-14 08:49:12 UTC
Hi, have there been any progress with testing? Note that you don't necessarily need to do a complete installation, you only need to enroll a node and verify that the schema is correct.

Comment 20 tali@redhat.com 2022-09-21 00:48:50 UTC
I was able to test the fix by patching the ironic image on a 4.11 hub cluster. This problem has been fixed.

apiVersion: metal3.io/v1alpha1
kind: FirmwareSchema
metadata:
  creationTimestamp: "2022-09-21T00:40:40Z"
  generation: 1
  name: schema-f7afa37b
  namespace: cnfde11
  ownerReferences:
  - apiVersion: metal3.io/v1alpha1
    kind: HostFirmwareSettings
    name: cnfde11.ptp.lab.eng.bos.redhat.com
    uid: 7e44cf2f-a294-47cc-b5ee-0315fa2e9d4a
  resourceVersion: "36820971"
  uid: f6058489-2918-4cea-9b44-416945232a0a
spec:
  schema:
    2xRefresh:
      allowable_values:
      - Auto
      - Enable
      attribute_type: Enumeration
      read_only: false
    ACSControl:
      allowable_values:
      - Enable
      - Disable
      attribute_type: Enumeration
      read_only: false
    AES-NI:
      allowable_values:
      - Disable
      - Enable
      attribute_type: Enumeration
      read_only: false
    ARISupport:
      allowable_values:
      - Disabled
      - Enabled
      attribute_type: Enumeration
      read_only: false
    ATS:
      allowable_values:
      - Enable
      - Disable
      attribute_type: Enumeration
      read_only: false
    Above4GDecoding:
      allowable_values:
      - Disabled
      - Enabled
      attribute_type: Enumeration
    ...

Comment 23 errata-xmlrpc 2023-01-17 19:51:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399