Bug 1983190

Summary: SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia
Product: OpenShift Container Platform Reporter: Marius Cornea <mcornea>
Component: Bare Metal Hardware ProvisioningAssignee: Derek Higgins <derekh>
Bare Metal Hardware Provisioning sub component: baremetal-operator QA Contact: Amit Ugol <augol>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: high CC: achernet, alosadag, aos-bugs, derekh, dphillip, lshilin, mavazque, tsedovic, yjoseph, zbitter
Version: 4.8Keywords: AutomationBlocker, Triaged
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AI-Team-Platform
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1997993 2000442 (view as bug list) Environment:
Last Closed: 2021-10-18 17:39:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1997993, 2000445    

Description Marius Cornea 2021-07-16 19:05:30 UTC 
Description of problem:

SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia. After the image is written to disk and the node reboots it keeps booting from virtualmedia.


Pasting below the efibootmgr entries before and after the image was written to drive:

before image was written to drive and reboot:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0017* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0018* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0019* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot001A* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.


after the image was written:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0015,0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0015* Red Hat Enterprise Linux  HD(2,GPT,3781ea19-6db6-4174-a202-1faae2eed326,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)


Version-Release number of selected component (if applicable):
2.3.0-DOWNSTREAM-2021-07-16-03-33-34

How reproducible:
100%

Steps to Reproduce:
1. Deploy SNO on HPE e910 server blade

Actual results:
The deployment doesn't succeed because the node always boots from virtualmedia.

Expected results:
After image is written to drive and the node reboots, it boots from the internal drive not from virtualmedia.

Additional info:
Comment 6 Steven Hardy 2021-08-06 15:45:00 UTC 
Related upstream PR https://github.com/metal3-io/baremetal-operator/pull/949 pushed since WIP https://github.com/openshift/baremetal-operator/pull/171 was proven to work around this problem.

When that merges we'll need a downstream backport, and a corresponding CBO PR to set LIVE_ISO_FORCE_PERSISTENT_BOOT_DEVICE

I'm not yet sure if we'll also need to make that configurable via the provisioning CR though (there are some use-cases for persistently booting appliance ISO images via metal3, which may require the existing behavior).
Comment 7 Marius Cornea 2021-08-13 08:24:44 UTC 
Note that I hit the same issue on a Dell R640 machine where I managed to workaround it by setting the `Hard-disk Drive Placeholder` Boot Setting to Disabled.
Comment 16 Marius Cornea 2021-09-01 14:54:34 UTC 
Verified on 4.9.0-0.nightly-2021-08-31-123131
Comment 19 errata-xmlrpc 2021-10-18 17:39:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759