Bug 1983190 - SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia
Summary: SNO deployment on HPE e910 blades fails because the node always boots from vi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.9.0
Assignee: Derek Higgins
QA Contact: Amit Ugol
URL:
Whiteboard: AI-Team-Platform
Depends On:
Blocks: 1997993 2000445
TreeView+ depends on / blocked
 
Reported: 2021-07-16 19:05 UTC by Marius Cornea
Modified: 2021-10-18 17:40 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1997993 2000442 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:39:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift baremetal-operator pull 173 0 None None None 2021-08-25 22:28:05 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:40:15 UTC

Description Marius Cornea 2021-07-16 19:05:30 UTC
Description of problem:

SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia. After the image is written to disk and the node reboots it keeps booting from virtualmedia.


Pasting below the efibootmgr entries before and after the image was written to drive:

before image was written to drive and reboot:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0017* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0018* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0019* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot001A* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.


after the image was written:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0015,0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0015* Red Hat Enterprise Linux  HD(2,GPT,3781ea19-6db6-4174-a202-1faae2eed326,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)


Version-Release number of selected component (if applicable):
2.3.0-DOWNSTREAM-2021-07-16-03-33-34

How reproducible:
100%

Steps to Reproduce:
1. Deploy SNO on HPE e910 server blade

Actual results:
The deployment doesn't succeed because the node always boots from virtualmedia.

Expected results:
After image is written to drive and the node reboots, it boots from the internal drive not from virtualmedia.

Additional info:

Comment 6 Steven Hardy 2021-08-06 15:45:00 UTC
Related upstream PR https://github.com/metal3-io/baremetal-operator/pull/949 pushed since WIP https://github.com/openshift/baremetal-operator/pull/171 was proven to work around this problem.

When that merges we'll need a downstream backport, and a corresponding CBO PR to set LIVE_ISO_FORCE_PERSISTENT_BOOT_DEVICE

I'm not yet sure if we'll also need to make that configurable via the provisioning CR though (there are some use-cases for persistently booting appliance ISO images via metal3, which may require the existing behavior).

Comment 7 Marius Cornea 2021-08-13 08:24:44 UTC
Note that I hit the same issue on a Dell R640 machine where I managed to workaround it by setting the `Hard-disk Drive Placeholder` Boot Setting to Disabled.

Comment 16 Marius Cornea 2021-09-01 14:54:34 UTC
Verified on 4.9.0-0.nightly-2021-08-31-123131

Comment 19 errata-xmlrpc 2021-10-18 17:39:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.