Bug 2000442

Summary: SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia
Product: OpenShift Container Platform Reporter: Marius Cornea <mcornea>
Component: Bare Metal Hardware ProvisioningAssignee: Derek Higgins <derekh>
Bare Metal Hardware Provisioning sub component: ironic QA Contact: Amit Ugol <augol>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: high CC: achernet, alosadag, aos-bugs, augol, derekh, lshilin, mavazque, rbartal, tsedovic, yjoseph, zbitter
Version: 4.8Keywords: AutomationBlocker, Triaged
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AI-Team-Platform
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1983190 Environment:
Last Closed: 2021-09-21 08:02:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2000445    
Bug Blocks:    

Description Marius Cornea 2021-09-02 07:41:31 UTC 
This is a clone to keep track of the 4.8 backport.

+++ This bug was initially created as a clone of Bug #1983190 +++

Description of problem:

SNO deployment on HPE e910 blades fails because the node always boots from virtualmedia. After the image is written to disk and the node reboots it keeps booting from virtualmedia.


Pasting below the efibootmgr entries before and after the image was written to drive:

before image was written to drive and reboot:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0017* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0018* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (HTTP(S) IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0019* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv4)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot001A* Slot 2 Port 1 : Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 (PXE IPv6)    PciRoot(0x3)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(d4f5ef433b1c,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.


after the image was written:

[root@localhost core]# efibootmgr -v
BootCurrent: 0014
Timeout: 20 seconds
BootOrder: 0015,0014,000A,000F,0011,0012,0010,0017,0018,0019,001A,000D,000E,000B,000C,0001,0002,0003,0004,0005,0006,0007,0008,0009,0000,0013
Boot0000* System Utilities  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(1fd631e5-44e0-2f91-10ab-f88f3568ef30)
Boot0001  Embedded UEFI Shell   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0002  Diagnose Error    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0849279d-40d5-53ea-e764-2496766f9844)
Boot0003  Intelligent Provisioning  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(4a433501-ddaa-490b-96b2-04f42d8669b8)
Boot0004  Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(d3fd6286-43c5-bb8d-0793-07b70aa9de36)
Boot0005  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(0334f316-c276-49f6-9879-aaf1ecffa5d5)
Boot0006  View Integrated Management Log    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(93c92423-d1c6-4286-be67-b76b6671047e)
Boot0007  HTTP Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(3f770860-3d63-4803-9ea3-df37144ab546)
Boot0008  PXE Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(ee8b26b0-37e9-11e1-b86c-0800200c9a66)
Boot0009  Embedded Diagnostics  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(b57fe6f1-4f49-d46e-4bba-0a8add34d2f3)
Boot000A* Generic USB Boot  UsbClass(ffff,ffff,255,255)
Boot000B* Embedded NVMe M.2 Drive 1 : NVM Express Controller - 704F715AFQD1-KXG60ZNV512G TOSHIBA-58EE38C    PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-05-00-39-26-28)N.....YM....R,Y.
Boot000C* Embedded NVMe M.2 Drive 2 : NVM Express Controller - 702Y102KYRX1-KXG60ZNV512G TOSHIBA-18EE38C    PciRoot(0x0)/Pci(0x1b,0x2)/Pci(0x0,0x0)/NVMe(0x1,8C-E3-8E-01-00-09-8C-71)N.....YM....R,Y.
Boot000D* NVMe M.2 Drive Riser 5 Bay 1 : NVM Express Controller - S436NA0NA20424-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000E* NVMe M.2 Drive Riser 5 Bay 2 : NVM Express Controller - S436NA0NA20412-SAMSUNG MZ1LB1T9HALS-00007-0   PciRoot(0x1)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot000F* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y.
Boot0010* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (HTTP(S) IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y.
Boot0011* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv4)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0012* Embedded LOM 1 Port 1 : Intel(R) I210 Gigabit  Backplane Connection (PXE IPv6)    PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)/MAC(b47af1ae9d26,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0013* iLO Virtual USB 3 : iLO Virtual CD-ROM    PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)N.....YM....R,Y.
Boot0014* Red Hat Enterprise Linux  PciRoot(0x0)/Pci(0x1c,0x4)/Pci(0x0,0x4)/USB(1,0)/CDROM(1,0x221,0xd2a)/File(\EFI\redhat\shimx64.efi)
Boot0015* Red Hat Enterprise Linux  HD(2,GPT,3781ea19-6db6-4174-a202-1faae2eed326,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)


Version-Release number of selected component (if applicable):
2.3.0-DOWNSTREAM-2021-07-16-03-33-34

How reproducible:
100%

Steps to Reproduce:
1. Deploy SNO on HPE e910 server blade

Actual results:
The deployment doesn't succeed because the node always boots from virtualmedia.

Expected results:
After image is written to drive and the node reboots, it boots from the internal drive not from virtualmedia.

Additional info:
Comment 2 Raviv Bar-Tal 2021-09-12 11:29:24 UTC 
Hey Marius, could you, please, verify this?
Comment 3 Marius Cornea 2021-09-13 07:36:33 UTC 
(In reply to Raviv Bar-Tal from comment #2)
> Hey Marius, could you, please, verify this?

Hey Raviv. Sure, I'll verify this BZ together with BZ#2000445 once that gets ON_QA as well. Leaving the needinfo set for me until then.
Comment 5 Marius Cornea 2021-09-16 11:47:28 UTC 
Verified on 4.8.0-0.nightly-2021-09-16-055919
Comment 7 errata-xmlrpc 2021-09-21 08:02:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.12 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3511
Comment 8 Christoffer Back 2021-10-07 07:55:58 UTC 
*** Bug 2008538 has been marked as a duplicate of this bug. ***