Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2009662

Summary: RHCOS fails to boot from fiber channel device
Product: OpenShift Container Platform Reporter: Roman Hodain <rhodain>
Component: RHCOSAssignee: Jonathan Lebon <jlebon>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Michael Nguyen <mnguyen>
Severity: low Docs Contact:
Priority: medium    
Version: 4.8CC: dornelas, jlebon, jligon, miabbott, mrussell, nstielau
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2011776 (view as bug list) Environment:
Last Closed: 2022-01-10 14:48:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
installTime-journal none

Description Roman Hodain 2021-10-01 08:04:23 UTC 
Created attachment 1827928 [details]
installTime-journal

OCP Version at Install Time:4.8
RHCOS Version at Install Time:4.8.2
Platform: bare metal
Architecture: x86_64

What are you trying to do? What is your use case?
Install RHCOS on a LUN provided over FCP

What happened? What went wrong or what did you expect?
The initial installation went fine:

mpathconf --enable && systemctl start multipathd.service
coreos-installer install /dev/disk/by-id/wwn-0x600c0ff00050b9fd628b4d6101000000 --append-karg rd.multipath=default --append-karg root=/dev/disk/by-label/dm-mpath-root --ignition-url http://FQDN:8080/bootstrap.ign  --insecure-ignition

The journal output is:
 
     installTime-journal

Finished successfully and the system was rebooted. 
First reboot:
   I used the FC device as the boot option and the system started to boot, but it ended up in the emergency shell. I suppose because of:

     Sep 30 13:17:59 localhost coreos-gpt-setup[1084]: Randomizing disk GUID
     Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: Warning: The kernel is still using the old partition table.
     Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: The new table will be used at the next reboot or after you
     Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: run partprobe(8) or kpartx(8)
     Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: The operation has completed successfully.
     Sep 30 13:20:00 localhost systemd[1]: coreos-gpt-setup.service: Main process exited, code=exited, status=1/FAILURE
     Sep 30 13:20:00 localhost systemd[1]: coreos-gpt-setup.service: Failed with result 'exit-code'.
     Sep 30 13:20:00 localhost systemd[1]: coreos-gpt-setup.service: Triggering OnFailure= dependencies.

The relates logs are:

     AfterFirstBoot-journal
     AfterFirstBoot-lsblk
     AfterFirstBoot-multipathd_status
     AfterFirstBoot-multipath-ll

Further reboots:
  All additional reboots end up in a loop without getting into the emergency shell. The error is:

    [  450.943611] systemctl[13018]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.

The related log is:
     LastReboot-serialLineOut
Comment 6 Jonathan Lebon 2021-10-01 14:19:11 UTC 
Looks like the error here is:

```
Sep 30 13:17:59 localhost coreos-gpt-setup[1084]: Randomizing disk GUID
Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: Warning: The kernel is still using the old partition table.
Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: The new table will be used at the next reboot or after you
Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: run partprobe(8) or kpartx(8)
Sep 30 13:18:00 localhost coreos-gpt-setup[1084]: The operation has completed successfully.
Sep 30 13:18:59 localhost systemd-udevd[546]: seq 2897 '/devices/pci0000:00/0000:00:01.0/0000:01:00.1/host9/rport-9:0-6/target9:0:4/9:0:4:0/block/sdn' is taking a long time
Sep 30 13:18:59 localhost systemd-udevd[546]: seq 2898 '/devices/pci0000:00/0000:00:01.0/0000:01:00.1/host9/rport-9:0-8/target9:0:6/9:0:6:0/block/sdq' is taking a long time
Sep 30 13:20:00 localhost systemd[1]: coreos-gpt-setup.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 13:20:00 localhost systemd[1]: coreos-gpt-setup.service: Failed with result 'exit-code'.
Sep 30 13:20:00 localhost systemd[1]: Failed to start Generate New UUID For Boot Disk GPT.
```

Which I think is from the `udevadm settle` failing after the 2 minute default timeout. Hmm, I think we should just ignore `udevadm settle` events in general. Submitted a patch for that. There's a good chance that that patch alone may fix the boot, but we can leave it open until it makes it to an RHCOS build you can test. At least we'll get a better error if that's not the case.

Looking at the logs more carefully though... am I reading correctly that there are 23 paths total for 4 different block devices? That's a nice multipathd stress test. :)
Comment 8 Jonathan Lebon 2021-10-07 16:17:25 UTC 
We'll try to have a 4.10 build soon you can test with.
Comment 9 Jonathan Lebon 2021-10-14 13:25:28 UTC 
The latest RHCOS 4.10 development builds have the patch above. Internally, you can access it via the release browser:

https://releases-rhcos-art.cloud.privileged.psi.redhat.com/

Here are externally-accessible links for the artifacts of the latest 4.10 build there (410.84.202110140201-0):

https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.10/410.84.202110140201-0/x86_64/rhcos-410.84.202110140201-0-live.x86_64.iso
https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.10/410.84.202110140201-0/x86_64/rhcos-410.84.202110140201-0-live-kernel-x86_64
https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.10/410.84.202110140201-0/x86_64/rhcos-410.84.202110140201-0-live-initramfs.x86_64.img
https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.10/410.84.202110140201-0/x86_64/rhcos-410.84.202110140201-0-live-rootfs.x86_64.img

To be clear, it's possible there is still an issue, but at least we should get a better error message.
Comment 10 Jonathan Lebon 2021-11-29 16:06:50 UTC 
@rhodain Any updates on this? Otherwise, I'll close the issue for now.
Comment 11 Jonathan Lebon 2022-01-10 14:48:08 UTC 
No updates. Closing.
Comment 12 Red Hat Bugzilla 2023-09-15 01:15:47 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days