After running: coreos-installer install -i /home/core/config.ign /dev/mmcblk0 and reboot, the node fail to start due to: Failed to start Ignition OSTree: Mount (firstboot) /sysroot Displaying logs from failed units: ignition-ostree-mount-firstboot-sysroot.service ignition-ostree-mount-var.service ignition-ostree-uuid-boot.service rdma-load-modules -- Logs begin at Fri 2018-06-22 11:11:49 UTC, end at Fri 2018-06-22 11:12:37 UTC. -- Jun 22 11:12:18 systemd[1]: Starting Ignition OSTree: Mount (firstboot) /sysroot... Jun 22 11:12:18 systemd[1]: ignition-ostree-mount-firstboot-sysroot.service: Main process exited, code=exited, status=32/n/a Jun 22 11:12:18 systemd[1]: ignition-ostree-mount-firstboot-sysroot.service: Failed with result 'exit-code'. Jun 22 11:12:18 systemd[1]: Failed to start Ignition OSTree: Mount (firstboot) /sysroot. Jun 22 11:12:35 ignition-ostree-mount-sysroot[2952]: Mounting /dev/disk/by-label/root (/dev/mmcblk0p4) to /sysroot Jun 22 11:12:35 ignition-ostree-mount-sysroot[2952]: mount: /sysroot: /dev/mmcblk0p4 already mounted on /sysroot/sysroot/sysroot. I got emergency shell and this is what we have on the node: :/# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT mmcblk0 179:0 0 59.3G 0 disk |-mmcblk0p2 179:2 0 127M 0 part |-mmcblk0p3 179:3 0 384M 0 part `-mmcblk0p4 179:4 0 58.8G 0 part /sysroot/sysroot/sysroot/sysroot mmcblk0boot0 179:8 0 31.5M 1 disk mmcblk0boot1 179:16 0 31.5M 1 disk Seems to be related to: https://github.com/coreos/fedora-coreos-tracker/issues/735
We believe this is the same as the FCOS issue: https://github.com/coreos/fedora-coreos-tracker/issues/735 The fix for that I think was pulled into current RHCOS via: https://github.com/openshift/os/pull/566 (see https://github.com/openshift/os/commit/28363f94d9371a77210390f2852414c28886b620 ) However, our internal 4.9 disk images (for x86_64) have been failing to build, which is likely inbound to fix.
I think https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.9-aarch64&release=49.84.202106301747-0#49.84.202106301747-0 has the fix.
This seems to be due to a bad/ancient time coming from the RTC. Indeed I can observe the same kind of boot failure on plain x86_64 qemu by setting the virtual RTC to a date way in the past, e.g. via `-rtc base=2010-10-10T10:10:10`. I tried with an older 4.7 bootimage that I already had locally and the first boot does not succeed, showing the same error as report here. I then tried with the latest 4.9 build (49.84.202107010027-0-qemu.x86_64.qcow2) and the boot succeeds as follows: ``` [ 5.950909] ignition-ostree-firstboot-uuid[643]: e2fsck 1.45.6 (20-Mar-2020) [ 5.956960] ignition-ostree-firstboot-uuid[643]: Pass 1: Checking inodes, blocks, and sizes [ 5.961087] ignition-ostree-firstboot-uuid[643]: Pass 2: Checking directory structure [ 5.964030] ignition-ostree-firstboot-uuid[643]: Pass 3: Checking directory connectivity [ 5.965069] ignition-ostree-firstboot-uuid[643]: Pass 4: Checking reference counts [ 5.968044] ignition-ostree-firstboot-uuid[643]: Pass 5: Checking group summary information [ 5.973402] ignition-ostree-firstboot-uuid[643]: boot: 321/98304 files (0.3% non-contiguous), 125074/393216 blocks [ 6.011170] ignition-ostree-firstboot-uuid[643]: Detected timestamp of last fsck is older than timestamp of last mount. [ 6.012486] ignition-ostree-firstboot-uuid[643]: Setting /dev/disk/by-label/boot timestamp of last fsck to same time as last mount. [ 6.028737] ignition-ostree-firstboot-uuid[643]: tune2fs 1.45.6 (20-Mar-2020) [ 6.029622] ignition-ostree-firstboot-uuid[643]: Setting time filesystem last checked to Thu Jul 1 00:31:19 2021 [ 6.046929] ignition-ostree-firstboot-uuid[643]: tune2fs 1.45.6 (20-Mar-2020) ... # hwclock --show 2010-10-10 10:26:03.465488+00:00 # tune2fs -l /dev/disk/by-label/boot | grep 'Last checked' Last checked: Thu Jul 1 00:31:19 2021 ```
> I think https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.9-aarch64&release=49.84.202106301747-0#49.84.202106301747-0 has the fix. OK and indeed it should. The build metadata says it's from 180d484163e46818700897b8d741a611d653927c so: ``` walters@toolbox /v/s/w/s/r/redhat-coreos (master)> git reset --hard 180d484163e46818700897b8d741a611d653927c HEAD is now at 180d484 Merge branch 'pvc_req_bump' into 'master' walters@toolbox /v/s/w/s/r/redhat-coreos (master)> git submodule update --init --recursive walters@toolbox /v/s/w/s/r/redhat-coreos (master)> git grep --recurse 'timestamp of last fsck' openshift-coreos-config/fedora-coreos-config/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid: echo "Detected timestamp of last fsck is older than timestamp of last mount." openshift-coreos-config/fedora-coreos-config/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid: echo "Setting "${target}" timestamp of last fsck to same time as last mount." walters@toolbox /v/s/w/s/r/redhat-coreos (master)> ``` There is a newer build https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.9-aarch64&release=49.84.202106302247-0#49.84.202106302247-0 to try, but AFAICS one (or more) of these is true: - You somehow used e.g. the old ISO and didn't test that fix - You installed an older metal image and not the osmet version - There is another bug that somehow results in the same symptom - Our fix still has a race condition?
To cross-check we have the fix, can you try: `grep -e "timestamp of last fsck" /usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid` on the booted ISO, and an even stronger check is mounting the target disk after running `coreos-installer` and verifying the file is present there.
I think you are right and that I didn't test it with the right RHCOS, (iso timestamp seems old) Tried it again with: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.9-aarch64/49.84.202107012247-0/aarch64/rhcos-49.84.202107012247-0-live.aarch64.iso Live ISO: [root@localhost core]# grep -e "timestamp of last fsck" /usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid echo "Detected timestamp of last fsck is older than timestamp of last mount." echo "Setting "${target}" timestamp of last fsck to same time as last mount." [root@localhost core]# cat /proc/driver/rtc rtc_time : 11:07:04 rtc_date : 2021-07-04 alrm_time : 00:00:00 alrm_date : 1970-01-01 alarm_IRQ : no alrm_pending : no update IRQ enabled : no periodic IRQ enabled : no periodic IRQ frequency : 1 max user IRQ frequency : 64 24hr : yes Time : 11:7:4.000000000 Date : 2021-7-4 Daylight : 0 Timezone : unspecified Alarm Time : 0:0:0.000000000 Alarm Date : 0-0-0 Alarm Daylight : 0 Enabled : no Pending : no Timezone : unspecified Resolution : 0 Accuracy : 0 SetstoZero : 0 [root@localhost core]# hwclock --show 2021-07-04 11:07:04.167310+00:00 [root@localhost core]# sudo systemctl stop chronyd.service [root@localhost core]# sudo chronyd -q 'server clock.redhat.com iburst' 2021-07-04T11:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG) 2021-07-04T11:07:18Z System clock wrong by -7205.838446 seconds (step) 2021-07-04T09:07:12Z chronyd exiting [root@localhost core]# hwclock --show 2021-07-04 11:07:26.049459+00:00 [root@localhost core]# date Sun Jul 4 09:07:23 UTC 2021
It seems that with this rhcos version (rhcos-49.84.202107012247-0-live.aarch64.iso) the node fails to boot from disk post-reboot (reproduced 3/3) I don't get to the grub menu and just get UEFI shell I see this in the console output while writing to disk: [ 1429.853136] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null) [ 1430.108056] GPT:Primary header thinks Alt. header is not at the end of the disk. [ 1430.122952] GPT:7571455 != 124321791 [ 1430.130109] GPT:Alternate GPT header not at the end of the disk. [ 1430.142169] GPT:7571455 != 124321791 [ 1430.149326] GPT: Use GNU Parted to correct GPT errors. [ 1430.159649] mmcblk0: p2 p3 p4
Note that I see the above output also when I use rhcos-48.84.202106121219-0-live.aarch64.iso, but with that release, I do get the grub menu post-reboot. How can I mount the target disk after running `coreos-installer`? Simple `mount /dev/mmcblk0 /mnt/rhcos` doesn't seem to work Getting: mount: /var/mnt/rhcos: wrong fs type, bad option, bad superblock on /dev/mmcblk0, missing codepage or helper program, or other error.
OK, this seems to be the mount I want mount /dev/mmcblk0p4 /mnt/foo/
And I see the fix is there: [root@localhost core]# grep -e "timestamp of last fsck" /mnt/foo/ostree/deploy/rhcos/deploy/acadb454d7d12d1a0a7a584242c289a0c51fa20dc2bf9c776bd2a7e74cc79d62.0/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid echo "Detected timestamp of last fsck is older than timestamp of last mount." echo "Setting "${target}" timestamp of last fsck to same time as last mount." But still it will not boot, after reboot I get to the UEFI shell
> I think you are right and that I didn't test it with the right RHCOS, (iso timestamp seems old) OK. What's the current state? > after reboot I get to the UEFI shell What do you see in the UEFI shell? That sounds like a different problem.
Current state (after syncing the hwclock): 1. https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.8-aarch64/48.84.202106121219-0/aarch64/rhcos-48.84.202106121219-0-live.aarch64.iso - works, I can boot the live ISO, use the coreos-installer to install to disk and after reboot the node start with the HCOS from disk 2. Doesn't work: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.8-aarch64/48.84.202106171757-0/aarch64/rhcos-48.84.202106171757-0-live.aarch64.iso https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.9-aarch64/48.84.202106181749-0/aarch64/rhcos-48.84.202106181749-0-live.aarch64.iso https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.9-aarch64/49.84.202106260247-0/aarch64/rhcos-49.84.202106260247-0-live.aarch64.iso https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.9-aarch64/49.84.202107032247-0/aarch64/rhcos-49.84.202107032247-0-live.aarch64.iso I can boot the live ISO, but after writing to disk I just get UEFI shell. no information really. 3. When I used the rhcos-48.84.202106121219-0-live.aarch64.iso for installing OCP 4.9, The OCP updated the rhcos and it booted! [core@localhost ~]$ rpm-ostree status State: idle Deployments: ● pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c872933d8d97d8b672ae6c9d95d367fa22b4e3995d50284d60695128af3d12 CustomOrigin: Managed by machine-config-operator Version: 49.84.202106272247-0 (2021-06-27T22:55:43Z) ostree://c6daa86173667503d04e66994901ea9e2619764579c9479bc665c0f72cfd4ae4 Version: 48.84.202106121219-0 (2021-06-12T12:31:09Z) Can't say what's different.
With rhcos-48.84.202106121219-0-live.aarch64.iso, I still have the BF-2 failing to boot.
I tried it again with rhcos-48.84.202106121219-0-live.aarch64.iso and it's failing as well. I also tried the latest 4.9 rhcos and got the same result- UEFI interactive shell
Please be more precise in your comments. Saying that the boot failed does not help us determine **when** the boot failed. When you get an EFI shell, can you manually boot?
I can manually boot, and the first thing I see is UEFI shell
Tentatively moving to multi-arch team. I think we need someone well-versed in UEFI/aarch64 to have a look at this.
Can you try to boot up a standard RHEL 8.4 image to see if that works ? That way if the issue is with RHEL, we can ask our extended kernel-hw team which manages aarch64 to look at it. Would also be good to isolate whether this is an RHCOS vs RHEL issue.
Adding my experience with this: * Whatever image I've used (either 48.84 or 49.84), there doesn't seem to be any correct EFI configuration applied, so no grub menu is presented and the use ends up in a standard UEFI Shell, firmware version is "BlueField:3.7.0-20-g98daf29" * We tried keeping the clock in sync and there is no change in behavior; * by manually selecting and booting the kernel from the UEFI Shell, we managed to boot the OS and get the cluster installation to complete (I used BLK1:/BOOT/redhat/grubaa64.efi) * rebooting node after boot doesn't seem to change anything in the bootloaders behavior. Best, Costin
Note that RHCOS does not add boot entries to the EFI config, it relies on the default bootloader path to boot the system. This might be an option in the firmware or a firmware bug.
At this point it does look like an issue with the firmware. Rom, Can we take this back to NVIDIA and ask them to check on the firmware to see if it needs an update or needs to be fixed? Thanks Prashanth
in the rhcos image what are all the files in /boot/efi/ ?
inside the ISO image /boot/efi has: /"oot@localhost efi]# find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/ . |-EFI | |-BOOT | | |-BOOTAA64.EFI | | |-fbaa64.efi | |-redhat | | |-BOOTAA64.CSV | | |-fonts | | |-grub.cfg | | |-grubaa64.efi | | |-mmaa64.efi | | |-shim.efi | | |-shimaa64-redhat.efi | | |-shimaa64.efi
also in my case i tried a virt-install on an ARMv8 ampere system i have and after i do a coreos-install and reboot, i see: System BootOrder not found. Initializing defaults. Creating boot entry "Boot0001" with label "Red Hat Enterprise Linux" for file "\EFI\redhat\shimaa64.efi" So the bootentry is automatically created after reboot.
Chatted with Rom and I understood that due to this bug, his team could not deploy RHCOS on DPU; and therefore, the team is blocked at the moment and due to that we are proposing the severity to be "High". However, we also agreed that this bug is not a 4.9 release Blocker. Therefore, I am also setting "Blocker-" flag as it does not block 4.9 target release. If folks that are closer to this bug disagrees, feel free to change the severity/blocker flag as appropriate.
(In reply to Dan Li from comment #30) > Chatted with Rom and I understood that due to this bug, his team could not > deploy RHCOS on DPU; and therefore, the team is blocked at the moment and > due to that we are proposing the severity to be "High". However, we also > agreed that this bug is not a 4.9 release Blocker. Therefore, I am also > setting "Blocker-" flag as it does not block 4.9 target release. If folks > that are closer to this bug disagrees, feel free to change the > severity/blocker flag as appropriate. +1 on making it a high priority. It is blocking me and a few other people in my team as well.
Note the needinfo requested in Comment 26.
The results from https://bugzilla.redhat.com/show_bug.cgi?id=1977983#c23 would be great too
(In reply to Timothée Ravier from comment #25) > Note that RHCOS does not add boot entries to the EFI config, it relies on > the default bootloader path to boot the system. Sorry, my comment here was partially incorrect. See the details from https://bugzilla.redhat.com/show_bug.cgi?id=1997805#c8: > RHCOS does create an EFI boot entry, but not in coreos-installer. On the first boot, shim is invoked as the fallback bootloader and it creates the entry: https://github.com/rhboot/shim/blob/main/README.fallback
In replay to comment 23 @cgament installed RHEL 8.4 on the same bluefiled-2 card and it booted successfully.
@alaa in regards to comment 26, Can you check on the firmware to see if it needs an update or needs to be fixed? This is how we currently update the firmware - https://gitlab.cee.redhat.com/egarver/smart-nic-poc/-/blob/master/provision/bf2/bluefield_provision.sh#L57
I just checked and that's the latest I can download.
Today, at the suggestion of Prashanth, I manually ran: $ efibootmgr -d /dev/mmcblk0 -p 2 -c -L "RHCOS works" -l \\EFI\\redhat\\shimx64.efi (this is what Assistent Installer does, see here: https://github.com/openshift/assisted-installer/blob/e346bc34fd2b3fb8d6c2cf6c0bb2125a565a198d/src/ops/ops.go#L235 ) After reboot, my new RHCOS installation booted just fine, with just this message on the console: System BootOrder not found. Initializing defaults. Creating boot entry "Boot0002" with label "Red Hat Enterprise Linux" for file "\EFI\redhat\shimaa64.efi" This last one seems to come from here: https://github.com/rhboot/shim/blob/1b30c2b9e5ee7d3e305a28a92805152d5cbfc9cb/fallback.c#L245 Subsequent reboots worked without any issue. Further, I removed all leftover entries in the EFI menu: $ efibootmgr -B -b <entry> And all subsequent CoreOS installs worked (booted into new system) without any additional changes. I used both the tricked imaged from Prashanth and this one https://mirror.openshift.com/pub/openshift-v4/aarch64/dependencies/rhcos/pre-release/latest/rhcos-4.9.0-fc.1-aarch64-live.aarch64.iso Maybe we were hitting a different bug?
The efibootmgr command in comment 38 shouldn't have made a difference, since you're not on x64. :-) What happens if you remove all the RHCOS boot entries, including your synthetic one, and reboot? The behavior described here seems similar to bug 1997805, where the firmware is being confused by stale boot entries.
> What happens if you remove all the RHCOS boot entries, including your synthetic one, and reboot? I get a message like this: System BootOrder not found. Initializing defaults. Creating boot entry "Boot0002" with label "Red Hat Enterprise Linux" for file "\EFI\redhat\shimaa64.efi" And indeed I have a new menu entry labeled "Red Hat Enterprise Linux" which correctly boots into RHCOS. Reading through https://bugzilla.redhat.com/show_bug.cgi?id=1997805 I believe that is exactly what we were hitting. Best, Cos
*** This bug has been marked as a duplicate of bug 1997805 ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days