Created attachment 1952857 [details] EC2 instance log 1. Please describe the problem: Rebooting aarch64 Fedora 36 system after applying updates fails to boot. I'm starting with ami-09c06fe16c4e3183d in us-west-2 region (official Fedora 36 cloud AMI). 2. What is the Version-Release number of the kernel: 6.2.7-100.fc36.aarch64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 6.2.7-100.fc36.aarch64 fails to boot after upgrading from a working 6.1.14-100.fc36.aarch64 installation. I've also verified that an upgrade from the original AMI listed above (with kernel 5.17.5-300) also fails to boot after upgrade. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes, launch an instance using AMI ami-09c06fe16c4e3183d in us-west-2 with aarch64 architecture. I used a t4g.medium instance type to verify. Apply upgrades with "dnf upgrade" and reboot. The instance fails to boot. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Have not tested this. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag.
We could reproduce this in AWS with a `t4g.xlarge` type machine, first running a version of Fedora from 2023-03-02. After running `dnf update && dnf upgrade` it fails to boot. The same behavior can be observed with `Fedora-Cloud-Base-36-20230327.0.aarch64-hvm-eu-central-1-standard-0 (ami-005ad2d9af8a613fc). It is never possible to even connect to this machine, once launched. As I've already seen this issue, I've captured the journal, while upgrading: ``` Mar 27 10:10:49 canary-aarch64 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Mar 27 10:10:58 canary-aarch64 dracut[1322]: dracut-057-3.fc36 Mar 27 10:10:58 canary-aarch64 dracut[1324]: Executing: /usr/bin/dracut -f /boot/initramfs-6.2.7-100.fc36.aarch64.img 6.2.7-100.fc36.aarch64 Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'modsign' will not be installed, because command 'keyctl' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'busybox' will not be installed, because command 'busybox' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'dbus-daemon' will not be installed, because command 'dbus-daemon' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvmmerge' will not be installed, because command 'lvm' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvmthinpool-monitor' will not be installed, because command 'lvm' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvm' will not be installed, because command 'lvm' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'multipath' will not be installed, because command 'multipath' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found! Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: systemd *** Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: systemd-initrd *** Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: nss-softokn *** Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: i18n *** Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: btrfs *** Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: crypt *** Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: dm *** Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 64-device-mapper.rules Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 60-persistent-storage-dm.rules Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 55-dm.rules Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: kernel-modules *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: kernel-modules-extra *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: kernel-modules-extra: configuration source "/run/depmod.d" does not exist Mar 27 10:11:03 canary-aarch64 dracut[1324]: kernel-modules-extra: configuration source "/etc/depmod.d" is ignored (directory or doesn't exist) Mar 27 10:11:03 canary-aarch64 dracut[1324]: kernel-modules-extra: configuration source "/lib/depmod.d" does not exist Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: mdraid *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 64-md-raid.rules Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: nvdimm *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: qemu *** Mar 27 10:11:03 canary-aarch64 filebeat[867]: {"log.level":"info","@timestamp":"2023-03-27T10:11:03.770Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":129269760}}}},"cpu":{"system":{"ticks":110},"total":{"ticks":480,"time":{"ms":40},"value":480},"user":{"ticks":370,"time":{"ms":40}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":15},"info":{"ephemeral_id":"e39d223b-3add-4ba4-a98e-5b3f81d7fc39","uptime":{"ms":120077},"version":"8.6.2"},"memstats":{"gc_next":32381656,"memory_alloc":16089328,"memory_sys":4194304,"memory_total":86635776,"rss":95449088},"runtime":{"goroutines":20}},"filebeat":{"events":{"active":0,"added":45,"done":45},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":39,"active":0,"batches":5,"total":39},"read":{"bytes":220},"write":{"bytes":10808}},"pipeline":{"clients":1,"events":{"active":0,"filtered":6,"published":39,"total":45},"queue":{"acked":39}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.5,"15":0.04,"5":0.15,"norm":{"1":0.125,"15":0.01,"5":0.0375}}}},"ecs.version":"1.6.0"}} Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: lunmask *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: resume *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: rootfs-block *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: terminfo *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: udev-rules *** Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 40-redhat.rules Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 50-firmware.rules Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 50-udev.rules Mar 27 10:11:04 canary-aarch64 dracut[1324]: Skipping udev rule: 91-permissions.rules Mar 27 10:11:04 canary-aarch64 dracut[1324]: Skipping udev rule: 80-drivers-modprobe.rules Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: virtiofs *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: dracut-systemd *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: usrmount *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: base *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: fs-lib *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: memstrack *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: shutdown *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including modules done *** Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Installing kernel module dependencies *** Mar 27 10:11:07 canary-aarch64 dracut[1324]: *** Installing kernel module dependencies done *** Mar 27 10:11:07 canary-aarch64 dracut[1324]: *** Resolving executable dependencies *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Resolving executable dependencies done *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Hardlinking files *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: Mode: real Mar 27 10:11:09 canary-aarch64 dracut[1324]: Method: sha256 Mar 27 10:11:09 canary-aarch64 dracut[1324]: Files: 2256 Mar 27 10:11:09 canary-aarch64 dracut[1324]: Linked: 8 files Mar 27 10:11:09 canary-aarch64 dracut[1324]: Compared: 0 xattrs Mar 27 10:11:09 canary-aarch64 dracut[1324]: Compared: 423 files Mar 27 10:11:09 canary-aarch64 dracut[1324]: Saved: 1.21 MiB Mar 27 10:11:09 canary-aarch64 dracut[1324]: Duration: 0.034221 seconds Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Hardlinking files done *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: Could not find 'strip'. Not stripping the initramfs. Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Store current command line parameters *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Creating image file '/boot/initramfs-6.2.7-100.fc36.aarch64.img' *** Mar 27 10:11:09 canary-aarch64 dracut[1324]: dracut: using auto-determined compression method 'pigz' Mar 27 10:11:12 canary-aarch64 dracut[1324]: Image: /var/tmp/dracut.RirYAW/initramfs.img: 30M Mar 27 10:11:12 canary-aarch64 dracut[1324]: ======================================================================== Mar 27 10:11:12 canary-aarch64 dracut[1324]: Version: Mar 27 10:11:12 canary-aarch64 dracut[1324]: Arguments: -f Mar 27 10:11:13 canary-aarch64 dracut[1324]: dracut modules: Mar 27 10:11:13 canary-aarch64 dracut[1324]: systemd Mar 27 10:11:13 canary-aarch64 dracut[1324]: systemd-initrd Mar 27 10:11:13 canary-aarch64 dracut[1324]: nss-softokn Mar 27 10:11:13 canary-aarch64 dracut[1324]: i18n Mar 27 10:11:13 canary-aarch64 dracut[1324]: btrfs Mar 27 10:11:13 canary-aarch64 dracut[1324]: crypt Mar 27 10:11:13 canary-aarch64 dracut[1324]: dm Mar 27 10:11:13 canary-aarch64 dracut[1324]: kernel-modules Mar 27 10:11:13 canary-aarch64 dracut[1324]: kernel-modules-extra Mar 27 10:11:13 canary-aarch64 dracut[1324]: mdraid Mar 27 10:11:13 canary-aarch64 dracut[1324]: nvdimm Mar 27 10:11:13 canary-aarch64 dracut[1324]: qemu Mar 27 10:11:13 canary-aarch64 dracut[1324]: lunmask Mar 27 10:11:13 canary-aarch64 dracut[1324]: resume Mar 27 10:11:13 canary-aarch64 dracut[1324]: rootfs-block Mar 27 10:11:13 canary-aarch64 dracut[1324]: terminfo Mar 27 10:11:13 canary-aarch64 dracut[1324]: udev-rules Mar 27 10:11:13 canary-aarch64 dracut[1324]: virtiofs Mar 27 10:11:13 canary-aarch64 dracut[1324]: dracut-systemd Mar 27 10:11:13 canary-aarch64 dracut[1324]: usrmount Mar 27 10:11:13 canary-aarch64 dracut[1324]: base Mar 27 10:11:13 canary-aarch64 dracut[1324]: fs-lib Mar 27 10:11:13 canary-aarch64 dracut[1324]: memstrack Mar 27 10:11:13 canary-aarch64 dracut[1324]: shutdown Mar 27 10:11:13 canary-aarch64 dracut[1324]: ======================================================================== Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r-- 1 root root 5, 1 Sep 26 2022 dev/console Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r-- 1 root root 1, 11 Sep 26 2022 dev/kmsg Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r-- 1 root root 1, 3 Sep 26 2022 dev/null Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r-- 1 root root 1, 8 Sep 26 2022 dev/random Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r-- 1 root root 1, 9 Sep 26 2022 dev/urandom Mar 27 10:11:14 canary-aarch64 dracut[1324]: drwxr-xr-x 1 root root ... Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x 1 root root 0 Sep 26 2022 usr/share/terminfo/l Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r-- 1 root root 1728 Jan 21 2022 usr/share/terminfo/l/linux Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x 1 root root 0 Sep 26 2022 usr/share/terminfo/v Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r-- 1 root root 1190 Jan 21 2022 usr/share/terminfo/v/vt100 Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r-- 1 root root 1184 Jan 21 2022 usr/share/terminfo/v/vt102 Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r-- 1 root root 1391 Jan 21 2022 usr/share/terminfo/v/vt220 Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx 1 root root 20 Sep 26 2022 usr/share/unimaps -> /usr/lib/kbd/unimaps Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x 1 root root 0 Sep 26 2022 var Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx 1 root root 11 Sep 26 2022 var/lock -> ../run/lock Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx 1 root root 6 Sep 26 2022 var/run -> ../run Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x 1 root root 0 Sep 26 2022 var/tmp Mar 27 10:11:23 canary-aarch64 dracut[1324]: ======================================================================== Mar 27 10:11:23 canary-aarch64 dracut[1324]: *** Creating initramfs image file '/boot/initramfs-6.2.7-100.fc36.aarch64.img' done *** Mar 27 10:11:24 canary-aarch64 systemd[1]: Started run-r16e2a41adcf74a6ba9bf8ee841b4799b.service - /usr/bin/systemctl start man-db-cache-update. ░░ Subject: A start job for unit run-r16e2a41adcf74a6ba9bf8ee841b4799b.service has finished successfully ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit run-r16e2a41adcf74a6ba9bf8ee841b4799b.service has finished successfully. ░░ ░░ The job identifier is 1703. ``` I'm not sure, but as its not booting this message seems suspicious: `Could not find 'strip'. Not stripping the initramfs.`
This is currently making us unable to deploy a fedora-based EC2 host on AWS; admittedly, we are using the dailies as base AMI, which is probably not the best idea.
The issue is around the EFI ZBOOT bits, and the fact that F36 grub does not support them. When the 6.2.9 kernel ships later this week, it should resolve the issue for F36. Note, the 6.2.8 in updates-testing right now will not.
FEDORA-2023-c46eb02bbf has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c46eb02bbf
FEDORA-2023-98251cef79 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-98251cef79
FEDORA-2023-c46eb02bbf has been pushed to the Fedora 38 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-c46eb02bbf See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-d525cf5272 has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-d525cf5272` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-d525cf5272 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-98251cef79 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-98251cef79` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-98251cef79 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-c46eb02bbf has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-98251cef79 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-d525cf5272 has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.