Bug 2180910

Summary:

kernel 6.2.7-100.fc36.aarch64 fails to boot after upgrade (AWS EC2)

Product:

[Fedora] Fedora

Reporter:

Greg Bailey <gbailey>

Component:

kernel

Assignee:

Kernel Maintainer List <kernel-maint>

Status:

CLOSED ERRATA

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

CC:

acaringi, adscvr, airlied, alciregi, bskeggs, diego, hdegoede, hpa, jarodwilson, jforbes, jglisse, josef, kernel-maint, lgoncalv, linville, masami256, max.wittig95, mchehab, ptalbert, steved

Target Milestone:

---

Target Release:

---

Hardware:

aarch64

OS:

Unspecified

Whiteboard:

Fixed In Version:

kernel-6.2.9-300.fc38 kernel-6.2.9-200.fc37 kernel-6.2.9-100.fc36

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2023-04-02 00:16:37 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
EC2 instance log	none

Description Greg Bailey 2023-03-22 15:32:29 UTC

Created attachment 1952857 [details]
EC2 instance log

1. Please describe the problem:

Rebooting aarch64 Fedora 36 system after applying updates fails to boot. I'm starting with ami-09c06fe16c4e3183d in us-west-2 region (official Fedora 36 cloud AMI).

2. What is the Version-Release number of the kernel:

6.2.7-100.fc36.aarch64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

6.2.7-100.fc36.aarch64 fails to boot after upgrading from a working 6.1.14-100.fc36.aarch64 installation.  I've also verified that an upgrade from the original AMI listed above (with kernel 5.17.5-300) also fails to boot after upgrade.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Yes, launch an instance using AMI ami-09c06fe16c4e3183d in us-west-2 with aarch64 architecture.  I used a t4g.medium instance type to verify.  Apply upgrades with "dnf upgrade" and reboot.  The instance fails to boot.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Have not tested this.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Comment 1 Max 2023-03-27 10:21:08 UTC

We could reproduce this in AWS with a `t4g.xlarge` type machine, first running a version of Fedora from 2023-03-02.
After running `dnf update && dnf upgrade` it fails to boot. 

The same behavior can be observed with `Fedora-Cloud-Base-36-20230327.0.aarch64-hvm-eu-central-1-standard-0 (ami-005ad2d9af8a613fc). It is never possible to even connect to this machine, once launched.

As I've already seen this issue, I've captured the journal, while upgrading:

```
Mar 27 10:10:49 canary-aarch64 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 27 10:10:58 canary-aarch64 dracut[1322]: dracut-057-3.fc36
Mar 27 10:10:58 canary-aarch64 dracut[1324]: Executing: /usr/bin/dracut -f /boot/initramfs-6.2.7-100.fc36.aarch64.img 6.2.7-100.fc36.aarch64
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'modsign' will not be installed, because command 'keyctl' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'dbus-daemon' will not be installed, because command 'dbus-daemon' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvmmerge' will not be installed, because command 'lvm' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvmthinpool-monitor' will not be installed, because command 'lvm' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'lvm' will not be installed, because command 'lvm' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: systemd ***
Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: systemd-initrd ***
Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: nss-softokn ***
Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: i18n ***
Mar 27 10:10:59 canary-aarch64 dracut[1324]: *** Including module: btrfs ***
Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: crypt ***
Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: dm ***
Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 64-device-mapper.rules
Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 60-persistent-storage-dm.rules
Mar 27 10:11:00 canary-aarch64 dracut[1324]: Skipping udev rule: 55-dm.rules
Mar 27 10:11:00 canary-aarch64 dracut[1324]: *** Including module: kernel-modules ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: kernel-modules-extra ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]:   kernel-modules-extra: configuration source "/run/depmod.d" does not exist
Mar 27 10:11:03 canary-aarch64 dracut[1324]:   kernel-modules-extra: configuration source "/etc/depmod.d" is ignored (directory or doesn't exist)
Mar 27 10:11:03 canary-aarch64 dracut[1324]:   kernel-modules-extra: configuration source "/lib/depmod.d" does not exist
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: mdraid ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 64-md-raid.rules
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: nvdimm ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: qemu ***
Mar 27 10:11:03 canary-aarch64 filebeat[867]: {"log.level":"info","@timestamp":"2023-03-27T10:11:03.770Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":129269760}}}},"cpu":{"system":{"ticks":110},"total":{"ticks":480,"time":{"ms":40},"value":480},"user":{"ticks":370,"time":{"ms":40}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":15},"info":{"ephemeral_id":"e39d223b-3add-4ba4-a98e-5b3f81d7fc39","uptime":{"ms":120077},"version":"8.6.2"},"memstats":{"gc_next":32381656,"memory_alloc":16089328,"memory_sys":4194304,"memory_total":86635776,"rss":95449088},"runtime":{"goroutines":20}},"filebeat":{"events":{"active":0,"added":45,"done":45},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":39,"active":0,"batches":5,"total":39},"read":{"bytes":220},"write":{"bytes":10808}},"pipeline":{"clients":1,"events":{"active":0,"filtered":6,"published":39,"total":45},"queue":{"acked":39}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.5,"15":0.04,"5":0.15,"norm":{"1":0.125,"15":0.01,"5":0.0375}}}},"ecs.version":"1.6.0"}}
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: lunmask ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: resume ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: rootfs-block ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: terminfo ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: *** Including module: udev-rules ***
Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 40-redhat.rules
Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 50-firmware.rules
Mar 27 10:11:03 canary-aarch64 dracut[1324]: Skipping udev rule: 50-udev.rules
Mar 27 10:11:04 canary-aarch64 dracut[1324]: Skipping udev rule: 91-permissions.rules
Mar 27 10:11:04 canary-aarch64 dracut[1324]: Skipping udev rule: 80-drivers-modprobe.rules
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: virtiofs ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: dracut-systemd ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: usrmount ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: base ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: fs-lib ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: memstrack ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including module: shutdown ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Including modules done ***
Mar 27 10:11:04 canary-aarch64 dracut[1324]: *** Installing kernel module dependencies ***
Mar 27 10:11:07 canary-aarch64 dracut[1324]: *** Installing kernel module dependencies done ***
Mar 27 10:11:07 canary-aarch64 dracut[1324]: *** Resolving executable dependencies ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Resolving executable dependencies done ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Hardlinking files ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Mode:                     real
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Method:                   sha256
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Files:                    2256
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Linked:                   8 files
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Compared:                 0 xattrs
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Compared:                 423 files
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Saved:                    1.21 MiB
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Duration:                 0.034221 seconds
Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Hardlinking files done ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: Could not find 'strip'. Not stripping the initramfs.
Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Store current command line parameters ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: *** Creating image file '/boot/initramfs-6.2.7-100.fc36.aarch64.img' ***
Mar 27 10:11:09 canary-aarch64 dracut[1324]: dracut: using auto-determined compression method 'pigz'
Mar 27 10:11:12 canary-aarch64 dracut[1324]: Image: /var/tmp/dracut.RirYAW/initramfs.img: 30M
Mar 27 10:11:12 canary-aarch64 dracut[1324]: ========================================================================
Mar 27 10:11:12 canary-aarch64 dracut[1324]: Version:
Mar 27 10:11:12 canary-aarch64 dracut[1324]: Arguments:  -f
Mar 27 10:11:13 canary-aarch64 dracut[1324]: dracut modules:
Mar 27 10:11:13 canary-aarch64 dracut[1324]: systemd
Mar 27 10:11:13 canary-aarch64 dracut[1324]: systemd-initrd
Mar 27 10:11:13 canary-aarch64 dracut[1324]: nss-softokn
Mar 27 10:11:13 canary-aarch64 dracut[1324]: i18n
Mar 27 10:11:13 canary-aarch64 dracut[1324]: btrfs
Mar 27 10:11:13 canary-aarch64 dracut[1324]: crypt
Mar 27 10:11:13 canary-aarch64 dracut[1324]: dm
Mar 27 10:11:13 canary-aarch64 dracut[1324]: kernel-modules
Mar 27 10:11:13 canary-aarch64 dracut[1324]: kernel-modules-extra
Mar 27 10:11:13 canary-aarch64 dracut[1324]: mdraid
Mar 27 10:11:13 canary-aarch64 dracut[1324]: nvdimm
Mar 27 10:11:13 canary-aarch64 dracut[1324]: qemu
Mar 27 10:11:13 canary-aarch64 dracut[1324]: lunmask
Mar 27 10:11:13 canary-aarch64 dracut[1324]: resume
Mar 27 10:11:13 canary-aarch64 dracut[1324]: rootfs-block
Mar 27 10:11:13 canary-aarch64 dracut[1324]: terminfo
Mar 27 10:11:13 canary-aarch64 dracut[1324]: udev-rules
Mar 27 10:11:13 canary-aarch64 dracut[1324]: virtiofs
Mar 27 10:11:13 canary-aarch64 dracut[1324]: dracut-systemd
Mar 27 10:11:13 canary-aarch64 dracut[1324]: usrmount
Mar 27 10:11:13 canary-aarch64 dracut[1324]: base
Mar 27 10:11:13 canary-aarch64 dracut[1324]: fs-lib
Mar 27 10:11:13 canary-aarch64 dracut[1324]: memstrack
Mar 27 10:11:13 canary-aarch64 dracut[1324]: shutdown
Mar 27 10:11:13 canary-aarch64 dracut[1324]: ========================================================================
Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r--   1 root     root       5,   1 Sep 26  2022 dev/console
Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r--   1 root     root       1,  11 Sep 26  2022 dev/kmsg
Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r--   1 root     root       1,   3 Sep 26  2022 dev/null
Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r--   1 root     root       1,   8 Sep 26  2022 dev/random
Mar 27 10:11:14 canary-aarch64 dracut[1324]: crw-r--r--   1 root     root       1,   9 Sep 26  2022 dev/urandom
Mar 27 10:11:14 canary-aarch64 dracut[1324]: drwxr-xr-x   1 root     root 

...


Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x   1 root     root            0 Sep 26  2022 usr/share/terminfo/l
Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r--   1 root     root         1728 Jan 21  2022 usr/share/terminfo/l/linux
Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x   1 root     root            0 Sep 26  2022 usr/share/terminfo/v
Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r--   1 root     root         1190 Jan 21  2022 usr/share/terminfo/v/vt100
Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r--   1 root     root         1184 Jan 21  2022 usr/share/terminfo/v/vt102
Mar 27 10:11:23 canary-aarch64 dracut[1324]: -rw-r--r--   1 root     root         1391 Jan 21  2022 usr/share/terminfo/v/vt220
Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx   1 root     root           20 Sep 26  2022 usr/share/unimaps -> /usr/lib/kbd/unimaps
Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x   1 root     root            0 Sep 26  2022 var
Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx   1 root     root           11 Sep 26  2022 var/lock -> ../run/lock
Mar 27 10:11:23 canary-aarch64 dracut[1324]: lrwxrwxrwx   1 root     root            6 Sep 26  2022 var/run -> ../run
Mar 27 10:11:23 canary-aarch64 dracut[1324]: drwxr-xr-x   1 root     root            0 Sep 26  2022 var/tmp
Mar 27 10:11:23 canary-aarch64 dracut[1324]: ========================================================================
Mar 27 10:11:23 canary-aarch64 dracut[1324]: *** Creating initramfs image file '/boot/initramfs-6.2.7-100.fc36.aarch64.img' done ***
Mar 27 10:11:24 canary-aarch64 systemd[1]: Started run-r16e2a41adcf74a6ba9bf8ee841b4799b.service - /usr/bin/systemctl start man-db-cache-update.
░░ Subject: A start job for unit run-r16e2a41adcf74a6ba9bf8ee841b4799b.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit run-r16e2a41adcf74a6ba9bf8ee841b4799b.service has finished successfully.
░░ 
░░ The job identifier is 1703.

```

I'm not sure, but as its not booting this message seems suspicious: `Could not find 'strip'. Not stripping the initramfs.`

Comment 2 Diego 2023-03-27 12:10:35 UTC

This is currently making us unable to deploy a fedora-based EC2 host on AWS; admittedly, we are using the dailies as base AMI, which is probably not the best idea.

Comment 3 Justin M. Forbes 2023-03-27 13:26:04 UTC

The issue is around the EFI ZBOOT bits, and the fact that F36 grub does not support them. When the 6.2.9 kernel ships later this week, it should resolve the issue for F36.  Note, the 6.2.8 in updates-testing right now will not.

Comment 4 Fedora Update System 2023-03-31 19:46:32 UTC

FEDORA-2023-c46eb02bbf has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c46eb02bbf

Comment 5 Fedora Update System 2023-03-31 19:53:14 UTC

FEDORA-2023-98251cef79 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-98251cef79

Comment 6 Fedora Update System 2023-04-01 01:34:15 UTC

FEDORA-2023-c46eb02bbf has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-c46eb02bbf

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2023-04-01 02:11:40 UTC

FEDORA-2023-d525cf5272 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-d525cf5272`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-d525cf5272

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2023-04-01 02:17:09 UTC

FEDORA-2023-98251cef79 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-98251cef79`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-98251cef79

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2023-04-02 00:16:37 UTC

FEDORA-2023-c46eb02bbf has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 10 Fedora Update System 2023-04-05 01:35:23 UTC

FEDORA-2023-98251cef79 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 Fedora Update System 2023-04-08 01:29:36 UTC

FEDORA-2023-d525cf5272 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.