Bug 2362273 - [kernel] upgrading from 6.15 rc2 -> rc3 hangs after a systemctl daemon-reload
Summary: [kernel] upgrading from 6.15 rc2 -> rc3 hangs after a systemctl daemon-reload
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/cockpit-project/co...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-04-25 09:14 UTC by Jelle van der Waa
Modified: 2025-05-12 09:00 UTC (History)
17 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Jelle van der Waa 2025-04-25 09:14:43 UTC
1. Please describe the problem:

Our Cockpit integration tests running on updates-testing started to fail a MDRAID test where a raid1 is assembled and deleted.

2. What is the Version-Release number of the kernel:

6.15.0-0.rc3.20250423gitbc3372351d0c.30.fc43.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

6.15.0-0.rc3.20250423gitbc3372351d0c.30.fc43.x86_64

Worked in 6.15.0-0.rc2.20250416git1a1d569a75f3.24.fc43.x86_64

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

truncate --size=50MB /var/tmp/member1; losetup -P --show --find /var/tmp/member1
truncate --size=50MB /var/tmp/member2; losetup -P --show --find /var/tmp/member2
mdadm --create SOMERAID --run --level=1 --metadata=1.0 --raid-devices=2 /dev/loop0 /dev/loop1
mdadm --stop /dev/md/SOMERAID
systemctl daemon-reload

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Yes

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

I will, but the whole machine freezes. In a separate ssh session I could run `strace ps` which also hangs:

openat(AT_FDCWD, "/proc/1016/cmdline", O_RDONLY) = 4
read(4, "/sbin/agetty\0-o\0-- \\u\0--noreset\0"..., 131072) = 86
read(4, "", 130986)                     = 0
close(4)                                = 0
openat(AT_FDCWD, "/proc/1016/ctty", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/ttyS64",

Reproducible: Always

Comment 1 Jelle van der Waa 2025-04-25 09:16:47 UTC
I will try to boot our immutable test image mutable to obtain the kernel log when this issue occurs. Enabling systemd's debug logging has no provided any useful clues as of yet.

Comment 2 Jelle van der Waa 2025-04-25 09:37:01 UTC
Tested 6.15.0-0.rc2.20250418gitfc96b232f8e7.25.fc43.x86_64 and this also does not have the issue so it was introduced in rc3.

Comment 3 Jelle van der Waa 2025-04-25 10:09:08 UTC
So as loop devices and raid can be an issue, I have tested this also with two qcow2 images attached to our test vm and:

mdadm --create SOMERAID --run --level=0 --raid-devices=2 /dev/sda /dev/sdb
mdadm --stop /dev/md/SOMERAID
systemctl daemon-reload

Still hangs on this scenario.

Comment 4 Jelle van der Waa 2025-04-25 10:12:24 UTC
So for a git bisect:

known bad: bc3372351d0c8b2726b7d4229b878342e3e6b0e8
known good: fc96b232f8e7c0a6c282f47726b2ff6a5fb341d2

Comment 5 Radek Vykydal 2025-04-25 14:42:06 UTC
We are hitting issues with raid and rc3 in Anaconda installer. The installation hangs during shutdown: https://github.com/rhinstaller/kickstart-tests/issues/1414

Comment 6 Jelle van der Waa 2025-04-28 08:22:05 UTC
In the weekend I bisected this to 777d0961ff95b26d5887fdae69900374364976f3 but I was unable to reproduce the issue on git master today 6.15-rc4 so a new kernel update should resolve this.

Comment 7 Marius Vollmer 2025-05-02 06:48:19 UTC
Looks like rawhide has started to pass again: https://artifacts.dev.testing-farm.io/13bef028-3f3f-410f-b562-9534fc15c451/  I didn't check what kernel that was.

Comment 8 Radek Vykydal 2025-05-12 09:00:58 UTC
(In reply to Radek Vykydal from comment #5)
> We are hitting issues with raid and rc3 in Anaconda installer. The
> installation hangs during shutdown:
> https://github.com/rhinstaller/kickstart-tests/issues/1414

Our tests are passing again as well.


Note You need to log in before you can comment on or make changes to this bug.