Unfortunately, I'm unable to reproduce the problem on Fedora 39 or Fedora rawhide.

This happens after

    systemctl enable kdump; kdumpctl reset-crashkernel; reboot

(in a situation where the current initrd doesn't have kdump, or at least an outdated config version). However, it's not reliable, I only get this in about one out of ten runs. Then this happens:

kdumpctl[901]: kdump: No kdump initial ramdisk found.
kdumpctl[901]: kdump: Rebuilding /boot/initramfs-6.5.6-300.fc39.x86_64kdump.img
dracut[1208]: dracut-059-15.fc39
dracut[1211]: Executing: /usr/bin/dracut --add kdumpbase --quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode strict --hostonly-nics  --aggressive-strip -o "plymouth resume ifcfg earlykdump" --mount "/dev/disk/by-uuid/377161b1-d8ce-45b2-84f3-bf5a7009cf94 /sysroot btrfs rw,relatime,seclabel,compress=zstd:1,discard=async,space_cache=v2,subvolid=256,subvol=/root" --squash-compressor zstd --no-hostonly-default-device -f /boot/initramfs-6.5.6-300.fc39.x86_64kdump.img 6.5.6-300.fc39.x86_64
dracut[1211]: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut[1211]: dracut module 'dbus-daemon' will not be installed, because command 'dbus-daemon' could not be found!
dracut[1211]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut[1211]: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut[1211]: dracut module 'ifcfg' will not be installed, because it's in the list to be omitted!
dracut[1211]: dracut module 'plymouth' will not be installed, because it's in the list to be omitted!
dracut[1211]: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut[1211]: dracut module 'cifs' will not be installed, because command 'mount.cifs' could not be found!
dracut[1211]: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut[1211]: dracut module 'resume' will not be installed, because it's in the list to be omitted!
dracut[1211]: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut[1211]: dracut module 'earlykdump' will not be installed, because it's in the list to be omitted!
dracut[1211]: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut[1211]: dracut module 'dbus-daemon' will not be installed, because command 'dbus-daemon' could not be found!
dracut[1211]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut[1211]: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut[1211]: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut[1211]: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut[1211]: dracut module 'cifs' will not be installed, because command 'mount.cifs' could not be found!
dracut[1211]: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut[1211]: *** Including module: systemd-initrd ***
dracut[1211]: *** Including module: nss-softokn ***
dracut[1211]: *** Including module: i18n ***
dracut[1211]: *** Including module: drm ***
systemd[1]: Reloading requested from client PID 2148 ('systemctl') (unit session-3.scope)...
systemd[1]: Reloading...
dracut[1211]: *** Including module: btrfs ***
dracut[1211]: *** Including module: kernel-modules ***
kernel: block vda: the capability attribute has been deprecated.
dracut[1211]: *** Including module: kernel-modules-extra ***
dracut[1211]:   kernel-modules-extra: configuration source "/run/depmod.d" does not exist
dracut[1211]:   kernel-modules-extra: configuration source "/etc/depmod.d" is ignored (directory or doesn't exist)
dracut[1211]:   kernel-modules-extra: configuration source "/lib/depmod.d" does not exist
dracut[1211]: *** Including module: qemu ***
dracut[1211]: *** Including module: fstab-sys ***
dracut[1211]: *** Including module: rootfs-block ***
dracut[1211]: *** Including module: terminfo ***
dracut[1211]: *** Including module: udev-rules ***
dracut[1211]: Skipping udev rule: 40-redhat.rules
dracut[1211]: Skipping udev rule: 50-firmware.rules
dracut[1211]: Skipping udev rule: 50-udev.rules
systemd[1]: Reloading finished in 931 ms.
dracut[1211]: Skipping udev rule: 91-permissions.rules
dracut[1211]: Skipping udev rule: 80-drivers-modprobe.rules
dracut[1211]: Skipping udev rule: 70-persistent-net.rules
dracut[1211]: *** Including module: virtiofs ***
dracut[1211]: *** Including module: dracut-systemd ***
dracut[1211]: *** Including module: usrmount ***
dracut[1211]: *** Including module: base ***
dracut[1211]: *** Including module: fs-lib ***
dracut[1211]: *** Including module: kdumpbase ***
dracut[1211]: *** Including module: memstrack ***
dracut[1211]: *** Including module: shutdown ***
dracut[1211]: *** Including module: squash ***
dracut[1211]: *** Including modules done ***
dracut[1211]: *** Installing kernel module dependencies ***
dracut[1211]: *** Installing kernel module dependencies done ***
dracut[1211]: *** Resolving executable dependencies ***
dracut[1211]: *** Resolving executable dependencies done ***
dracut[1211]: *** Hardlinking files ***
dracut[1211]: Mode:                     real
dracut[1211]: Method:                   sha256
dracut[1211]: Files:                    427
dracut[1211]: Linked:                   1 files
dracut[1211]: Compared:                 0 xattrs
dracut[1211]: Compared:                 9 files
dracut[1211]: Saved:                    56.46 KiB
dracut[1211]: Duration:                 0.013916 seconds
dracut[1211]: *** Hardlinking files done ***
dracut[1211]: *** Generating early-microcode cpio image ***
dracut[1211]: *** Store current command line parameters ***
dracut[1211]: Stored kernel commandline:
dracut[1211]: rd.driver.pre=btrfs
dracut[1211]: *** Install squash loader ***
dracut[1211]: *** Stripping files done ***
dracut[1211]: *** Squashing the files inside the initramfs ***
kdumpctl[883]: Terminated
systemd[1]: kdump.service: Main process exited, code=killed, status=15/TERM
systemd[1]: kdump.service: Failed with result 'signal'.
audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kdump comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
audit[3843]: AVC avc:  denied  { rmdir } for  pid=3843 comm="(sd-rmrf)" name="mkdumprd.v9d2uO" dev="tmpfs" ino=35 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0
audit[3844]: AVC avc:  denied  { remove_name } for  pid=3844 comm="(sd-rmrf)" name="bin" dev="vda5" ino=200398 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0
systemd[1]: Stopped kdump.service - Crash recovery kernel arming.
systemd[1]: kdump.service: Consumed 5.847s CPU time.
audit: PROCTITLE proctitle="(sd-rmrf)"
audit[3844]: AVC avc:  denied  { remove_name } for  pid=3844 comm="(sd-rmrf)" name="null" dev="vda5" ino=200436 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0
audit[3844]: SYSCALL arch=c000003e syscall=263 success=no exit=-13 a0=9 a1=563fb408a803 a2=0 a3=4 items=0 ppid=1 pid=3844 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="(sd-rmrf)" exe="/usr/lib/systemd/systemd" subj=system_u:system_r:init_t:s0 key=(null)
audit: PROCTITLE proctitle="(sd-rmrf)"
audit[3844]: AVC avc:  denied  { remove_name } for  pid=3844 comm="(sd-rmrf)" name="kmsg" dev="vda5" ino=200437 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0
audit[3844]: SYSCALL arch=c000003e syscall=263 success=no exit=-13 a0=9 a1=563fb408a81b a2=0 a3=4 items=0 ppid=1 pid=3844 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="(sd-rmrf)" exe="/usr/lib/systemd/systemd" subj=system_u:system_r:init_t:s0 key=(null)
[ ... a gazillion similar errors for all file paths in the temp dir)

However, this doesn't run in isolation. Lots of other services start during this, and as ssh is already active, our tests also already log in and start a user session. Indeed I can run `systemctl list-jobs` after reboot and ssh-login, and see "kdump.service start running".

The notable thing in the log is the systemctl daemon-reload. Perhaps this somehow interferes with the running `kdumpctl start`, i.e. the initrd build? Reloading systemd also re-runs systemd-tmpfiles and whatnot.

I tried to open two terminals with

 while true; do systemctl daemon-reload; sleep 1; done
 touch /etc/kdump.conf; systemctl restart kdump.service; systemctl status kdump

but didn't catch it that way.

So, sorry this doesn't have a simple reproducer. So far this only happens with the cockpit test:

git clone https://github.com/cockpit-project/cockpit
cd cockpit
test/image-prepare -q fedora-39
test/verify/check-kdump TestKdump.testBasic  # repeat many times

But even that isn't terribly useful, as it's difficult to do interactive observations while kdump.service is starting (i.e. rebuilding initrd) during boot.

Can I enable more debug logging for this somehow?

How to enable full auditing:
https://fedoraproject.org/wiki/SELinux/Debugging#Enable_full_auditing

Unfortunately, I am unable to setup a system to run the tests.

Created attachment 1997424 [details]
journal with full auditing

Note to self: As auditd kept failing, it first needs this:

mkdir -p /var/log/audit
restorecon -v /var/log/audit/

After that, the ausearch command gives this:

# ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today | head -n30
----
type=PROCTITLE msg=audit(11/06/2023 12:19:26.379:309) : proctitle=(sd-rmrf) 
type=PATH msg=audit(11/06/2023 12:19:26.379:309) : item=1 name=overlayfs inode=201209 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=PATH msg=audit(11/06/2023 12:19:26.379:309) : item=0 name=/ inode=201206 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(11/06/2023 12:19:26.379:309) : cwd=/ 
type=SYSCALL msg=audit(11/06/2023 12:19:26.379:309) : arch=x86_64 syscall=unlinkat success=no exit=EACCES(Permission denied) a0=0xc a1=0x5637b4f35100 a2=0x200 a3=0x4 items=2 ppid=1 pid=3872 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=(sd-rmrf) exe=/usr/lib/systemd/systemd subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(11/06/2023 12:19:26.379:309) : avc:  denied  { remove_name } for  pid=3872 comm=(sd-rmrf) name=overlayfs dev="vda5" ino=201209 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(11/06/2023 12:19:26.381:310) : proctitle=(sd-rmrf) 
type=PATH msg=audit(11/06/2023 12:19:26.381:310) : item=1 name=mkdumprd.gCauPu inode=36 dev=00:24 mode=dir,700 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=PATH msg=audit(11/06/2023 12:19:26.381:310) : item=0 name=/ inode=34 dev=00:24 mode=dir,sticky,777 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(11/06/2023 12:19:26.381:310) : cwd=/ 
type=SYSCALL msg=audit(11/06/2023 12:19:26.381:310) : arch=x86_64 syscall=unlinkat success=no exit=EACCES(Permission denied) a0=0x4 a1=0x5637b4d56a40 a2=0x200 a3=0x4 items=2 ppid=1 pid=3871 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=(sd-rmrf) exe=/usr/lib/systemd/systemd subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(11/06/2023 12:19:26.381:310) : avc:  denied  { rmdir } for  pid=3871 comm=(sd-rmrf) name=mkdumprd.gCauPu dev="tmpfs" ino=36 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(11/06/2023 12:19:26.386:311) : proctitle=(sd-rmrf) 
type=PATH msg=audit(11/06/2023 12:19:26.386:311) : item=1 name=fs inode=201206 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=PATH msg=audit(11/06/2023 12:19:26.386:311) : item=0 name=/ inode=201202 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(11/06/2023 12:19:26.386:311) : cwd=/ 
type=SYSCALL msg=audit(11/06/2023 12:19:26.386:311) : arch=x86_64 syscall=unlinkat success=no exit=EACCES(Permission denied) a0=0xb a1=0x5637b4caeb70 a2=0x200 a3=0x5637b4f35100 items=2 ppid=1 pid=3872 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=(sd-rmrf) exe=/usr/lib/systemd/systemd subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(11/06/2023 12:19:26.386:311) : avc:  denied  { remove_name } for  pid=3872 comm=(sd-rmrf) name=fs dev="vda5" ino=201206 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(11/06/2023 12:19:26.387:312) : proctitle=(sd-rmrf) 
type=PATH msg=audit(11/06/2023 12:19:26.387:312) : item=1 name=kernel inode=201202 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=PATH msg=audit(11/06/2023 12:19:26.387:312) : item=0 name=/ inode=201201 dev=00:1e mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:kdumpctl_tmp_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(11/06/2023 12:19:26.387:312) : cwd=/ 
type=SYSCALL msg=audit(11/06/2023 12:19:26.387:312) : arch=x86_64 syscall=unlinkat success=no exit=EACCES(Permission denied) a0=0xa a1=0x5637b4f4c240 a2=0x200 a3=0x5637b4caeb70 items=2 ppid=1 pid=3872 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=(sd-rmrf) exe=/usr/lib/systemd/systemd subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(11/06/2023 12:19:26.387:312) : avc:  denied  { remove_name } for  pid=3872 comm=(sd-rmrf) name=kernel dev="vda5" ino=201202 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kdumpctl_tmp_t:s0 tclass=dir permissive=0 
----

There's 500 more lines, but as they all have the same rot cause, the first three paragraphs hopefully suffice?

I also attach the corresponding full journal.

Thank you, Martin, that was sufficient.

We don't have a straightforward reproducer, but hopefully the fix is complete.

FEDORA-2023-a2dacfbdcb has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a2dacfbdcb

FEDORA-2023-a2dacfbdcb has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a2dacfbdcb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a2dacfbdcb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-a2dacfbdcb has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.