Actually perhaps not on that diff, that was from koji logs, but I expect that failure going away did not break things.

cpio: etc/gshadow: Cannot open: Permission denied seems a new error just before the failure.

Bit more context, I noticed the above, and let's put it together with what Justin noticed. Good build with dracut 105:

dracut[I]: Executing: /usr/bin/dracut --conf=/builddir/build/SOURCES/dracut-virt.conf --confdir=/tmp/tmp.tHXs7AFLSc --verbose --kver 6.16.0-0.rc5.65.fc43.x86_64+debug --kmoddir /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.65.fc43.x86_64+debug/ --logfile=/tmp/tmp.y2XMbKfTPD --uefi --kernel-image /builddir/build/BUILD/kernel-6.16.0-build/kernel-6.16-rc5/linux-6.16.0-0.rc5.65.fc43.x86_64/arch/x86/boot/bzImage --kernel-cmdline "console=tty0 console=ttyS0" /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.65.fc43.x86_64+debug/vmlinuz-virt.efi
/usr/lib/dracut/dracut-functions.sh: line 816: btrfs: command not found
dracut[I]: *** Including module: bash ***
dracut[I]: *** Including module: shell-interpreter ***
dracut[I]: *** Including module: systemd ***
dracut[I]: *** Including module: fips ***
dracut[I]: *** Including module: systemd-ask-password ***
dracut[I]: *** Including module: systemd-cryptsetup ***
dracut[I]: *** Including module: systemd-initrd ***
dracut[I]: *** Including module: systemd-journald ***
dracut[I]: *** Including module: systemd-modules-load ***
dracut[I]: *** Including module: systemd-pcrphase ***
dracut[I]: *** Including module: systemd-sysctl ***
dracut[I]: *** Including module: systemd-sysext ***
dracut[I]: *** Including module: systemd-sysusers ***
dracut[I]: *** Including module: systemd-tmpfiles ***
dracut[I]: *** Including module: systemd-udevd ***
dracut[I]: *** Including module: systemd-veritysetup ***
dracut[I]: *** Including module: nss-softokn ***
dracut[I]: *** Including module: i18n ***
dracut[I]: *** Including module: crypt ***
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.206 (2025-05-05) and kernel driver (unknown version).
Command failed.
dracut[I]: *** Including module: dm ***
dracut[I]: *** Including module: lvm ***
dracut[I]: *** Including module: crypt-loop ***
dracut[I]: *** Including module: tpm2-tss ***
dracut[I]: *** Including module: rootfs-block ***
dracut[I]: *** Including module: udev-rules ***
dracut[I]: *** Including module: virtiofs ***
dracut[I]: *** Including module: dracut-systemd ***
dracut[I]: *** Including module: base ***
grep: /etc/shadow: Permission denied
dracut[I]: *** Including module: fs-lib ***
dracut[I]: *** Including module: openssl ***
dracut[I]: *** Including module: shutdown ***
dracut[I]: *** Including modules done ***
dracut[I]: *** Installing kernel module dependencies ***
dracut[I]: *** Installing kernel module dependencies done ***
dracut[I]: *** Resolving executable dependencies ***
dracut[I]: *** Resolving executable dependencies done ***
dracut[I]: *** Hardlinking files ***
dracut[I]: *** Hardlinking files done ***
dracut[I]: *** Generating early-microcode cpio image ***
dracut[I]: *** Store current command line parameters ***
ldconfig: need absolute file name for configuration file when using -r
dracut[E]: ldconfig might need uid=0 (root) for chroot()
dracut[I]: *** Creating image file '/builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.65.fc43.x86_64+debug/vmlinuz-virt.efi' ***
dracut[I]: Using UEFI kernel cmdline:
dracut[I]:  console=tty0 console=ttyS0 
dracut[I]: *** Creating UEFI image file '/builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.65.fc43.x86_64+debug/vmlinuz-virt.efi' done ***

Bad build with dracut 107:

dracut[I]: Executing: /usr/bin/dracut --conf=/builddir/build/SOURCES/dracut-virt.conf --confdir=/tmp/tmp.YoohsNpweC --verbose --kver 6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug --kmoddir /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/ --logfile=/tmp/tmp.j3tkowPfdT --uefi --kernel-image /builddir/build/BUILD/kernel-6.16.0-build/kernel-6.16-rc5-38-g733923397fd9/linux-6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64/arch/x86/boot/bzImage --kernel-cmdline "console=tty0 console=ttyS0" /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/vmlinuz-virt.efi
/usr/lib/dracut/dracut-functions.sh: line 816: btrfs: command not found
dracut[I]: *** Including module: systemd ***
dracut[I]: *** Including module: fips ***
dracut[I]: *** Including module: systemd-ask-password ***
dracut[I]: *** Including module: systemd-cryptsetup ***
dracut[I]: *** Including module: systemd-initrd ***
dracut[I]: *** Including module: systemd-journald ***
dracut[I]: *** Including module: systemd-modules-load ***
dracut[I]: *** Including module: systemd-pcrphase ***
dracut[I]: *** Including module: systemd-sysctl ***
dracut[I]: *** Including module: systemd-sysext ***
dracut[I]: *** Including module: systemd-tmpfiles ***
dracut[I]: *** Including module: systemd-udevd ***
dracut[I]: *** Including module: systemd-veritysetup ***
dracut[I]: *** Including module: nss-softokn ***
dracut[I]: *** Including module: i18n ***
dracut[I]: *** Including module: crypt ***
dracut[I]: *** Including module: dm ***
dracut[I]: *** Including module: lvm ***
dracut[I]: *** Including module: crypt-loop ***
dracut[I]: *** Including module: tpm2-tss ***
dracut[I]: *** Including module: rootfs-block ***
dracut[I]: *** Including module: udev-rules ***
dracut[I]: *** Including module: virtiofs ***
dracut[I]: *** Including module: dracut-systemd ***
dracut[I]: *** Including module: base ***
grep: /etc/shadow: Permission denied
dracut[I]: *** Including module: fs-lib ***
dracut[I]: *** Including module: openssl ***
dracut[I]: *** Including module: shutdown ***
dracut[I]: *** Including module: systemd-sysusers ***
dracut[I]: *** Including modules done ***
dracut[I]: *** Installing kernel module dependencies ***
dracut[I]: *** Installing kernel module dependencies done ***
dracut[I]: *** Resolving executable dependencies ***
dracut[I]: *** Resolving executable dependencies done ***
dracut[I]: *** Hardlinking files ***
dracut[I]: *** Hardlinking files done ***
dracut[I]: *** Generating early-microcode cpio image ***
dracut[I]: *** Store current command line parameters ***
ldconfig: need absolute file name for configuration file when using -r
dracut[E]: ldconfig might need uid=0 (root) for chroot()
dracut[I]: *** Creating image file '/builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/vmlinuz-virt.efi' ***
cpio: etc/gshadow: Cannot open: Permission denied
dracut[F]: Creation of /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/vmlinuz-virt.efi failed

So:

* there's differences in the modules included
* as Justin noted, the errors around device-mapper stuff *went away* with 107
* finally, when the image creation actually happens, we get `cpio: etc/gshadow: Cannot open: Permission denied` with 107 and it fails

Hello, I've looked into this, and can't find any obvious commit, which might be causing this. Can you share the logs with `-vvv` (triple verbose) on dracut command line? Or maybe even `--debug` (if `-vvv` doesn’t help)? Alternatively, please provide a reproducer which is not part of kernel build? I tried to reproduce on regular system, the arguments as close as I could without the configs, and it did work fine: https://gist.github.com/pvalena/61a2d842a63dccd8a05195ddfcd034cf

FYI, the full change diff is here: https://gist.github.com/pvalena/12de812cc0ae7829c9d5479f2b44359e

I'd kinda guess you at least need to be running it inside a mock to hit this...

OK, so yeah, I can reproduce this by building a kernel in a mock. I can also reproduce it by shelling into the mock and running the same command **as the mockbuild user** - when run as root, it works.

-vvv doesn't change anything. --debug gives us very verbose output that ends:

/usr/bin/dracut@2447(): [[ -n xz ]]
/usr/bin/dracut@2463(): case $compress in
/usr/bin/dracut@2475(): compress='xz --check=crc32 --lzma2=dict=1MiB -T0'
/usr/bin/dracut@2497(): [[ -n '' ]]
/usr/bin/dracut@2523(): umask 077
/usr/bin/dracut@2524(): cd /var/tmp/dracut.dBoccTL/initramfs
/usr/bin/dracut@2525(): find . -print0
/usr/bin/dracut@2525(): sed -e 's,\./,,g'
/usr/bin/dracut@2525(): sort -z
/usr/bin/dracut@2526(): cpio -o --reproducible --null -R 0:0 -H newc --quiet
/usr/bin/dracut@2527(): xz --check=crc32 --lzma2=dict=1MiB -T0
cpio: etc/gshadow: Cannot open: Permission denied
cpio: etc/shadow: Cannot open: Permission denied
/usr/bin/dracut@2529(): dfatal 'Creation of /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/vmlinuz-virt.efi failed'
/usr/lib/dracut/dracut-logger.sh@457(dfatal): set +x
dracut[F]: Creation of /builddir/build/BUILD/kernel-6.16.0-build/BUILDROOT/lib/modules/6.16.0-0.rc5.250709g733923397fd9.47.fc43.x86_64+debug/vmlinuz-virt.efi failed
/usr/bin/dracut@2530(): exit 1
/usr/bin/dracut@2(): ret=1
/usr/bin/dracut@3(): [[ -n '' ]]
/usr/bin/dracut@3(): [[ -n /var/tmp/dracut.dBoccTL ]]
/usr/bin/dracut@3(): rm -rf -- /var/tmp/dracut.dBoccTL
/usr/bin/dracut@4(): [[ -n '' ]]
/usr/bin/dracut@7(): exit 1

I'll poke about at that a bit...

If I downgrade dracut to 105, it works, and this is how that bit looks:

/usr/bin/dracut@2412(): [[ -n xz ]]
/usr/bin/dracut@2428(): case $compress in
/usr/bin/dracut@2440(): compress='xz --check=crc32 --lzma2=dict=1MiB -T0'
/usr/bin/dracut@2462(): [[ -n '' ]]
/usr/bin/dracut@2488(): umask 077
/usr/bin/dracut@2489(): cd /var/tmp/dracut.U4znLc/initramfs
/usr/bin/dracut@2490(): find . -print0
/usr/bin/dracut@2490(): sed -e 's,\./,,g'
/usr/bin/dracut@2490(): sort -z
/usr/bin/dracut@2491(): cpio -o --reproducible --null -R 0:0 -H newc --quiet
/usr/bin/dracut@2492(): xz --check=crc32 --lzma2=dict=1MiB -T0
/usr/bin/dracut@2499(): (( maxloglvl >= 5 ))
/usr/bin/dracut@2507(): umask 077

so the *commands* are the same. I get the feeling it's trying to *include* etc/shadow and etc/gshadow in this archive now where it wasn't before, and that's the problem.

Created attachment 2096978 [details]
full --verbose "good" log (dracut 105, succeeded)

Created attachment 2096979 [details]
full --verbose "bad" log (dracut 107, failed)

OK, I bisected this. The cause is:

commit f3dacc013d90bd2c0bbfa04f5f9b167b65298440 (HEAD)
Author: Jo Zzsi <jozzsicsataban>
Date:   Sat Jan 4 18:45:32 2025 -0500

    feat(systemd-sysusers): run systemd-sysusers as part of the build process
    
    This PR makes the boot process faster and the generated initrd smaller.
    
    It also make the code easier to maintain.
    
    The primary goal of this PR is refactoring and removing code,
    this is why it is not marked as perf() in the commit.
    
    That said, with this PR systemd-sysusers no longer needs to be copied
    into the initrd, so that is about 50KB saving both on when initrd gets
    saved/compressed and when it gets loaded and uncompressed.
    
    This PR also make the debugging slightly easier as one can just
    inspect /etc/passwd with lsinitrd instead of trying to figure out
    which users are created runtime.

oh hey, somebody else bisected it in March. Sigh.

I've done a build of dracut with that commit reverted, so kernel builds will be possible again. That's probably not the 'real fix', though.

Likely the real fix - available since April - https://github.com/dracut-ng/dracut-ng/pull/1268

Yeah, I found that, but it's sitting around unmerged. It feels kinda icky to me - is running the command then wiping the files really the best idea? - but eh, dracut is one big hack, sooo...

Fixed upstream: https://github.com/dracut-ng/dracut-ng/pull/1445

Fixed via rebase to dracut 108: https://koji.fedoraproject.org/koji/taskinfo?taskID=136795166

Relevant part was reverted as of downstream patch; reopening for full fix.

PR: https://src.fedoraproject.org/rpms/dracut/pull-request/88