This was being discussed at https://pagure.io/releng/failed-composes/issue/2515 , but now we know more or less what is going on, I'm filing a bug for better tracking. Rawhide composes have been failing since 2021-05-18; the last successful compose was 20210517.n.0. The failure coincides with dracut-054-5.git20210517.fc35 being tagged. The failure mode is that when building disk images, anaconda fails to boot, instead we are dumped in the initramfs environment. This is happening because curl cannot download stage2 via https, because the CA cert bundle curl needs for SSL transactions - /etc/pki/tls/certs/ca-bundle.crt - is missing. This in turn appears to be happening because the dracut url-lib module is going wrong when it attempts to parse curl library files to find out and include the location of the certificate bundle. We see these errors in the dracut output when building the initramfs: 2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: dracut: Couldn't find SSL CA cert bundle or libnssckbi.so; HTTPS won't work. Those errors come exactly from the block which should be including the certificate bundle in the initramfs: for _dir in $libdirs; do [[ -d $dracutsysrootdir$_dir ]] || continue for _lib in "$dracutsysrootdir$_dir"/libcurl.so.*; do [[ -e $_lib ]] || continue if ! [[ $_nssckbi ]]; then read -r -d '' _nssckbi < <(grep -F --binary-files=text -z libnssckbi "$_lib") fi read -r -d '' _crt < <(grep -F --binary-files=text -z .crt "$_lib") [[ $_crt ]] || continue [[ $_crt == /*/* ]] || continue if ! inst "${_crt#$dracutsysrootdir}"; then dwarn "Couldn't install '$_crt' SSL CA cert bundle; HTTPS might not work." continue fi _found=1 done done lines 35 and 37 are the two lines that do `read -r -d '' somevar < <(somecommand)` to try and initialize a variable with the output from a command. Note we see similar errors elsewhere in the dracut output from two lines in other modules that use the same `< <(somecommand)` construction: 2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/00systemd/module-setup.sh: line 168: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/90kernel-modules-extra/module-setup.sh: line 77: /dev/fd/63: No such file or directory so that is what seems to be the problem. These lines were changed in https://github.com/dracutdevs/dracut/commit/8e84fa7262d5d3273684473538d94a008b5f583e . They used to just do `somevar=$(somecommand)`. Not sure exactly why this is happening, yet.
Marking as an automatic F35 Beta blocker per https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers - "Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release".
Maybe anaconda or the image builder should setup a proper chroot for scripts to run correctly, instead of fixing all scripts, which could possibly run during an image build. I would assign this to anaconda or to the environment setting up the chroot for the dracut process to run in.
Basically nowadays I refuse to make workarounds just for incomplete chroot environments.
Anyway, the real fix would be to fix the chroot setup, where the symlink `/dev/fd -> /proc/self/fd` is missing.
See: https://github.com/weldr/lorax/pull/1149 with my POC patch, lorax builds.. # lorax -p Fedora -v 34 -r 34 -s http://dl.fedoraproject.org/pub/fedora/linux/releases/34/Everything/x86_64/os/ -s https://dl.fedoraproject.org/pub/fedora/linux/updates/34/Everything/x86_64/ -s https://dl.fedoraproject.org/pub/fedora/linux/updates/testing/34/Everything/x86_64/ ./results/ # lsinitrd ./results/isolinux/initrd.img|fgrep ca-bundle -r--r--r-- 1 root root 216090 May 18 10:36 etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem lrwxrwxrwx 1 root root 46 May 18 10:36 etc/pki/tls/certs/ca-bundle.crt -> ../../ca-trust/extracted/pem/tls-ca-bundle.pem
The lorax build installed dracut-054-6.git20210518.fc34.x86_64
https://github.com/weldr/lorax/pull/1151
FEDORA-2021-4764329a16 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-4764329a16
FEDORA-2021-4764329a16 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
We've retagged the newer dracut build, we'll see if things go ok with the next compose.