Bug 1962975 - use of `< <(somecommand)` in dracut modules doesn't work (causes "/dev/fd/63: No such file or directory" errors), breaks stage2 retrieval and hence Rawhide composes
Summary: use of `< <(somecommand)` in dracut modules doesn't work (causes "/dev/fd/63:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lorax
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Brian Lane
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F35BetaBlocker 1982271
TreeView+ depends on / blocked
 
Reported: 2021-05-20 23:24 UTC by Adam Williamson
Modified: 2021-07-14 15:17 UTC (History)
11 users (show)

Fixed In Version: lorax-35.3-1 lorax-35.3-1.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1982271 (view as bug list)
Environment:
Last Closed: 2021-05-25 19:54:36 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2021-05-20 23:24:35 UTC
This was being discussed at https://pagure.io/releng/failed-composes/issue/2515 , but now we know more or less what is going on, I'm filing a bug for better tracking. Rawhide composes have been failing since 2021-05-18; the last successful compose was 20210517.n.0. The failure coincides with dracut-054-5.git20210517.fc35 being tagged.

The failure mode is that when building disk images, anaconda fails to boot, instead we are dumped in the initramfs environment. This is happening because curl cannot download stage2 via https, because the CA cert bundle curl needs for SSL transactions - /etc/pki/tls/certs/ca-bundle.crt - is missing. This in turn appears to be happening because the dracut url-lib module is going wrong when it attempts to parse curl library files to find out and include the location of the certificate bundle. We see these errors in the dracut output when building the initramfs:

2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: dracut: Couldn't find SSL CA cert bundle or libnssckbi.so; HTTPS won't work.

Those errors come exactly from the block which should be including the certificate bundle in the initramfs:

    for _dir in $libdirs; do
        [[ -d $dracutsysrootdir$_dir ]] || continue
        for _lib in "$dracutsysrootdir$_dir"/libcurl.so.*; do
            [[ -e $_lib ]] || continue
            if ! [[ $_nssckbi ]]; then
                read -r -d '' _nssckbi < <(grep -F --binary-files=text -z libnssckbi "$_lib")
            fi
            read -r -d '' _crt < <(grep -F --binary-files=text -z .crt "$_lib")
            [[ $_crt ]] || continue
            [[ $_crt == /*/* ]] || continue
            if ! inst "${_crt#$dracutsysrootdir}"; then
                dwarn "Couldn't install '$_crt' SSL CA cert bundle; HTTPS might not work."
                continue
            fi
            _found=1
        done
    done

lines 35 and 37 are the two lines that do `read -r -d '' somevar < <(somecommand)` to try and initialize a variable with the output from a command. Note we see similar errors elsewhere in the dracut output from two lines in other modules that use the same `< <(somecommand)` construction:

2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/00systemd/module-setup.sh: line 168: /dev/fd/63: No such file or directory
2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/90kernel-modules-extra/module-setup.sh: line 77: /dev/fd/63: No such file or directory

so that is what seems to be the problem.

These lines were changed in https://github.com/dracutdevs/dracut/commit/8e84fa7262d5d3273684473538d94a008b5f583e . They used to just do `somevar=$(somecommand)`.

Not sure exactly why this is happening, yet.

Comment 1 Adam Williamson 2021-05-20 23:25:21 UTC
Marking as an automatic F35 Beta blocker per https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers - "Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release".

Comment 2 Harald Hoyer 2021-05-21 07:30:04 UTC
Maybe anaconda or the image builder should setup a proper chroot for scripts to run correctly, instead of fixing all scripts, which could possibly run during an image build.

I would assign this to anaconda or to the environment setting up the chroot for the dracut process to run in.

Comment 3 Harald Hoyer 2021-05-21 07:35:56 UTC
Basically nowadays I refuse to make workarounds just for incomplete chroot environments.

Comment 4 Harald Hoyer 2021-05-21 08:02:26 UTC
Anyway, the real fix would be to fix the chroot setup, where the symlink `/dev/fd -> /proc/self/fd` is missing.

Comment 5 Harald Hoyer 2021-05-21 15:26:42 UTC
See: https://github.com/weldr/lorax/pull/1149

with my POC patch, lorax builds..

# lorax -p Fedora -v 34 -r 34           -s http://dl.fedoraproject.org/pub/fedora/linux/releases/34/Everything/x86_64/os/           -s https://dl.fedoraproject.org/pub/fedora/linux/updates/34/Everything/x86_64/      -s https://dl.fedoraproject.org/pub/fedora/linux/updates/testing/34/Everything/x86_64/     ./results/

# lsinitrd ./results/isolinux/initrd.img|fgrep ca-bundle
-r--r--r--   1 root     root       216090 May 18 10:36 etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx   1 root     root           46 May 18 10:36 etc/pki/tls/certs/ca-bundle.crt -> ../../ca-trust/extracted/pem/tls-ca-bundle.pem

Comment 6 Harald Hoyer 2021-05-21 15:28:01 UTC
The lorax build installed dracut-054-6.git20210518.fc34.x86_64

Comment 7 Brian Lane 2021-05-24 22:04:44 UTC
https://github.com/weldr/lorax/pull/1151

Comment 8 Fedora Update System 2021-05-25 16:37:43 UTC
FEDORA-2021-4764329a16 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-4764329a16

Comment 9 Fedora Update System 2021-05-25 19:54:36 UTC
FEDORA-2021-4764329a16 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 10 Adam Williamson 2021-05-25 23:31:21 UTC
We've retagged the newer dracut build, we'll see if things go ok with the next compose.


Note You need to log in before you can comment on or make changes to this bug.