curl-7.53.1-7 changed the default curl source of trust from /etc/pki/tls/certs/ca-bundle.crt to (libdir)/libnssckbi.so: http://pkgs.fedoraproject.org/cgit/rpms/curl.git/commit/?id=9549974a4cbd8216ce694e85ce69aa128ff939dc since that change, no TLS transactions using curl have worked in the initramfs environment, because that file is missing from it. The "obvious" fix is this: [adamw@adam dracut (curl-nssckbi %)]$ git diff diff --git a/modules.d/45url-lib/module-setup.sh b/modules.d/45url-lib/module-setup.sh index 1ece400f..0fcf7f91 100755 --- a/modules.d/45url-lib/module-setup.sh +++ b/modules.d/45url-lib/module-setup.sh @@ -20,6 +20,7 @@ install() { inst_multiple -o ctorrent inst_multiple curl # also install libs for curl https + inst_libdir_file "libnssckbi.so*" inst_libdir_file "libnsspem.so*" inst_libdir_file "libnsssysinit.so*" inst_libdir_file "libsoftokn3.so*" but I'm not sure if that will be tripped up by the file being a symlink. I'm going to run some tests now and will file a dracut PR if I can verify that i have a working fix. Nominating as Fedora 26 Beta release blocker, because this prevents the use of any kickstart or updates image on an HTTPS server: "The scripted installation mechanism must provide a working function for creating local user accounts, including the ability to specify a hashed password, and for specifying a hashed password for the root account." "The installer must be able to download and use an installer update image from an HTTP server." https://fedoraproject.org/wiki/Fedora_26_Alpha_Release_Criteria#Scripted_user_creation
This is the bug that's breaking the openQA install_kickstart_firewall_configured and install_kickstart_firewall_disabled tests - they are the ones that use an HTTPS-hosted kickstart - and install_scsi_updates_img - that one uses an HTTPS-hosted updates image.
Good news: inst_libdir_file manages to resolve the symlinks and install the file. Bad news: the nssckbi route still needs a certificate trust source of some kind, and we're not including it, whatever it is (I think it's /etc/pki/nssdb/cert9.db and /etc/pki/nssdb/key4.db ). This vaguely terrifying snippet is, apparently, how dracut tries to include the appropriate cert bundle for curl: for _dir in $libdirs; do [[ -d $_dir ]] || continue for _lib in $_dir/libcurl.so.*; do [[ -e $_lib ]] || continue _crt=$(grep -F --binary-files=text -z .crt $_lib) [[ $_crt ]] || continue [[ $_crt == /*/* ]] || continue if ! inst "$_crt"; then dwarn "Couldn't install '$_crt' SSL CA cert bundle; HTTPS might not work." continue fi _found=1 done done [[ $_found ]] || dwarn "Couldn't find SSL CA cert bundle; HTTPS won't work." from https://git.kernel.org/pub/scm/boot/dracut/dracut.git/tree/modules.d/45url-lib/module-setup.sh#n32 - that's just not doing the right thing for this (new?) codepath in curl at all, I don't think.
https://github.com/dracutdevs/dracut/pull/226 is my proposal to fix this. It does work in my testing, but there may be improvements that could be made to it.
(In reply to Adam Williamson from comment #0) > but I'm not sure if that will be tripped up by the file being a symlink. Symlink to what? > Nominating as Fedora 26 Beta release blocker, because this prevents the use > of any kickstart or updates image on an HTTPS server: Sounds reasonable.
*** Bug 1447276 has been marked as a duplicate of this bug. ***
kdudka: libnssckbi.so is handled by alternatives (for some reason) so /usr/lib64/libnssckbi.so is a symlink to /etc/alternatives/libnssckbi.so.x86_64 , which is a symlink to /usr/lib64/pkcs11/p11-kit-trust.so . But just using dracut's 'inst_libdir_file' function works okay, it installs the actual file into the initramfs.
CCing anaconda folks for information - if you notice / get reports of kickstart installs failing in recent F26, this is likely why.
OK, I did not realize it was a symlink. Then it could probably be switched to the legacy implementation of libnssckbi.so. Nevertheless, if you made it work with p11-kit-trust.so, it is even better because it provides a more reliable source of trust.
I'm seeing this when trying to use the mirrors as a repo for the anaconda install. Sometimes I get an https mirror (in which case I see a failure) and sometimes I get an http mirror (in which case it succeeds). Even worse is the failures happen at different times during the install because i could get one mirror for downloading stage2 and a different one when anaconda is doing software selection and thus if one of those is https things will fail.
actually, this should theoretically only affect initramfs so failures later in the install shouldn't be related to this bug
Well, it's possible the same issue exists in the installer environment I guess, as that gets stripped by lorax in some ways. But I haven't checked on that, and it would theoretically be a separate bug if it was the case (as it won't be fixed by fixing dracut).
Where can I download an installation image to debug this?
https://www.happyassassin.net/nightlies.html
I was not able to reproduce the problem using the following image: https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170508.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170508.n.0.iso After it booted, I switched to terminal using Ctrl + Alt + F2 and typed the following command: # curl -svo/dev/null https://google.com It connected over TLS successfully. The verbose output contained: * loaded libnssckbi.so Please provide details on how to trigger the bug mentioned in comment #9.
A command like this should do it: ``` virt-install --name builder --ram 1500 --vcpus 2 --disk path=.//builder.img,size=12 --accelerate --location https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os --graphics none --force --noreboot --network network=default --extra-args=console=ttyS0 ks=https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks inst.sshd ``` which ends up with a VM that has these errors: ``` [ 10.326047] dracut-initqueue[632]: Warning: can't find installer mainimage path in .treeinfo [ 10.342555] dracut-initqueue[632]: % Total % Received % Xferd Average Speed Time Time Time Current [ 10.345015] dracut-initqueue[632]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 10.367653] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?) [ 10.374227] dracut-initqueue[632]: Warning: Downloading 'https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os/images/install.img' failed! [ 10.387305] dracut-initqueue[632]: % Total % Received % Xferd Average Speed Time Time Time Current [ 10.389286] dracut-initqueue[632]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 10.473252] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?) [ 10.476304] dracut-initqueue[632]: Warning: Downloading 'https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os/LiveOS/squashfs.img' failed! [ 10.478516] dracut-initqueue[632]: Warning: Could not retrieve stage2 image from https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os using ens2 [ 11.022875] dracut-initqueue[632]: % Total % Received % Xferd Average Speed Time Time Time Current [ 11.024902] dracut-initqueue[632]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 11.126322] dracut-initqueue[632]: curl: (77) Problem with the SSL CA cert (path? access rights?) [ 11.127208] dracut-initqueue[632]: Warning: failed to fetch kickstart from https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks ```
note that you did not see an error because you were already into anaconda installer. I'm seeing the error before we get to anaconda, in dracut. I had some issues with quoting in that previous output. Use this instead: ``` virt-install --name builder --ram 1500 --vcpus 2 --disk path=.//builder.img,size=12 --accelerate --location https://download.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os --graphics none --force --noreboot --network network=default --extra-args='console=ttyS0 ks=https://kojipkgs.fedoraproject.org//work/tasks/748/19420748/koji-f26-build-19420748-base.ks inst.sshd' ```
Kamil: that's not the initramfs environment. That's the installer environment. The easiest way to observe the problem is to just try and use a kickstart or updates image hosted on HTTPS, e.g. using this kernel parameter: inst.ks=https://fedorapeople.org/groups/qa/kickstarts/firewall-configured-net.ks it should result in a curl error, anaconda will fail to start, and eventually the system should dump you at a bash prompt in the initramfs environment, from where you can poke around and confirm the problem.
Using virt-manager, as described here: https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot Using vmlinuz and initrd.img from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/images/pxeboot/ And using console=ttyS0 inst.repo=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os as kernel args I hit the same issue [ 5.463750] 8139cp 0000:00:03.0 ens3: renamed from eth0 [ 5.577739] 8139cp 0000:00:03.0 ens3: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.625156] dracut-initqueue[658]: Warning: can't find installer mainimage path in .treeinfo [ 8.636791] dracut-initqueue[658]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.637492] dracut-initqueue[658]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 8.842588] dracut-initqueue[658]: curl: (77) Problem with the SSL CA cert (path? access rights?) [ 8.845857] dracut-initqueue[658]: Warning: Downloading 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/images/install.img' failed! [ 8.860042] dracut-initqueue[658]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.863912] dracut-initqueue[658]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 9.129313] dracut-initqueue[658]: curl: (77) Problem with the SSL CA cert (path? access rights?) [ 9.133672] dracut-initqueue[658]: Warning: Downloading 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os/LiveOS/squashfs.img' failed! [ 9.135731] dracut-initqueue[658]: Warning: Could not retrieve stage2 image from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170504.n.1/compose/Workstation/x86_64/os using ens3 [ 94.732084] random: crng init done [ 132.395467] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts [ 132.914615] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts [ 133.440413] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts [ 133.949967] dracut-initqueue[658]: Warning: dracut-initqueue timeout - starting timeout scripts ... [ 193.311689] dracut-initqueue[658]: Warning: Could not boot. [ 193.423458] dracut-initqueue[658]: Warning: /dev/root does not exist Starting Setup Virtual Console... [ OK ] Started Setup Virtual Console. ... Warning: /dev/root does not exist Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. dracut:/#
Yes, that would be expected. This bug will affect any codepath which involves downloading via HTTPS from the initramfs environment (before pivot to the installer environment), which obviously includes retrieving the installer environment itself via HTTPS.
OK. I was confused by comment #11 where a (theoretical?) bug in the installer environment was mentioned. Do I understand it correctly that only dracut is affected by this bug and the bug is going to be fixed by the patch mentioned in comment #3? Anyway, thank all of you for providing the steps to reproduce! I am able to debug it locally now: dracut:/# curl -svo/dev/null https://google.com * Rebuilt URL to: https://google.com/ * Trying 172.217.22.14... * TCP_NODELAY set * Connected to google.com (172.217.22.14) port 443 (#0) * Initializing NSS with certpath: none * CAfile: none CApath: none * failed to load libnssckbi.so * Closing connection 0 dracut:/# stat /usr/lib64/libnssckbi.so stat: cannot stat '/usr/lib64/libnssckbi.so': No such file or directory dracut:/# stat /etc/pki/tls/certs/ca-bundle.crt stat: cannot stat '/etc/pki/tls/certs/ca-bundle.crt': No such file or directory
Yeah, in #c11 we were discussing that the same issue *may* exist in the installer environment, but it doesn't look like that's the case. Note that /etc/pki/tls/certs/ca-bundle.crt is not, in fact, the important bundle - that's the bundle curl *previously* used, but with libnssckbi , the important bundle file is /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit (at least, on Fedora that's where it lives), so that's why my patch pulls in *that* bundle file.
Maybe it's not yet fully clear to you how libnssckbi.so and the replacement p11-kit-trust.so work. The symlink named libnssckbi.so can alternatively point to either (a) the file libnssckbi.so provided by a nss.rpm package. This is a statically operating module. It doesn't read any other files. It contains the set of CA trust as provided by Mozilla, both positive trust and distrust. It ignores any local system configuration. Your guess in comment 2 is wrong, libnssckbi.so doesn't access /etc/pki/nssdb/ It's an application decision if an application uses /etc/pki/nssdb/ as the NSS database init path. I don't know how curl initializes NSS. If curl chooses to use the read-only NSS init APIs, which don't take any database parameter, then directory /etc/pki/nssdb/ isn't accessed. If curl uses read-only NSS init APIs, and curl loads the module named libnssckbi.so, then having the nss.rpm package installed should be sufficient to give you the default trust, and allow the loading of libnssckbi.so to work. (b) p11-kit-trust.so from the p11-kit-trust.rpm package. If this is installed, you need much more to operate correctly. Please don't try to guess what files you should manually copy to make it work. If p11-kit-trust is installed, you should make sure that the ca-certificates package is installed, too. The ca-certificates package installs multiple files. There is no guarantee that any single file is sufficient for p11-kit-trust.so to work. In order to get the intended behavior, you need the full set of files that the p11-kit-trust package installs. p11-kit-trust.so will dynamically scan the /usr/share/pki/ca-trust-source and /etc/pki/ca-trust/source directories (both), to learn about the shipped CAs. Some files inside thoes directories might be symlinks, which point to files outside these two directories. This is used by the ca-legacy configuration, to potentially enable an additional set of CAs for backward compatibility. The files /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt are *NOT* read by p11-kit-trust.so Those are backwards compatibility filenames, that point to a converted and combined result of all trust information. For its internal operation, sufficient for applications that load the p11-kit-trust.so module as their source of CA trust, it's sufficient for it to be able to read all the trust source directories (plus the result of navigating symbolic links inside those trust source directories). For compatibility with applications that don't support loading p11-kit-trust.so, it's possible to run the utility update-ca-trust, which will call into p11-kit-trust.so, and ask it to produce the set union of all trust, and write it to different file formats, e.g. java keystore file format, and openssl plain certificate file format (ca-bundle.crt is a symlink to that), and openssl extended certificate file format that includes trust/distrust flags (ca-bundle-trust.crt is a symlink to that).
So, if your intention is to create an environment, where curl works to access https sites, and you don't need any local CA configuration in that environment, then it should be sufficient to pull in the libnssckbi.so file from the nss.rpm package. (Whatever way you use to obtain libnssckbi.so from nss.rpm, don't make it a one time copy that you check in somewhere. It should be a dynamic operation, to pull libnssckbi.so from the most recently published nss.rpm package.) I believe libnssckbi.so doesn't need anything else from NSS, but you must have installed nspr.rpm
Thanks for the explanation, Kai! In order to make it easier to maintain (although less secure), I propose to pick only %{libdir}/nss/libnssckbi.so and create %{libdir}/libnssckbi.so symlink to it.
I talked it through in quite a lot of detail with Kai, and we agreed to an approach where we install whatever libnssckbi.so we find in a libdir (dracut already handles the problem of differing libdirs across distros) and then grep it for p11 trust directories and install any we find. I have updated the pull request to implement this. We decided against just trying to always find and install the original NSS libnssckbi.so for a couple of reasons: i) it *is* less configurable, and this is technically a regression from the previous behaviour - previously curl was using ca-bundle.crt , which respects local configuration of trust, so if we go from that to the NSS libnssckbi.so on a system where p11-kit-trust.so is available, we're going backwards ii) we don't actually have any particularly reliable way to *find* the NSS libnssckbi.so (or even a cast iron guarantee it'll always be there); sure, right now on Fedora it's in libdir/nss , but what about other distros, if they also switch away from using a CA bundle file? what if it gets moved? We can search through all the subdirs of all libdirs with `find`, or something, but with a system like alternatives, it doesn't even really *have* to be under a libdir. The PR has been updated, please do take a look and see what you think! Thanks.
Discussed in the 2017-05-02 Blocker Review meeting [0]. This is a violation of the following criterion: "The installer must be able to download and use an installer update image from an HTTP server" (note: 'HTTP' there is meant as 'HTTP not FTP or NFS', it does not mean 'we don't expect HTTPS to work'), and Accepted as a Blocker. [0] Sorry for the late update on this.
dracut-044-181.fc26
dracut-044-181.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fa869268a2
Dusty created a test image for anyone who wants to verify this fix: https://dustymabe.fedorapeople.org/new-dracut.iso if anyone else can check, the update needs one more +1. thanks!
dracut-044-181.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fa869268a2
(In reply to Adam Williamson from comment #29) > https://dustymabe.fedorapeople.org/new-dracut.iso It works. In virt-manager, using Direct kernel boot, providing vmlinuz and initrd.img taken from this ISO, and using "console=ttyS0 inst.repo=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170513.n.0/compose/Workstation/x86_64/os" as kernel parameters I am able to successfully complete QA:Testcase Boot Methods Pxeboot (as stated before, it fails using vmlinuz and initrd.img from 20170513.n.0 compose).
dracut-044-181.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.