Bug 1980910

Summary: install fails when using https for installation source because curl cannot load ca-bundle.crt
Product: Red Hat Enterprise Linux 9 Reporter: Alex Schultz <aschultz>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED DUPLICATE QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0CC: bstinson, dracut-maint-list, dtardon, jkonecny, jstodola, jwboyer, mlewando
Target Milestone: betaKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-14 15:27:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Schultz 2021-07-09 19:57:14 UTC
Description of problem:
Attempting to use an https source for the install.img results in the installation hanging because it cannot verify the ssl certificate due to:

[   11.753820] dracut-initqueue[1203]: curl: (77) error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt


Version-Release number of selected component (if applicable):


How reproducible:

100% if using https source


Steps to Reproduce:
1. Use virt install to attempt to install from the odcs composes
WORK_DIR=$(pwd)
MIRROR="https://odcs.stream.centos.org/test/latest-CentOS-Stream/compose"
DISK_SIZE=10


virt-install \
    --transient \
    --name=centos-9-stream \
    --ram=4096 \
    --arch=x86_64 \
    --cpu=host \
    --vcpus=4 \
    --os-variant=rhel8.0 \
    --extra-args="ipv6.disable=1 inst.text console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH" \
    --disk="${WORK_DIR}/centos-9-stream.img,size=${DISK_SIZE},sparse=true,format=qcow2" \
    --location="${MIRROR}/BaseOS/x86_64/os" \
    --serial=pty \
    --nographics


Actual results:

Install haults because it cannot fetch the install.img

[   10.239528] dracut-initqueue[1071]: Warning: can't find installer main image path in .treeinfo
[   10.264253] dracut-initqueue[1209]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   10.266804] dracut-initqueue[1209]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[   10.327793] dracut-initqueue[1209]: curl: (77) error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt
[   10.333588] dracut-initqueue[1203]: Warning: Downloading 'https://odcs.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os/images/install.img' failed!
[   10.361152] dracut-initqueue[1219]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   10.361407] dracut-initqueue[1219]:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[   10.418645] dracut-initqueue[1219]: curl: (77) error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt
[   10.435797] dracut-initqueue[1213]: Warning: Downloading 'https://odcs.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os/LiveOS/squashfs.img' failed!
[   10.436456] dracut-initqueue[1071]: Warning: anaconda: failed to fetch stage2 from https://odcs.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os
<info>  [1625860137.9996] policy: set-hostname: set hostname to 'localhost.localdomain' (no hostname found)
[  148.798622] dracut-initqueue[1071]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[  148.805771] dracut-initqueue[1071]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2froot.sh: "[ -e "/dev/root" ]"
[  148.813008] dracut-initqueue[1071]: Warning: /lib/dracut/hooks/initqueue/finished/nm.sh: "[ -f /tmp/nm.done ]"
[  148.819697] dracut-initqueue[1071]: Warning: /lib/dracut/hooks/initqueue/finished/wait_for_settle.sh: "[ -f /tmp/settle.done ]"
[  148.829479] dracut-initqueue[1071]: Warning: dracut-initqueue: starting timeout scripts
...SNIP...
[  219.804591] dracut-initqueue[1071]: Warning: /lib/dracut/hooks/initqueue/finished/wait_for_settle.sh: "[ -f /tmp/settle.done ]"
[  219.808806] dracut-initqueue[1071]: Warning: dracut-initqueue: starting timeout scripts
[  219.809385] dracut-initqueue[1071]: Warning: Could not boot.
         Starting Dracut Emergency Shell...
Warning: /dev/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


Press Enter for maintenance
(or press Control-D to continue): 



Expected results:

install.img should be downloaded and the installation should continue.

Additional info:

If you configure a passthrough proxy (e.g. via nginx) that handles the http/https transition the installation works fine.

Comment 1 Alex Schultz 2021-07-09 20:07:54 UTC
switched component to anaconda as it appears the request is coming from https://github.com/rhinstaller/anaconda/blob/master/dracut/anaconda-lib.sh

Comment 2 Alex Schultz 2021-07-09 20:11:17 UTC
Adding inst.noverifyssl to the extra-args seems to disable the ssl verification and allows it to continue

Comment 3 Jan Stodola 2021-07-12 15:19:52 UTC
The whole /etc/pki is missing in the RHEL-9 installation initrd:

$ lsinitrd /mnt/redhat/rhel-9/nightly/RHEL-9-Beta/RHEL-9.0.0-20210709.2/compose/BaseOS/x86_64/os/images/pxeboot/initrd.img | grep etc/pki
$

Compare with RHEL-8:
$ lsinitrd /mnt/redhat/rhel-8/nightly/RHEL-8/RHEL-8.5.0-20210712.n.1/compose/BaseOS/x86_64/os/images/pxeboot/initrd.img | grep etc/pki
drwxr-xr-x   4 root     root            0 Apr 26 03:30 etc/pki
drwxr-xr-x   3 root     root            0 Apr 26 03:30 etc/pki/ca-trust
drwxr-xr-x   3 root     root            0 Apr 26 03:30 etc/pki/ca-trust/extracted
drwxr-xr-x   2 root     root            0 Apr 26 03:30 etc/pki/ca-trust/extracted/pem
-r--r--r--   1 root     root       200578 Apr 26 03:30 etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
drwxr-xr-x   3 root     root            0 Apr 26 03:30 etc/pki/tls
drwxr-xr-x   2 root     root            0 Apr 26 03:30 etc/pki/tls/certs
lrwxrwxrwx   1 root     root           46 Apr 26 03:30 etc/pki/tls/certs/ca-bundle.crt -> ../../ca-trust/extracted/pem/tls-ca-bundle.pem
$

Comment 4 David Tardon 2021-07-14 08:42:57 UTC
*** Bug 1975278 has been marked as a duplicate of this bug. ***

Comment 5 Jiri Konecny 2021-07-14 15:27:43 UTC
Hi, this issue is happening because newer version of Dracut is using bash features which needs a bit different setup. 
This issue is already fixed in upstream Lorax in the bug 1962975. There is now clone for RHEL-9 bug 1982271 . Closing this one as duplicate of the upstream clone because it has more details.

*** This bug has been marked as a duplicate of bug 1982271 ***