Bug 1956807 - dnf segfaults during kickstart (libcurl.so.4.7.0) with ssl repositories
Summary: dnf segfaults during kickstart (libcurl.so.4.7.0) with ssl repositories
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 34
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Lukáš Hrázký
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-04 13:05 UTC by bbk
Modified: 2021-05-12 12:27 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)

Description bbk 2021-05-04 13:05:36 UTC
Description of problem:

During the kickstart in a post-install script calling dnf segfaults when having ssl encrypted repositories:

```
[Tue May  4 14:26:48 2021] traps: dnf[37008] general protection fault ip:7faa4cf34457 sp:7ffd1fa12900 error:0 in libcurl.so.4.7.0[7faa4cedd000+70000]
```


How reproducible:

I think the simplest way to reproduce is to create a %post block in the kickstart file adding a `sleep 10000` and then:

1. Go the the kickstart console
2. chroot /mnt/sysroot
3. dnf clean all
4. dnf check-update

Actual results:

```
Segmentation fault (core dumped)                 [               ===                         ] ---  B/s |   0  B     --:-- ETA
```

```
[Tue May  4 14:26:48 2021] traps: dnf[37008] general protection fault ip:7faa4cf34457 sp:7ffd1fa12900 error:0 in libcurl.so.4.7.0[7faa4cedd000+70000]
```

Expected results:

dnf should be able to install packages during %post installation blocks.

Comment 1 bbk 2021-05-04 13:39:58 UTC
I tried to get more information with coredumpctl:

```
Program terminated with signal SIGSEGV, Segmentation fault.
Python Exception <class 'NameError'> Installation error: gdb._execute_unwinders function is missing: 
#0  0x00007faa4cf34457 in ossl_new_session_cb () from /lib64/libcurl.so.4
```

Comment 2 bbk 2021-05-04 14:11:23 UTC
as ossl_new_session_cb() suggests, you get only affected by this bug when using repositories with ssl.

I use foreman/katello to provision my hosts https is the default, so i have to workaround this bug in using only http:// repositories.

Comment 3 Lukáš Hrázký 2021-05-10 13:58:20 UTC
Hello, the reproducer is quite vague for people not familiar with kickstart, can you please provide the kickstart file and the command you run? Thank you.

Comment 4 bbk 2021-05-11 09:34:49 UTC
Ok, while preparing a kickstart file i was not able to reproduce the error without having a foreman/katello server setup. Maybe it is also possible to reproduce the error on a satlite server.



However the process of using a kickstart file requires at for example a pxe boot setup (dhcp, tftp, ipxe).

Then i used libvirt, i created a bride interface where the VM can connect and PXE boot.

The kickstart file i used, but where i could not reproduce the error is:

```

##########################################################
# Fedorabuger Kickstart
##########################################################

#
# Agree EULA
eula --agreed
#
# System language
lang en_US.UTF-8
#
# System keyboard
keyboard us
#
# System timezone
timezone --utc Europe/Zurich 
#
# Reboot after installation
reboot
#
# Network configuration
network --bootproto dhcp
#
# Disable Initial-Setup Service
firstboot --disable
#
# Root password
# create with python -c 'import crypt; print(crypt.crypt("password", "$6$My Salt"))'
rootpw --iscrypted $6$fjdfioefjdklfjio$uA6HHPzJFBBR6uv9CczoRfjHlthAMxsaXkYnWp3GQc7K5BEqcIp8zPtqG2JeUcj.m3AhvEx/G2qThEmKHa/rS0
#
# Root ssh password during kickstart
# create with python -c 'import crypt; print(crypt.crypt("password", "$6$My Salt"))'
sshpw --username root --iscrypted "$6$fjdfioefjdklfjio$uA6HHPzJFBBR6uv9CczoRfjHlthAMxsaXkYnWp3GQc7K5BEqcIp8zPtqG2JeUcj.m3AhvEx/G2qThEmKHa/rS0"
#
#
#disable SELinux
selinux --permissive
#
# Firewall configuration
firewall --service=ssh
#
# System bootloader configuration
# use grub2-mkpasswd-pbkdf2 to generate the password password
bootloader --append="rhgb quiet splash=quiet" --iscrypted --password=grub.pbkdf2.sha512.10000.3526A56082F76B0A78B10A8CF874D1C8E3AE3FE19AD7909CFF7B7EEE0E6678A269D862FF9A093494E60736683A1E07FD087E2D74406BF3B3919073FDC137CCF2.1570F3AF48E6A6B86306AF82A3C6D69A98A9E9AC3E54B2889934110BA722969B9ED984E0C69886B6F9436356D83ED82CB2387734C0B5E912F7E834414F62D7CC
#
# Installation mode
text
#
# Add repos during kickstart
repo --name "Everything" --mirrorlist https://mirrors.fedoraproject.org/metalink?repo=fedora-33&arch=x86_64

#
# Storage
clearpart --all
autopart

#########################################
# START packagegroups-minimal template
# Package Groups
#
%packages --ignoremissing
@minimal-environment
@admin-tools
@system-tools
%end
#
# END packagegroups-minimal template
#########################################

#########################################
# START packages-minimal template
# minimal packages
#
%packages --ignoremissing
atop
bash-completion
bc
binutils-devel
bzr
bzrtools
ccache
cmake
dmidecode
efibootmgr
fedora-cert
ftp
git
htop
iftop
iotop
iptables
jq
kernel
kernel-devel
kernel-headers
kernel-tools
mc
mutt
nmap
postfix
redhat-lsb-core
strace
tftp
tmux
unrar
wget
xmlstarlet
%end
#
# END packages-minimal template
#########################################


#########################################
# START tail template
# the last template to run
#

%post --erroronfail --log /root/kickstart-log/template-tail.log


# permit ssh root login
echo "PermitRootLogin yes" >> /etc/ssh/sshd_config





#########################################
# the debug section
cat << EOF > /etc/yum.repos.d/eddy33.repo
[eddy33]
name=eddy33's repo for Fedora \$releasever
baseurl=https://kadionik.vvv.enseirb-matmeca.fr/fedora/\$releasever/\$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/GPG-KEY-eddy33

[eddy33-updates]
name=eddy33's updates for Fedora \$releasever
baseurl=https://kadionik.vvv.enseirb-matmeca.fr/fedora/updates/\$releasever/\$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/GPG-KEY-eddy33
EOF

dnf clean all
dnf check-update


sleep 100000
#########################################





%end
#
# END tail template
#########################################

```

Comment 5 Lukáš Hrázký 2021-05-11 14:17:09 UTC
Well, thanks for the kickstart file, but you say it does not reproduce it in the end? And you are mentioning several other prerequisites, which I'm not sure how to set up and it's not even clear which are ultimately required and which not.

Comment 6 Lukáš Hrázký 2021-05-11 14:44:40 UTC
Also, can you post full backtrace from the coredump? (via `coredumpctl debug ID` and `backtrace full` in gdb console)

Comment 7 bbk 2021-05-12 09:47:58 UTC
For my provisioning i use Foreman with the Katello plugin:

* https://theforeman.org/
* https://theforeman.org/plugins/katello/

It is quite a complex setup, but concering the repositories it basically provides them over https:// with a self signed certificate. Before registering (during kickstart) a new host with the subscription-manager (https://linux.die.net/man/8/subscription-manager), it runs a script placing all the necessary certificates in the new installation.

As it seems only to affect the foreman/katello provisioning i started a forum thread in the foreman forum: https://community.theforeman.org/t/fedora-34-dnf-segfault-with-ssl-repositories/23530

Here is the full backtrace (i hope i did it the right way):

```
#0  0x00007f7f29d61427 in ossl_new_session_cb () from /lib64/libcurl.so.4
No symbol table info available.
#1  0x00007f7f2987ce85 in ssl_update_cache () from /lib64/libssl.so.1.1
No symbol table info available.
#2  0x00007f7f298afd87 in tls_finish_handshake.constprop () from /lib64/libssl.so.1.1
No symbol table info available.
#3  0x00007f7f2988f998 in state_machine.part () from /lib64/libssl.so.1.1
No symbol table info available.
#4  0x00007f7f29863fd6 in ssl3_read_bytes () from /lib64/libssl.so.1.1
No symbol table info available.
#5  0x00007f7f29872aca in ssl3_read_internal () from /lib64/libssl.so.1.1
No symbol table info available.
#6  0x00007f7f29876f67 in SSL_read () from /lib64/libssl.so.1.1
No symbol table info available.
#7  0x00007f7f29d6b416 in ossl_recv () from /lib64/libcurl.so.4
No symbol table info available.
#8  0x00007f7f29d49b3a in Curl_read () from /lib64/libcurl.so.4
No symbol table info available.
#9  0x00007f7f29d5aea9 in Curl_readwrite () from /lib64/libcurl.so.4
No symbol table info available.
#10 0x00007f7f29d44004 in multi_runsingle () from /lib64/libcurl.so.4
No symbol table info available.
#11 0x00007f7f29d47026 in curl_multi_perform () from /lib64/libcurl.so.4
No symbol table info available.
#12 0x00007f7f2a972312 in lr_download () from /lib64/librepo.so.0
No symbol table info available.
#13 0x00007f7f2a973f50 in lr_download_single_cb () from /lib64/librepo.so.0
No symbol table info available.
#14 0x00007f7f2a983cf1 in lr_yum_download_repo () from /lib64/librepo.so.0
No symbol table info available.
#15 0x00007f7f2a984584 in lr_yum_perform () from /lib64/librepo.so.0
No symbol table info available.
#16 0x00007f7f2a97a448 in lr_handle_perform () from /lib64/librepo.so.0
No symbol table info available.
#17 0x00007f7f2aaaef02 in libdnf::Repo::Impl::lrHandlePerform(_LrHandle*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /lib64/libdnf.so.2
No symbol table info available.
#18 0x00007f7f2aab0aa1 in libdnf::Repo::Impl::fetch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<_LrHandle, std::default_delete<_LrHandle> >&&) () from /lib64/libdnf.so.2
No symbol table info available.
#19 0x00007f7f2aab182f in libdnf::Repo::Impl::load() () from /lib64/libdnf.so.2
#20 0x00007f7f291ecf4c in _wrap_Repo_load.lto_priv.0 () from /usr/lib64/python3.9/site-packages/libdnf/_repo.so
No symbol table info available.
#21 0x00007f7f2b96969b in cfunction_vectorcall_O () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#22 0x00007f7f2b962362 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#23 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#24 0x00007f7f2b95d8e9 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#25 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#26 0x00007f7f2b972501 in method_vectorcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#27 0x00007f7f2b962362 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#28 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#29 0x00007f7f2b95d8e9 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#30 0x00007f7f2b95c2fd in _PyEval_EvalCode () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#31 0x00007f7f2b96a01e in _PyFunction_Vectorcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#32 0x00007f7f2b972501 in method_vectorcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#33 0x00007f7f2b95e66a in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#34 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#35 0x00007f7f2b95d8e9 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#36 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#37 0x00007f7f2b95d8e9 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#38 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#39 0x00007f7f2b95d680 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#40 0x00007f7f2b96a343 in function_code_fastcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#41 0x00007f7f2b95d680 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#42 0x00007f7f2b95c2fd in _PyEval_EvalCode () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#43 0x00007f7f2b96a01e in _PyFunction_Vectorcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#44 0x00007f7f2b95d680 in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#45 0x00007f7f2b95c2fd in _PyEval_EvalCode () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#46 0x00007f7f2b96a01e in _PyFunction_Vectorcall () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#47 0x00007f7f2b95e66a in _PyEval_EvalFrameDefault () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#48 0x00007f7f2b95c2fd in _PyEval_EvalCode () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#49 0x00007f7f2b9d8fa5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#50 0x00007f7f2b9d8f3d in PyEval_EvalCodeEx () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#51 0x00007f7f2b9d8eef in PyEval_EvalCode () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#52 0x00007f7f2ba05703 in run_eval_code_obj () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#53 0x00007f7f2ba01516 in run_mod () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#54 0x00007f7f2b8d94e3 in pyrun_file.cold () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#55 0x00007f7f2b9fbc33 in PyRun_SimpleFileExFlags () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#56 0x00007f7f2b9f9188 in Py_RunMain () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#57 0x00007f7f2b9cb9bd in Py_BytesMain () from /lib64/libpython3.9.so.1.0
No symbol table info available.
#58 0x00007f7f2b6acb75 in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#59 0x000055a9bbc5609e in _start ()
No symbol table info available.
(gdb) 
```

Comment 8 Lukáš Hrázký 2021-05-12 12:27:27 UTC
Okay, I forgot to mention you should install *-debuginfo packages so that gdb prints the symbols in the back trace. gdb should ask you this at the start and give you the dnf command to install those.

Alternatively, `dnf debuginfo-install libdnf librepo libcurl openssl-libs` should install those that matter here.

Then you repeat the steps to get the backtrace and it should have some more info in it. Hopefully it will be a bit more helpful :)

Is there a way for you to force older versions of dnf, libdnf and librepo for the installation, so that we can check the crash is really caused by new versions of the dnf stack? (and then if not, you could do the same for libcurl and openssl-libs, those seem to be the only packages through which this goes)


Note You need to log in before you can comment on or make changes to this bug.