+++ This bug was initially created as a clone of Bug #1436158 +++

Description of problem:

My Fedora client is managed by Katello 3.3 server. When I run dnf update -y on it, dnf will crash.


Version-Release number of selected component (if applicable):

$ rpm -qa | grep dnf
python3-dnf-1.1.10-6.fc25.noarch
dnf-1.1.10-6.fc25.noarch
python3-dnf-plugins-core-0.1.21-4.fc25.noarch
dnf-yum-1.1.10-6.fc25.noarch
dnf-conf-1.1.10-6.fc25.noarch
dnf-plugins-core-0.1.21-4.fc25.noarch



Steps to Reproduce:
sudo dnf update -y

Actual results:

[SKIPPED] qemu-kvm-2.7.1-4.fc25.x86_64.rpm: Already downloaded                                                                                                                         
[SKIPPED] qemu-guest-agent-2.7.1-4.fc25.x86_64.rpm: Already downloaded                                                                                                                 
(153/258): realmd-0.16.2-5.fc25_0.16.2-8.fc25.x86_64.drpm                                                                                               398 kB/s |  91 kB     00:00    
[DRPM] openldap-2.4.44-2.fc25_2.4.44-7.fc25.x86_64.drpm: done                                                                                                                          
[DRPM] openslp-2.0.0-9.fc25_2.0.0-10.fc25.x86_64.drpm: done                                                                                                                            
[DRPM] openvas-scanner-5.0.6-1.fc25_5.0.7-1.fc25.x86_64.drpm: done                                                                                                                     
[DRPM] opus-1.1.3-1.fc25_1.1.3-2.fc25.x86_64.drpm: done                                                                                                                                
[DRPM] oz-0.15.0-2.fc25_0.15.0-5.fc25.noarch.drpm: done                                                                                                                                
(154/258): redhat-rpm-config-44-1.fc25_45-1.fc25.noarch.drpm                                                                                            121 kB/s |  29 kB     00:00    
(155/258): redland-1.0.17-7.fc25_1.0.17-8.fc25.x86_64.drpm                                                                                              220 kB/s |  54 kB     00:00    
[DRPM] pam-kwallet-5.8.1-1.fc25_5.8.6-1.fc25.x86_64.drpm: done                                                                                                                         
Segmentation fault (core dumped)8.fc25_2.3.3-61.1.fc25.x86_64.drpm       57% [=========================================-                              ] 413 kB/s |  95 MB     02:51 ETA



$ dmesg -T | grep dnf
[Mon Mar 27 09:40:14 2017] dnf[13839]: segfault at 8 ip 00007f3489082af0 sp 00007ffc88c176e0 error 4 in libc-2.24.so[7f3489013000+1bd000]
[Mon Mar 27 09:43:34 2017] dnf[14322]: segfault at 8 ip 00007fbdff0bfaf0 sp 00007ffc523ed950 error 4 in libc-2.24.so[7fbdff050000+1bd000]
[Mon Mar 27 09:44:29 2017] dnf[14766]: segfault at ffffffffffffffff ip ffffffffffffffff sp 00007ffe428fbd98 error 15
[Mon Mar 27 09:54:31 2017] dnf[18951]: segfault at 7f006e69616d ip 00007f006e69616d sp 00007ffcc31b59b8 error 14
[Mon Mar 27 09:54:51 2017] dnf[19128]: segfault at 0 ip 00007f3ebb285ace sp 00007ffdb55b3fa0 error 4 in libc-2.24.so[7f3ebb216000+1bd000]
[Mon Mar 27 10:02:15 2017] dnf[22116]: segfault at 8 ip 00007f7a97cafaf0 sp 00007ffeb85392d0 error 4 in libc-2.24.so[7f7a97c40000+1bd000]
[Mon Mar 27 10:20:19 2017] dnf[28485]: segfault at 300010080 ip 0000000300010080 sp 00007ffdba6ffee8 error 14 in system-python[55997cd79000+1000]
[Mon Mar 27 10:22:33 2017] dnf[28634]: segfault at 300010080 ip 0000000300010080 sp 00007ffe278ff178 error 14 in system-python[5588b1965000+1000]
[Mon Mar 27 11:00:46 2017] dnf[29962]: segfault at 8 ip 00007faaa53dcaf0 sp 00007ffc5a027650 error 4 in libc-2.24.so[7faaa536d000+1bd000]
[Mon Mar 27 11:38:27 2017] dnf[11741]: segfault at 300010080 ip 0000000300010080 sp 00007fff12a53858 error 14 in system-python[5643fb95d000+1000]
[Mon Mar 27 11:43:32 2017] dnf[28404]: segfault at 55adbbd2a2d0 ip 000055adbbd2a2d0 sp 00007ffd0d4e0528 error 15
[Mon Mar 27 11:43:42 2017] traps: dnf[28505] general protection ip:7f3f0b9a6093 sp:7ffd27a0ba98 error:0


Additional info:

Katello server logs at the time crash happens:

Mar 27 11:38:23 puppet.teamwpc.local pulp_streamer[7694]: [-] 127.0.0.1 - - [27/Mar/2017:10:38:23 +0000] "GET /var/lib/pulp/content/units/rpm/f8/11c045bd847e72db972379806893a549596f111bc8150d68bf376acb066d76/qemu-system-cris-2.7.1-4.fc25.x86_64.rpm HTTP/1.1" 200 1524246 "-" "dnf/1.1.10"
Mar 27 11:38:24 puppet.teamwpc.local pulp_streamer[7694]: [-] 127.0.0.1 - - [27/Mar/2017:10:38:24 +0000] "GET /var/lib/pulp/content/units/rpm/94/e1891fc8e9ae091dac2630af92df0119db3c9828b2b0fdc2f4c5617267b760/qemu-kvm-2.7.1-4.fc25.x86_64.rpm HTTP/1.1" 200 67682 "-" "dnf/1.1.10"
Mar 27 11:38:24 puppet.teamwpc.local pulp_streamer[7694]: [-] 127.0.0.1 - - [27/Mar/2017:10:38:24 +0000] "GET /var/lib/pulp/content/units/rpm/df/7ed0caedf48386aad7eaef5c7fa002256ba9ec3dde8a4e77fa58aa7220ec41/qemu-guest-agent-2.7.1-4.fc25.x86_64.rpm HTTP/1.1" 200 204150 "-" "dnf/1.1.10"
Mar 27 11:38:24 puppet.teamwpc.local pulp_streamer[7694]: [-] 127.0.0.1 - - [27/Mar/2017:10:38:24 +0000] "GET /var/lib/pulp/content/units/rpm/42/39ad5e40d634c1650e874933ea69200c431c912ee8faac7c3402b00d449ac6/qt5-srpm-macros-5.7.1-1.fc25.noarch.rpm HTTP/1.1" 200 8066 "-" "dnf/1.1.10"

nothing our of ordinary here.

FYI, the same Katello server serves all my CentOS 6 and 7 servers and there are no problems with it at all. The problem is Fedora specific.

There are few things here you should be aware of:

1) If I run dnf update against Fedora mirror - everything works. It only crashes when DOWNLOADING from Katello server. Once the download stage is finished - it installs updates fine.

2) When I run smaller updates - like dnf update o* p* q* -y it WILL complete successfully. It only crashes when I try to upgrade the entire system (or a significantly high number of packages - 300+). The workstation I'm installing updates on has 64GB or RAM and it's NOT running out of memory during the upgrade process.

--- Additional comment from Kamil Dudka on 2017-07-19 19:34:10 CEST ---

I have identified a possible bug in libcurl source code.  Could you please try libcurl from the following scratch build?

https://koji.fedoraproject.org/koji/taskinfo?taskID=20614836

--- Additional comment from Rob Sanders on 2017-07-19 20:03:22 CEST ---

Looks like you've nailed it!

I can no longer re-produce this issue after upgrading curl/libcurl and trust me, I've tried. Previously I could reproduce it every few seconds:

[Wed Jul 19 17:06:16 2017] yum[26175]: segfault at a1 ip 00000000000000a1 sp 00007ffd5a113e18 error 14 in system-python[55e3ef392000+2000]
[Wed Jul 19 17:06:28 2017] traps: yum[26320] general protection ip:7ff60b330aee sp:7ffe5c0a8040 error:0 in libc-2.25.so[7ff60b2be000+1c7000]
[Wed Jul 19 17:06:35 2017] traps: yum[26354] general protection ip:7fe03c3a1063 sp:7ffcf206fa28 error:0 in libcurl.so.4.4.0[7fe03c389000+7c000]
[Wed Jul 19 17:06:41 2017] traps: yum[26413] general protection ip:7fa9483df063 sp:7ffc4bbe5a08 error:0 in libcurl.so.4.4.0[7fa9483c7000+7c000]



Is it possible to push this change to F25? This will save me - and others - lots of manual patching before bigger F26 rollout.

--- Additional comment from Kamil Dudka on 2017-07-19 23:04:58 CEST ---

Perfect.  Thanks for confirmation!  I will prepare builds for f25/f26 tomorrow.  It is interesting that you discovered the bug now because it was introduced in 2009 (exactly 8 years ago) by this commit:

https://github.com/curl/curl/commit/curl-7_19_5-204-g5f0cae803

--- Additional comment from Kamil Dudka on 2017-07-20 08:23:35 CEST ---

Fix pushed upstream:

https://github.com/curl/curl/commit/curl-7_54_0-406-gc89eb6d
https://github.com/curl/curl/commit/curl-7_54_0-407-g42a4cd4