All ftp-related tests of python-pycurl fail on s390x on F40 and Rawhide due to vsftpd crashing. Reproducible: Always Steps to Reproduce: $ fedpkg clone python-pycurl $ cd python-pycurl # dnf builddep -y python-pycurl.spec $ fedpkg local Actual Results: All ftp python-pycurl tests fail on s390x due to vsftpd crashing. Expected Results: All ftp python-pycurl tests pass on s390x. The culprit is vsftpd: # coredumpctl info 7369 PID: 7369 (vsftpd) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Tue 2024-02-13 04:25:33 EST (18min ago) Command Line: vsftpd /root/python-pycurl/pycurl-7.45.2/tests/vsftpd.conf -oanon_root=/root/python-pycurl/pycurl-7.45.2/tests/.. Executable: /usr/sbin/vsftpd Control Group: /user.slice/user-0.slice/session-2.scope Unit: session-2.scope Slice: user-0.slice Session: 2 Owner UID: 0 (root) Boot ID: aba3857f7ff349efafdbd70c1296ba6f Machine ID: dda48676128e4244ab2f8488944bc74b Hostname: ibm-z-508.lab.eng.rdu2.redhat.com Storage: /var/lib/systemd/coredump/core.vsftpd.0.aba3857f7ff349efafdbd70c1296ba6f.7369.1707816333000000.zst (present) Size on Disk: 124.9K Package: vsftpd/3.0.5-6.fc40 build-id: fb298e5b944a23cabf0e89fb69be4c9c4f4e74ac Message: Process 7369 (vsftpd) of user 0 dumped core. Module libcap-ng.so.0 from rpm libcap-ng-0.8.4-4.fc40.s390x Module libz.so.1 from rpm zlib-ng-2.1.6-2.fc40.s390x Module libeconf.so.0 from rpm libeconf-0.5.2-3.fc40.s390x Module libaudit.so.1 from rpm audit-4.0-8.fc40.s390x Module libcrypto.so.3 from rpm openssl-3.2.1-2.fc40.s390x Module libcap.so.2 from rpm libcap-2.69-3.fc40.s390x Module libpam.so.0 from rpm pam-1.6.0-1.fc40.s390x Module libssl.so.3 from rpm openssl-3.2.1-2.fc40.s390x Module vsftpd from rpm vsftpd-3.0.5-6.fc40.s390x Stack trace of thread 7369: #0 0x000002aa30b93d00 hash_pid.lto_priv.0 (vsftpd + 0x13d00) #1 0x000002aa30b942d2 hash_get_node_by_key (vsftpd + 0x142d2) #2 0x000002aa30b976ea hash_add_entry (vsftpd + 0x176ea) #3 0x000002aa30b8833a main (vsftpd + 0x833a) #4 0x000003ffab7b4a5c __libc_start_call_main (libc.so.6 + 0x34a5c) #5 0x000003ffab7b4b5e __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x34b5e) #6 0x000002aa30b897b0 _start (vsftpd + 0x97b0) ELF object binary architecture: IBM S/390 GDB: (gdb) bt #0 hash_pid (buckets=256, p_key=0x0) at standalone.c:338 #1 0x000002aa30b942d2 in hash_get_bucket (p_key=0x0, p_hash=p_hash@entry=0x2aa3267f720) at hash.c:123 #2 hash_get_node_by_key (p_hash=p_hash@entry=0x2aa3267f720, p_key=p_key@entry=0x0) at hash.c:134 #3 0x000002aa30b976ea in hash_lookup_entry (p_key=0x0, p_hash=0x2aa3267f720) at hash.c:54 #4 hash_add_entry (p_hash=0x2aa3267f720, p_key=0x0, p_value=<optimized out>) at hash.c:67 #5 0x000002aa30b8833a in vsf_standalone_main () at standalone.c:202 #6 main (argc=<optimized out>, argv=<optimized out>) at main.c:158 Frame 0 looks like this: │ 333 │ 334 static unsigned int │ 335 hash_pid(unsigned int buckets, void* p_key) │ 336 { │ 337 unsigned int* p_pid = (unsigned int*)p_key; │ > 338 return (*p_pid) % buckets; │ 339 } │ 340 So this is a null pointer dereference. Frame 5 looks like this: │ 198 /* Parent context */ │ 199 vsf_sysutil_close(new_client_sock); │ 200 if (new_child > 0) │ 201 { │ > 202 hash_add_entry(s_p_pid_ip_hash, (void*)&new_child, p_raw_addr); │ 203 } │ 204 else │ 205 { There is no way the second argument can be 0x0 but it is still somehow mangled to that value as shown in the backtrace above. The following spec change workarounds the bug. So this is apparently a miscompilation when LTO is enabled on s390x. diff --git a/vsftpd.spec b/vsftpd.spec index a05c3f7..b65492f 100644 --- a/vsftpd.spec +++ b/vsftpd.spec @@ -1,3 +1,4 @@ +%global _lto_cflags %nil %global _generatorsdir %{_prefix}/lib/systemd/system-generators Name: vsftpd
Reassigning to GCC as this is probably a compiler bug. Note that the same bug may also affect ELN and the current CentOS Stream 10 bootstrap.
Is there some simple way to test this? Building another package and running its testsuite is significantly more costly than running some simple command which I'd need to run on the vsftpd binaries in mock building that package...
Sure. I was able to reduce the reproducer to this on a fresh Fedora Rawhide s390x VM: # dnf install -y vsftpd # vsftpd -oanon_root=. -oallow_anon_ssl=true & [1] 3561 # curl ftp://localhost/ curl: (67) Access denied: 530 [1]+ Done vsftpd -oanon_root=. -oallow_anon_ssl=true # coredumpctl info PID: 3562 (vsftpd) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Mon 2024-02-19 04:35:54 EST (1min 44s ago) Command Line: vsftpd -oanon_root=. -oallow_anon_ssl=true Executable: /usr/sbin/vsftpd Control Group: /user.slice/user-0.slice/session-2.scope Unit: session-2.scope Slice: user-0.slice Session: 2 Owner UID: 0 (root) Boot ID: bf7ce2ca4dd94947bcdbfe7ef6f60bc7 Machine ID: ec24cdedf45647eaa122dfe2a9444a6c Hostname: s390x-kvm-056.lab.eng.rdu2.redhat.com Storage: /var/lib/systemd/coredump/core.vsftpd.0.bf7ce2ca4dd94947bcdbfe7ef6f60bc7.3562.1708335354000000.zst (p> Size on Disk: 125.3K Package: vsftpd/3.0.5-6.fc40 build-id: fb298e5b944a23cabf0e89fb69be4c9c4f4e74ac Message: Process 3562 (vsftpd) of user 0 dumped core. Module libcap-ng.so.0 from rpm libcap-ng-0.8.4-4.fc40.s390x Module libz.so.1 from rpm zlib-ng-2.1.6-2.fc40.s390x Module libeconf.so.0 from rpm libeconf-0.5.2-3.fc40.s390x Module libaudit.so.1 from rpm audit-4.0-8.fc40.s390x Module libcrypto.so.3 from rpm openssl-3.2.1-2.fc40.s390x Module libcap.so.2 from rpm libcap-2.69-3.fc40.s390x Module libpam.so.0 from rpm pam-1.6.0-2.fc41.s390x Module libssl.so.3 from rpm openssl-3.2.1-2.fc40.s390x Module vsftpd from rpm vsftpd-3.0.5-6.fc40.s390x Stack trace of thread 3562: #0 0x000002aa13d93d00 hash_pid.lto_priv.0 (vsftpd + 0x13d00) #1 0x000002aa13d942d2 hash_get_node_by_key (vsftpd + 0x142d2) #2 0x000002aa13d976ea hash_add_entry (vsftpd + 0x176ea) #3 0x000002aa13d8833a main (vsftpd + 0x833a) #4 0x000003ff9b0b4a5c __libc_start_call_main (libc.so.6 + 0x34a5c) #5 0x000003ff9b0b4b5e __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x34b5e) #6 0x000002aa13d897b0 _start (vsftpd + 0x97b0) ELF object binary architecture: IBM S/390
This is a glibc bug.
Upstream patch under review: https://patchwork.sourceware.org/project/glibc/patch/20240221161307.2821102-1-stli@linux.ibm.com/
This is now fixed upstream, and the next Rawhide update will pull this in: commit 02782fd12849b6673cb5c2728cb750e8ec295aa3 Author: Stefan Liebler <stli.com> Date: Thu Feb 22 15:03:27 2024 +0100 S390: Do not clobber r7 in clone [BZ #31402] Starting with commit e57d8fc97b90127de4ed3e3a9cdf663667580935 "S390: Always use svc 0" clone clobbers the call-saved register r7 in error case: function or stack is NULL. This patch restores the saved registers also in the error case. Furthermore the existing test misc/tst-clone is extended to check all error cases and that clone does not clobber registers in this error case.
The Fedora Rawhide glibc rebase for glibc-2.39.9000-5.fc41 includes the fix. We need bring in the fix from upstream for Fedora 40 and Fedora 39 to fix this across. The fix has beeen backported upstream into glibc 2.38, and glibc 2.39 so the next sync will bring these into Fedora.
FEDORA-2024-0ccb7fb6a6 (glibc-2.39-4.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2024-0ccb7fb6a6
FEDORA-2024-8e1ba83d6b (glibc-2.38-17.fc39) has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-8e1ba83d6b
FEDORA-2024-0ccb7fb6a6 has been pushed to the Fedora 40 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-0ccb7fb6a6` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-0ccb7fb6a6 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-8e1ba83d6b has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-8e1ba83d6b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-8e1ba83d6b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-0ccb7fb6a6 (glibc-2.39-4.fc40) has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2024-8e1ba83d6b (glibc-2.38-17.fc39) has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.