The external package cisco AnyConnect VPN crashes permanently after upgrading glibc to glibc-2.37-5.fc38; Downgrading glibc to glibc-2.37-1.fc38 (or even to glibc-2.37-4.fc38) helps. Here is the coredump of the latest crash: coredumpctl dump /opt/cisco/anyconnect/bin/vpnagentd PID: 17985 (vpnagentd) UID: 1000 (<user>) GID: 1000 (<user>) Signal: 11 (SEGV) Timestamp: Sun 2023-09-17 09:31:32 CEST (6s ago) Command Line: vpnagentd -verify_certs /opt/cisco/anyconnect/pem.V5zCuC 4 1 <URL> <config>.xml Executable: /opt/cisco/anyconnect/bin/vpnagentd Control Group: /system.slice/vpnagentd.service Unit: vpnagentd.service Slice: system.slice Boot ID: a72177313ccc4048a8c099654b0df0b5 Machine ID: 90ae3673a52f497c89e5d7fee71a9244 Hostname: <hostname> Storage: /var/lib/systemd/coredump/core.vpnagentd.1000.a72177313ccc4048a8c099654b0df0b5.17985.1694935892000000.zst (present) Size on Disk: 3.2M Message: Process 17985 (vpnagentd) of user 1000 dumped core. Module libpcsclite.so.1 from rpm pcsc-lite-1.9.9-3.fc38.x86_64 Module libcrypto.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libopensc.so.8 from rpm opensc-0.23.0-5.fc38.x86_64 Module opensc-pkcs11.so from rpm opensc-0.23.0-5.fc38.x86_64 Module libtasn1.so.6 from rpm libtasn1-4.19.0-2.fc38.x86_64 Module p11-kit-trust.so from rpm p11-kit-0.25.0-1.fc38.x86_64 Module p11-kit-proxy.so from rpm p11-kit-0.25.0-1.fc38.x86_64 Module libnspr4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libplds4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libplc4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libblkid.so.1 from rpm util-linux-2.38.1-4.fc38.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc38.1.x86_64 Module libffi.so.8 from rpm libffi-3.4.4-2.fc38.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-1.fc38.x86_64 Module libmount.so.1 from rpm util-linux-2.38.1-4.fc38.x86_64 Module libgmodule-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module liblzma.so.5 from rpm xz-5.4.1-1.fc38.x86_64 Module libglib-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libgobject-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libgio-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64 Module libxml2.so.2 from rpm libxml2-2.10.4-1.fc38.x86_64 Stack trace of thread 17985: #0 0x0000000000000000 n/a (n/a + 0x0) #1 0x00007fdffa52fc80 SECMOD_SlotDestroyModule (libnss3.so + 0x69c80) #2 0x00007fdffa5300bf SECMOD_DestroyModuleListElement (libnss3.so + 0x6a0bf) #3 0x00007fdffa530535 SECMOD_DestroyModuleList (libnss3.so + 0x6a535) #4 0x00007fdffa5305bd SECMOD_Shutdown (libnss3.so + 0x6a5bd) #5 0x00007fdffa4e5421 nss_Shutdown (libnss3.so + 0x1f421) #6 0x00007fdffa4e5520 NSS_Shutdown (libnss3.so + 0x1f520) #7 0x00007fdffca7d689 _ZN13CNSSCertUtilsD2Ev (libvpncommoncrypt.so + 0x7d689) #8 0x00007fdffb061c3d __cxa_finalize (libc.so.6 + 0x3fc3d) #9 0x00007fdffca2a2d3 n/a (libvpncommoncrypt.so + 0x2a2d3) #10 0x00007fdffd2e70f2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x10f2) #11 0x00007fdffd2eae0e _dl_fini (ld-linux-x86-64.so.2 + 0x4e0e) #12 0x00007fdffb0621e6 __run_exit_handlers (libc.so.6 + 0x401e6) #13 0x00007fdffb06232e exit (libc.so.6 + 0x4032e) #14 0x0000562ef802f401 n/a (vpnagentd + 0x2f401) #15 0x00007fdffb049b8a __libc_start_call_main (libc.so.6 + 0x27b8a) #16 0x00007fdffb049c4b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c4b) #17 0x0000562ef802f638 n/a (vpnagentd + 0x2f638) ELF object binary architecture: AMD x86-64 Reproducible: Always Steps to Reproduce: 1. Start VPN 2. Make an authentication attempt 3. Got certificate download failure Cisco AnyConnect Secure Mobility Client Version 4.10.02086 is used
Here is corresponding call stack: (gdb) bt #0 0x0000000000000000 in () #1 0x00007fdffa51422d in SECMOD_UnloadModule (mod=mod@entry=0x562ef9e59220) at pk11load.c:664 #2 0x00007fdffa52fc80 in SECMOD_SlotDestroyModule (module=0x562ef9e59220, fromSlot=<optimized out>) at pk11util.c:954 #3 0x00007fdffa5300bf in SECMOD_DestroyModuleListElement (element=0x562ef9e81890) at pk11util.c:971 #4 0x00007fdffa530535 in SECMOD_DestroyModuleList (list=<optimized out>) at pk11util.c:986 #5 0x00007fdffa5305bd in SECMOD_Shutdown () at pk11util.c:68 #6 0x00007fdffa4e5421 in nss_Shutdown () at nssinit.c:1163 #7 0x00007fdffa4e5520 in NSS_Shutdown () at nssinit.c:1221 #8 0x00007fdffca7d689 in CNSSCertUtils::~CNSSCertUtils() () at /opt/cisco/anyconnect/lib/libvpncommoncrypt.so #9 0x00007fdffb061c3d in __cxa_finalize (d=0x7fdffcc9ea08) at cxa_finalize.c:82 #10 0x00007fdffca2a2d3 in () at /opt/cisco/anyconnect/lib/libvpncommoncrypt.so #11 0x00007ffc989020a0 in () #12 0x00007fdffd2e70f2 in _dl_call_fini (closure_map=0x7fdffd2e45b0) at dl-call_fini.c:43 (gdb) l Downloading source file /usr/src/debug/nss-3.93.0-1.fc38.x86_64/nss/lib/pk11wrap/<built-in> 1 /usr/src/debug/nss-3.93.0-1.fc38.x86_64/nss/lib/pk11wrap/<built-in>: Directory not empty. (gdb) up #1 0x00007fdffa51422d in SECMOD_UnloadModule (mod=mod@entry=0x562ef9e59220) at pk11load.c:664 Downloading source file /usr/src/debug/nss-3.93.0-1.fc38.x86_64/nss/lib/pk11wrap/pk11load.c 664 PK11_GETTAB(mod)->C_Finalize(NULL); (gdb) l 659 if (!mod->loaded) { 660 return SECFailure; 661 } 662 if (finalizeModules) { 663 if (mod->functionList && !mod->moduleDBOnly) { 664 PK11_GETTAB(mod)->C_Finalize(NULL); 665 } 666 } 667 mod->moduleID = 0; 668 mod->loaded = PR_FALSE;
Thank you for tracking this. We encountered it on friday. One thing to add : the crash won't happen if the user has no firefox profile in ~/.mozilla/firefox/... Cisco AnyConnect VPN Client looks into firefox profiles explicitly, as referenced in /opt/cisco/anyconnect/lib/libvpncommoncrypt.so.
(In reply to Guillaume Cassonnet from comment #2) > Thank you for tracking this. We encountered it on friday. > > One thing to add : the crash won't happen if the user has no firefox profile > in ~/.mozilla/firefox/... > Cisco AnyConnect VPN Client looks into firefox profiles explicitly, as > referenced in /opt/cisco/anyconnect/lib/libvpncommoncrypt.so. This is also true in my case. But I observe other weird side effects: Usually I need to restart `systemd-resolved` after shutting the VPN down. This time restarting of the `systemd-resolved` ended with timeout. I had to reboot my machine to get it back to stable state.
Do you use smartcard authentication with the VPN? It would be helpful if you could run the crashing process with the LD_DEBUG=all environment variable both under glibc-2.37-5.fc38 and glibc-2.37-4.fc38. We changed the order in which ELF destructors are invoked in glibc-2.37-5.fc38, but I don't know yet if the new implementation has a bug, or if it merely exposed a pre-existing bug in a system library or the VPN client.
Created attachment 1989459 [details] LD_DEBUG=all /opt/cisco/anyconnect/bin/vpnui 2>&1 | tee > /tmp/vpnui_glibc-2.37-4.fc38.lo
Created attachment 1989460 [details] LD_DEBUG=all /opt/cisco/anyconnect/bin/vpnui 2>&1 | tee > /tmp/vpnui_glibc-2.37-5.fc38.log
We used to do this: /* Now we have to do the sorting. We can skip looking for the binary itself which is at the front of the search list for the main namespace. */ _dl_sort_maps (maps, nmaps, (ns == LM_ID_BASE), true); That is, force_first is true for the base namespace. Force-first handling is gone from the new version. Instead, we might call destructors of objects first which have been dlopen'ed after main has been called. I suspect this is what is causing this bug. The other major difference is that we interleave destructors from different namespaces. We used to process namespaces in reverse ID order. Hopefully this does not matter because dlmopen is still rarely used, and reverse namespace ID order is quite arbitrary—it matches reverse dlmopen order only without out-of-order dlclose calls for namespaces. This really doesn't bode well for fixing the (different, but superficially similar) C++ destructor ordering issue.
Created attachment 1989465 [details] Candidate patch
(In reply to Florian Weimer from comment #12) > Created attachment 1989465 [details] > Candidate patch I agree with the core concept of this patch that the executable has to be the last thing constructed and the first thing destructed, with follow-on effects applied like dlclose() calls from the shutdown APIs of various directly linked shared objects. Not because it is technically the correct thing to do, but because applications have come to expected that the executable destructors run first because the executable is seen as a "last constructed, first destructed" sentinel of sorts; even though dlopen()-ed objects after main() should be the first to be destructed in proper LIFO fashion. I tested this against a reproducer I have with AnyConnect 4.10.06090 and.... it resolves the crash. New glibc which runs executable destructors first: LD_DEBUG=all ./glibc-2.38.9000-10.fc40.x86_64/lib64/ld-linux-x86-64.so.2 --library-path /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64:/home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn ./anyconnect-linux64-4.10.06090/vpn/vpnagentd -d -n 10 -verify_certs ./anyconnect-linux64-4.10.06090/vpn/DigiCertAssuredIDRootCA.pem >& unload_debug_all_new.txt grep "calling fini:" unload_debug_all_new.txt 7840: calling fini: [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnagentutilities.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnapi.so [0] 7840: calling fini: /usr/lib64/libsmime3.so [0] 7840: calling fini: /usr/lib64/libnss3.so [0] 7840: calling fini: /lib64/libnssutil3.so [0] 7840: calling fini: /lib64/libplc4.so [0] 7840: calling fini: /lib64/libplds4.so [0] 7840: calling fini: /lib64/libnspr4.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpncommoncrypt.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpncommon.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libaccurl.so.4 [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libacciscossl.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libacciscocrypto.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_thread.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_system.so [0] 7840: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_regex.so [0] 7840: calling fini: /lib64/libxml2.so.2 [0] 7840: calling fini: /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64/libpthread.so.0 [0] 7840: calling fini: /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64/librt.so.1 [0] 7840: calling fini: /lib64/libstdc++.so.6 [0] 7840: calling fini: /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64/libm.so.6 [0] 7840: calling fini: /lib64/libgcc_s.so.1 [0] 7840: calling fini: /lib64/libgio-2.0.so.0 [0] 7840: calling fini: /lib64/libz.so.1 [0] 7840: calling fini: /lib64/libgobject-2.0.so.0 [0] 7840: calling fini: /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64/libdl.so.2 [0] 7840: calling fini: /lib64/liblzma.so.5 [0] 7840: calling fini: /lib64/libgmodule-2.0.so.0 [0] 7840: calling fini: /lib64/libglib-2.0.so.0 [0] 7840: calling fini: /lib64/libmount.so.1 [0] 7840: calling fini: /lib64/libselinux.so.1 [0] 7840: calling fini: /lib64/libffi.so.8 [0] 7840: calling fini: /lib64/libpcre2-8.so.0 [0] 7840: calling fini: /lib64/libblkid.so.1 [0] 7840: calling fini: /home/carlos/cisco/glibc-2.38.9000-10.fc40.x86_64/lib64/libc.so.6 [0] 7840: calling fini: ./glibc-2.38.9000-10.fc40.x86_64/lib64/ld-linux-x86-64.so.2 [0] [SIC no crash] Current glibc which strictly runs in reverse order and causes a crash: LD_DEBUG=all LD_LIBRARY_PATH=/home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn ./anyconnect-linux64-4.10.06090/vpn/vpnagentd -d -n 10 -verify_certs ./anyconnect-linux64-4.10.06090/vpn/DigiCertAssuredIDRootCA.pem >& unload_debug_all.txt Segmentation fault (core dumped) grep 'calling fini:' unload_debug_all.txt 7872: calling fini: /lib64/libpcsclite.so.1 [0] 7872: calling fini: /usr/lib64/pkcs11/opensc-pkcs11.so [0] 7872: calling fini: /lib64/libopensc.so.8 [0] 7872: calling fini: /lib64/libcrypto.so.3 [0] 7872: calling fini: /usr/lib64/pkcs11/p11-kit-trust.so [0] 7872: calling fini: /lib64/libtasn1.so.6 [0] 7872: calling fini: /lib64/p11-kit-proxy.so [0] 7872: calling fini: /usr/lib64/libfreeblpriv3.so [0] 7872: calling fini: /usr/lib64/libsoftokn3.so [0] 7872: calling fini: /lib64/libsqlite3.so.0 [0] 7872: calling fini: /lib64/libnss_systemd.so.2 [0] 7872: calling fini: /lib64/libcap.so.2 [0] 7872: calling fini: /lib64/libnss_sss.so.2 [0] 7872: calling fini: [0] 7872: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnagentutilities.so [0] 7872: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnapi.so [0] 7872: calling fini: /usr/lib64/libsmime3.so [0] 7872: calling fini: /usr/lib64/libnss3.so [0] 7872: calling fini: /lib64/libnssutil3.so [0] 7872: calling fini: /lib64/libplc4.so [0] 7872: calling fini: /lib64/libplds4.so [0] 7872: calling fini: /lib64/libnspr4.so [0] 7872: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpncommoncrypt.so [0] [SIC crashes here] Older version again without the strict ordering change: LD_DEBUG=all ./glibc-2.38-3.fc39/lib64/ld-linux-x86-64.so.2 --library-path /home/carlos/cisco/glibc-2.38-3.fc39/lib64:/home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn ./anyconnect-linux64-4.10.06090/vpn/vpnagentd -d -n 10 -verify_certs ./anyconnect-linux64-4.10.06090/vpn/DigiCertAssuredIDRootCA.pem >& unload_debug_all_old.txt grep 'calling fini:' unload_debug_all_old.txt 7920: calling fini: [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnagentutilities.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpnapi.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpncommoncrypt.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libvpncommon.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libaccurl.so.4 [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libacciscossl.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libacciscocrypto.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_thread.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_system.so [0] 7920: calling fini: /home/carlos/cisco/anyconnect-linux64-4.10.06090/vpn/libboost_regex.so [0] 7920: calling fini: /lib64/libxml2.so.2 [0] 7920: calling fini: /home/carlos/cisco/glibc-2.38-3.fc39/lib64/libpthread.so.0 [0] 7920: calling fini: /home/carlos/cisco/glibc-2.38-3.fc39/lib64/librt.so.1 [0] 7920: calling fini: /lib64/libstdc++.so.6 [0] 7920: calling fini: /home/carlos/cisco/glibc-2.38-3.fc39/lib64/libm.so.6 [0] 7920: calling fini: /lib64/libgcc_s.so.1 [0] 7920: calling fini: /lib64/libgio-2.0.so.0 [0] 7920: calling fini: /lib64/libz.so.1 [0] 7920: calling fini: /lib64/libgobject-2.0.so.0 [0] 7920: calling fini: /home/carlos/cisco/glibc-2.38-3.fc39/lib64/libdl.so.2 [0] 7920: calling fini: /lib64/liblzma.so.5 [0] 7920: calling fini: /lib64/libgmodule-2.0.so.0 [0] 7920: calling fini: /lib64/libglib-2.0.so.0 [0] 7920: calling fini: /lib64/libmount.so.1 [0] 7920: calling fini: /lib64/libselinux.so.1 [0] 7920: calling fini: /lib64/libffi.so.8 [0] 7920: calling fini: /lib64/libpcre2-8.so.0 [0] 7920: calling fini: /lib64/libblkid.so.1 [0] 7920: calling fini: /usr/lib64/libsmime3.so [0] 7920: calling fini: /usr/lib64/libnss3.so [0] 7920: calling fini: /lib64/libnssutil3.so [0] 7920: calling fini: /lib64/libplds4.so [0] 7920: calling fini: /lib64/libplc4.so [0] 7920: calling fini: /lib64/libnspr4.so [0] 7920: calling fini: /home/carlos/cisco/glibc-2.38-3.fc39/lib64/libc.so.6 [0] 7920: calling fini: ./glibc-2.38-3.fc39/lib64/ld-linux-x86-64.so.2 [0] [SIC no crash] Again, we're back to running the executables destructors first.
Patches posted upstream: [PATCH 0/3] Fine-tune ELF destructor ordering (bug 30869) <https://inbox.sourceware.org/libc-alpha/cover.1695113064.git.fweimer@redhat.com/>
FEDORA-2023-7f0a294b1a has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7f0a294b1a
FEDORA-2023-7f0a294b1a has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7f0a294b1a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7f0a294b1a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
For me this was working with the build that was published under the above advisory. I did updates earlier today and it crashes again, but with a different stack trace. The new build published (2.37-7.fc38) doesn't seem to resolve the issue. stack trace: #0 0x00007f4e2cf2b211 PK11_DestroySlot (libnss3.so + 0x65211) #1 0x00007f4e2cf2fd53 SECMOD_DestroyModule (libnss3.so + 0x69d53) #2 0x00007f4e2cf300bf SECMOD_DestroyModuleListElement (libnss3.so + 0x6a0bf) #3 0x00007f4e2cf30535 SECMOD_DestroyModuleList (libnss3.so + 0x6a535) #4 0x00007f4e2cf305bd SECMOD_Shutdown (libnss3.so + 0x6a5bd) #5 0x00007f4e2cee5421 nss_Shutdown (libnss3.so + 0x1f421) #6 0x00007f4e2cee5520 NSS_Shutdown (libnss3.so + 0x1f520) #7 0x00007f4e2f87b009 _ZN13CNSSCertUtilsD2Ev (libvpncommoncrypt.so + 0x7b009) #8 0x00007f4e2de61c3d __cxa_finalize (libc.so.6 + 0x3fc3d) #9 0x00007f4e2f823063 n/a (libvpncommoncrypt.so + 0x23063) #10 0x00007f4e300840f2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x10f2) #11 0x00007f4e30087e0d _dl_fini (ld-linux-x86-64.so.2 + 0x4e0d) #12 0x00007f4e2de621e6 __run_exit_handlers (libc.so.6 + 0x401e6) #13 0x00007f4e2de6232e exit (libc.so.6 + 0x4032e) #14 0x000055ade602e101 n/a (vpnagentd + 0x2e101) #15 0x00007f4e2de49b8a __libc_start_call_main (libc.so.6 + 0x27b8a) #16 0x00007f4e2de49c4b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c4b) #17 0x000055ade602e2a8 n/a (vpnagentd + 0x2e2a8) I'm going to attach the output of LD_DEBUG=all /opt/cisco/secureclient/bin/vpnagentd -d -n 10 -verify_certs vpn.pem for both a working version (when downgraded to 2.37-1.fc38) and the crashing version (2.37-7.fc38)
Created attachment 1990655 [details] LD_DEBUG=all vpnagentd -d -n10 -verify_certs vpn.pem working
Created attachment 1990656 [details] LD_DEBUG=all vpnagentd -d -n10 -verify_certs vpn.pem broken
Well, now I'm not sure the other updates are in play at all. I just applied the 2.37-7 to a system that hasn't had updates applied since I previously applied the 2.37-6 build and it crashes the same. Have downgraded to 2.37-1 there too.
Created attachment 1990839 [details] LD_DEBUG=all /opt/cisco/anyconnect/bin/vpnui 2>&1 | tee > /tmp/vpnui_glibc-2.37-7.fc38.log AnyConnect crashes with glibc-2.37-7.fc38 again oredumpctl info /opt/cisco/anyconnect/bin/vpnagentd PID: 12448 (vpnagentd) UID: 1000 (<user>) GID: 1000 (<user>) Signal: 11 (SEGV) Timestamp: Wed 2023-09-20 21:55:34 CEST (6 days ago) Command Line: vpnagentd -verify_certs /opt/cisco/anyconnect/pem.HIZQ96 4 1 <URL> <config>.xml Executable: /opt/cisco/anyconnect/bin/vpnagentd Control Group: /system.slice/vpnagentd.service Unit: vpnagentd.service Slice: system.slice Boot ID: 19ec891fffbe461c9b2b8ade3ba523f2 Machine ID: d7bede7ee67844e89dd831134442d733 Hostname: WSETT03-3Z8WV42 Storage: /var/lib/systemd/coredump/core.vpnagentd.1000.19ec891fffbe461c9b2b8ade3ba523f2.12448.1695239734000000.zst (missing) Message: Process 12448 (vpnagentd) of user 1000 dumped core. Module libpcsclite.so.1 from rpm pcsc-lite-1.9.9-3.fc38.x86_64 Module libcrypto.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libopensc.so.8 from rpm opensc-0.23.0-5.fc38.x86_64 Module opensc-pkcs11.so from rpm opensc-0.23.0-5.fc38.x86_64 Module p11-kit-proxy.so from rpm p11-kit-0.25.0-1.fc38.x86_64 Module libtasn1.so.6 from rpm libtasn1-4.19.0-2.fc38.x86_64 Module libnssckbi.so from rpm p11-kit-0.25.0-1.fc38.x86_64 Module libnspr4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libplds4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libplc4.so from rpm nss-3.93.0-1.fc38.x86_64 Module libblkid.so.1 from rpm util-linux-2.38.1-4.fc38.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc38.1.x86_64 Module libffi.so.8 from rpm libffi-3.4.4-2.fc38.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-1.fc38.x86_64 Module libmount.so.1 from rpm util-linux-2.38.1-4.fc38.x86_64 Module libgmodule-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module liblzma.so.5 from rpm xz-5.4.1-1.fc38.x86_64 Module libglib-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libgobject-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libgio-2.0.so.0 from rpm glib2-2.76.5-2.fc38.x86_64 Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64 Module libxml2.so.2 from rpm libxml2-2.10.4-1.fc38.x86_64 Stack trace of thread 12448: #0 0x0000557c69653490 n/a (n/a + 0x0) #1 0x00007f249272fc80 SECMOD_SlotDestroyModule (libnss3.so + 0x69c80) #2 0x00007f24927300bf SECMOD_DestroyModuleListElement (libnss3.so + 0x6a0bf) #3 0x00007f2492730535 SECMOD_DestroyModuleList (libnss3.so + 0x6a535) #4 0x00007f24927305bd SECMOD_Shutdown (libnss3.so + 0x6a5bd) #5 0x00007f24926e5421 nss_Shutdown (libnss3.so + 0x1f421) #6 0x00007f24926e5520 NSS_Shutdown (libnss3.so + 0x1f520) #7 0x00007f2494c7d689 _ZN13CNSSCertUtilsD2Ev (libvpncommoncrypt.so + 0x7d689) #8 0x00007f2493261c3d __cxa_finalize (libc.so.6 + 0x3fc3d) #9 0x00007f2494c2a2d3 n/a (libvpncommoncrypt.so + 0x2a2d3) #10 0x00007f249554e0f2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x10f2) #11 0x00007f2495551e0e _dl_fini (ld-linux-x86-64.so.2 + 0x4e0e) #12 0x00007f24932621e6 __run_exit_handlers (libc.so.6 + 0x401e6) #13 0x00007f249326232e exit (libc.so.6 + 0x4032e) #14 0x0000557c67a2f401 n/a (vpnagentd + 0x2f401) #15 0x00007f2493249b8a __libc_start_call_main (libc.so.6 + 0x27b8a) #16 0x00007f2493249c4b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c4b) #17 0x0000557c67a2f638 n/a (vpnagentd + 0x2f638) ELF object binary architecture: AMD x86-64 PID: 11376 (vpnagentd) UID: 1000 (<user>) GID: 1000 (<user>) Signal: 11 (SEGV) Timestamp: Wed 2023-09-27 18:57:46 CEST (5min ago) Command Line: vpnagentd -verify_certs /opt/cisco/anyconnect/pem.9F0wav 4 1 <URL> <config>.xml Executable: /opt/cisco/anyconnect/bin/vpnagentd Control Group: /system.slice/vpnagentd.service Unit: vpnagentd.service Slice: system.slice Boot ID: ba3a3a69e2d846d091d13a8ebc2c412f Machine ID: d7bede7ee67844e89dd831134442d733
The problem also occurs in Fedora 39. coredumpctl dump /opt/cisco/anyconnect/bin/vpnagentd PID: 7817 (vpnagentd) UID: 1000 (<user>) GID: 1000 (<user>) Signal: 11 (SEGV) Timestamp: Mon 2023-10-02 08:23:44 CEST (27min ago) Command Line: vpnagentd -verify_certs /opt/cisco/anyconnect/pem.4J7ubT 3 1 <gateway> Executable: /opt/cisco/anyconnect/bin/vpnagentd Control Group: /system.slice/vpnagentd.service Unit: vpnagentd.service Slice: system.slice Boot ID: df2ea2c7908e4b29bd840177cd2ca501 Machine ID: 613c986b77a74369beda616ba56562d2 Hostname: <hostname> Storage: /var/lib/systemd/coredump/core.vpnagentd.1000.df2ea2c7908e4b29bd840177cd2ca501.7817.1696227824000000.zst (present) Size on Disk: 3.8M Message: Process 7817 (vpnagentd) of user 1000 dumped core. Module libpcsclite.so.1 from rpm pcsc-lite-2.0.0-2.fc39.x86_64 Module libcrypto.so.3 from rpm openssl-3.1.1-4.fc39.x86_64 Module libopensc.so.8 from rpm opensc-0.23.0-5.fc39.x86_64 Module opensc-pkcs11.so from rpm opensc-0.23.0-5.fc39.x86_64 Module p11-kit-proxy.so from rpm p11-kit-0.25.0-2.fc39.x86_64 Module libtasn1.so.6 from rpm libtasn1-4.19.0-3.fc39.x86_64 Module libnssckbi.so from rpm p11-kit-0.25.0-2.fc39.x86_64 Module libcap.so.2 from rpm libcap-2.48-7.fc39.x86_64 Module libnss_systemd.so.2 from rpm systemd-254.5-2.fc39.x86_64 Module libnss_sss.so.2 from rpm sssd-2.9.2-1.fc39.x86_64 Module libnspr4.so from rpm nss-3.93.0-1.fc39.x86_64 Module libplds4.so from rpm nss-3.93.0-1.fc39.x86_64 Module libplc4.so from rpm nss-3.93.0-1.fc39.x86_64 Module libblkid.so.1 from rpm util-linux-2.39.2-1.fc39.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc39.2.x86_64 Module libffi.so.8 from rpm libffi-3.4.4-4.fc39.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-5.fc39.x86_64 Module libmount.so.1 from rpm util-linux-2.39.2-1.fc39.x86_64 Module libgmodule-2.0.so.0 from rpm glib2-2.78.0-3.fc39.x86_64 Module liblzma.so.5 from rpm xz-5.4.4-1.fc39.x86_64 Module libglib-2.0.so.0 from rpm glib2-2.78.0-3.fc39.x86_64 Module libgobject-2.0.so.0 from rpm glib2-2.78.0-3.fc39.x86_64 Module libgio-2.0.so.0 from rpm glib2-2.78.0-3.fc39.x86_64 Module libz.so.1 from rpm zlib-1.2.13-4.fc39.x86_64 Module libxml2.so.2 from rpm libxml2-2.10.4-3.fc39.x86_64 Stack trace of thread 7817: #0 0x0000000000000000 n/a (n/a + 0x0) #1 0x00007f9d9792fc80 SECMOD_SlotDestroyModule (libnss3.so + 0x69c80) #2 0x00007f9d979300bf SECMOD_DestroyModuleListElement (libnss3.so + 0x6a0bf) #3 0x00007f9d97930535 SECMOD_DestroyModuleList (libnss3.so + 0x6a535) #4 0x00007f9d979305bd SECMOD_Shutdown (libnss3.so + 0x6a5bd) #5 0x00007f9d978e5421 nss_Shutdown (libnss3.so + 0x1f421) #6 0x00007f9d978e5520 NSS_Shutdown (libnss3.so + 0x1f520) #7 0x00007f9d99e81a19 _ZN13CNSSCertUtilsD2Ev (libvpncommoncrypt.so + 0x81a19) #8 0x00007f9d9845ea2d __cxa_finalize (libc.so.6 + 0x40a2d) #9 0x00007f9d99e2b673 n/a (libvpncommoncrypt.so + 0x2b673) #10 0x00007f9d9a67a0f2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x10f2) #11 0x00007f9d9a67de0e _dl_fini (ld-linux-x86-64.so.2 + 0x4e0e) #12 0x00007f9d9845efd6 __run_exit_handlers (libc.so.6 + 0x40fd6) #13 0x00007f9d9845f11e exit (libc.so.6 + 0x4111e) #14 0x00005586cc42f9f1 n/a (vpnagentd + 0x2f9f1) #15 0x00007f9d9844614a __libc_start_call_main (libc.so.6 + 0x2814a) #16 0x00007f9d9844620b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820b) #17 0x00005586cc42fb98 n/a (vpnagentd + 0x2fb98) ELF object binary architecture: AMD x86-64
FEDORA-2023-2b8c11ee75 has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2b8c11ee75` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2b8c11ee75 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Proposed as a Blocker for 39-final by Fedora user jnk0 using the blocker tracking app because: The bug in glibc makes VPN login with the Cisco AnyConnect client impossible. Depending on the desktop environment selected, there is also no alternative available for a VPN login. Under Gnome, at least the SSO (MFA) login with NetworkManager-openconnect-gnome works.
FEDORA-2023-2b8c11ee75 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
So, there's a couple of things here. 1: the bug was proposed as a blocker for F39 but then auto-closed by an F38 update going stable. So we need to reopen it. 2: I don't think either glibc-2.37-10.fc38 or glibc-2.38-6.fc39 actually have a working fix for this. In the F38 path, 0d684079d2c7ac610261c92c8032342b4c61fa08 (which was the 2.37-7.fc38 build) reverted the patch that tried to fix this (because of the problems it caused for FreeIPA and iSCSI). No commit since then looks obviously like it reintroduced any attempt at a fix for this. On the F39 path, 4dd992f514279ad79ccf3b55e8cf882dc226fc39 mentions "elf: Always call destructors in reverse constructor order (bug 30785)" - which I believe is the change that *causes* this problem - but no commit after that seems to have tried to fix it. Upstream, Florian posted the initial fix attempt that was backported for F38 - https://inbox.sourceware.org/libc-alpha/cover.1695113064.git.fweimer@redhat.com/ - but it does not seem to have been merged, and the "V2" that was mentioned does not seem to have appeared. I can't find any later upstream proposal that attempts to fix the same problem (without breaking FreeIPA and iSCSI). Conclusion: I'm setting this back to POST.
Created attachment 1992281 [details] D_DEBUG=all /opt/cisco/anyconnect/bin/vpnui 2>&1 | tee > /tmp/vpnui_glibc-2.37-10.fc38.log This bug is still reproducible with glibc-2.37-10 on FC38.
Yes. As I explained above, this is expected. We don't really need any more logs, the nature of the issue is understood, but the initial attempt to fix it broke other things that are more important, so it was reverted.
(In reply to Adam Williamson from comment #27) > Upstream, Florian posted the initial fix attempt that was backported for F38 > - > https://inbox.sourceware.org/libc-alpha/cover.1695113064.git.fweimer@redhat.com/ > - but it does not seem to have been merged, and the "V2" that was > mentioned does not seem to have appeared. I can't find any later upstream > proposal that attempts to fix the same problem (without breaking FreeIPA and > iSCSI). We currently do not have a patch that makes all three or four impacted pieces of software work. In particular, the upstream patch or a variant of that (we tried both in F38 updates) does not really help. It's not just a matter of merging some existing patch.
Revert posted upstream: Revert "elf: Always call destructors in reverse constructor order (bug 30785)" <https://inbox.sourceware.org/libc-alpha/87zg0wf67p.fsf@oldenburg.str.redhat.com/T/>
FEDORA-2023-b59c306ba1 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1
Proposed as a Freeze Exception for 39-final by Fedora user jnk0 using the blocker tracking app because: Fedora 39 (beta) currently ships a glibc version that prevents the use of the Cisco AnyConnect client. Shipping the fixed version would be preferable.
So I think this has gotten a bit confusing, so let me try to summarize. AFAICT, we have these choices: 1. Leave "elf: Always call destructors in reverse constructor order (bug 30785)" out - result: AnyConnect, FreeIPA and iSCSI all work, but the unspecified "application problems" it was attempting to fix are still present (Florian, what *were* these "problems"?) 2. Include only "elf: Always call destructors in reverse constructor order (bug 30785)" - result: AnyConnect is broken, FreeIPA and iSCSI work 3. Include ""elf: Always call destructors in reverse constructor order (bug 30785)" and "Fine-tune ELF destructor ordering" - result: AnyConnect works, FreeIPA and iSCSI are broken At present, F38 and F39 are both in state 2 and the proposal is to move them to state 1. (Right?)
-5 votes for blocker in https://pagure.io/fedora-qa/blocker-review/issue/1374 , so marking rejected as a blocker. FE vote is still open.
+3 for FE in https://pagure.io/fedora-qa/blocker-review/issue/1374 , so marking accepted FE.
FEDORA-2023-b59c306ba1 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-b59c306ba1` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
The assumption for F39 in comment #34 is (was) correct, but F38 is in a different state: 4. contains "Fine-tune ELF destructor ordering" (bug 30869) - rolled out with https://bodhi.fedoraproject.org/updates/FEDORA-2023-2b8c11ee75 - result: AnyConnect works (?) My propolsal only applies to F39, as https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 does not include builds for F38. But I can confirm that the included builds for F39 fixed the problem with AnyConnect.
Um. I already listed that as state 3. And no, F38 is not in that state. The update notes are misleading: all attempts to patch the destructor stuff were dropped in https://src.fedoraproject.org/rpms/glibc/c/0d684079d2c7ac610261c92c8032342b4c61fa08?branch=f38 , and the update is from two commits after that. I guess nobody remembered to change the update notes.
(In reply to Jan Siml from comment #38) > The assumption for F39 in comment #34 is (was) correct, but F38 is in a > different state: > > 4. contains "Fine-tune ELF destructor ordering" (bug 30869) - rolled out > with https://bodhi.fedoraproject.org/updates/FEDORA-2023-2b8c11ee75 - > result: AnyConnect works (?) > On my F38 system with glibc-2.37-10 AnyConnect still fails with a complaint that the certificate on the VPN head-end is not valid.
(In reply to Mike Iglesias from comment #40) > (In reply to Jan Siml from comment #38) > > The assumption for F39 in comment #34 is (was) correct, but F38 is in a > > different state: > > > > 4. contains "Fine-tune ELF destructor ordering" (bug 30869) - rolled out > > with https://bodhi.fedoraproject.org/updates/FEDORA-2023-2b8c11ee75 - > > result: AnyConnect works (?) > > > > On my F38 system with glibc-2.37-10 AnyConnect still fails with a complaint > that the certificate on the VPN head-end is not valid. upgrading to glibc.x86_64 2.37-6 from updates-testing repo a few days ago fixed this issue, but normal update to 2.37-10 brought it back . The only option I see right now is to downgrade to 2.37-1 , because all other versions are removed from repos.
I focused on F39 because it was labeled a release blocker. But as correctly pointed out here, it is not a regression from F38, so that designation is a bit odd. The revert of the problematic commits has not happened upstream yet. I suppose we could carry the revert downstream-only, like we now do for F39. On the other hand, I'm awaiting confirmation that another fix attempt for the original problem (not reported against Fedora, but undoubtedly present as well) works for that piece of software. If that fix works, it would have to go into Fedora eventually, too. The new fix is quite conservative, but I thought that about the previous attempts as well …
It's not a blocker - note it says "RejectedBlocker" in the whiteboard field. :) It's accepted as an FE, with the intent that we can land https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 if that's the best idea. Per my summary above, that would put us in state 1), right?
(In reply to Adam Williamson from comment #43) > It's not a blocker - note it says "RejectedBlocker" in the whiteboard field. > :) It's accepted as an FE, with the intent that we can land > https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 if that's the > best idea. Per my summary above, that would put us in state 1), right? Correct. I've linked the relevant downstream issue.
FEDORA-2023-b59c306ba1 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
Anyconnect problem stil exists on Fedora 38 with glibc.x86_64 2.37-10.fc38. I can't make a connection to the vpn because of the false certificate error. VPN Client Cisco AnyConnect Secure Mobility Client Version 4.10.07061 and same for Cisco Secure Client Version 5.0.0.5040
May I ask if there is already a short-term decision regarding F38? I myself would prefer the fix already rolled out in F39, and not want to wait for confirmation in #42 for a different approach or upstream integration. Because F38 users currently have the choice between plague and cholera: Downgrading to glibc-2.37-4.fc38, thus reopening the flank of CVE-2023-4911 or an update to glibc-2.37-10.fc38, which fixes the vulnerability but prevents the use of the 3rd party VPN client. If the fix is good enough for the upcoming release (implicitly thanks to Florian again!), why not for the current version (F38)?
https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 has been pushed stable. Can anyone confirm that AnyConnect works OK with this version?
(In reply to Adam Williamson from comment #48) > https://bodhi.fedoraproject.org/updates/FEDORA-2023-b59c306ba1 has been > pushed stable. Can anyone confirm that AnyConnect works OK with this version? I can confirm that Cisco AnyConnect Secure Mobility Client Version 4.10.07061 and Cisco Secure Client Version 5.0.0.5040 works with the updated packages on F39.
Thanks. Since it's not done for F38 yet, let's not close this, instead i'll bump it to F38 and clear the F39 metadata.
Perhaps this is a useful work-around. Fedora 37 - My main system [1] glibc-2.36-14.fc37.x86_64 Cisco Secure Client 5.0.02075 I was using Cisco Secure Client successfully until about 2 weeks ago. Since then certificate errors are displayed and the VPN connection fails. I have not tried reverting to an older version of glibc. After I changed ExcludeFirefoxNSSCertStore to true, no error messages are displayed and the VPN connects. $ diff /opt/cisco/secureclient/AnyConnectLocalPolicy.xml.BAK /opt/cisco/secureclient/AnyConnectLocalPolicy.xml 4c4 < <ExcludeFirefoxNSSCertStore>false</ExcludeFirefoxNSSCertStore> --- > <ExcludeFirefoxNSSCertStore>true</ExcludeFirefoxNSSCertStore> When I moved .mozilla out of the way on an F38 VM, the same failure occurred. When I changed ExcludeFirefoxNSSCertStore as above on F38, the VPN connected. I don't yet have a F39 system. IMHO, the Linux Cisco Secure Client does not play well with others. It doesn't clean up after itself (DNS, in particular). Very unimpressed... [1] I skip every other Fedora release, and I'll upgrade to F39 when it's available.
(In reply to jimstaffer from comment #51) > Perhaps this is a useful work-around. > > Fedora 37 - My main system [1] > glibc-2.36-14.fc37.x86_64 > Cisco Secure Client 5.0.02075 > > I was using Cisco Secure Client successfully until about 2 weeks ago. Since > then certificate errors are displayed and the VPN connection fails. I have > not tried reverting to an older version of glibc. After I changed > ExcludeFirefoxNSSCertStore to true, no error messages are displayed and the > VPN connects. > > $ diff /opt/cisco/secureclient/AnyConnectLocalPolicy.xml.BAK > /opt/cisco/secureclient/AnyConnectLocalPolicy.xml > 4c4 > < <ExcludeFirefoxNSSCertStore>false</ExcludeFirefoxNSSCertStore> > --- > > <ExcludeFirefoxNSSCertStore>true</ExcludeFirefoxNSSCertStore> > Thanks! It works for me as well.
The reverts have landed upstream. I will integrate them into Fedora. Bug 2244992 tracks the alternative fix that we need for compatibility with other software.
FEDORA-2023-7dfa1f6429 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7dfa1f6429
FEDORA-2023-eb57cdaf88 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-eb57cdaf88
I have prepared new RPM builds and Bodhi updates with a different fix. It reverts the reverse constructor ordering changes that were so problematic. glibc-2.38-9.fc39: https://bodhi.fedoraproject.org/updates/FEDORA-2023-cec30b1297 glibc-2.37-12.fc38: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7dfa1f6429 glibc-2.36-16.fc37: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eb57cdaf88 I would appreciate if people could test any of these updates with the previously impacted VPN software, without any workarounds applied (that is, the configuration that was crashing previously). Thank you.
The new F38 build worked for me with AnyConnect.
I also tested the new F37 build and it worked as well.
Confirmed cisco secure client is good with this latest build on f38.
FEDORA-2023-7dfa1f6429 has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7dfa1f6429` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7dfa1f6429 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-eb57cdaf88 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-eb57cdaf88` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eb57cdaf88 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
The Cisco Secure Client is working with the new build under Fedora 39 too.
FEDORA-2023-7dfa1f6429 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
It seems that after this fix libcall functionality used in vim-xkbswitch plugin began to cause crashes in both vim and neovim. Vim backtrace (a part of): #0 dfs_traversal (rpo=rpo@entry=0x7fffe5376c80, map=0x564b0744f6b0, do_reldeps=do_reldeps@entry=0x7fffe5376c7f) at dl-sort-maps.c:165 #1 0x00007f853188c4db in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=<optimized out>) at dl-sort-maps.c:178 #2 dfs_traversal (rpo=rpo@entry=0x7fffe5376c80, map=0x564b0743f530, do_reldeps=do_reldeps@entry=0x7fffe5376c7f) at dl-sort-maps.c:157 #3 0x00007f853188c52c in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=<optimized out>) at dl-sort-maps.c:169 #4 dfs_traversal (rpo=rpo@entry=0x7fffe5376c80, map=0x564b0744dcb0, do_reldeps=do_reldeps@entry=0x7fffe5376c7f) at dl-sort-maps.c:172 #5 0x00007f853188c8f4 in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=0x7fffe5376c80) at dl-sort-maps.c:145 #6 _dl_sort_maps_dfs (for_fini=true, force_first=true, nmaps=<optimized out>, maps=0x7fffe5376cd0) at dl-sort-maps.c:228 #7 _dl_sort_maps (maps=maps@entry=0x7fffe5376cd0, nmaps=nmaps@entry=17, force_first=true, for_fini=for_fini@entry=true) at dl-sort-maps.c:314 #8 0x00007f853187cb07 in _dl_close_worker (map=<optimized out>, map@entry=0x564b0743f530, force=force@entry=false) at dl-close.c:240 #9 0x00007f853187d67b in _dl_close (_map=0x564b0743f530) at dl-close.c:791 #10 0x00007f853187c523 in __GI__dl_catch_exception (exception=exception@entry=0x7fffe5376f60, operate=0x7f853187d640 <_dl_close>, args=0x564b0743f530) at dl-catch.c:237 #11 0x00007f853187c679 in _dl_catch_error (objname=0x7fffe5376fc8, errstring=0x7fffe5376fd0, mallocedp=0x7fffe5376fc7, operate=<optimized out>, args=<optimized out>) at dl-catch.c:256 #12 0x00007f85315581f3 in _dlerror_run (operate=<optimized out>, args=<optimized out>) at dlerror.c:138 #13 0x00007f8531557f26 in __dlclose (handle=<optimized out>) at dlclose.c:31 #14 0x0000564b05256dc8 in mch_libcall (libname=<optimized out>, funcname=0x564b07516470 "Xkb_Switch_getXkbLayout", argstring=0x564b06c4e890 "", argint=113567888, string_result=0x7fffe5377e28, number_result=0x7fffe5377094) at /usr/src/debug/vim-9.0.2048-1.fc38.x86_64/src/os_unix.c:7867 #15 0x0000564b051662ab in libcall_common (argvars=0x7fffe5377550, rettv=0x7fffe5377e20, type=7) at /usr/src/debug/vim-9.0.2048-1.fc38.x86_64/src/evalfunc.c:7585 Neovim backtrace (a part of): #0 dfs_traversal (rpo=rpo@entry=0x7ffe2a979a70, map=0x5615455f3060, do_reldeps=do_reldeps@entry=0x7ffe2a979a6f) at dl-sort-maps.c:152 #1 0x00007fb13faf54db in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=<optimized out>) at dl-sort-maps.c:178 #2 dfs_traversal (rpo=rpo@entry=0x7ffe2a979a70, map=0x5615455eeee0, do_reldeps=do_reldeps@entry=0x7ffe2a979a6f) at dl-sort-maps.c:157 #3 0x00007fb13faf552c in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=<optimized out>) at dl-sort-maps.c:169 #4 dfs_traversal (rpo=rpo@entry=0x7ffe2a979a70, map=0x5615455f2b80, do_reldeps=do_reldeps@entry=0x7ffe2a979a6f) at dl-sort-maps.c:172 #5 0x00007fb13faf58f4 in dfs_traversal (do_reldeps=<optimized out>, map=<optimized out>, rpo=0x7ffe2a979a70) at dl-sort-maps.c:145 #6 _dl_sort_maps_dfs (for_fini=true, force_first=true, nmaps=<optimized out>, maps=0x7ffe2a979ac0) at dl-sort-maps.c:228 #7 _dl_sort_maps (maps=maps@entry=0x7ffe2a979ac0, nmaps=nmaps@entry=18, force_first=true, for_fini=for_fini@entry=true) at dl-sort-maps.c:314 #8 0x00007fb13fae5b07 in _dl_close_worker (map=<optimized out>, map@entry=0x5615455eeee0, force=force@entry=false) at dl-close.c:240 #9 0x00007fb13fae667b in _dl_close (_map=0x5615455eeee0) at dl-close.c:791 #10 0x00007fb13fae5523 in __GI__dl_catch_exception (exception=exception@entry=0x7ffe2a979d50, operate=0x7fb13fae6640 <_dl_close>, args=0x5615455eeee0) at dl-catch.c:237 #11 0x00007fb13fae5679 in _dl_catch_error (objname=0x7ffe2a979db8, errstring=0x7ffe2a979dc0, mallocedp=0x7ffe2a979db7, operate=<optimized out>, args=<optimized out>) at dl-catch.c:256 #12 0x00007fb13f7281f3 in _dlerror_run (operate=<optimized out>, args=<optimized out>) at dlerror.c:138 #13 0x00007fb13f727f26 in __dlclose (handle=<optimized out>) at dlclose.c:31 #14 0x00007fb13f894aa3 in uv_dlclose (lib=lib@entry=0x7ffe2a979e70) at src/unix/dl.c:47 #15 0x0000561543646f11 in os_libcall (libname=<optimized out>, funcname=0x5615454f9ba0 "Xkb_Switch_getXkbLayout", argv=0x561545326940 "", argi=1160931648, str_out=0x7ffe2a97a868, int_out=0x7ffe2a979ed4) at /usr/src/debug/neovim-0.9.2-1.fc38.x86_64/src/nvim/os/dl.c:86 #16 0x00005615435588d4 in libcall_common (argvars=0x7ffe2a97a1b0, rettv=0x7ffe2a97a860, out_type=out_type@entry=2) at /usr/src/debug/neovim-0.9.2-1.fc38.x86_64/src/nvim/eval/funcs.c:4469 They look similar and crash ar dlclose().
I mean this fix: * Fri Oct 20 2023 Florian Weimer <fweimer> - 2.37-12 - Fix force-first handling in dlclose (#2244992)
Alexey, could you provide a reproducer? I don't quite see how the code changes could introduce this bug. I wonder if the change exposed a latent, preexisting bug.
Sorry, got confused with the two bugs. I thought Bugzilla had eaten my previous comment. Let's continue on bug 2244992.
Cisco Secure Client Version 5.0.0.5040 and glibc 2.37-12.fc38 are working fine on Fedora 38. Thank you for the fix.
FEDORA-2023-3b9da12381 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-3b9da12381` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-3b9da12381 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-3b9da12381 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.