Bug 2256083 - Sporadic segv in imapd after upgrading to FC39
Summary: Sporadic segv in imapd after upgrading to FC39
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cyrus-imapd
Version: 39
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Martin Osvald 🛹
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-12-28 12:12 UTC by Chris Palmer
Modified: 2024-03-23 00:25 UTC (History)
5 users (show)

Fixed In Version: cyrus-imapd-3.8.1-11.fc41 cyrus-imapd-3.8.1-11.fc39 cyrus-imapd-3.8.1-11.fc40
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-03-02 08:05:19 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cyrusimap cyrus-imapd issues 4785 0 None open imapd/pop3d crashing on TLS shutdown 2024-02-06 05:44:19 UTC

Description Chris Palmer 2023-12-28 12:12:23 UTC
On a low-usage instance we have started getting a few SEGV logged each day. There is no obvious user impact. The stack trace is always the same, as given below. There were no problems under FC38, and the SEGV started immediately following upgrade to FC39. Now running cyrus-imapd-3.8.1-4.fc39.

In all of the cases I have just looked at the client was an IOS device. I don't know whether that is always the case.

Reproducible: Sometimes

Steps to Reproduce:
1. Seems to happen when a client connects to check for new mail.
2.
3.
Actual Results:  
Module libnss_systemd.so.2 from rpm systemd-254.7-1.fc39.x86_64
Module libnss_sss.so.2 from rpm sssd-2.9.3-1.fc39.x86_64
Module libnss_resolve.so.2 from rpm systemd-254.7-1.fc39.x86_64
Module libcap.so.2 from rpm libcap-2.48-9.fc39.x86_64
Module libnss_myhostname.so.2 from rpm systemd-254.7-1.fc39.x86_64
Module libplain.so from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module liblogin.so from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libgssapiv2.so from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libsasldb.so from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libanonymous.so from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libevent-2.1.so.7 from rpm libevent-2.1.12-9.fc39.x86_64
Module liblzma.so.5 from rpm xz-5.4.4-1.fc39.x86_64
Module libselinux.so.1 from rpm libselinux-3.5-5.fc39.x86_64
Module libkeyutils.so.1 from rpm keyutils-1.6.1-7.fc39.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-2.fc39.x86_64
Module libicalvcal.so.3 from rpm libical-3.0.17-1.fc39.x86_64
Module libicalss.so.3 from rpm libical-3.0.17-1.fc39.x86_64
Module libical.so.3 from rpm libical-3.0.17-1.fc39.x86_64
Module libxml2.so.2 from rpm libxml2-2.10.4-3.fc39.x86_64
Module libpcre2-posix.so.3 from rpm pcre2-10.42-1.fc39.2.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc39.2.x86_64
Module libpq.so.5 from rpm libpq-15.3-1.fc39.x86_64
Module libmariadb.so.3 from rpm mariadb-connector-c-3.3.5-2.fc39.x86_64
Module libkrb5support.so.0 from rpm krb5-1.21.2-2.fc39.x86_64
Module libk5crypto.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libkrb5.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libgssapi_krb5.so.2 from rpm krb5-1.21.2-2.fc39.x86_64
Module libjansson.so.4 from rpm jansson-2.13.1-7.fc39.x86_64
Module libshp.so.2 from rpm shapelib-1.5.0-16.fc39.x86_64
Module libzstd.so.1 from rpm zstd-1.5.5-4.fc39.x86_64
Module libnghttp2.so.14 from rpm nghttp2-1.55.1-4.fc39.x86_64
Module libicudata.so.73 from rpm icu-73.2-2.fc39.x86_64
Module libicuuc.so.73 from rpm icu-73.2-2.fc39.x86_64
Module libicui18n.so.73 from rpm icu-73.2-2.fc39.x86_64
Module libuuid.so.1 from rpm util-linux-2.39.2-1.fc39.x86_64
Module libz.so.1 from rpm zlib-1.2.13-4.fc39.x86_64
Module libcom_err.so.2 from rpm e2fsprogs-1.47.0-2.fc39.x86_64
Module libcrypto.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libssl.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libcyrus.so.0 from rpm cyrus-imapd-3.8.1-4.fc39.x86_64
Module libcyrus_min.so.0 from rpm cyrus-imapd-3.8.1-4.fc39.x86_64
Module libcyrus_sieve.so.0 from rpm cyrus-imapd-3.8.1-4.fc39.x86_64
Module libcyrus_imap.so.0 from rpm cyrus-imapd-3.8.1-4.fc39.x86_64
Module imapd from rpm cyrus-imapd-3.8.1-4.fc39.x86_64
Stack trace of thread 3580:
#0  0x00007f02670b9127 unlink_chunk.isra.0 (libc.so.6 + 0x9b127)
#1  0x00007f02670bc0ad _int_malloc (libc.so.6 + 0x9e0ad)
#2  0x00007f02670bce22 __libc_malloc (libc.so.6 + 0x9ee22)
#3  0x00007f0267457936 sha384_dupctx.lto_priv.0 (libcrypto.so.3 + 0x2579>
#4  0x00007f026737cfab EVP_MD_CTX_copy_ex (libcrypto.so.3 + 0x17cfab)
#5  0x00007f02673b0e78 HMAC_CTX_copy (libcrypto.so.3 + 0x1b0e78)
#6  0x00007f026748104e hmac_dup.lto_priv.0 (libcrypto.so.3 + 0x28104e)
#7  0x00007f026739e9a2 EVP_MAC_CTX_dup (libcrypto.so.3 + 0x19e9a2)
#8  0x00007f0267472068 tls1_prf_P_hash (libcrypto.so.3 + 0x272068)
#9  0x00007f0267472326 kdf_tls1_prf_derive.lto_priv.0 (libcrypto.so.3 + >
#10 0x00007f0267867038 tls1_PRF.constprop.0 (libssl.so.3 + 0x72038)
#11 0x00007f0267831173 tls1_setup_key_block (libssl.so.3 + 0x3c173)
#12 0x00007f0267868c23 tls_process_change_cipher_spec.isra.0 (libssl.so.>
#13 0x00007f026785737f state_machine (libssl.so.3 + 0x6237f)
#14 0x00007f0267b6a80e tls_start_servertls (libcyrus_imap.so.0 + 0xcb80e)
#15 0x000056446fd9b730 cmd_starttls.lto_priv.0 (imapd + 0x34730)
#16 0x000056446fdb48bc service_main.isra.0 (imapd + 0x4d8bc)
#17 0x000056446fd795ee main (imapd + 0x125ee)
#18 0x00007f026704614a __libc_start_call_main (libc.so.6 + 0x2814a)
#19 0x00007f026704620b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820>
#20 0x000056446fd79ac5 _start (imapd + 0x12ac5)
ELF object binary architecture: AMD x86-64


Expected Results:  
No error

Comment 1 Chris Palmer 2023-12-31 15:24:33 UTC
And very occasionally there is a General Protection Fault in a slightly different place in the same libc unlink_chunk_isra method, with a different stacktrace:

Stack trace of thread 39075:
#0  0x00007f6e32ab912d unlink_chunk.isra.0 (libc.so.6 + 0x9b12d)
#1  0x00007f6e32abc0ad _int_malloc (libc.so.6 + 0x9e0ad)
#2  0x00007f6e32abc755 _int_realloc (libc.so.6 + 0x9e755)
#3  0x00007f6e32abd815 realloc (libc.so.6 + 0x9f815)
#4  0x00007f6e333dbbb2 xrealloc (libcyrus_min.so.0 + 0x14bb2)
#5  0x00007f6e3332ae41 seqset_add (libcyrus.so.0 + 0x9ae41)
#6  0x00007f6e3353b32f search_folder_get_seqset (libcyrus_imap.so.0 + 0xc332f)
#7  0x00007f6e334e5bb6 index_search (libcyrus_imap.so.0 + 0x6dbb6)
#8  0x000055914ac5f462 cmd_search.lto_priv.0 (imapd + 0x25462)
#9  0x000055914ac5843d cmdloop.lto_priv.0 (imapd + 0x1e43d)
#10 0x000055914ac87455 service_main.isra.0 (imapd + 0x4d455)
#11 0x000055914ac4c5ee main (imapd + 0x125ee)
#12 0x00007f6e32a4614a __libc_start_call_main (libc.so.6 + 0x2814a)
#13 0x00007f6e32a4620b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820b)

Comment 2 Martin Osvald 🛹 2024-02-06 05:44:19 UTC
Thank you for reporting this issue.

This looks very similar to this TLS upstream issue:

https://github.com/cyrusimap/cyrus-imapd/issues/4785

The second backtrace in comment 1 looks like a different issue though.

Comment 3 Martin Osvald 🛹 2024-02-29 17:54:22 UTC
I can see the upstream has merged the fix for these TLS shutdown crashes, I will be backporting this commit soon:

https://github.com/cyrusimap/cyrus-imapd/commit/30eec6e793d8dd85e3ef8cb744ccfdfe2ba90513

Comment 4 Fedora Update System 2024-03-02 06:28:58 UTC
FEDORA-2024-db42e37012 (cyrus-imapd-3.8.1-11.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-db42e37012

Comment 5 Fedora Update System 2024-03-02 07:06:06 UTC
FEDORA-2024-3f18b03659 (cyrus-imapd-3.8.1-11.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-3f18b03659

Comment 6 Fedora Update System 2024-03-02 07:29:15 UTC
FEDORA-2024-7a959bc16d (cyrus-imapd-3.8.1-11.fc39) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-7a959bc16d

Comment 7 Fedora Update System 2024-03-02 08:05:19 UTC
FEDORA-2024-db42e37012 (cyrus-imapd-3.8.1-11.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Fedora Update System 2024-03-03 01:12:39 UTC
FEDORA-2024-7a959bc16d has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-7a959bc16d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-7a959bc16d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2024-03-03 02:09:47 UTC
FEDORA-2024-3f18b03659 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-3f18b03659`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-3f18b03659

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2024-03-11 01:36:37 UTC
FEDORA-2024-7a959bc16d (cyrus-imapd-3.8.1-11.fc39) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 Fedora Update System 2024-03-23 00:25:56 UTC
FEDORA-2024-3f18b03659 (cyrus-imapd-3.8.1-11.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.