Bug 1836069

Summary: [abrt] Use-after-free on thread exit
Product: [Fedora] Fedora Reporter: David Juran <djuran>
Component: softhsmAssignee: Paul Wouters <pwouters>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: abokovoy, caillon+fedoraproject, lucilanga, mcrha, nmavrogi, pwouters, rhughes, rstrode, sandmann
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/62f250c112373f6925e0c0988e942851f1bff85b
Whiteboard: abrt_hash:ea16b35e6d91b3b9f25ace83145f6cf268dfc321;VARIANT_ID=workstation;
Fixed In Version: softhsm-2.6.1-3.fc31 softhsm-2.6.1-3.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-24 03:15:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: core_backtrace
none
File: cpuinfo
none
File: dso_list
none
File: environ
none
File: limits
none
File: maps
none
File: mountinfo
none
File: open_fds
none
File: proc_pid_status
none
File: var_log_messages
none
valgrind output
none
valgrind output with openssl and softhsm debuginfo none

Description David Juran 2020-05-15 05:05:20 UTC
Description of problem:
Crash reported when shutting down evolution

Version-Release number of selected component:
evolution-3.34.4-1.fc31

Additional info:
reporter:       libreport-2.12.0
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/gnome-launched-org.gnome.Evolution.desktop-2705.scope
cmdline:        evolution
crash_function: magazine_cache_trim
executable:     /usr/bin/evolution
journald_cursor: s=c82e0286ac834277b3b9b30808fa3cf7;i=52924d;b=a2b0b1edd9194a42ab87d17327626e47;m=b5b0bcff7;t=5a5a02c85a2e1;x=d7649965112f0df1
kernel:         5.6.11-200.fc31.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000
xsession_errors:

Comment 1 David Juran 2020-05-15 05:05:25 UTC
Created attachment 1688751 [details]
File: backtrace

Comment 2 David Juran 2020-05-15 05:05:27 UTC
Created attachment 1688752 [details]
File: core_backtrace

Comment 3 David Juran 2020-05-15 05:05:28 UTC
Created attachment 1688753 [details]
File: cpuinfo

Comment 4 David Juran 2020-05-15 05:05:31 UTC
Created attachment 1688754 [details]
File: dso_list

Comment 5 David Juran 2020-05-15 05:05:33 UTC
Created attachment 1688755 [details]
File: environ

Comment 6 David Juran 2020-05-15 05:05:34 UTC
Created attachment 1688756 [details]
File: limits

Comment 7 David Juran 2020-05-15 05:05:37 UTC
Created attachment 1688757 [details]
File: maps

Comment 8 David Juran 2020-05-15 05:05:39 UTC
Created attachment 1688758 [details]
File: mountinfo

Comment 9 David Juran 2020-05-15 05:05:40 UTC
Created attachment 1688759 [details]
File: open_fds

Comment 10 David Juran 2020-05-15 05:05:42 UTC
Created attachment 1688760 [details]
File: proc_pid_status

Comment 11 David Juran 2020-05-15 05:05:43 UTC
Created attachment 1688761 [details]
File: var_log_messages

Comment 12 Milan Crha 2020-05-15 06:42:44 UTC
Thanks for a bug report. I do not see much from the backtrace, it only shows that one of the threads (probably a dedicated thread, not the main thread) detected memory corruption:

> malloc_consolidate(): invalid chunk size

when cleaning up after itself. It's really hard to guess where the actual memory corruption could happen in the code, without having a reproducer or any clue.

Are you able to reproduce this or any similar crash, please? The memory corruption can sometimes cause a crash in various places, while being still caused by the same coding error.

Comment 13 David Juran 2020-05-15 07:04:52 UTC
Running evolution under valgrind, it crashed on startup, producing the attached output

export GIGACAGE_ENABLED=0
G_SLICE=always-malloc valgrind --show-leak-kinds=definite --num-callers=30 --leak-check=no --aspace-minaddr=0x100000000 evolution &>/var/tmp/log.txt

Comment 14 David Juran 2020-05-15 07:07:21 UTC
Created attachment 1688796 [details]
valgrind output

Comment 15 David Juran 2020-05-15 07:21:32 UTC
Once again, with openssl and softhsm debuginfo installed

This time, evolution actually started and I could press the exit button

Comment 16 David Juran 2020-05-15 07:22:16 UTC
Created attachment 1688800 [details]
valgrind output with openssl and softhsm debuginfo

Comment 17 Milan Crha 2020-05-15 07:28:25 UTC
Thanks for the update. The valgrind log claims a use-after-free in libcrypto, but most likely caused by softhsm, which expects to have its thread exit handlers run before libcrypto, which didn't happen in your case. From the more detailed valgrind log:

==10739== Thread 1:
==10739== Invalid read of size 4
==10739==    at 0x100A1B7D6: pthread_rwlock_wrlock (in /usr/lib64/libpthread-2.30.so)
==10739==    by 0x108DD5E7C: CRYPTO_THREAD_write_lock (threads_pthread.c:78)
==10739==    by 0x108D3B744: ENGINE_finish (eng_init.c:101)
==10739==    by 0x11E49460A: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:240)
==10739==    by 0x11E49469C: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:259)
==10739==    by 0x104C1EE86: __run_exit_handlers (in /usr/lib64/libc-2.30.so)
==10739==    by 0x104C1F03F: exit (in /usr/lib64/libc-2.30.so)
==10739==    by 0x104C071A9: (below main) (in /usr/lib64/libc-2.30.so)
==10739==  Address 0x11e0ebcb8 is 24 bytes inside a block of size 56 free'd
==10739==    at 0x10083AA0C: free (vg_replace_malloc.c:540)
==10739==    by 0x108D67BE1: OPENSSL_cleanup (init.c:601)
==10739==    by 0x108D67BE1: OPENSSL_cleanup (init.c:497)
==10739==    by 0x104C1EE86: __run_exit_handlers (in /usr/lib64/libc-2.30.so)
==10739==    by 0x104C1F03F: exit (in /usr/lib64/libc-2.30.so)
==10739==    by 0x104C071A9: (below main) (in /usr/lib64/libc-2.30.so)
==10739==  Block was alloc'd at
==10739==    at 0x10083980B: malloc (vg_replace_malloc.c:309)
==10739==    by 0x108D7059D: CRYPTO_zalloc (mem.c:230)
==10739==    by 0x108DD5E0B: CRYPTO_THREAD_lock_new (threads_pthread.c:29)
==10739==    by 0x108D3B7B9: do_engine_lock_init (eng_lib.c:25)
==10739==    by 0x108D3B7B9: do_engine_lock_init_ossl_ (eng_lib.c:21)
==10739==    by 0x100A1E97E: __pthread_once_slow (in /usr/lib64/libpthread-2.30.so)
==10739==    by 0x108DD5EFC: CRYPTO_THREAD_run_once (threads_pthread.c:118)
==10739==    by 0x108D3B878: ENGINE_new (eng_lib.c:33)
==10739==    by 0x108D3D2FD: ENGINE_rdrand (eng_rdrand.c:70)

And there are also some invalid writes, causing the crash.

That's with:
softhsm-2.6.1-1.fc31.x86_64
openssl-libs-1:1.1.1g-1.fc31.x86_64

I move this to softhsm for further investigation.

Comment 18 Alexander Bokovoy 2020-05-15 07:30:34 UTC
We need to push https://bodhi.fedoraproject.org/updates/FEDORA-2020-c95b9c3b99 to older releases.

Comment 19 Alexander Bokovoy 2020-05-15 07:41:59 UTC
I backported changes to f32 and f31.

F32: https://koji.fedoraproject.org/koji/taskinfo?taskID=44517170
F31: https://koji.fedoraproject.org/koji/taskinfo?taskID=44517242

will submit updates as soon as they are built.

Comment 20 Fedora Update System 2020-05-15 07:55:24 UTC
FEDORA-2020-77604f07ae has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-77604f07ae

Comment 21 Fedora Update System 2020-05-15 07:56:05 UTC
FEDORA-2020-901be5b51e has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-901be5b51e

Comment 22 Fedora Update System 2020-05-16 04:44:10 UTC
FEDORA-2020-901be5b51e has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-901be5b51e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-901be5b51e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 23 Fedora Update System 2020-05-16 05:06:49 UTC
FEDORA-2020-77604f07ae has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-77604f07ae`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-77604f07ae

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Fedora Update System 2020-05-24 03:15:59 UTC
FEDORA-2020-901be5b51e has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Fedora Update System 2020-05-24 03:27:41 UTC
FEDORA-2020-77604f07ae has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.