Bug 2010690

Summary: 'malloc_consolidate(): unaligned fastbin chunk detected' going from openssl_backend_exit()
Product: Red Hat Enterprise Linux 9 Reporter: Tomáš Bžatek <tbzatek>
Component: cryptsetupAssignee: Ondrej Kozina <okozina>
Status: CLOSED NOTABUG QA Contact: Storage QE <storage-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0CC: agk, jbrassow, okozina, prajnoha
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-07 14:17:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2001549    
Attachments:
Description Flags
gdb bt none

Description Tomáš Bžatek 2021-10-05 11:09:02 UTC
Created attachment 1829358 [details]
gdb bt

Description of problem:
udisksd crashes on exit with

malloc_consolidate(): unaligned fastbin chunk detected
Aborted (core dumped)

Version-Release number of selected component (if applicable):
openssl-3.0.0-2.el9.x86_64
cryptsetup-2.4.1-1.el9.x86_64
glibc-2.34-6.el9.x86_64
redhat-rpm-config-188-1.el9.noarch

How reproducible:
Always.
Reproduces against multiple versions of udisks and libblockdev.
See bug 2001549#c5 for detailed reproducer.

Udisks has been compiled from git master with the following CFLAGS:
CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 '


Thread 1 (Thread 0x7f716db83900 (LWP 12970)):
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f716e5838a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:80
#2  0x00007f716e536686 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f716e5207d3 in __GI_abort () at abort.c:79
#4  0x00007f716e577a07 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f716e6b4693 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007f716e58d82c in malloc_printerr (str=str@entry=0x7f716e6b7078 "malloc_consolidate(): unaligned fastbin chunk detected") at malloc.c:5543
#6  0x00007f716e58e59c in malloc_consolidate (av=av@entry=0x7f716e6eeaa0 <main_arena>) at malloc.c:4637
#7  0x00007f716e58f5a0 in _int_free (av=0x7f716e6eeaa0 <main_arena>, p=0x5574458fe400, have_lock=<optimized out>) at malloc.c:4561
#8  0x00007f716e591bc5 in __GI___libc_free (mem=<optimized out>) at malloc.c:3278
#9  0x00007f716de01cfd in sa_doall (sa=0x5574458ca7e0, node=0x7f716de00df0 <sa_free_node>, leaf=0x0, arg=0x0) at crypto/sparse_array.c:86
#10 0x00007f716de269ed in ossl_sa_free (sa=<optimized out>) at crypto/sparse_array.c:117
#11 ossl_sa_ALGORITHM_free (sa=<optimized out>) at crypto/property/property.c:75
#12 ossl_method_store_free (store=0x5574458ca7b0) at crypto/property/property.c:232
#13 0x00007f716ddfc9a5 in CRYPTO_free_ex_data (class_index=<optimized out>, obj=0x0, ad=0x5574458c11d8) at crypto/ex_data.c:402
#14 0x00007f716ddf572f in context_deinit.part.0.lto_priv.0 (ctx=ctx@entry=0x5574458c11d0) at crypto/context.c:132
#15 0x00007f716ddf57dc in context_deinit (ctx=0x5574458c11d0) at crypto/context.c:120
#16 OSSL_LIB_CTX_free (ctx=0x5574458c11d0) at crypto/context.c:243
#17 OSSL_LIB_CTX_free (ctx=0x5574458c11d0) at crypto/context.c:238
#18 0x00007f716c33973b in openssl_backend_exit () at lib/crypto_backend/crypto_openssl.c:146
#19 0x00007f716c2ef349 in crypt_backend_destroy () at lib/crypto_backend/crypto_openssl.c:226
#20 libcryptsetup_exit () at lib/setup.c:6218
#21 0x00007f716ed3badc in _dl_fini () at dl-fini.c:139
#22 0x00007f716e538dc5 in __run_exit_handlers (status=0, listp=0x7f716e6ee658 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:113
#23 0x00007f716e538f40 in __GI_exit (status=<optimized out>) at exit.c:143
#24 0x00007f716e521567 in __libc_start_call_main (main=main@entry=0x55744479f360 <main>, argc=argc@entry=3, argv=argv@entry=0x7ffe7c161b48) at ../sysdeps/nptl/libc_start_call_main.h:74
#25 0x00007f716e52160c in __libc_start_main_impl (main=0x55744479f360 <main>, argc=3, argv=0x7ffe7c161b48, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe7c161b38) at ../csu/libc-start.c:409
#26 0x000055744479f775 in _start ()

Comment 1 Tomáš Bžatek 2021-10-06 15:23:18 UTC
Well... maybe it was really udisksd issue after all with a double-free that remained hidden. Cryptsetup might be innocent here, just taking a hit from a corrupted memory.

Found this by reading the docs and checking memory ownership by hand: https://github.com/storaged-project/udisks/pull/926

Comment 2 Tomáš Bžatek 2021-10-07 14:17:24 UTC
Heavy testing overnight revealed no further issues. Considering the mentioned udisks patch has fixed the issue altogether.

Closing this bugreport for the moment, no further work or investigation needed from cryptsetup side. It's crazy to realize how seemingly innocent valgrind warning can cause such major consequences somewhere in a completely unrelated foreign code.