Hide Forgot
In the change between version 5 and 6 of gcc/asan, under certain conditions false positives by ASAN can be displayed. This can be seen in the output of Fedora 26 and 24 which use gcc 6. The code in question is the following: backend_manager.c:364 void be_unbindall(Connection *conn, Operation *op) { int i; Slapi_PBlock pb = {0}; for ( i = 0; i < maxbackends; i++ ) { if ( backends[i] && (backends[i]->be_unbind != NULL) ) { /* This is the modern, and faster way to do pb memset(0) * It also doesn't trigger the HORRIBLE stack overflows I found ... */ pblock_init_common( &pb, backends[i], conn, op ); if ( plugin_call_plugins( &pb, SLAPI_PLUGIN_PRE_UNBIND_FN ) == 0 ) { int rc = 0; slapi_pblock_set( &pb, SLAPI_PLUGIN, backends[i]->be_database ); if(backends[i]->be_state != BE_STATE_DELETED && backends[i]->be_unbind!=NULL) { rc = (*backends[i]->be_unbind)( &pb ); } slapi_pblock_set( &pb, SLAPI_PLUGIN_OPRETURN, &rc ); (void) plugin_call_plugins( &pb, SLAPI_PLUGIN_POST_UNBIND_FN ); } } } } The call to pblock_init_common then causes the "overflow". void pblock_init_common( Slapi_PBlock *pb, Slapi_Backend *be, Connection *conn, Operation *op ) { PR_ASSERT( NULL != pb ); memset( pb, '\0', sizeof(Slapi_PBlock) ); pb->pb_backend = be; pb->pb_conn = conn; pb->pb_op = op; } The issue of using memset on a stack allocated value aside (this will change in the future), we observe a false buffer overflow. The output is: ================================================================= ==23660==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f63e4428508 at pc 0x7f64005fbd74 bp 0x7f63e4428310 sp 0x7f63e4427ab8 WRITE of size 712 at 0x7f63e4428508 thread T10 #0 0x7f64005fbd73 in ?? _asan_rtl_:0 #1 0x7f64001ddee1 in pblock_init_common /home/william/development/389ds/ds/ldap/servers/slapd/pblock.c:38:0 #2 0x7f6400120c9c in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:363:0 #3 0x45ab7e in do_unbind /home/william/development/389ds/ds/ldap/servers/slapd/unbind.c:86:0 #4 0x41e3e4 in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:601:0 #5 0x4236fe in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #6 0x7f63fd6135cb in _pt_root /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #7 0x7f63fd3d36f9 in start_thread /usr/src/debug/glibc-2.24-256-g5140d03/nptl/pthread_create.c:333:0 #8 0x7f63fd10e2ae in __GI___clone /usr/src/debug////////glibc-2.24-256-g5140d03/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:105:0 Address 0x7f63e4428508 is located in stack of thread T10 at offset 408 in frame #0 0x7f6400120aa9 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:351:0 This frame has 2 object(s): [32, 36) 'rc' [96, 808) 'pb' <== Memory access at offset 408 partially overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) Thread T10 created by T0 here: #0 0x7f64005e5458 in pthread_create _asan_rtl_:0 #1 0x7f63fd6132aa in _PR_CreateThread /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.3+0x47d73) Shadow bytes around the buggy address: 0x0fecfc87d050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 0x0fecfc87d070: f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 00 00 00 00 00 0x0fecfc87d080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fecfc87d0a0: 00[f4]f4 f4 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d0b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d0d0: 00 00 00 f4 f4 f4 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d0e0: 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 00 00 00 00 0x0fecfc87d0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==23660==ABORTING If we examine the struct *before* the memset: $1 = (Slapi_PBlock *) 0x7f63e44283d0 (gdb) print sizeof(pb) $2 = 712 (gdb) x/712xb 0x7f63e44283d0 0x7f63e44283d0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ... repeats a lot 0x7f63e4428690: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 This means there are no poison bytes *in* the struct, so we should not be seeing a partial overflow as the space is already zeroed. If we pay close attention: ASAN output: 0x0fecfc87d090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fecfc87d0a0: 00[f4]f4 f4 00 00 00 00 00 00 00 00 00 00 00 00 0x0fecfc87d0b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Note the address 0x7f63e4428508 which is where the "issue" is, is not actually where asan reports the fault (0x0fecfc87d0a0). The bytes are in a different range. This also lends to some curious behaviour. If you alter the code to: void be_unbindall(Connection *conn, Operation *op) { int i; Slapi_PBlock pb = {0}; Slapi_PBlock pb_second = {0}; for ( i = 0; i < maxbackends; i++ ) { if ( backends[i] && (backends[i]->be_unbind != NULL) ) { /* This is the modern, and faster way to do pb memset(0) * It also doesn't trigger the HORRIBLE stack overflows I found ... */ pblock_init_common( &pb, backends[i], conn, op ); pblock_init_common( &pb_second, backends[i], conn, op ); if ( plugin_call_plugins( &pb, SLAPI_PLUGIN_PRE_UNBIND_FN ) == 0 ) { int rc = 0; slapi_pblock_set( &pb, SLAPI_PLUGIN, backends[i]->be_database ); if(backends[i]->be_state != BE_STATE_DELETED && backends[i]->be_unbind!=NULL) { rc = (*backends[i]->be_unbind)( &pb ); } slapi_pblock_set( &pb, SLAPI_PLUGIN_OPRETURN, &rc ); (void) plugin_call_plugins( &pb, SLAPI_PLUGIN_POST_UNBIND_FN ); } } } } Now with this alteration, we are adding a second copy of the slapi_pblock struct. Logically if we were overflowing, we should still see the overflow manifest in pb, not in pb_second. We should see the crash on the line "pblock_init_common( &pb, backends[i], conn, op );". However, this is not the case. We recieve the result: ================================================================= ==15233==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f5387434508 at pc 0x7f53a462bd74 bp 0x7f5387434010 sp 0x7f53874337b8 WRITE of size 712 at 0x7f5387434508 thread T11 #0 0x7f53a462bd73 in ?? _asan_rtl_:0 #1 0x7f53a420dea3 in pblock_init_common /home/william/development/389ds/ds/ldap/servers/slapd/pblock.c:38:0 #2 0x7f53a4150d28 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:364:0 #3 0x45abca in do_unbind /home/william/development/389ds/ds/ldap/servers/slapd/unbind.c:86:0 #4 0x41e3e4 in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:601:0 #5 0x4236fe in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #6 0x7f53a16435cb in _pt_root /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #7 0x7f53a14036f9 in start_thread /usr/src/debug/glibc-2.24-256-g5140d03/nptl/pthread_create.c:333:0 #8 0x7f53a113e2ae in __GI___clone /usr/src/debug////////glibc-2.24-256-g5140d03/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:105:0 Address 0x7f5387434508 is located in stack of thread T11 at offset 1176 in frame #0 0x7f53a4150aa9 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:351:0 This frame has 3 object(s): [32, 36) 'rc' [96, 808) 'pb' [864, 1576) 'pb_second' <== Memory access at offset 1176 partially overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) Thread T11 created by T0 here: #0 0x7f53a4615458 in pthread_create _asan_rtl_:0 #1 0x7f53a16432aa in _PR_CreateThread /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.3+0x47d73) Shadow bytes around the buggy address: 0x0feaf0e7e850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e870: 00 00 00 f4 f4 f4 f2 f2 f2 f2 00 00 00 00 00 00 0x0feaf0e7e880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0feaf0e7e8a0: 00[f4]f4 f4 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8d0: 00 00 00 f4 f4 f4 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8e0: 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 00 00 00 00 0x0feaf0e7e8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==15233==ABORTING It is always occuring when you access ~400 bytes from the end of the current stack allocation. Testing this with an array of various sizes yielded the same behaviour if the array was size 200, 1000, 2000 or 4000, the overflow would always occur when you memset the item that was 400 bytes from the end of the stack. ########################### Other outputs and comparisons of the errors from various versions: ############################## This was tested with a stack size of 8mb, 16mb and 32mb, which did not alter the results. The overflow always occurs on the same address for a build (it's repeatable). This overflow began to happen as we added an extra library into our application (ie we link to another library). During the operation with that library this error shows up more frequently, which makes me consider that there is some kind of extra stack pressure. The extra library is also produced with ASAN instrumentation. Setting a hardware watch point at the address in question shows this address is touched frequently during start up of ASAN. For example.: awatch *(int *) 0x7fffde516288 <<-- This is the problem address for this build br __asan::ReportGenericError run Thread 1 #0 __sanitizer::internal_sched_yield () at ../../../../libsanitizer/sanitizer_common/sanitizer_linux.cc:304 #1 0x00007ffff6e66585 in __interceptor_pthread_create (thread=thread@entry=0x7fffffffdec8, attr=attr@entry=0x7fffffffded0, start_routine=start_routine@entry=0x7ffff3e944c0 <_pt_root>, arg=arg@entry=0x612000097e40) at ../../../../libsanitizer/asan/asan_interceptors.cc:256 #2 0x00007ffff3e942ab in _PR_CreateThread (type=PR_USER_THREAD, start=0x42269f <connection_threadmain>, arg=0x0, priority=PR_PRIORITY_NORMAL, scope=<optimized out>, state=PR_UNJOINABLE_THREAD, stackSize=0, isGCAble=0) at ../../../nspr/pr/src/pthreads/ptthread.c:457 #3 0x00007ffff3e94960 in PR_CreateThread (type=<optimized out>, start=<optimized out>, arg=<optimized out>, priority=<optimized out>, scope=<optimized out>, state=<optimized out>, stackSize=0) at ../../../nspr/pr/src/pthreads/ptthread.c:548 #4 0x000000000041d854 in init_op_threads () at /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:448 #5 0x000000000042b925 in slapd_daemon (ports=0x7fffffffe4d0) at /home/william/development/389ds/ds/ldap/servers/slapd/daemon.c:1023 #6 0x000000000043dfa9 in main (argc=5, argv=0x7fffffffe668) at /home/william/development/389ds/ds/ldap/servers/slapd/main.c:1145 Thread 13 (gdb) bt #0 __asan::Allocator::Allocate (this=0x7ffff7164d20 <__asan::instance>, size=<optimized out>, alignment=8, stack=0x7fffde516330, alloc_type=__asan::FROM_MALLOC, can_fill=<optimized out>) at ../../../../libsanitizer/sanitizer_common/sanitizer_atomic_clang_x86.h:47 #1 0x00007ffff6efc191 in __interceptor_realloc (ptr=0x0, size=32) at ../../../../libsanitizer/asan/asan_malloc_linux.cc:83 #2 0x00007ffff3c56305 in pthread_getattr_np (thread_id=<optimized out>, attr=attr@entry=0x7fffde516c90) at pthread_getattr_np.c:174 #3 0x00007ffff6f159bd in __sanitizer::GetThreadStackTopAndBottom (at_initialization=at_initialization@entry=false, stack_top=stack_top@entry=0x7fffde516d18, stack_bottom=stack_bottom@entry=0x7fffde516d20) at ../../../../libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:112 #4 0x00007ffff6f15f1e in __sanitizer::GetThreadStackAndTls (main=<optimized out>, stk_addr=stk_addr@entry=0x7fffddcfc020, stk_size=stk_size@entry=0x7fffddcfc028, tls_addr=tls_addr@entry=0x7fffddcfc030, tls_size=tls_size@entry=0x7fffde516d70) at ../../../../libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:388 #5 0x00007ffff6f09635 in __asan::AsanThread::SetThreadStackAndTls (this=this@entry=0x7fffddcfc000) at ../../../../libsanitizer/asan/asan_thread.cc:197 #6 0x00007ffff6f09862 in __asan::AsanThread::Init (this=this@entry=0x7fffddcfc000) at ../../../../libsanitizer/asan/asan_thread.cc:153 #7 0x00007ffff6f09a28 in __asan::AsanThread::ThreadStart (this=0x7fffddcfc000, os_id=25860, signal_thread_is_registered=0x7fffffffd648) at ../../../../libsanitizer/asan/asan_thread.cc:166 #8 0x00007ffff3c546fa in start_thread (arg=0x7fffde517700) at pthread_create.c:333 #9 0x00007ffff398f2af in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105 cont: Old value = 1 New value = 8835024 __asan::Allocator::Allocate (this=0x7ffff7164d20 <__asan::instance>, size=<optimized out>, alignment=<optimized out>, stack=0x7fffde516330, alloc_type=<optimized out>, can_fill=<optimized out>) at ../../../../libsanitizer/asan/asan_allocator.cc:420 420 m->alloc_context_id = StackDepotPut(*stack); There are about 20 or so places where this address is "read or written to" in the start up of the thread 13 in this case. Eventually in this build with the test, thread 13 does eventually become the thread that causes the fake buffer overflow. ######################## Versions tested ############################## Fedora Rawhide (26) libasan-6.2.1-2.fc26.x86_64 gcc-6.2.1-2.fc26.x86_64 ================================================================= ==15233==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f5387434508 at pc 0x7f53a462bd74 bp 0x7f5387434010 sp 0x7f53874337b8 WRITE of size 712 at 0x7f5387434508 thread T11 #0 0x7f53a462bd73 in ?? _asan_rtl_:0 #1 0x7f53a420dea3 in pblock_init_common /home/william/development/389ds/ds/ldap/servers/slapd/pblock.c:38:0 #2 0x7f53a4150d28 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:364:0 #3 0x45abca in do_unbind /home/william/development/389ds/ds/ldap/servers/slapd/unbind.c:86:0 #4 0x41e3e4 in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:601:0 #5 0x4236fe in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #6 0x7f53a16435cb in _pt_root /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #7 0x7f53a14036f9 in start_thread /usr/src/debug/glibc-2.24-256-g5140d03/nptl/pthread_create.c:333:0 #8 0x7f53a113e2ae in __GI___clone /usr/src/debug////////glibc-2.24-256-g5140d03/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:105:0 Address 0x7f5387434508 is located in stack of thread T11 at offset 1176 in frame #0 0x7f53a4150aa9 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:351:0 This frame has 3 object(s): [32, 36) 'rc' [96, 808) 'pb' [864, 1576) 'pb_second' <== Memory access at offset 1176 partially overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) Thread T11 created by T0 here: #0 0x7f53a4615458 in pthread_create _asan_rtl_:0 #1 0x7f53a16432aa in _PR_CreateThread /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.3+0x47d73) Shadow bytes around the buggy address: 0x0feaf0e7e850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e870: 00 00 00 f4 f4 f4 f2 f2 f2 f2 00 00 00 00 00 00 0x0feaf0e7e880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0feaf0e7e8a0: 00[f4]f4 f4 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8d0: 00 00 00 f4 f4 f4 00 00 00 00 00 00 00 00 00 00 0x0feaf0e7e8e0: 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 00 00 00 00 0x0feaf0e7e8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==15233==ABORTING Fedora 24 libasan-6.2.1-2.fc24.x86_64 gcc-6.2.1-2.fc24.x86_64 ================================================================= ==7778==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6437922508 at pc 0x7f644f96bdb4 bp 0x7f6437922010 sp 0x7f64379217b8 WRITE of size 712 at 0x7f6437922508 thread T11 #0 0x7f644f96bdb3 in ?? _asan_rtl_:0 #1 0x7f644f54ded3 in pblock_init_common /home/william/development/389ds/ds/ldap/servers/slapd/pblock.c:38:0 #2 0x7f644f490d58 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:364:0 #3 0x45ac2e in do_unbind /home/william/development/389ds/ds/ldap/servers/slapd/unbind.c:86:0 #4 0x41e494 in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:601:0 #5 0x4237ae in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #6 0x7f644c9845eb in _pt_root /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #7 0x7f644c7465c9 in start_thread /usr/src/debug/glibc-2.23-81-g2eda04e/nptl/pthread_create.c:333:0 #8 0x7f644c47ff6c in __clone /usr/src/debug////////glibc-2.23-81-g2eda04e/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0 Address 0x7f6437922508 is located in stack of thread T11 at offset 1176 in frame #0 0x7f644f490ad9 in be_unbindall /home/william/development/389ds/ds/ldap/servers/slapd/backend_manager.c:351:0 This frame has 3 object(s): [32, 36) 'rc' [96, 808) 'pb' [864, 1576) 'pb_second' <== Memory access at offset 1176 partially overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) Thread T11 created by T0 here: #0 0x7f644f955498 in pthread_create _asan_rtl_:0 #1 0x7f644c9842ca in _PR_CreateThread /usr/src/debug/nspr-4.13.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.3+0x47db3) Shadow bytes around the buggy address: 0x0fed06f1c450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c470: 00 00 00 f4 f4 f4 f2 f2 f2 f2 00 00 00 00 00 00 0x0fed06f1c480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0fed06f1c4a0: 00[f4]f4 f4 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c4d0: 00 00 00 f4 f4 f4 00 00 00 00 00 00 00 00 00 00 0x0fed06f1c4e0: 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 00 00 00 00 0x0fed06f1c4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==7778==ABORTING Fedora 23 libasan-5.3.1-6.fc23.x86_64 gcc-5.3.1-6.fc23.x86_64 PASS (both variants of the code) EL7 libasan-4.8.5-4.el7.x86_64 gcc-4.8.5-4.el7.x86_64 PASS (both variants of the code) ############ SUMMARY ################# Between version 5 -> 6 of GCC and ASAN, there exist conditions which cause a stack buffer overflow to be detected incorrectly. There is no overflow present in the code in question, and appears to be some kind of error in the stack interception of ASAN with regards to memset. I'm happy to provide more details to this error and assist in solving it in anyway possible. Thanks,
######### Follow up ############ Directory Server (the application in question) when it links to nunc-stans (the second library as mentioned) and both have ASAN enabled, the fake overflow occurs. When I build Directory Server with ASAN, and nunc-stans without ASAN, the fake overflow does *not* occur. I believe that this issue is that any application built with gcc 6 and ASAN can NOT link to another library which in turn links to ASAN. I have just tested this on rawhide and this works around the issue.
Extra note: DS uses a lot of plugins that are dlopen()ed, each of these is linked to libasan also. Could be it due to the number of parallel asan instances being started?
You haven't really provided any instruction on how to reproduce this issue, judging from snippets is hard or impossible. Can you provide preprocessed sources backend_manager.i and pblock.i + full gcc command line for compilation of backend_manager.c and pblock.c? The shadow memory really doesn't correspond to how large the variables are, I guess the important question is if the shadow memory is clear upon entry to the be_unbindall function or if, due to some unsupportable stuff in the program (stack switching, *context, custom stack unwinding, ...) or due to some gcc bug somewhere, upon entry to the function the shadow memory corresponding to stack is not fully cleared. ASAN relies on the shadow memory to be clear upon entry, it just stores the non-zero bytes in the padding in between variables, and on exit from the function clears it afterwards, longjmp and exception handling clears then everything in between the bottom and top frames in between which it is jumping.
Created attachment 1214153 [details] Commandlines
Created attachment 1214154 [details] backend_manager.i
Created attachment 1214155 [details] pblock.i
Since I have been working, this error has moved. Obviously as our application has changed, the location of the false positive has changed too depending on circumstance. I'm (for the moment) able to reproduce this reliably in our SASL authentication code. The existing error has now "vanished". 2881 pblock->pb_op->o_params.target_address.sdn = (Slapi_DN *)value; (gdb) print pblock->pb_op->o_params.target_address.sdn $12 = (Slapi_DN *) 0x0 ================================================================= ==6592==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffcecf2748 at pc 0x7ffff6a77ef6 bp 0x7fffcecf2480 sp 0x7fffcecf2470 WRITE of size 8 at 0x7fffcecf2748 thread T41 #0 0x7ffff6a77ef5 in slapi_pblock_set /home/william/development/389ds/ds/ldap/servers/slapd/pblock.c:2881:0 #1 0x45532d in ids_sasl_userdb_checkpass /home/william/development/389ds/ds/ldap/servers/slapd/saslbind.c:563:0 #2 0x7ffff47826ad in _sasl_checkpass /home/william/development/cyrus-sasl/lib/server.c:1902:0 #3 0x7fffea878497 in _init /usr/src/debug/cyrus-sasl-2.1.26/plugins/plain.c:172:0 #4 0x7ffff47847fd in sasl_server_step /home/william/development/cyrus-sasl/lib/server.c:1618:0 #5 0x7ffff4784d2f in sasl_server_start /home/william/development/cyrus-sasl/lib/server.c:1533:0 #6 0x456d0c in ids_sasl_check_bind /home/william/development/389ds/ds/ldap/servers/slapd/saslbind.c:946:0 #7 0x41418a in do_bind /home/william/development/389ds/ds/ldap/servers/slapd/bind.c:354:0 #8 0x41dfbf in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:596:0 #9 0x423301 in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #10 0x7ffff35b45cb in _pt_root /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #11 0x7ffff33765c9 in start_thread /usr/src/debug/glibc-2.23-85-g422facf/nptl/pthread_create.c:333:0 #12 0x7ffff30b00ec in __clone /usr/src/debug////////glibc-2.23-85-g422facf/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0 Address 0x7fffcecf2748 is located in stack of thread T41 at offset 456 in frame #0 0x455023 in ids_sasl_userdb_checkpass /home/william/development/389ds/ds/ldap/servers/slapd/saslbind.c:514:0 This frame has 7 object(s): [32, 40) 'mech' [96, 104) 'referral' [160, 168) 'sdn' [224, 232) 'op' [288, 296) 'be' [352, 368) 'cred' [416, 448) 'sv_cred' <== Memory access at offset 456 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) Thread T41 created by T0 here: #0 0x7ffff6e68498 in pthread_create _asan_rtl_:0 #1 0x7ffff35b42aa in _PR_CreateThread /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: stack-buffer-overflow (/opt/dirsrv/lib/dirsrv/libslapd.so.0+0x1a1ef5) Shadow bytes around the buggy address: 0x100079d96490: 00 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 00 00 0x100079d964a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x100079d964b0: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 0x100079d964c0: f2 f2 f2 f2 00 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 0x100079d964d0: f2 f2 f2 f2 00 f4 f4 f4 f2 f2 f2 f2 00 00 f4 f4 =>0x100079d964e0: f2 f2 f2 f2 00 00 00 00 f3[f3]f3 f3 00 00 00 00 0x100079d964f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x100079d96500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x100079d96510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x100079d96520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x100079d96530: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 04 f4 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==6592==ABORTING There is something very odd going on here with ASAN.
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle. Changing version to '26'.
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.