1441522 – AddressSanitizer: heap-use-after-free in libreplication-plugin.so

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1441522 - AddressSanitizer: heap-use-after-free in libreplication-plugin.so

Summary: AddressSanitizer: heap-use-after-free in libreplication-plugin.so

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	389-ds-base
Sub Component:
Version:	7.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	mreynolds
QA Contact:	Viktor Ashirov
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-04-12 07:39 UTC by Viktor Ashirov
Modified:	2017-08-01 21:16 UTC (History)
CC List:	6 users (show)
Fixed In Version:	389-ds-base-1.3.6.1-14.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-01 21:16:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:2086	0	normal	SHIPPED_LIVE	389-ds-base bug fix and enhancement update	2017-08-01 18:37:38 UTC

Description Viktor Ashirov 2017-04-12 07:39:02 UTC

Description of problem:
Issue was found during stress test from TET AutoMembers test suite.

=================================================================
==1711== ERROR: AddressSanitizer: heap-use-after-free on address 0x600407c9c4d0 at pc 0x7f215143c462 bp 0x7f210d6cdf90 sp 0x7f210d6cdf80
READ of size 8 at 0x600407c9c4d0 thread T45
    #0 0x7f215143c461 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0xaa461)
    #1 0x7f21457ebbac (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x66bac)
    #2 0x7f21457ec0e9 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x670e9)
    #3 0x7f214584904a (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0xc404a)
    #4 0x7f214581bd27 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x96d27)
    #5 0x7f214581e617 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x99617)
    #6 0x7f21514f2ec2 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x160ec2)
    #7 0x7f21514f3348 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x161348)
    #8 0x7f21514c1688 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x12f688)
    #9 0x7f21514c46de (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x1326de)
    #10 0x55f9d7cc88c7 (/usr/sbin/ns-slapd+0x3b8c7)
    #11 0x7f214fa109ba (/usr/lib64/libnspr4.so+0x289ba)
    #12 0x7f2151ceba97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
    #13 0x7f214f3b0dc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
    #14 0x7f214ec9234c (/usr/lib64/libc-2.17.so+0xf834c)
0x600407c9c4d0 is located 0 bytes inside of 16-byte region [0x600407c9c4d0,0x600407c9c4e0)
freed by thread T45 here:
    #0 0x7f2151ce8009 (/usr/lib64/libasan.so.0.0.0+0x16009)
    #1 0x7f2151438b28 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0xa6b28)
    #2 0x7f21457ed0c4 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x680c4)
    #3 0x7f214f9fc395 (/usr/lib64/libnspr4.so+0x14395)
previously allocated by thread T45 here:
    #0 0x7f2151ce8225 (/usr/lib64/libasan.so.0.0.0+0x16225)
    #1 0x7f21514386e8 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0xa66e8)
    #2 0x7f215143c093 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0xaa093)
    #3 0x7f214581ae47 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x95e47)
    #4 0x7f2145849170 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0xc4170)
    #5 0x7f214581bd27 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x96d27)
    #6 0x7f214581e617 (/usr/lib64/dirsrv/plugins/libreplication-plugin.so+0x99617)
    #7 0x7f21514f2ec2 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x160ec2)
    #8 0x7f21514f3348 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x161348)
    #9 0x7f21514c1688 (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x12f688)
    #10 0x7f21514c46de (/usr/lib64/dirsrv/libslapd.so.0.1.0+0x1326de)
    #11 0x55f9d7cc88c7 (/usr/sbin/ns-slapd+0x3b8c7)
    #12 0x7f214fa109ba (/usr/lib64/libnspr4.so+0x289ba)
Thread T45 created by T0 here:
    #0 0x7f2151cdcc3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x7f214fa1068b (/usr/lib64/libnspr4.so+0x2868b)
    #2 0x0
Shadow bytes around the buggy address:
  0x0c0100f8b840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b850: fa fa 04 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b880: fa fa 00 00 fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0100f8b890: fa fa fa fa fa fa fa fa fa fa[fd]fd fa fa fa fa
  0x0c0100f8b8a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa 00 05
  0x0c0100f8b8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8e0: fa fa fa fa fa fa fa fa fa fa 00 00 fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==1711== ABORTING


Version-Release number of selected component (if applicable):
389-ds-base-1.3.6.1-6.el7.x86_64

How reproducible:
Deterministically

Comment 3 wibrown@redhat.com 2017-04-13 04:17:25 UTC

Hey mate,

If I can give you a python script, can you run it against this asan trace on the machine with debug info installed?

https://github.com/Firstyear/ds-devel-root/blob/master/asan_symbolize_el7.py

Put the trace into a text file on the machine, ie:

/tmp/output.trace

python asan_symbolize_el7.py < /tmp/output.trace

Then provide that to this ticket. That will give us the line numbers and code that caused the issue. From there it's a short step to a fix I hope! 

Thanks so much!

Comment 4 Viktor Ashirov 2017-04-13 06:02:21 UTC

Oh, sorry, I forgot about symbolizing. Here you go:

==1711== ERROR: AddressSanitizer: heap-use-after-free on address 0x600407c9c4d0 at pc 0x7f215143c462 bp 0x7f210d6cdf90 sp 0x7f210d6cdf80
READ of size 8 at 0x600407c9c4d0 thread T45
    #0 0x7f215143c461 in csn_as_string /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/csn.c:209
    #1 0x7f21457ebbac in csnpldata_free /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:387
    #2 0x7f21457ec0e9 in csnplInsert /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:140
    #3 0x7f214584904a in ruv_add_csn_inprogress /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1651
    #4 0x7f214581bd27 in replica_get_for_backend /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:1489
    #5 0x7f214581e617 in multimaster_preop_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:428
    #6 0x7f21514f2ec2 in slapi_plugin_op_finished /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2072 (discriminator 1)
    #7 0x7f21514f3348 in plugin_call_plugins /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2014
    #8 0x7f21514c1688 in slapi_matchingrule_can_use_compare_fn /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:1036
    #9 0x7f21514c46de in do_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:388
    #10 0x55f9d7cc88c7 in ?? /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/connection.c:633
    #11 0x7f214fa109ba in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216
    #12 0x7f2151ceba97 in _ZN6__asan10AsanThread11ThreadStartEv _asan_rtl_
    #13 0x7f214f3b0dc4 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308
    #14 0x7f214ec9234c in __clone /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113
0x600407c9c4d0 is located 0 bytes inside of 16-byte region [0x600407c9c4d0,0x600407c9c4e0)
freed by thread T45 here:
    #0 0x7f2151ce8009 in __interceptor_free _asan_rtl_
    #1 0x7f2151438b28 in slapi_ch_free /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/ch_malloc.c:274
    #2 0x7f21457ed0c4 in csnplFreeCSN /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:400
    #3 0x7f214f9fc395 in PR_SetThreadPrivate /usr/src/debug/nspr-4.13.1/pr/src/threads/../../../nspr/pr/src/threads/prtpd.c:184
previously allocated by thread T45 here:
    #0 0x7f2151ce8225 in calloc ??:?
    #1 0x7f21514386e8 in slapi_ch_calloc /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/ch_malloc.c:183
    #2 0x7f215143c093 in csn_dup ??:?
    #3 0x7f214581ae47 in set_thread_primary_csn /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_init.c:170
    #4 0x7f2145849170 in ruv_add_csn_inprogress /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1648
    #5 0x7f214581bd27 in replica_get_for_backend /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:1489
    #6 0x7f214581e617 in multimaster_preop_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:428
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534963) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534950) greater than or equal to .debug_str size (132596).
    #7 0x7f21514f2ec2 in plugin_call_func /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2072 (discriminator 1)
    #8 0x7f21514f3348 in plugin_call_plugins /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2014
    #9 0x7f21514c1688 in op_shared_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:1036
    #10 0x7f21514c46de in do_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:388
    #11 0x55f9d7cc88c7 in ?? /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/connection.c:633
    #12 0x7f214fa109ba in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216
Thread T45 created by T0 here:
    #0 0x7f2151cdcc3a in __interceptor_pthread_create _asan_rtl_
    #1 0x7f214fa1068b in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457
    #2 0x0
Shadow bytes around the buggy address:
  0x0c0100f8b840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b850: fa fa 04 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b880: fa fa 00 00 fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0100f8b890: fa fa fa fa fa fa fa fa fa fa[fd]fd fa fa fa fa
  0x0c0100f8b8a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa 00 05
  0x0c0100f8b8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0100f8b8e0: fa fa fa fa fa fa fa fa fa fa 00 00 fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==1711== ABORTING

Comment 5 Ludwig 2017-04-13 08:57:29 UTC

Viktor, 

the stack trace looks strange and cannot be correct.

we have eg:
    #5 0x7f214581e617 in multimaster_preop_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:428
    #6 0x7f21514f2ec2 in slapi_plugin_op_finished /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2072 (discriminator 1)

but slapi_plugin_op_finished just decrements a counter

or: 
   #3 0x7f214584904a in ruv_add_csn_inprogress /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1651
    #4 0x7f214581bd27 in replica_get_for_backend /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:1489

replica_get_for_backend does not perform any op, just looks up a replica for a backend.

Is this an issue only reported by ASAN or do you have crashes as well ?

Comment 6 Viktor Ashirov 2017-04-13 09:36:39 UTC

I was investigating this crash: https://bugzilla.redhat.com/show_bug.cgi?id=1431278
I rerun the test with ASAN and got these errors, not related to the original crash. So it might be a false positive.

Comment 7 Viktor Ashirov 2017-04-13 13:58:19 UTC

Ludwig, does this one make more sense?

==30282== ERROR: AddressSanitizer: heap-use-after-free on address 0x6004009330d0 at pc 0x7f19131df8f1 bp 0x7f18d0899790 sp 0x7f18d0899780
READ of size 8 at 0x6004009330d0 thread T43
    #0 0x7f19131df8f0 in csn_is_equal /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/csn.c:277
    #1 0x7f190758f897 in csnplCommitAll /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:233
    #2 0x7f19075e5e1a in replica_enumerate_replicas /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1770
    #3 0x7f19075ec3e4 in ruv_update_ruv /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1745
    #4 0x7f19075cae1e in replica_update_ruv /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_replica.c:898
    #5 0x7f19075c07ca in multimaster_get_local_purl /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:1014
    #6 0x7f1913295ec2 in slapi_plugin_op_finished /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2072 (discriminator 1)
    #7 0x7f1913296348 in plugin_call_plugins /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2014
    #8 0x7f19079b2ea4 in ldbm_back_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/back-ldbm/ldbm_modify.c:835
    #9 0x7f1913264b5e in slapi_matchingrule_can_use_compare_fn /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:1059
    #10 0x7f19132676de in do_modify /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/modify.c:388
    #11 0x55a1e36ef8c7 in ?? /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/connection.c:633
    #12 0x7f19117b39ba in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216
    #13 0x7f1913a8ea97 in _ZN6__asan10AsanThread11ThreadStartEv _asan_rtl_
    #14 0x7f1911153dc4 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308
    #15 0x7f1910a3534c in __clone /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113
0x6004009330d0 is located 0 bytes inside of 16-byte region [0x6004009330d0,0x6004009330e0)
freed by thread T39 here:
    #0 0x7f1913a8b009 in __interceptor_free _asan_rtl_
    #1 0x7f19131dbb28 in slapi_ch_free /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/ch_malloc.c:274
    #2 0x7f19075900c4 in csnplFreeCSN /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:400
    #3 0x7f191179f395 in PR_SetThreadPrivate /usr/src/debug/nspr-4.13.1/pr/src/threads/../../../nspr/pr/src/threads/prtpd.c:184
previously allocated by thread T39 here:
    #0 0x7f1913a8b225 in calloc ??:?
    #1 0x7f19131db6e8 in slapi_ch_calloc /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/ch_malloc.c:183
    #2 0x7f19131df093 in csn_dup ??:?
    #3 0x7f19075bde47 in set_thread_primary_csn /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_init.c:170
    #4 0x7f19075ec170 in ruv_add_csn_inprogress /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_ruv.c:1648
    #5 0x7f19075bed27 in replica_get_for_backend /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:1489
    #6 0x7f19075c130b in multimaster_preop_delete /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/repl5_plugins.c:329
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534962) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
addr2line: Dwarf Error: Offset (534949) greater than or equal to .debug_str size (132596).
    #7 0x7f1913295ec2 in plugin_call_func /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2072 (discriminator 1)
    #8 0x7f1913296348 in plugin_call_plugins /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/plugin.c:2014
    #9 0x7f19131e38cd in op_shared_delete /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/delete.c:322
    #10 0x7f19131e3f2e in do_delete ??:?
    #11 0x55a1e36ef887 in ?? /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/connection.c:623
    #12 0x7f19117b39ba in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216
Thread T43 created by T0 here:
    #0 0x7f1913a7fc3a in __interceptor_pthread_create _asan_rtl_
    #1 0x7f19117b368b in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457
    #2 0x0
Thread T39 created by T0 here:
    #0 0x7f1913a7fc3a in __interceptor_pthread_create ??:?
    #1 0x7f19117b368b in PR_Select /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457
    #2 0x0
Shadow bytes around the buggy address:
  0x0c010011e5c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c010011e5d0: fa fa fa fa fa fa fa fa fa fa fd fd fa fa fd fd
  0x0c010011e5e0: fa fa fa fa fa fa fa fa fa fa fd fd fa fa fd fd
  0x0c010011e5f0: fa fa fd fd fa fa fd fd fa fa fd fa fa fa fa fa
  0x0c010011e600: fa fa fd fa fa fa fa fa fa fa fd fd fa fa fd fd
=>0x0c010011e610: fa fa fa fa fa fa fd fd fa fa[fd]fd fa fa fd fd
  0x0c010011e620: fa fa fd fd fa fa fa fa fa fa fd fd fa fa fd fd
  0x0c010011e630: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c010011e640: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c010011e650: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fd
  0x0c010011e660: fa fa fd fa fa fa fd fd fa fa fd fd fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==30282== ABORTING

Comment 8 Ludwig 2017-04-13 14:34:50 UTC

it is not really better, the calls cannot happen in this sequence.

It clearly indicates a problem of reuse of a csn that was stored in the thread local data, but unfortunatly the stack of the freeing thread is truncated, so we also cannot really guess where it happend. 

freed by thread T39 here:
    #0 0x7f1913a8b009 in __interceptor_free _asan_rtl_
    #1 0x7f19131dbb28 in slapi_ch_free /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/slapd/ch_malloc.c:274
    #2 0x7f19075900c4 in csnplFreeCSN /usr/src/debug/389-ds-base-1.3.6.1/ldap/servers/plugins/replication/csnpl.c:400
    #3 0x7f191179f395 in PR_SetThreadPrivate /usr/src/debug/nspr-4.13.1/pr/src/threads/../../../nspr/pr/src/threads/prtpd.c:184
previously allocated by thread T39 here:

the code was introduce with the fix for ticket #49008, I will review it again

Comment 9 thierry bordaz 2017-04-13 17:13:21 UTC

Looking at the following code (ruv_add_csn_inprogress) , there is something I am unsure:

1    prim_csn = get_thread_primary_csn();
2    if (prim_csn == NULL) {
3        set_thread_primary_csn(csn);
4        prim_csn = get_thread_primary_csn();
5    }
6    rc = csnplInsert (replica->csnpl, csn, prim_csn);


In 3, we allocate a csn (that is the primary csn) and store it into a thread private area

In 6, 'prim_csn' will be referred into a csnplnode->prim_csn

So the same csn is referred in a csnplnode and in the thread private area.

This csn is only freed from thread private area, but how is it enforced that it will not be read from the csnplnode after it was freed (csnplRemoveAll/csnplCommitAll) ?

Comment 12 thierry bordaz 2017-04-13 17:59:12 UTC

Thanks Viktor.
The scenario looks complex. Error logs report and DB_DEADLOCK while writing the changelog but https://bugzilla.redhat.com/show_bug.cgi?id=1441522#c7 shows a csnplCommitAll that should happen only after successful changelog update.
Possibly this two events are not related but I doubt.

Comment 22 Ludwig 2017-04-27 13:30:41 UTC

Victor: I created upstream ticket 49238 for this and attache a patch,

could you run your tests with this patch ?

Comment 23 Viktor Ashirov 2017-04-27 14:07:57 UTC

(In reply to Ludwig from comment #22)
> Victor: I created upstream ticket 49238 for this and attache a patch,
> 
> could you run your tests with this patch ?

Yes, I will schedule test run today.

Comment 30 thierry bordaz 2017-05-18 16:16:44 UTC

Upstream ticket pushed -> POST

Comment 33 errata-xmlrpc 2017-08-01 21:16:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2086

Note You need to log in before you can comment on or make changes to this bug.