Bug 1905581

Summary: Memory leak reported by 389-ds cmocka tests
Product: [Fedora] Fedora Reporter: Viktor Ashirov <vashirov>
Component: p11-kitAssignee: Daiki Ueno <dueno>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 33CC: crypto-team, dueno, kai-engert-fedora, kdudka, patrick, stefw, tmraz
Target Milestone: ---Keywords: TestBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: p11-kit-0.23.22-2.fc33 p11-kit-0.23.22-2.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-30 01:54:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Viktor Ashirov 2020-12-08 15:34:42 UTC
Description of problem:
After upgrading from p11-kit-0.23.20-1.fc32.x86_64 to p11-kit-0.23.21-2.fc33.x86_64 cmocka tests for 389-ds-base with ASAN started to fail.

Version-Release number of selected component (if applicable):
p11-kit-0.23.21-2.fc33.x86_64
nss-3.58.0-3.fc33.x86_64

How reproducible:
always

Steps to Reproduce:
1. git clone https://github.com/389ds/389-ds-base && cd 389-ds-base
2. ./autogen.sh && ./configure --enable-asan --enable-debug --enable-cmocka && make check

Actual results:

$ ASAN_OPTIONS=fast_unwind_on_malloc=0 ./test_slapd
=================================================================
==237383==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8194 byte(s) in 2 object(s) allocated from:
    #0 0x7fc362ddb667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667)
    #1 0x7fc362d6f900  (/lib64/libasan.so.6+0x44900)
    #2 0x7fc35e3c57dd  (<unknown module>)
    #3 0x7fc35e3ca338  (<unknown module>)
    #4 0x7fc35e3ce149  (<unknown module>)
    #5 0x7fc3625272ca in secmod_LoadPKCS11Module /usr/src/debug/nss-3.58.0-3.fc33.x86_64/nss/lib/pk11wrap/pk11load.c:518
    #6 0x7fc36253411c in SECMOD_LoadModule /usr/src/debug/nss-3.58.0-3.fc33.x86_64/nss/lib/pk11wrap/pk11pars.c:1840
    #7 0x7fc362534257 in SECMOD_LoadModule /usr/src/debug/nss-3.58.0-3.fc33.x86_64/nss/lib/pk11wrap/pk11pars.c:1876
    #8 0x7fc3624fd830 in nss_Init /usr/src/debug/nss-3.58.0-3.fc33.x86_64/nss/lib/nss/nssinit.c:712
    #9 0x7fc3624fdd21 in NSS_Initialize /usr/src/debug/nss-3.58.0-3.fc33.x86_64/nss/lib/nss/nssinit.c:889
    #10 0x405fef in test_plugin_pwdstorage_nss_setup test/plugins/pwdstorage/pbkdf2.c:17
    #11 0x7fc362d23e42  (/lib64/libcmocka.so.0+0x5e42)
    #12 0x7fc362d24434 in _cmocka_run_group_tests (/lib64/libcmocka.so.0+0x6434)
    #13 0x405f0e in run_plugin_tests test/plugins/test.c:30
    #14 0x402558 in main test/main.c:16
    #15 0x7fc36228e041 in __libc_start_main (/lib64/libc.so.6+0x27041)
    #16 0x40247d in _start (/workspace/ds/.libs/lt-test_slapd+0x40247d)

SUMMARY: AddressSanitizer: 8194 byte(s) leaked in 2 allocation(s).


Expected results:
No memory leak

Additional info:

Comment 1 Viktor Ashirov 2020-12-08 15:46:48 UTC
This also happens in RHEL8 with p11-kit-0.23.21-4.el8.x86_64

Comment 2 Patrick Monnerat 2021-01-12 00:23:55 UTC
Probably the same bug occurs within curl compiled with nss:

test 0067...[HTTP with NTLM authorization]
 valgrind ERROR ==3387== 4,096 bytes in 1 blocks are definitely lost in loss record 28 of 29
==3387==    at 0x4839809: malloc (vg_replace_malloc.c:307)
==3387==    by 0x4E14206: realpath@@GLIBC_2.3 (in /usr/lib64/libc-2.32.so)
==3387==    by 0x134767CD: ???
==3387==    by 0x1347B318: ???
==3387==    by 0x1347F129: ???
==3387==    by 0x4ACC34A: secmod_LoadPKCS11Module (pk11load.c:518)
==3387==    by 0x4AD957C: SECMOD_LoadModule (pk11pars.c:1938)
==3387==    by 0x4AD96B7: SECMOD_LoadModule (pk11pars.c:1974)
==3387==    by 0x4AA1EC0: nss_Init (nssinit.c:712)
==3387==    by 0x4AA245F: NSS_InitContext (nssinit.c:910)
==3387==    by 0x48C791F: nss_init_core (nss.c:1337)
==3387==    by 0x48C791F: nss_init.part.0 (nss.c:1408)
==3387==    by 0x48CAAC7: nss_init (nss.c:1458)
==3387==    by 0x48CAAC7: Curl_nss_force_init (nss.c:1454)
==3387==    by 0x48C556D: Curl_auth_decode_ntlm_type2_message (ntlm.c:282)
==3387==    by 0x4882FE9: Curl_input_ntlm (http_ntlm.c:82)
==3387==    by 0x487A857: Curl_http_input_auth (http.c:968)
==3387==    by 0x487DD49: Curl_http_header (http.c:3518)

Comment 3 Daiki Ueno 2021-01-25 17:31:02 UTC
I suspect this is caused by the newly added progname detection logic, which caches executable path in a static variable:
https://github.com/p11-glue/p11-kit/pull/307

If this is the case, I would rather suggest adding it into the suppressions file, but in case it's not easy, I've filed:
https://github.com/p11-glue/p11-kit/pull/349

Comment 4 Daiki Ueno 2021-01-26 18:51:57 UTC
With the following scratch build incorporating the patch, I confirm that the leak with the 'realpath' code path disappears:
https://koji.fedoraproject.org/koji/taskinfo?taskID=60579346

However, the test_slapd test seems to be still leaking, in a different path, which I'm not sure p11-kit related:

==310263== 192 bytes in 1 blocks are definitely lost in loss record 31 of 49
==310263==    at 0x4839809: malloc (vg_replace_malloc.c:307)
==310263==    by 0x52718ED: CRYPTO_zalloc (mem.c:230)
==310263==    by 0x523F33C: ENGINE_new (eng_lib.c:34)
==310263==    by 0x526B855: UnknownInlinedFun (eng_rdrand.c:70)
...


Viktor, could you try the scratch build and check whether this is my local configuration problem or not?

Comment 5 Viktor Ashirov 2021-01-26 19:54:20 UTC
Hi Daiki,

After upgrading p11-kit packages to p11-kit-0.23.22-2.fc33.x86_64 I no longer see the memleak:

$ ASAN_OPTIONS=fast_unwind_on_malloc=0 ./test_slapd
[==========] Running 12 test(s).
[ RUN      ] test_libslapd_hello
[       OK ] test_libslapd_hello
[ RUN      ] test_libslapd_pblock_analytics
[       OK ] test_libslapd_pblock_analytics
[ RUN      ] test_libslapd_pblock_v3c_target_dn
[       OK ] test_libslapd_pblock_v3c_target_dn
[ RUN      ] test_libslapd_pblock_v3c_target_sdn
[       OK ] test_libslapd_pblock_v3c_target_sdn
[ RUN      ] test_libslapd_pblock_v3c_original_target_dn
[       OK ] test_libslapd_pblock_v3c_original_target_dn
[ RUN      ] test_libslapd_pblock_v3c_target_uniqueid
[       OK ] test_libslapd_pblock_v3c_target_uniqueid
[ RUN      ] test_libslapd_schema_filter_validate_simple
[       OK ] test_libslapd_schema_filter_validate_simple
[ RUN      ] test_libslapd_operation_v3c_target_spec
[       OK ] test_libslapd_operation_v3c_target_spec
[ RUN      ] test_libslapd_counters_atomic_usage
[       OK ] test_libslapd_counters_atomic_usage
[ RUN      ] test_libslapd_counters_atomic_overflow
[       OK ] test_libslapd_counters_atomic_overflow
[ RUN      ] test_libslapd_pal_meminfo
[       OK ] test_libslapd_pal_meminfo
[ RUN      ] test_libslapd_util_cachesane
[       OK ] test_libslapd_util_cachesane
[==========] 12 test(s) run.
[  PASSED  ] 12 test(s).
[==========] Running 3 test(s).
[ RUN      ] test_plugin_hello
[       OK ] test_plugin_hello
[ RUN      ] test_plugin_pwdstorage_pbkdf2_auth
[       OK ] test_plugin_pwdstorage_pbkdf2_auth
[ RUN      ] test_plugin_pwdstorage_pbkdf2_rounds
[       OK ] test_plugin_pwdstorage_pbkdf2_rounds
[==========] 3 test(s) run.
[  PASSED  ] 3 test(s).

But I'm using AddressSanitizer, not Valgrind. 
Thank you!

Comment 6 Daiki Ueno 2021-01-28 10:12:09 UTC
Thank you for confirming. I've submitted a build.

Comment 7 Fedora Update System 2021-01-28 11:10:52 UTC
FEDORA-2021-b848e92d08 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-b848e92d08

Comment 8 Fedora Update System 2021-01-28 11:10:53 UTC
FEDORA-2021-5f917a244e has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-5f917a244e

Comment 9 Fedora Update System 2021-01-29 02:01:43 UTC
FEDORA-2021-b848e92d08 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-b848e92d08`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-b848e92d08

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2021-01-29 02:52:40 UTC
FEDORA-2021-5f917a244e has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-5f917a244e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-5f917a244e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2021-01-30 01:54:46 UTC
FEDORA-2021-b848e92d08 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 12 Fedora Update System 2021-02-13 01:17:49 UTC
FEDORA-2021-5f917a244e has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.