After a debug session, attempting to discover why the global variable 'root_cert_store' was being corrupted, we found this: [...] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff7ff8700 (LWP 27991)] [New Thread 0x7ffff4a96700 (LWP 27992)] [New Thread 0x7ffff4295700 (LWP 27993)] [New Thread 0x7ffff3a94700 (LWP 27994)] [New Thread 0x7ffff3293700 (LWP 27995)] [New Thread 0x7ffff2a92700 (LWP 27996)] [New Thread 0x7ffff2291700 (LWP 27997)] [New Thread 0x7ffff1a90700 (LWP 27998)] [New Thread 0x7ffff128f700 (LWP 27999)] Breakpoint 1, SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1e74380) at ssl_lib.c:3315 3315 { (gdb) bt #0 SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1e74380) at ssl_lib.c:3315 #1 0x0000000000f0089d in node::crypto::SecureContext::AddRootCerts (args=...) at ../src/node_crypto.cc:964 #2 0x00000000007c9a74 in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffd1c0, f=f@entry=0xf00840 <node::crypto::SecureContext::AddRootCerts(v8::FunctionCallbackInfo<v8::Value> const&)>) at ../deps/v8/src/api-arguments.cc:16 #3 0x000000000080f6ab in v8::internal::(anonymous namespace)::HandleApiCallHelper (isolate=isolate@entry=0x1aa6d70, args=...) at ../deps/v8/src/builtins.cc:4311 #4 0x000000000080fe9e in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x1aa6d70) at ../deps/v8/src/builtins.cc:4329 #5 0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326 #6 0x00001d119c2092a7 in ?? () #7 0x00001d119c2091e1 in ?? () #8 0x00007fffffffd370 in ?? () #9 0x0000000300000000 in ?? () #10 0x00007fffffffd410 in ?? () #11 0x00001d119d1e892a in ?? () #12 0x000006de98704381 in ?? () #13 0x00002ad14ec21719 in ?? () #14 0x000018061c77d069 in ?? () #15 0x00002ad14ec21719 in ?? () #16 0x00002ad14ec3d0e9 in ?? () #17 0x000006de98704381 in ?? () #18 0x000006de98704381 in ?? () #19 0x000006de98704381 in ?? () #20 0x000018061c77cfe1 in ?? () #21 0x000006de98704381 in ?? () #22 0x000006de98704381 in ?? () #23 0x000006de98704381 in ?? () #24 0x000006de98704381 in ?? () #25 0x00002ad14ec3d0e9 in ?? () #26 0x000013fd38484f91 in ?? () #27 0x00007fffffffd450 in ?? () #28 0x00001d119c209895 in ?? () #29 0x000006de98704381 in ?? () #30 0x000018061c779571 in ?? () #31 0x000013fd384e1069 in ?? () ---Type <return> to continue, or q <return> to quit---q Quit (gdb) p root_cert_store No symbol "root_cert_store" in current context. (gdb) f 5 #5 0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326 4326 BUILTIN(HandleApiCall) { (gdb) p root_cert_store No symbol "root_cert_store" in current context. (gdb) f 1 #1 0x0000000000f0089d in node::crypto::SecureContext::AddRootCerts (args=...) at ../src/node_crypto.cc:964 964 SSL_CTX_set_cert_store(sc->ctx_, root_cert_store); (gdb) p root_cert_store $1 = (X509_STORE *) 0x1e74380 (gdb) p* root_cert_store $2 = {cache = 1, objs = 0x1bcecb0, get_cert_methods = 0x1e76c00, param = 0x1e77270, verify = 0x0, verify_cb = 0x0, get_issuer = 0x0, check_issued = 0x0, check_revocation = 0x0, get_crl = 0x0, check_crl = 0x0, cert_crl = 0x0, lookup_certs = 0x0, lookup_crls = 0x0, cleanup = 0x0, ex_data = {sk = 0x0, dummy = -1515870811}, references = 1} (gdb) p& root_cert_store.cache $3 = (int *) 0x1e74380 (gdb) watch *$3 Hardware watchpoint 2: *$3 (gdb) p &root_cert_store.references $4 = (int *) 0x1e74408 (gdb) watch *$4 Hardware watchpoint 3: *$4 (gdb) info b Num Type Disp Enb Address What 1 breakpoint keep y 0x00007ffff7110510 in SSL_CTX_set_cert_store at ssl_lib.c:3315 breakpoint already hit 1 time 2 hw watchpoint keep y *$3 3 hw watchpoint keep y *$4 (gdb) d 1 (gdb) cont Continuing. Hardware watchpoint 3: *$4 Old value = 1 New value = 0 CRYPTO_add_lock (pointer=0x1e74408, amount=<optimized out>, type=11, file=0x7ffff75261ef "x509_lu.c", line=241) at cryptlib.c:634 634 CRYPTO_lock(CRYPTO_UNLOCK | CRYPTO_WRITE, type, file, line); (gdb) bt #0 CRYPTO_add_lock (pointer=0x1e74408, amount=<optimized out>, type=11, file=0x7ffff75261ef "x509_lu.c", line=241) at cryptlib.c:634 #1 0x00007ffff74a38e5 in X509_STORE_free (vfy=0x1e74380) at x509_lu.c:241 #2 0x00007ffff711052a in SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1cee2b0) at ssl_lib.c:3317 #3 0x0000000000f00cd6 in node::crypto::SecureContext::LoadPKCS12 (args=...) at ../src/node_crypto.cc:1194 #4 0x00000000007c9a74 in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffd1b0, f=f@entry=0xf00a00 <node::crypto::SecureContext::LoadPKCS12(v8::FunctionCallbackInfo<v8::Value> const&)>) at ../deps/v8/src/api-arguments.cc:16 #5 0x000000000080f6ab in v8::internal::(anonymous namespace)::HandleApiCallHelper (isolate=isolate@entry=0x1aa6d70, args=...) at ../deps/v8/src/builtins.cc:4311 #6 0x000000000080fe9e in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x1aa6d70) at ../deps/v8/src/builtins.cc:4329 #7 0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326 #8 0x00001d119c2092a7 in ?? () #9 0x00001d119c2091e1 in ?? () #10 0x00007fffffffd360 in ?? () #11 0x0000000300000000 in ?? () #12 0x00007fffffffd410 in ?? () #13 0x00001d119d1e997e in ?? () #14 0x000006de98704381 in ?? () #15 0x00002ad14ec22059 in ?? () #16 0x000018061c77d0c1 in ?? () #17 0x000018061c74a399 in ?? () #18 0x000018061c77d069 in ?? () #19 0x00002ad14ec22059 in ?? () #20 0x00002ad14ec3d0e9 in ?? () #21 0x000018061c77d0c1 in ?? () #22 0x000018061c74a399 in ?? () #23 0x000006de98704381 in ?? () #24 0x000018061c77cfe1 in ?? () #25 0x000006de98704381 in ?? () #26 0x000006de98704381 in ?? () #27 0x000006de98704381 in ?? () #28 0x000006de98704381 in ?? () #29 0x00002ad14ec3d0e9 in ?? () #30 0x000013fd38484f91 in ?? () ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 3 #3 0x0000000000f00cd6 in node::crypto::SecureContext::LoadPKCS12 (args=...) at ../src/node_crypto.cc:1194 1194 SSL_CTX_set_cert_store(sc->ctx_, cert_store); (gdb) list 1189 for (int i = 0; i < sk_X509_num(extra_certs); i++) { 1190 X509* ca = sk_X509_value(extra_certs, i); 1191 1192 if (cert_store == root_cert_store) { 1193 cert_store = NewRootCertStore(); 1194 SSL_CTX_set_cert_store(sc->ctx_, cert_store); 1195 } 1196 X509_STORE_add_cert(cert_store, ca); 1197 SSL_CTX_add_client_CA(sc->ctx_, ca); 1198 } (gdb) p cert_store $5 = (X509_STORE *) 0x1cee2b0 (gdb) p root_cedrt_store No symbol "root_cedrt_store" in current context. (gdb) p root_cert_store $6 = (X509_STORE *) 0x1e74380 (gdb) f 2 #2 0x00007ffff711052a in SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1cee2b0) at ssl_lib.c:3317 3317 X509_STORE_free(ctx->cert_store); (gdb) f 1 #1 0x00007ffff74a38e5 in X509_STORE_free (vfy=0x1e74380) at x509_lu.c:241 241 i = CRYPTO_add(&vfy->references, -1, CRYPTO_LOCK_X509_STORE); (gdb) cont Continuing. Hardware watchpoint 2: *$3 Old value = 1 New value = 1515870810 __memset_sse2 () at ../sysdeps/x86_64/memset.S:84 84 andq $-64, %rcx First a breakpoint was set in SSL_CTX_set_cert_store, then, at frame #1 root_cert_store became visible, so some watchpoints were added. As noticed above, it would be required to patch around ../src/node_crypto.cc:1194 to increment the 'references' field when if (cert_store == root_cert_store) { is true. Or a variant of the patch added at https://bugzilla.redhat.com/show_bug.cgi?id=1436445, that is to set ctx->cert_store to NULL.
Created attachment 1512210 [details] node-v6.11.3-sfdc02251159.patch Suggested patch.
Customer confirms that the patch worked for them.
Red Hat does not currently plan to provide any further changes to this collection in a Red Hat Software Collections update release. This software collection is nearing the retirement date (April 2019) after which customers are encouraged to upgrade to a later release. Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/
In accordance with the Red Hat Software Collections Product Life Cycle, the support period for this collection has ended. New bug fix, enhancement, and security errata updates, as well as technical support services will no longer be made available for this collection. Customers are encouraged to upgrade to a later release. Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/