Bug 1656156

Summary: apiconnect software not working after latest set of patch upgrades which included rh-nodej6 upgrade
Product: Red Hat Software Collections Reporter: Paulo Andrade <pandrade>
Component: rh-nodejs6Assignee: Zuzana Svetlikova <zsvetlik>
Status: CLOSED EOL QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: medium Docs Contact:
Priority: medium    
Version: rh-nodejs6CC: jwright, webstack-team
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-24 12:52:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
node-v6.11.3-sfdc02251159.patch none

Description Paulo Andrade 2018-12-04 20:51:17 UTC
After a debug session, attempting to discover why the global variable
'root_cert_store' was being corrupted, we found this:

[...]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7ff8700 (LWP 27991)]
[New Thread 0x7ffff4a96700 (LWP 27992)]
[New Thread 0x7ffff4295700 (LWP 27993)]
[New Thread 0x7ffff3a94700 (LWP 27994)]
[New Thread 0x7ffff3293700 (LWP 27995)]
[New Thread 0x7ffff2a92700 (LWP 27996)]
[New Thread 0x7ffff2291700 (LWP 27997)]
[New Thread 0x7ffff1a90700 (LWP 27998)]
[New Thread 0x7ffff128f700 (LWP 27999)]

Breakpoint 1, SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1e74380) at ssl_lib.c:3315
3315    {
(gdb) bt
#0  SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1e74380) at ssl_lib.c:3315
#1  0x0000000000f0089d in node::crypto::SecureContext::AddRootCerts (args=...) at ../src/node_crypto.cc:964
#2  0x00000000007c9a74 in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffd1c0, 
    f=f@entry=0xf00840 <node::crypto::SecureContext::AddRootCerts(v8::FunctionCallbackInfo<v8::Value> const&)>)
    at ../deps/v8/src/api-arguments.cc:16
#3  0x000000000080f6ab in v8::internal::(anonymous namespace)::HandleApiCallHelper (isolate=isolate@entry=0x1aa6d70, 
    args=...) at ../deps/v8/src/builtins.cc:4311
#4  0x000000000080fe9e in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x1aa6d70)
    at ../deps/v8/src/builtins.cc:4329
#5  0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, 
    args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326
#6  0x00001d119c2092a7 in ?? ()
#7  0x00001d119c2091e1 in ?? ()
#8  0x00007fffffffd370 in ?? ()
#9  0x0000000300000000 in ?? ()
#10 0x00007fffffffd410 in ?? ()
#11 0x00001d119d1e892a in ?? ()
#12 0x000006de98704381 in ?? ()
#13 0x00002ad14ec21719 in ?? ()
#14 0x000018061c77d069 in ?? ()
#15 0x00002ad14ec21719 in ?? ()
#16 0x00002ad14ec3d0e9 in ?? ()
#17 0x000006de98704381 in ?? ()
#18 0x000006de98704381 in ?? ()
#19 0x000006de98704381 in ?? ()
#20 0x000018061c77cfe1 in ?? ()
#21 0x000006de98704381 in ?? ()
#22 0x000006de98704381 in ?? ()
#23 0x000006de98704381 in ?? ()
#24 0x000006de98704381 in ?? ()
#25 0x00002ad14ec3d0e9 in ?? ()
#26 0x000013fd38484f91 in ?? ()
#27 0x00007fffffffd450 in ?? ()
#28 0x00001d119c209895 in ?? ()
#29 0x000006de98704381 in ?? ()
#30 0x000018061c779571 in ?? ()
#31 0x000013fd384e1069 in ?? ()
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) p root_cert_store
No symbol "root_cert_store" in current context.
(gdb) f 5
#5  0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, 
    args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326
4326    BUILTIN(HandleApiCall) {
(gdb) p root_cert_store
No symbol "root_cert_store" in current context.
(gdb) f 1
#1  0x0000000000f0089d in node::crypto::SecureContext::AddRootCerts (args=...) at ../src/node_crypto.cc:964
964       SSL_CTX_set_cert_store(sc->ctx_, root_cert_store);
(gdb) p root_cert_store
$1 = (X509_STORE *) 0x1e74380
(gdb) p* root_cert_store
$2 = {cache = 1, objs = 0x1bcecb0, get_cert_methods = 0x1e76c00, param = 0x1e77270, verify = 0x0, verify_cb = 0x0, 
  get_issuer = 0x0, check_issued = 0x0, check_revocation = 0x0, get_crl = 0x0, check_crl = 0x0, cert_crl = 0x0, 
  lookup_certs = 0x0, lookup_crls = 0x0, cleanup = 0x0, ex_data = {sk = 0x0, dummy = -1515870811}, references = 1}
(gdb) p& root_cert_store.cache
$3 = (int *) 0x1e74380
(gdb) watch *$3
Hardware watchpoint 2: *$3
(gdb) p &root_cert_store.references
$4 = (int *) 0x1e74408
(gdb) watch *$4
Hardware watchpoint 3: *$4
(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007ffff7110510 in SSL_CTX_set_cert_store at ssl_lib.c:3315
        breakpoint already hit 1 time
2       hw watchpoint  keep y                      *$3
3       hw watchpoint  keep y                      *$4
(gdb) d 1
(gdb) cont
Continuing.
Hardware watchpoint 3: *$4

Old value = 1
New value = 0
CRYPTO_add_lock (pointer=0x1e74408, amount=<optimized out>, type=11, file=0x7ffff75261ef "x509_lu.c", line=241)
    at cryptlib.c:634
634             CRYPTO_lock(CRYPTO_UNLOCK | CRYPTO_WRITE, type, file, line);
(gdb) bt
#0  CRYPTO_add_lock (pointer=0x1e74408, amount=<optimized out>, type=11, file=0x7ffff75261ef "x509_lu.c", line=241)
    at cryptlib.c:634
#1  0x00007ffff74a38e5 in X509_STORE_free (vfy=0x1e74380) at x509_lu.c:241
#2  0x00007ffff711052a in SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1cee2b0) at ssl_lib.c:3317
#3  0x0000000000f00cd6 in node::crypto::SecureContext::LoadPKCS12 (args=...) at ../src/node_crypto.cc:1194
#4  0x00000000007c9a74 in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffd1b0, 
    f=f@entry=0xf00a00 <node::crypto::SecureContext::LoadPKCS12(v8::FunctionCallbackInfo<v8::Value> const&)>)
    at ../deps/v8/src/api-arguments.cc:16
#5  0x000000000080f6ab in v8::internal::(anonymous namespace)::HandleApiCallHelper (isolate=isolate@entry=0x1aa6d70, 
    args=...) at ../deps/v8/src/builtins.cc:4311
#6  0x000000000080fe9e in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x1aa6d70)
    at ../deps/v8/src/builtins.cc:4329
#7  0x000000000081d83e in v8::internal::Builtin_HandleApiCall (args_length=<optimized out>, 
    args_object=0x7fffffffd3a8, isolate=0x1aa6d70) at ../deps/v8/src/builtins.cc:4326
#8  0x00001d119c2092a7 in ?? ()
#9  0x00001d119c2091e1 in ?? ()
#10 0x00007fffffffd360 in ?? ()
#11 0x0000000300000000 in ?? ()
#12 0x00007fffffffd410 in ?? ()
#13 0x00001d119d1e997e in ?? ()
#14 0x000006de98704381 in ?? ()
#15 0x00002ad14ec22059 in ?? ()
#16 0x000018061c77d0c1 in ?? ()
#17 0x000018061c74a399 in ?? ()
#18 0x000018061c77d069 in ?? ()
#19 0x00002ad14ec22059 in ?? ()
#20 0x00002ad14ec3d0e9 in ?? ()
#21 0x000018061c77d0c1 in ?? ()
#22 0x000018061c74a399 in ?? ()
#23 0x000006de98704381 in ?? ()
#24 0x000018061c77cfe1 in ?? ()
#25 0x000006de98704381 in ?? ()
#26 0x000006de98704381 in ?? ()
#27 0x000006de98704381 in ?? ()
#28 0x000006de98704381 in ?? ()
#29 0x00002ad14ec3d0e9 in ?? ()
#30 0x000013fd38484f91 in ?? ()
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) f 3
#3  0x0000000000f00cd6 in node::crypto::SecureContext::LoadPKCS12 (args=...) at ../src/node_crypto.cc:1194
1194            SSL_CTX_set_cert_store(sc->ctx_, cert_store);
(gdb) list
1189        for (int i = 0; i < sk_X509_num(extra_certs); i++) {
1190          X509* ca = sk_X509_value(extra_certs, i);
1191
1192          if (cert_store == root_cert_store) {
1193            cert_store = NewRootCertStore();
1194            SSL_CTX_set_cert_store(sc->ctx_, cert_store);
1195          }
1196          X509_STORE_add_cert(cert_store, ca);
1197          SSL_CTX_add_client_CA(sc->ctx_, ca);
1198        }
(gdb) p cert_store
$5 = (X509_STORE *) 0x1cee2b0
(gdb) p root_cedrt_store
No symbol "root_cedrt_store" in current context.
(gdb) p root_cert_store
$6 = (X509_STORE *) 0x1e74380
(gdb) f 2
#2  0x00007ffff711052a in SSL_CTX_set_cert_store (ctx=0x1dc65c0, store=0x1cee2b0) at ssl_lib.c:3317
3317            X509_STORE_free(ctx->cert_store);
(gdb) f 1
#1  0x00007ffff74a38e5 in X509_STORE_free (vfy=0x1e74380) at x509_lu.c:241
241         i = CRYPTO_add(&vfy->references, -1, CRYPTO_LOCK_X509_STORE);
(gdb) cont
Continuing.
Hardware watchpoint 2: *$3

Old value = 1
New value = 1515870810
__memset_sse2 () at ../sysdeps/x86_64/memset.S:84
84              andq    $-64, %rcx


  First a breakpoint was set in SSL_CTX_set_cert_store, then, at frame #1 root_cert_store
became visible, so some watchpoints were added. As noticed above, it would be required to
patch around ../src/node_crypto.cc:1194 to increment the 'references' field when
        if (cert_store == root_cert_store) {
is true.

  Or a variant of the patch added at https://bugzilla.redhat.com/show_bug.cgi?id=1436445,
that is to set ctx->cert_store to NULL.

Comment 2 Paulo Andrade 2018-12-06 16:58:14 UTC
Created attachment 1512210 [details]
node-v6.11.3-sfdc02251159.patch

Suggested patch.

Comment 3 Joe Wright 2018-12-21 14:26:02 UTC
Customer confirms that the patch worked for them.

Comment 5 Joe Orton 2019-03-14 13:08:37 UTC
Red Hat does not currently plan to provide any further changes to this collection in a Red Hat Software Collections update release.

This software collection is nearing the retirement date (April 2019) after which customers are encouraged to upgrade to a later release.

Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/

Comment 6 Joe Orton 2019-05-24 12:52:45 UTC
In accordance with the Red Hat Software Collections Product Life Cycle, the support period for this collection has ended.

New bug fix, enhancement, and security errata updates, as well as technical support services will no longer be made available for this collection.

Customers are encouraged to upgrade to a later release.

Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/