Bug 1277613

Summary: Segmentation fault in httpd/mod_nss in the parent process while reloading via SIGHUP
Product: Red Hat Enterprise Linux 6 Reporter: Benoit Donneaux <benoit>
Component: mod_nssAssignee: Matthew Harmsen <mharmsen>
Status: CLOSED ERRATA QA Contact: Kaleem <ksiddiqu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7CC: akasurde, dpal, mharmsen, mjamis, moshiro, mpoole, nkinder, rbost, rcritten
Target Milestone: rc   
Target Release: 6.8   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: mod_nss-1.0.10-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1296685 (view as bug list) Environment:
Last Closed: 2016-05-10 19:40:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1296685    
Attachments:
Description Flags
mod_nss 'nss.conf' configuration file
none
Shell script used for testing this bug on RHEL 6.7
none
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized mharmsen: review+

Description Benoit Donneaux 2015-11-03 16:29:47 UTC 
WARNING: This bug has been found on CentOS 6.7, but I'm pretty sure it is not related to this distro. Though, I can't verify this on Redhat 6.7.


Description of problem:

Apache httpd dies once in a while reporting a seg fault after receiving a SIGHUP:

[Nov 03 16:03:11 2015] [notice] SIGHUP received.  Attempting to restart
ERR [Nov 03 16:03:11 2015] [notice] Digest: generating secret for digest authentication ...
ERR [Nov 03 16:03:11 2015] [notice] Digest: done
ERR [Nov 03 16:03:12 2015] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indic
ation support (RFC 4366)
ERR [Nov 03 16:03:15 2015] [notice] seg fault or similar nasty error detected in the parent process


Version-Release number of selected component (if applicable):

Server Version: Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_nss/2.2.15 NSS/3.19.1 Basic ECC

httpd-2.2.15-47.el6.centos.x86_64
mod_nss-1.0.10-1.el6.x86_64

all package versions are matching the one provided by Redhat/CentOS 6.7


How reproducible:

Looks random on production conditions, but can be reproduce when reloading in a loop.


Steps to Reproduce:

service httpd restart; sleep 1; time { RET=0; while [ $RET -ne 1 ]; do service httpd reload; sleep 1; service httpd status; RET=$?; done;

Usually occurred around a minute after the restart (it does not seems to matter if I remove the sleep 1 between reload and status).


Actual results:

httpd dies on seg fault

Expected results:

httpd keeps running even after thousands on SIGHUP.


Additional info:
Got some core dump, but can not find my way through.
We are using a couple of other mods (mod_ldap, mod_proxy, mod_rewrite...) and some externally maintained like (modsec).
Comment 1 Benoit Donneaux 2015-11-03 16:33:05 UTC 
Workaround : I've downgraded mod_nss to 1.0.8-21.el6 and I can not longer reproduce this seg fault using the reload loop.

According to my findings, looks like this issue has been introduced with this fix (add support for TLSv1.2): https://bugzilla.redhat.com/show_bug.cgi?id=1166316
Comment 4 Matthew Harmsen 2015-12-03 01:54:00 UTC 
Created attachment 1101667 [details]
mod_nss 'nss.conf' configuration file
Comment 5 Matthew Harmsen 2015-12-03 01:54:47 UTC 
Created attachment 1101668 [details]
Shell script used for testing this bug on RHEL 6.7
Comment 6 Matthew Harmsen 2015-12-07 22:19:00 UTC 
I was unable to reproduce this bug on a RHEL 6.7 system:
    
    # cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 6.7 (Santiago)

    # rpm -q httpd mod_nss mod_ssl openssl nss
    httpd-2.2.15-47.el6_7.x86_64
    mod_nss-1.0.10-1.el6.x86_64
    mod_ssl-2.2.15-47.el6_7.x86_64
    openssl-1.0.1e-42.el6.x86_64
    nss-3.19.1-3.el6_6.x86_64

    # httpd -M
    Loaded Modules:
     core_module (static)
     mpm_prefork_module (static)
     http_module (static)
     so_module (static)
     auth_basic_module (shared)
     auth_digest_module (shared)
     authn_file_module (shared)
     authn_alias_module (shared)
     authn_anon_module (shared)
     authn_dbm_module (shared)
     authn_default_module (shared)
     authz_host_module (shared)
     authz_user_module (shared)
     authz_owner_module (shared)
     authz_groupfile_module (shared)
     authz_dbm_module (shared)
     authz_default_module (shared)
     ldap_module (shared)
     authnz_ldap_module (shared)
     include_module (shared)
     log_config_module (shared)
     logio_module (shared)
     env_module (shared)
     ext_filter_module (shared)
     mime_magic_module (shared)
     expires_module (shared)
     deflate_module (shared)
     headers_module (shared)
     usertrack_module (shared)
     setenvif_module (shared)
     mime_module (shared)
     dav_module (shared)
     status_module (shared)
     autoindex_module (shared)
     info_module (shared)
     dav_fs_module (shared)
     vhost_alias_module (shared)
     negotiation_module (shared)
     dir_module (shared)
     actions_module (shared)
     speling_module (shared)
     userdir_module (shared)
     alias_module (shared)
     substitute_module (shared)
     rewrite_module (shared)
     proxy_module (shared)
     proxy_balancer_module (shared)
     proxy_ftp_module (shared)
     proxy_http_module (shared)
     proxy_ajp_module (shared)
     proxy_connect_module (shared)
     cache_module (shared)
     suexec_module (shared)
     disk_cache_module (shared)
     cgi_module (shared)
     version_module (shared)
     auth_kerb_module (shared)
     nss_module (shared)
     ssl_module (shared)
     wsgi_module (shared)
    Syntax OK

From '/etc/httpd/conf.d/nss.conf' (attached):
    .
    .
    .
    LoadModule nss_module modules/libmodnss.so
    .
    .
    .
    NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2
    .
    .
    .

Ran the attached shell script as a test:

    # sudo ./1277613.sh 
    [20151203021947] BEGIN Test of Bugzilla Bug #1277613:

    Stopping httpd:                                            [  OK  ]
    Starting httpd:                                            [  OK  ]
    Test Iteration #1 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #2 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #3 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    .
    .
    .
    Test Iteration #223 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #224 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #225 . . .
    Reloading httpd: 
    ^C** Trapped CTRL-C to end test! **

    [20151203022504] END Test of Bugzilla Bug #1277613.

    Duration of test:  557 seconds.

Please provide any additional configuration information that would possibly conflict with this test.
Comment 7 Matthew Harmsen 2015-12-07 22:20:27 UTC 
Benoit,

Additionally, if you could attach a stacktrace to this bug from the platform that failed, it may be helpful.

Thanks,
-- Matt
Comment 8 Martin Poole 2015-12-13 09:20:16 UTC 
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
1697		    if (pLock->timeStamp   < then &&
(gdb) bt
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

(gdb) bt full
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
        pid = <value optimized out>
        cache = 0x7fffef572e40
        sharedCache = 0x7fff11605000
        pLock = <value optimized out>
        timeout = 30000
        now = 1449821565
        then = <value optimized out>
        locks_polled = <value optimized out>
        locks_to_poll = 10
        expiration = 30
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
        rv = <value optimized out>
        thred = 0x7ffff8590e60
        detached = 0
        id = 140735487686400
        tid = 17108
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7fff88c03700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140735487686400, 3339883259476957173, 140737488346400, 140735487687104, 0, 3,
                -3339780170127648779, -3339899399347775499}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
              prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.




(gdb) thread apply all bt
Thread 31 (Thread 0x7ffff7f937e0 (LWP 11596)):
#0  __memset_sse2 () at ../sysdeps/x86_64/memset.S:908
#1  0x00007fffef35b596 in InitCache (cache=0x7fffef572e40, ssl2_timeout=<value optimized out>, ssl3_timeout=<value optimized out>,
    directory=0x7fffef368930 "/tmp", shared=1, maxCacheEntries=10000, maxCertCacheEntries=10000, maxSrvNameCacheEntries=-1)
    at /usr/include/bits/string3.h:85
#2  ssl_ConfigServerSessionIDCacheInstanceWithOpt (cache=0x7fffef572e40, ssl2_timeout=<value optimized out>,
    ssl3_timeout=<value optimized out>, directory=0x7fffef368930 "/tmp", shared=1, maxCacheEntries=10000,
    maxCertCacheEntries=10000, maxSrvNameCacheEntries=-1) at sslsnce.c:1319
#3  0x00007fffef35b8ac in ssl_ConfigMPServerSIDCacheWithOpt (ssl2_timeout=<value optimized out>,
    ssl3_timeout=<value optimized out>, directory=<value optimized out>, maxCacheEntries=<value optimized out>,
    maxCertCacheEntries=<value optimized out>, maxSrvNameCacheEntries=<value optimized out>) at sslsnce.c:1400
#4  0x00007fffef582cf0 in nss_init_Module (p=0x7ffff820b158, plog=<value optimized out>, ptemp=0x7ffff82c0568,
    base_server=0x7ffff8210870) at nss_engine_init.c:511
#5  0x00007ffff7fd6c49 in ap_run_post_config (pconf=0x7ffff820b158, plog=0x7ffff823d2e8, ptemp=0x7ffff82c0568, s=0x7ffff8210870)
    at /usr/src/debug/httpd-2.2.15/server/config.c:94
#6  0x00007ffff7fc29c7 in main (argc=1, argv=0x7fffffffe228) at /usr/src/debug/httpd-2.2.15/server/main.c:752

Thread 30 (Thread 0x7fff99604700 (LWP 16828)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff340008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff340008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85fed20) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff99604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 29 (Thread 0x7fffa1604700 (LWP 14784)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff440008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff440008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861aea0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 28 (Thread 0x7fffa0c03700 (LWP 14824)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff3c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff3c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85a1190) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 27 (Thread 0x7fff98c03700 (LWP 16932)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff2c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff2c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85db420) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff98c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 26 (Thread 0x7fffd1604700 (LWP 11902)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffa40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
---Type <return> to continue, or q <return> to quit---
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffa40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8678ec0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 25 (Thread 0x7fffa9604700 (LWP 14710)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff540008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff540008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff86168b0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 24 (Thread 0x7fffc0c03700 (LWP 14521)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff7c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff7c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8687c10) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 23 (Thread 0x7fffa8c03700 (LWP 14747)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff4c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff4c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff867f3a0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 22 (Thread 0x7fff91604700 (LWP 16975)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff240008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff240008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff860d440) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff91604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 21 (Thread 0x7fffd0c03700 (LWP 11939)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff9c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff9c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff864f3d0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 20 (Thread 0x7fffb0c03700 (LWP 14673)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff5c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff5c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861b870) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 19 (Thread 0x7fffd8c03700 (LWP 11865)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffac0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffac0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85870d0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 18 (Thread 0x7fffc9604700 (LWP 11976)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff940008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff940008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff866c570) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 17 (Thread 0x7fffb1604700 (LWP 14636)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff640008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff640008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff866fb50) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 16 (Thread 0x7fffd9604700 (LWP 11828)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffb40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffb40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff867ea60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 15 (Thread 0x7fff90c03700 (LWP 17029)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff1c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff1c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8273b20) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff90c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 14 (Thread 0x7fffc1604700 (LWP 13641)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff840008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff840008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff860f740) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 13 (Thread 0x7fffeb325700 (LWP 11597)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffe40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffe40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff859b740) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffeb325700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 12 (Thread 0x7fffb8c03700 (LWP 14599)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff6c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff6c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff859c880) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 11 (Thread 0x7fffe0c03700 (LWP 11791)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffbc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffbc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff826b500) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fff89604700 (LWP 17066)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff140008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff140008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8589680) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff89604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fffe97db700 (LWP 11680)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffd40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffd40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8599c50) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe97db700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7fffe1604700 (LWP 11754)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffc40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffc40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff857d2e0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7fffc8c03700 (LWP 12013)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff8c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff8c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85d9a90) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7fffb9604700 (LWP 14562)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff740008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff740008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8619500) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7fff80c03700 (LWP 19232)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7ffefc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7ffefc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8620aa0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff80c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7fffe8dda700 (LWP 11717)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffcc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffcc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8631d80) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe8dda700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fffea1dc700 (LWP 11639)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffdc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffdc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861ee90) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffea1dc700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fff81604700 (LWP 19079)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff040008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff040008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff86266f0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff81604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fff88c03700 (LWP 17108)):
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 10 Rob Crittenden 2016-01-13 22:34:23 UTC 
I've been able to reproduce the crash, matching the backtrace that Martin provided.

Will continue investigating.
Comment 11 Rob Crittenden 2016-01-15 15:22:41 UTC 
Created attachment 1115180 [details]
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized

The SessionIDCache has its own set of threads and objects so it is safe to call even in the case that NSS_Initialized() returns FALSE.

I was able to run 30k+ iterations within Matt's test script. Without this I could reliably kill Apache with < 150 iterations.
Comment 12 Matthew Harmsen 2016-01-19 16:56:16 UTC 
Comment on attachment 1115180 [details]
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized

For future clarification, in nss_init_ModuleKill(), please add a preceding comment as to why it was necessary to add the additional check for NSPR (i. e. - our IRC discussion regarding Parent vs. Child processes).
Comment 14 Abhijeet Kasurde 2016-03-03 06:37:28 UTC 
Verified 'SanityOnly' using mod_nss version ::

mod_nss-1.0.10-6.el6.x86_64
Comment 16 mjamis 2016-04-18 17:05:34 UTC 
Any ETA on when this fix might be released, or is there a package that can be used to test it?   I'm seeing what I'm pretty certain is this exact same issue on a RHEL6.7 system.
Comment 17 mjamis 2016-04-18 21:56:04 UTC 
(In reply to mjamis from comment #16)
> Any ETA on when this fix might be released, or is there a package that can
> be used to test it?   I'm seeing what I'm pretty certain is this exact same
> issue on a RHEL6.7 system.

fyi, I found the fix in mod_nss-1.0.10-5.el6 the 6.8 beta release channel and can confirm that it does fix this issue for me.
Comment 19 errata-xmlrpc 2016-05-10 19:40:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0751.html