Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1277613

Summary: Segmentation fault in httpd/mod_nss in the parent process while reloading via SIGHUP
Product: Red Hat Enterprise Linux 6 Reporter: Benoit Donneaux <benoit>
Component: mod_nssAssignee: Matthew Harmsen <mharmsen>
Status: CLOSED ERRATA QA Contact: Kaleem <ksiddiqu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7CC: akasurde, dpal, mharmsen, mjamis, moshiro, mpoole, nkinder, rbost, rcritten
Target Milestone: rc   
Target Release: 6.8   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: mod_nss-1.0.10-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1296685 (view as bug list) Environment:
Last Closed: 2016-05-10 19:40:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1296685    
Attachments:
Description Flags
mod_nss 'nss.conf' configuration file
none
Shell script used for testing this bug on RHEL 6.7
none
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized mharmsen: review+

Description Benoit Donneaux 2015-11-03 16:29:47 UTC 
WARNING: This bug has been found on CentOS 6.7, but I'm pretty sure it is not related to this distro. Though, I can't verify this on Redhat 6.7.


Description of problem:

Apache httpd dies once in a while reporting a seg fault after receiving a SIGHUP:

[Nov 03 16:03:11 2015] [notice] SIGHUP received.  Attempting to restart
ERR [Nov 03 16:03:11 2015] [notice] Digest: generating secret for digest authentication ...
ERR [Nov 03 16:03:11 2015] [notice] Digest: done
ERR [Nov 03 16:03:12 2015] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indic
ation support (RFC 4366)
ERR [Nov 03 16:03:15 2015] [notice] seg fault or similar nasty error detected in the parent process


Version-Release number of selected component (if applicable):

Server Version: Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_nss/2.2.15 NSS/3.19.1 Basic ECC

httpd-2.2.15-47.el6.centos.x86_64
mod_nss-1.0.10-1.el6.x86_64

all package versions are matching the one provided by Redhat/CentOS 6.7


How reproducible:

Looks random on production conditions, but can be reproduce when reloading in a loop.


Steps to Reproduce:

service httpd restart; sleep 1; time { RET=0; while [ $RET -ne 1 ]; do service httpd reload; sleep 1; service httpd status; RET=$?; done;

Usually occurred around a minute after the restart (it does not seems to matter if I remove the sleep 1 between reload and status).


Actual results:

httpd dies on seg fault

Expected results:

httpd keeps running even after thousands on SIGHUP.


Additional info:
Got some core dump, but can not find my way through.
We are using a couple of other mods (mod_ldap, mod_proxy, mod_rewrite...) and some externally maintained like (modsec).
Comment 1 Benoit Donneaux 2015-11-03 16:33:05 UTC 
Workaround : I've downgraded mod_nss to 1.0.8-21.el6 and I can not longer reproduce this seg fault using the reload loop.

According to my findings, looks like this issue has been introduced with this fix (add support for TLSv1.2): https://bugzilla.redhat.com/show_bug.cgi?id=1166316
Comment 4 Matthew Harmsen 2015-12-03 01:54:00 UTC 
Created attachment 1101667 [details]
mod_nss 'nss.conf' configuration file
Comment 5 Matthew Harmsen 2015-12-03 01:54:47 UTC 
Created attachment 1101668 [details]
Shell script used for testing this bug on RHEL 6.7
Comment 6 Matthew Harmsen 2015-12-07 22:19:00 UTC 
I was unable to reproduce this bug on a RHEL 6.7 system:
    
    # cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 6.7 (Santiago)

    # rpm -q httpd mod_nss mod_ssl openssl nss
    httpd-2.2.15-47.el6_7.x86_64
    mod_nss-1.0.10-1.el6.x86_64
    mod_ssl-2.2.15-47.el6_7.x86_64
    openssl-1.0.1e-42.el6.x86_64
    nss-3.19.1-3.el6_6.x86_64

    # httpd -M
    Loaded Modules:
     core_module (static)
     mpm_prefork_module (static)
     http_module (static)
     so_module (static)
     auth_basic_module (shared)
     auth_digest_module (shared)
     authn_file_module (shared)
     authn_alias_module (shared)
     authn_anon_module (shared)
     authn_dbm_module (shared)
     authn_default_module (shared)
     authz_host_module (shared)
     authz_user_module (shared)
     authz_owner_module (shared)
     authz_groupfile_module (shared)
     authz_dbm_module (shared)
     authz_default_module (shared)
     ldap_module (shared)
     authnz_ldap_module (shared)
     include_module (shared)
     log_config_module (shared)
     logio_module (shared)
     env_module (shared)
     ext_filter_module (shared)
     mime_magic_module (shared)
     expires_module (shared)
     deflate_module (shared)
     headers_module (shared)
     usertrack_module (shared)
     setenvif_module (shared)
     mime_module (shared)
     dav_module (shared)
     status_module (shared)
     autoindex_module (shared)
     info_module (shared)
     dav_fs_module (shared)
     vhost_alias_module (shared)
     negotiation_module (shared)
     dir_module (shared)
     actions_module (shared)
     speling_module (shared)
     userdir_module (shared)
     alias_module (shared)
     substitute_module (shared)
     rewrite_module (shared)
     proxy_module (shared)
     proxy_balancer_module (shared)
     proxy_ftp_module (shared)
     proxy_http_module (shared)
     proxy_ajp_module (shared)
     proxy_connect_module (shared)
     cache_module (shared)
     suexec_module (shared)
     disk_cache_module (shared)
     cgi_module (shared)
     version_module (shared)
     auth_kerb_module (shared)
     nss_module (shared)
     ssl_module (shared)
     wsgi_module (shared)
    Syntax OK

From '/etc/httpd/conf.d/nss.conf' (attached):
    .
    .
    .
    LoadModule nss_module modules/libmodnss.so
    .
    .
    .
    NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2
    .
    .
    .

Ran the attached shell script as a test:

    # sudo ./1277613.sh 
    [20151203021947] BEGIN Test of Bugzilla Bug #1277613:

    Stopping httpd:                                            [  OK  ]
    Starting httpd:                                            [  OK  ]
    Test Iteration #1 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #2 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #3 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    .
    .
    .
    Test Iteration #223 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #224 . . .
    Reloading httpd: 
    httpd (pid  7828) is running...
    Test Iteration #225 . . .
    Reloading httpd: 
    ^C** Trapped CTRL-C to end test! **

    [20151203022504] END Test of Bugzilla Bug #1277613.

    Duration of test:  557 seconds.

Please provide any additional configuration information that would possibly conflict with this test.
Comment 7 Matthew Harmsen 2015-12-07 22:20:27 UTC 
Benoit,

Additionally, if you could attach a stacktrace to this bug from the platform that failed, it may be helpful.

Thanks,
-- Matt
Comment 8 Martin Poole 2015-12-13 09:20:16 UTC 
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
1697		    if (pLock->timeStamp   < then &&
(gdb) bt
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

(gdb) bt full
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
        pid = <value optimized out>
        cache = 0x7fffef572e40
        sharedCache = 0x7fff11605000
        pLock = <value optimized out>
        timeout = 30000
        now = 1449821565
        then = <value optimized out>
        locks_polled = <value optimized out>
        locks_to_poll = 10
        expiration = 30
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
        rv = <value optimized out>
        thred = 0x7ffff8590e60
        detached = 0
        id = 140735487686400
        tid = 17108
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7fff88c03700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140735487686400, 3339883259476957173, 140737488346400, 140735487687104, 0, 3,
                -3339780170127648779, -3339899399347775499}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
              prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.




(gdb) thread apply all bt
Thread 31 (Thread 0x7ffff7f937e0 (LWP 11596)):
#0  __memset_sse2 () at ../sysdeps/x86_64/memset.S:908
#1  0x00007fffef35b596 in InitCache (cache=0x7fffef572e40, ssl2_timeout=<value optimized out>, ssl3_timeout=<value optimized out>,
    directory=0x7fffef368930 "/tmp", shared=1, maxCacheEntries=10000, maxCertCacheEntries=10000, maxSrvNameCacheEntries=-1)
    at /usr/include/bits/string3.h:85
#2  ssl_ConfigServerSessionIDCacheInstanceWithOpt (cache=0x7fffef572e40, ssl2_timeout=<value optimized out>,
    ssl3_timeout=<value optimized out>, directory=0x7fffef368930 "/tmp", shared=1, maxCacheEntries=10000,
    maxCertCacheEntries=10000, maxSrvNameCacheEntries=-1) at sslsnce.c:1319
#3  0x00007fffef35b8ac in ssl_ConfigMPServerSIDCacheWithOpt (ssl2_timeout=<value optimized out>,
    ssl3_timeout=<value optimized out>, directory=<value optimized out>, maxCacheEntries=<value optimized out>,
    maxCertCacheEntries=<value optimized out>, maxSrvNameCacheEntries=<value optimized out>) at sslsnce.c:1400
#4  0x00007fffef582cf0 in nss_init_Module (p=0x7ffff820b158, plog=<value optimized out>, ptemp=0x7ffff82c0568,
    base_server=0x7ffff8210870) at nss_engine_init.c:511
#5  0x00007ffff7fd6c49 in ap_run_post_config (pconf=0x7ffff820b158, plog=0x7ffff823d2e8, ptemp=0x7ffff82c0568, s=0x7ffff8210870)
    at /usr/src/debug/httpd-2.2.15/server/config.c:94
#6  0x00007ffff7fc29c7 in main (argc=1, argv=0x7fffffffe228) at /usr/src/debug/httpd-2.2.15/server/main.c:752

Thread 30 (Thread 0x7fff99604700 (LWP 16828)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff340008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff340008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85fed20) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff99604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 29 (Thread 0x7fffa1604700 (LWP 14784)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff440008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff440008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861aea0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 28 (Thread 0x7fffa0c03700 (LWP 14824)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff3c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff3c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85a1190) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 27 (Thread 0x7fff98c03700 (LWP 16932)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff2c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff2c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85db420) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff98c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 26 (Thread 0x7fffd1604700 (LWP 11902)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffa40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
---Type <return> to continue, or q <return> to quit---
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffa40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8678ec0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 25 (Thread 0x7fffa9604700 (LWP 14710)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff540008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff540008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff86168b0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 24 (Thread 0x7fffc0c03700 (LWP 14521)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff7c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff7c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8687c10) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 23 (Thread 0x7fffa8c03700 (LWP 14747)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff4c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff4c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff867f3a0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffa8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 22 (Thread 0x7fff91604700 (LWP 16975)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff240008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff240008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff860d440) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff91604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 21 (Thread 0x7fffd0c03700 (LWP 11939)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff9c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff9c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff864f3d0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 20 (Thread 0x7fffb0c03700 (LWP 14673)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff5c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff5c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861b870) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 19 (Thread 0x7fffd8c03700 (LWP 11865)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffac0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffac0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85870d0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 18 (Thread 0x7fffc9604700 (LWP 11976)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff940008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff940008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff866c570) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 17 (Thread 0x7fffb1604700 (LWP 14636)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff640008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff640008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff866fb50) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 16 (Thread 0x7fffd9604700 (LWP 11828)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffb40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffb40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff867ea60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffd9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 15 (Thread 0x7fff90c03700 (LWP 17029)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff1c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff1c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8273b20) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff90c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 14 (Thread 0x7fffc1604700 (LWP 13641)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff840008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff840008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff860f740) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 13 (Thread 0x7fffeb325700 (LWP 11597)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffe40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffe40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff859b740) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffeb325700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 12 (Thread 0x7fffb8c03700 (LWP 14599)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff6c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff6c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff859c880) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 11 (Thread 0x7fffe0c03700 (LWP 11791)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffbc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffbc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff826b500) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe0c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fff89604700 (LWP 17066)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff140008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff140008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8589680) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff89604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fffe97db700 (LWP 11680)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffd40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffd40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8599c50) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe97db700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7fffe1604700 (LWP 11754)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffc40008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffc40008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff857d2e0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe1604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7fffc8c03700 (LWP 12013)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff8c0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff8c0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff85d9a90) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffc8c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7fffb9604700 (LWP 14562)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff740008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff740008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8619500) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffb9604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7fff80c03700 (LWP 19232)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7ffefc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7ffefc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8620aa0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff80c03700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7fffe8dda700 (LWP 11717)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffcc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffcc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8631d80) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffe8dda700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fffea1dc700 (LWP 11639)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fffdc0008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fffdc0008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff861ee90) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fffea1dc700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fff81604700 (LWP 19079)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007fffee7a5f99 in pt_TimedWait (cv=0x7fff040008c8, ml=0x7ffff8326a90, timeout=<value optimized out>)
    at ../../../nspr/pr/src/pthreads/ptsynch.c:264
#2  0x00007fffee7a6f1f in PR_WaitCondVar (cvar=0x7fff040008c0, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387
#3  0x00007fffee7ac6e8 in PR_Sleep (ticks=30000) at ../../../nspr/pr/src/pthreads/ptthread.c:834
#4  0x00007fffef35bb0f in LockPoller (arg=0x7fffef572e40) at sslsnce.c:1686
#5  0x00007fffee7aca83 in _pt_root (arg=0x7ffff86266f0) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#6  0x00007ffff6880a51 in start_thread (arg=0x7fff81604700) at pthread_create.c:301
#7  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fff88c03700 (LWP 17108)):
#0  LockPoller (arg=0x7fffef572e40) at sslsnce.c:1697
#1  0x00007fffee7aca83 in _pt_root (arg=0x7ffff8590e60) at ../../../nspr/pr/src/pthreads/ptthread.c:212
#2  0x00007ffff6880a51 in start_thread (arg=0x7fff88c03700) at pthread_create.c:301
#3  0x00007ffff65cd96d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 10 Rob Crittenden 2016-01-13 22:34:23 UTC 
I've been able to reproduce the crash, matching the backtrace that Martin provided.

Will continue investigating.
Comment 11 Rob Crittenden 2016-01-15 15:22:41 UTC 
Created attachment 1115180 [details]
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized

The SessionIDCache has its own set of threads and objects so it is safe to call even in the case that NSS_Initialized() returns FALSE.

I was able to run 30k+ iterations within Matt's test script. Without this I could reliably kill Apache with < 150 iterations.
Comment 12 Matthew Harmsen 2016-01-19 16:56:16 UTC 
Comment on attachment 1115180 [details]
Call SSL_ShutdownServerSessionIDCache() if NSS or NSPR was initialized

For future clarification, in nss_init_ModuleKill(), please add a preceding comment as to why it was necessary to add the additional check for NSPR (i. e. - our IRC discussion regarding Parent vs. Child processes).
Comment 14 Abhijeet Kasurde 2016-03-03 06:37:28 UTC 
Verified 'SanityOnly' using mod_nss version ::

mod_nss-1.0.10-6.el6.x86_64
Comment 16 mjamis 2016-04-18 17:05:34 UTC 
Any ETA on when this fix might be released, or is there a package that can be used to test it?   I'm seeing what I'm pretty certain is this exact same issue on a RHEL6.7 system.
Comment 17 mjamis 2016-04-18 21:56:04 UTC 
(In reply to mjamis from comment #16)
> Any ETA on when this fix might be released, or is there a package that can
> be used to test it?   I'm seeing what I'm pretty certain is this exact same
> issue on a RHEL6.7 system.

fyi, I found the fix in mod_nss-1.0.10-5.el6 the 6.8 beta release channel and can confirm that it does fix this issue for me.
Comment 19 errata-xmlrpc 2016-05-10 19:40:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0751.html