Bug 1435017

Created attachment 1265709 [details]
Stack trace captured during segfault occurred on 389-ds-base process

(In reply to Matteo Piva from comment #6)
> Created attachment 1265709 [details]
> Stack trace captured during segfault occurred on 389-ds-base process

This stacktrace still doesn't provide more details in the stack:

#1  0x00007f53786af85e in ssl3_GatherCompleteHandshake () from /lib64/libssl3.so
No symbol table info available.

It's necessary to have the nss-debuginfo package installed, prior to reproducing the problem again and creating the new stack.

We have two 389 servers with replication configured and active. Some Java applications served by both.
When we run a Qualys vulnerability scan on the VMs that runs the services one or both ns-slapd daemons dies unexpectedly:

Mar 22 13:42:52 ips1.devenv.dev kernel: ns-slapd[2321]: segfault at 0 ip 00007f244de1a4e8 sp 00007f24237e57e8 error 6 in libc-2.17.so 7f244dccf000+1b6000]
Mar 22 13:42:53 ips1.devenv.dev systemd[1]: dirsrv @ips_ips1.service: main process exited, code=killed, status=11/SEGV

We run on CentOS Linux release 7.3.1611 (Core). Every packages is kept up to date to centos-updates and epel repos

The version of 389-ds-base installed is 1.3.5.10-18.el7_3 (latest)
The version of nss we use is 3.28.2-1.6.el7_3 (latest)


I attacked a stack trace captured during the sefault showing that the cause should be found in the libssl3.so code:

Thread 1 (Thread 0x7f53577fe700 (LWP 25383)):
#0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2183
No locals.
#1  0x00007f53786af85e in ssl3_GatherCompleteHandshake () from /lib64/libssl3.so
No symbol table info available.
#2  0x00007f53786aff79 in ssl_GatherRecord1stHandshake () from /lib64/libssl3.so
No symbol table info available.
#3  0x00007f53786b5182 in ssl_Do1stHandshake () from /lib64/libssl3.so
No symbol table info available.
#4  0x00007f53786b5f7e in ssl_SecureRecv () from /lib64/libssl3.so
No symbol table info available.
#5  0x00007f53786b9be1 in ssl_Recv () from /lib64/libssl3.so
No symbol table info available.
#6  0x00007f5379e4e5c7 in connection_read_operation ()
No symbol table info available.
#7  0x00007f5379e4edde in connection_threadmain ()
No symbol table info available.
#8  0x00007f5377b459bb in _pt_root () from /lib64/libnspr4.so
No symbol table info available.
#9  0x00007f53774e5dc5 in start_thread (arg=0x7f53577fe700) at pthread_create.c:308
        __res = <optimized out>
        pd = 0x7f53577fe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139995927013120, 169946331390907139, 0, 139995927013824, 139995927013120, 1, -216898777570635005, -216969013174627581}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#10 0x00007f537721473d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Hello,

I reproduced again the segfault and created a new stack for you as attachment.

Installed Packages
Name        : nss-debuginfo
Arch        : x86_64
Version     : 3.21.3
Release     : 2.el7_3
Size        : 24 M
Repo        : installed
From repo   : base-debuginfo
Summary     : Debug information for package nss
URL         : http://www.mozilla.org/projects/security/pki/nss/
License     : MPLv2.0
Description : This package provides debug information for package nss.
            : Debug information is useful when developing applications that use this
            : package or when debugging this package.

Installed Packages
Name        : nss
Arch        : x86_64
Version     : 3.28.2
Release     : 1.6.el7_3
Size        : 2.5 M
Repo        : installed
From repo   : updates
Summary     : Network Security Services
URL         : http://www.mozilla.org/projects/security/pki/nss/
License     : MPLv2.0
Description : Network Security Services (NSS) is a set of libraries designed to
            : support cross-platform development of security-enabled client and
            : server applications. Applications built with NSS can support SSL v2
            : and v3, TLS, PKCS #5, PKCS #7, PKCS #11, PKCS #12, S/MIME, X.509
            : v3 certificates, and other security standards.

Thank you

Created attachment 1265776 [details]
Stack trace captured during segfault occurred on 389-ds-base process (nss-debuginfo installed)

This didn't work yet. Unfortunately you have a version mismatch in the installed nss.rpm and nss-debuginfo.rpm packages. 

warning: the debug information found in "/usr/lib/debug//lib64/libssl3.so.debug" does not match "/lib64/libssl3.so" (CRC mismatch).

Name        : nss-debuginfo
Version     : 3.21.3
Release     : 2.el7_3

Name        : nss
Version     : 3.28.2
Release     : 1.6.el7_3

It will only work to create a detailed stack trace, if both version and release numbers of the installed packages match exactly.

For some reason at http://debuginfo.centos.org/7/x86_64/ we only have the nss-debuginfo-3.21.3-2.el7_3.x86_64.rpm version as latest, but it's not related with the issue.
I've manually got it by the last build at https://buildlogs.centos.org/c7.1611.u/nss/20170308180428/3.28.2-1.6.el7_3.x86_64/nss-debuginfo-3.28.2-1.6.el7_3.x86_64.rpm.

Installed Packages
Name        : nss
Arch        : x86_64
Version     : 3.28.2
Release     : 1.6.el7_3
Size        : 2.5 M
Repo        : installed
From repo   : updates
Summary     : Network Security Services
URL         : http://www.mozilla.org/projects/security/pki/nss/
License     : MPLv2.0
Description : Network Security Services (NSS) is a set of libraries designed to
            : support cross-platform development of security-enabled client and
            : server applications. Applications built with NSS can support SSL v2
            : and v3, TLS, PKCS #5, PKCS #7, PKCS #11, PKCS #12, S/MIME, X.509
            : v3 certificates, and other security standards.

Installed Packages
Name        : nss-debuginfo
Arch        : x86_64
Version     : 3.28.2
Release     : 1.6.el7_3
Size        : 25 M
Repo        : installed
Summary     : Debug information for package nss
URL         : http://www.mozilla.org/projects/security/pki/nss/
License     : MPLv2.0
Description : This package provides debug information for package nss.
            : Debug information is useful when developing applications that use this
            : package or when debugging this package.


New stack trace attached

Created attachment 1265825 [details]
Stack trace captured during segfault occurred on 389-ds-base process (nss-debuginfo installed)

*** Bug 1449228 has been marked as a duplicate of this bug. ***

*** Bug 1449227 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1977