Bug 1013006 - php-fpm Segfault in libnss3.so
php-fpm Segfault in libnss3.so
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: nss (Show other bugs)
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Elio Maldonado Batiz
BaseOS QE Security Team
Depends On:
  Show dependency treegraph
Reported: 2013-09-27 11:03 EDT by Christian Becker
Modified: 2014-07-01 15:31 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-07-01 15:31:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
modified reproducer (896 bytes, application/x-php)
2014-06-11 11:39 EDT, Hubert Kario
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
CentOS 0006618 None None None Never

  None (edit)
Description Christian Becker 2013-09-27 11:03:30 EDT
Description of problem:
php-fpm Segfaults in libnss3.so on curl SSL connection with client certificate

Version-Release number of selected component (if applicable):
Version     : 3.14.3                            Vendor: Red Hat, Inc.
Release     : 4.el6_4                       Build Date: Mon Apr 29 18:02:58 2013

Version     : 7.19.7                            Vendor: Red Hat, Inc.
Release     : 37.el6_4                      Build Date: Fri Jun 14 19:17:24 2013

How reproducible:
See http://bugs.centos.org/view.php?id=6618
See https://abrt.fedoraproject.org/faf/reports/144663/

Additional info:
php-fpm[2315]: segfault at 0 ip 00000038fee2ce52 sp 00007fff3f520080 error 4 in libnss3.so[38fee00000+135000]

#0  PK11_GetPrivateModulusLen (key=<value optimized out>) at pk11akey.c:815
#1  0x0000003cb2c48f25 in PK11_SignatureLen (key=0x202c2b0) at pk11obj.c:528
#2  0x0000003cb440f9c9 in ssl3_SignHashes (hash=0x7fff07a3e300, key=0x202c2b0, buf=0x7fff07a3e2e0, isTLS=1) at ssl3con.c:811
#3  0x0000003cb44130b9 in ssl3_SendCertificateVerify (ss=0x241f3e0) at ssl3con.c:5349
#4  ssl3_SendClientSecondRound (ss=0x241f3e0) at ssl3con.c:6321
#5  0x0000003cb4414792 in ssl3_HandleServerHelloDone (ss=0x241f3e0, b=0x240a9a4 "\001\001", length=0) at ssl3con.c:6242
#6  ssl3_HandleHandshakeMessage (ss=0x241f3e0, b=0x240a9a4 "\001\001", length=0) at ssl3con.c:9491
#7  0x0000003cb4415efe in ssl3_HandleHandshake (ss=<value optimized out>, cText=<value optimized out>, databuf=0x241f778) at ssl3con.c:9592
#8  ssl3_HandleRecord (ss=<value optimized out>, cText=<value optimized out>, databuf=0x241f778) at ssl3con.c:10230
#9  0x0000003cb4416d77 in ssl3_GatherCompleteHandshake (ss=0x241f3e0, flags=0) at ssl3gthr.c:361
#10 0x0000003cb4419522 in ssl_GatherRecord1stHandshake (ss=0x241f3e0) at sslcon.c:1227
#11 0x0000003cb441f7d5 in ssl_Do1stHandshake (ss=0x241f3e0) at sslsecur.c:120
#12 0x0000003cb442100f in SSL_ForceHandshake (fd=<value optimized out>) at sslsecur.c:390
#13 0x0000003cb7c410e5 in Curl_nss_connect () from /usr/lib64/libcurl.so.4
#14 0x0000003cb7c38482 in Curl_ssl_connect () from /usr/lib64/libcurl.so.4
#15 0x0000003cb7c16ecb in Curl_http_connect () from /usr/lib64/libcurl.so.4
#16 0x0000003cb7c1d682 in Curl_protocol_connect () from /usr/lib64/libcurl.so.4
#17 0x0000003cb7c23b3c in Curl_connect () from /usr/lib64/libcurl.so.4
#18 0x0000003cb7c2bbb0 in Curl_perform () from /usr/lib64/libcurl.so.4
#19 0x00007fe751ceada3 in ?? () from /usr/lib64/php/modules/curl.so
#20 0x000000000069620b in ?? ()
#21 0x000000000066deb0 in execute ()
#22 0x00000000006474dd in zend_execute_scripts ()
#23 0x00000000005f41c8 in php_execute_script ()
#24 0x00000000006da2a8 in ?? ()
#25 0x0000003cafc1ecdd in __libc_start_main (main=0x6d9210, argc=1, ubp_av=0x7fff07a43538, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff07a43528)
    at libc-start.c:226
#26 0x00000000004226b9 in _start ()
Comment 2 Christian Becker 2013-09-27 13:53:59 EDT
This issue is not triggered with every request. 

If we do our test case on the command line it always works fine. As soon as we switch to php-fpm we have about a 50/50 chance to hit this issue.

So we think there could be a memory corruption caused by nss which is also causing https://bugzilla.redhat.com/show_bug.cgi?id=1013014 since the segfault in php-fpm triggers the NMI with abrtd.

The probability for a hardware issue is pretty low here, since this happens on two different physical machines.
Comment 3 RHEL Product and Program Management 2013-10-13 21:53:31 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 6 Hubert Kario 2014-06-11 11:39:41 EDT
Created attachment 907695 [details]
modified reproducer
Comment 7 Hubert Kario 2014-06-11 11:47:00 EDT
Christian unfortunately I'm unable to reproduce this issue.

I obviously can't use the reproducer from CentOS bug tracker as I don't have request.xml, pem_AICCq3 or key_10IoMb.

I've modified the reproducer (see attachment 907695 [details]) to work with just openssl s_server working in -Verify mode with openssl generated certificates/keys.
As far as I can see from the stacktrace, those changes shouldn't affect the issue, but please double check.

Also, the current version of NSS is 3.16.1-2, could you please try reproducing it with newer NSS?
Comment 8 Christian Becker 2014-07-01 15:20:11 EDT

unfortunately we had to do a quick fix and compiled libcurl against OpenSSL because we dealt with a very important system.

We suspect a broken SSL Certificate at one of the endpoints we're connecting here.

We also did some tests in the last couple of months, but had no chance in reproducing it again. It might be even harder since we have no copy of the original SSL Certificate which caused this bug and their server admins weren't any help.

I guess the only chance to fix this is the information from the stacktrace, but reproducing it is very difficult. 

Since i guess i can't help you any further with solving this, it's up to you to decide wether this thicket should be closed or not.

Note You need to log in before you can comment on or make changes to this bug.