Red Hat Bugzilla – Bug 1013006
php-fpm Segfault in libnss3.so
Last modified: 2014-07-01 15:31:15 EDT
Description of problem:
php-fpm Segfaults in libnss3.so on curl SSL connection with client certificate
Version-Release number of selected component (if applicable):
Version : 3.14.3 Vendor: Red Hat, Inc.
Release : 4.el6_4 Build Date: Mon Apr 29 18:02:58 2013
Version : 7.19.7 Vendor: Red Hat, Inc.
Release : 37.el6_4 Build Date: Fri Jun 14 19:17:24 2013
php-fpm: segfault at 0 ip 00000038fee2ce52 sp 00007fff3f520080 error 4 in libnss3.so[38fee00000+135000]
#0 PK11_GetPrivateModulusLen (key=<value optimized out>) at pk11akey.c:815
#1 0x0000003cb2c48f25 in PK11_SignatureLen (key=0x202c2b0) at pk11obj.c:528
#2 0x0000003cb440f9c9 in ssl3_SignHashes (hash=0x7fff07a3e300, key=0x202c2b0, buf=0x7fff07a3e2e0, isTLS=1) at ssl3con.c:811
#3 0x0000003cb44130b9 in ssl3_SendCertificateVerify (ss=0x241f3e0) at ssl3con.c:5349
#4 ssl3_SendClientSecondRound (ss=0x241f3e0) at ssl3con.c:6321
#5 0x0000003cb4414792 in ssl3_HandleServerHelloDone (ss=0x241f3e0, b=0x240a9a4 "\001\001", length=0) at ssl3con.c:6242
#6 ssl3_HandleHandshakeMessage (ss=0x241f3e0, b=0x240a9a4 "\001\001", length=0) at ssl3con.c:9491
#7 0x0000003cb4415efe in ssl3_HandleHandshake (ss=<value optimized out>, cText=<value optimized out>, databuf=0x241f778) at ssl3con.c:9592
#8 ssl3_HandleRecord (ss=<value optimized out>, cText=<value optimized out>, databuf=0x241f778) at ssl3con.c:10230
#9 0x0000003cb4416d77 in ssl3_GatherCompleteHandshake (ss=0x241f3e0, flags=0) at ssl3gthr.c:361
#10 0x0000003cb4419522 in ssl_GatherRecord1stHandshake (ss=0x241f3e0) at sslcon.c:1227
#11 0x0000003cb441f7d5 in ssl_Do1stHandshake (ss=0x241f3e0) at sslsecur.c:120
#12 0x0000003cb442100f in SSL_ForceHandshake (fd=<value optimized out>) at sslsecur.c:390
#13 0x0000003cb7c410e5 in Curl_nss_connect () from /usr/lib64/libcurl.so.4
#14 0x0000003cb7c38482 in Curl_ssl_connect () from /usr/lib64/libcurl.so.4
#15 0x0000003cb7c16ecb in Curl_http_connect () from /usr/lib64/libcurl.so.4
#16 0x0000003cb7c1d682 in Curl_protocol_connect () from /usr/lib64/libcurl.so.4
#17 0x0000003cb7c23b3c in Curl_connect () from /usr/lib64/libcurl.so.4
#18 0x0000003cb7c2bbb0 in Curl_perform () from /usr/lib64/libcurl.so.4
#19 0x00007fe751ceada3 in ?? () from /usr/lib64/php/modules/curl.so
#20 0x000000000069620b in ?? ()
#21 0x000000000066deb0 in execute ()
#22 0x00000000006474dd in zend_execute_scripts ()
#23 0x00000000005f41c8 in php_execute_script ()
#24 0x00000000006da2a8 in ?? ()
#25 0x0000003cafc1ecdd in __libc_start_main (main=0x6d9210, argc=1, ubp_av=0x7fff07a43538, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff07a43528)
#26 0x00000000004226b9 in _start ()
This issue is not triggered with every request.
If we do our test case on the command line it always works fine. As soon as we switch to php-fpm we have about a 50/50 chance to hit this issue.
So we think there could be a memory corruption caused by nss which is also causing https://bugzilla.redhat.com/show_bug.cgi?id=1013014 since the segfault in php-fpm triggers the NMI with abrtd.
The probability for a hardware issue is pretty low here, since this happens on two different physical machines.
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Created attachment 907695 [details]
Christian unfortunately I'm unable to reproduce this issue.
I obviously can't use the reproducer from CentOS bug tracker as I don't have request.xml, pem_AICCq3 or key_10IoMb.
I've modified the reproducer (see attachment 907695 [details]) to work with just openssl s_server working in -Verify mode with openssl generated certificates/keys.
As far as I can see from the stacktrace, those changes shouldn't affect the issue, but please double check.
Also, the current version of NSS is 3.16.1-2, could you please try reproducing it with newer NSS?
unfortunately we had to do a quick fix and compiled libcurl against OpenSSL because we dealt with a very important system.
We suspect a broken SSL Certificate at one of the endpoints we're connecting here.
We also did some tests in the last couple of months, but had no chance in reproducing it again. It might be even harder since we have no copy of the original SSL Certificate which caused this bug and their server admins weren't any help.
I guess the only chance to fix this is the information from the stacktrace, but reproducing it is very difficult.
Since i guess i can't help you any further with solving this, it's up to you to decide wether this thicket should be closed or not.