Bug 1309665

Summary:	php54-php-fpm segfaults after glibc-2.12-1.166.el6_7.7 upgrade
Product:	Red Hat Enterprise Linux 6	Reporter:	Justin Albstmeijer <justin>
Component:	glibc	Assignee:	Carlos O'Donell <codonell>
Status:	CLOSED NOTABUG	QA Contact:	qe-baseos-tools-bugs
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	6.7	CC:	ashankar, fweimer, justin, mnewsome, pfrankli
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-18 15:17:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Albstmeijer 2016-02-18 12:03:46 UTC

Description of problem:

After upgrading to glibc-2.12-1.166.el6_7.7 php54-php-fpm started randomly generating http 500 errors.
In the logs we see segfaults.

Downgrading to glibc-2.12-1.166.el6_7.3 fixes this problem.

Further investigation shows the segfaults only occur with chroot'ed php-fpm sites.

Version-Release number of selected component (if applicable):

glibc-2.12-1.166.el6_7.7.x86_64.rpm
glibc-common-2.12-1.166.el6_7.7.x86_64.rpm

How reproducible:

Steps to Reproduce:

1. upgrade to glibc-2.12-1.166.el6_7.7
2. restart php-fpm
3. visit site  and see randmon http 500 errors and check /opt/rh/php54/root/var/log/php-fpm/error.log and see SIGSEGV errors

Actual results:

[18-Feb-2016 11:15:11] WARNING: [pool web567] child 16144 exited on signal 11 (SIGSEGV) after 289.643337 seconds from start
[18-Feb-2016 11:15:11] NOTICE: [pool web567] child 22916 started
[18-Feb-2016 11:17:14] WARNING: [pool web567] child 22916 exited on signal 11 (SIGSEGV) after 123.002904 seconds from start
[18-Feb-2016 11:17:14] NOTICE: [pool web567] child 25614 started


Expected results:

No http 500 and SIGSEGV errors

Additional info:

two ways found till now to workaround it;

- downgrade to glibc-2.12-1.166.el6_7.3
- do not use "chroot =" in website fpm config

Comment 1 Florian Weimer 2016-02-18 12:19:36 UTC

(In reply to Justin Albstmeijer from comment #0)
> - do not use "chroot =" in website fpm config

Did you upgrade the chroot environment after upgrading glibc?

Comment 3 Justin Albstmeijer 2016-02-18 12:34:26 UTC

> Did you upgrade the chroot environment after upgrading glibc?

Yes, using 

jk_update --hardlink --jail=/xxx/web567

I did not do this again after downgrading glibc, but is still solves the issue.

Comment 4 Florian Weimer 2016-02-18 13:39:12 UTC

(In reply to Justin Albstmeijer from comment #3)
> > Did you upgrade the chroot environment after upgrading glibc?
> 
> Yes, using 
> 
> jk_update --hardlink --jail=/xxx/web567
> 
> I did not do this again after downgrading glibc, but is still solves the
> issue.

The most likely explanation is that this update script does not work correctly, and it did not update nss_dns or libresolv in the chroot.  You can verify this by comparing sha256sums before and after the upgrade, both against the previous chroot contents and the file system tree of the main system outside the chroot.

One possible explanation is that libresolv.so.2 was not updated.  It is not listed in the jk_update upstream default configuration, and I don't know if it is automatically recognized at as a dependency of libnss_dns.so.2.

Comment 5 Justin Albstmeijer 2016-02-18 15:16:45 UTC

Hi,

You are right.

The jail was partly updated, seems lib64 got skipped.

Till now previous glibc updates did not break this, but good to find out the updates where done partly.

Now using:

jk_update  --hardlink --jail=$x $x/usr $x/lib $x/lib64 $x/bin

Al gets updated and we don't see the segfault any more.

Thank you for pointing me in the right direction.

Gr, J

Comment 6 Florian Weimer 2016-02-18 15:17:56 UTC

(In reply to Justin Albstmeijer from comment #5)
> The jail was partly updated, seems lib64 got skipped.

Thanks for confirming that this was not an issue with our packages.