Bug 1149991

Summary: mod_php crashes httpd while reloading httpd if mod_gnutls is loaded also
Product: Red Hat Enterprise Linux 7 Reporter: Django <django>
Component: phpAssignee: Remi Collet <rcollet>
Status: CLOSED ERRATA QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: jorton, nmavrogi, rcollet
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-05 12:40:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1110700, 1191021    

Description Django 2014-10-07 07:24:34 UTC
Description of problem:

If I install httpd, php and mod_gnutls, httpd dies while reloading httpd.

Version-Release number of selected component (if applicable):
httpd-2.4.6-18.el7.centos.x86_64 (from CentOS-Base-Repository)
mod_gnutls-0.5.10-13.el7.x86_64  (from EPEL-Repository)
php-5.4.16-23.el7_0.x86_64       (from CentOS-Base-Repository)


How reproducible:
I have mad many tests last day with different CentOS 7 hosts. 

Steps to Reproduce:
1. Install httpd-2.4.6-18.el7.centos.x86_64 and php-5.4.16-23.el7_0.x86_64 from CentOS-Base-Repo:
 Step 1a) Start Apache 2.4:     # systemctl start httpd.service
 Step 2b) Ask status of httpd:  # systemctl status httpd.service
 Step 2c) Reload apache-config: # systemctl reload httpd.service

I can do step "2c" many times, there's no problem- everything works as I expected!

2. Install mod_gnutls-0.5.10-13.el7.x86_64 (from EPEL-Repository)

3. Reload apache-daemon: 
 Step 3a) Restart Apache 2.4:   # systemctl restart httpd.service
 Step 3b) Ask status of httpd:  # systemctl status httpd.service

# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: active (running) since Thu 2014-09-18 10:36:53 CEST; 4s ago
  Process: 6812 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 6808 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 6843 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─6843 /usr/sbin/httpd -DFOREGROUND
           ├─6845 /usr/sbin/httpd -DFOREGROUND
           ├─6846 /usr/sbin/httpd -DFOREGROUND
           ├─6847 /usr/sbin/httpd -DFOREGROUND
           ├─6848 /usr/sbin/httpd -DFOREGROUND
           └─6849 /usr/sbin/httpd -DFOREGROUND

Sep 18 10:36:53 vml000097.dmz.example.com systemd[1]: Started The Apache HTTP Server.

 Step 3c) Reload apache-config: # systemctl reload httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: failed (Result: signal) since Thu 2014-09-18 10:38:12 CEST; 18s ago
  Process: 6856 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 6855 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
  Process: 6843 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=killed, signal=SEGV)
 Main PID: 6843 (code=killed, signal=SEGV)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"

Sep 18 10:36:53 vml000097.dmz.example.com systemd[1]: Started The Apache HTTP Server.
Sep 18 10:38:11 vml000097.dmz.example.com systemd[1]: Reloading The Apache HTTP Server.
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: httpd.service: main process exited, code=killed, status=11/SEGV
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: Reload failed for The Apache HTTP Server.
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: Unit httpd.service entered failed state.

# journalctl -xn
-- Logs begin at Wed 2014-09-17 12:47:28 CEST, end at Thu 2014-09-18 10:38:12 CEST. --
Sep 18 10:01:01 vml000097.dmz.example.com run-parts(/etc/cron.hourly)[6823]: starting 0anacron
Sep 18 10:01:01 vml000097.dmz.example.com run-parts(/etc/cron.hourly)[6829]: finished 0anacron
Sep 18 10:01:01 vml000097.dmz.example.com run-parts(/etc/cron.hourly)[6831]: starting 0yum-hourly.cron
Sep 18 10:01:01 vml000097.dmz.example.com run-parts(/etc/cron.hourly)[6835]: finished 0yum-hourly.cron
Sep 18 10:36:53 vml000097.dmz.example.com systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has begun starting up.
Sep 18 10:36:53 vml000097.dmz.example.com systemd[1]: Started The Apache HTTP Server.
-- Subject: Unit httpd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has finished starting up.
-- 
-- The start-up result is done.
Sep 18 10:38:11 vml000097.dmz.example.com systemd[1]: Reloading The Apache HTTP Server.
-- Subject: Unit httpd.service has begun with reloading its configuration
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has begun with reloading its configuration
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: httpd.service: main process exited, code=killed, status=11/SEGV
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: Reload failed for The Apache HTTP Server.
-- Subject: Unit httpd.service has finished reloading its configuration
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has finished reloading its configuration
-- 
-- The result is failed.
Sep 18 10:38:12 vml000097.dmz.example.com systemd[1]: Unit httpd.service entered failed state.


# cat /var/log/httpd/error_log
[Wed Sep 17 14:53:05.706634 2014] [suexec:notice] [pid 4737] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Sep 17 14:53:05.867889 2014] [auth_digest:notice] [pid 4737] AH01757: generating secret for digest authentication ...
[Wed Sep 17 14:53:05.869450 2014] [lbmethod_heartbeat:notice] [pid 4737] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 17 14:53:06.063338 2014] [core:warn] [pid 4737] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Sep 17 14:53:06.098697 2014] [mpm_prefork:notice] [pid 4737] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 mod_gnutls/0.5.10 configured -- resuming normal operations
[Wed Sep 17 14:53:06.098804 2014] [core:notice] [pid 4737] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Sep 17 14:53:23.408399 2014] [mpm_prefork:notice] [pid 4737] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Sep 17 14:53:24.426419 2014] [core:notice] [pid 4737] AH00060: seg fault or similar nasty error detected in the parent process
[Wed Sep 17 14:53:24.520783 2014] [suexec:notice] [pid 4751] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Sep 17 14:53:24.582705 2014] [auth_digest:notice] [pid 4751] AH01757: generating secret for digest authentication ...
[Wed Sep 17 14:53:24.583596 2014] [lbmethod_heartbeat:notice] [pid 4751] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 17 14:53:24.669171 2014] [mpm_prefork:notice] [pid 4751] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 mod_gnutls/0.5.10 configured -- resuming normal operations
[Wed Sep 17 14:53:24.669216 2014] [core:notice] [pid 4751] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Sep 17 14:53:38.628624 2014] [mpm_prefork:notice] [pid 4751] AH00171: Graceful restart requested, doing restart
[Wed Sep 17 14:53:38.649444 2014] [core:notice] [pid 4751] AH00060: seg fault or similar nasty error detected in the parent process 

*** Error in `/usr/sbin/httpd': munmap_chunk(): invalid pointer: 0x00007fa75e624b60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7bc07)[0x7fa75baadc07]
/lib64/libapr-1.so.0(apr_pool_destroy+0x1a7)[0x7fa75c22c2d7]
/etc/httpd/modules/mod_mpm_prefork.so(+0x321e)[0x7fa752ed421e]
/etc/httpd/modules/mod_mpm_prefork.so(+0x325b)[0x7fa752ed425b]
/lib64/libpthread.so.0(+0xf130)[0x7fa75c006130]
/lib64/libc.so.6(+0x7c4f2)[0x7fa75baae4f2]
/lib64/libc.so.6(+0x7d097)[0x7fa75baaf097]
/lib64/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fa75c22b97d]
/etc/httpd/modules/mod_mpm_prefork.so(+0x321e)[0x7fa752ed421e]
/etc/httpd/modules/mod_mpm_prefork.so(+0x36c7)[0x7fa752ed46c7]
/etc/httpd/modules/mod_mpm_prefork.so(+0x3a26)[0x7fa752ed4a26]
/etc/httpd/modules/mod_mpm_prefork.so(+0x3a86)[0x7fa752ed4a86]
/etc/httpd/modules/mod_mpm_prefork.so(+0x4790)[0x7fa752ed5790]
/usr/sbin/httpd(ap_run_mpm+0x4e)[0x7fa75d5580fe]
/usr/sbin/httpd(main+0xa86)[0x7fa75d551726]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa75ba53af5]
/usr/sbin/httpd(+0x1e861)[0x7fa75d551861]
======= Memory map: ========
7fa730000000-7fa730021000 rw-p 00000000 00:00 0
7fa730021000-7fa734000000 ---p 00000000 00:00 0
7fa7360ab000-7fa7360ac000 ---p 00000000 00:00 0
7fa7360ac000-7fa7368ac000 rw-p 00000000 00:00 0
7fa736cb8000-7fa736cbd000 r-xp 00000000 fd:01 16787613                   /usr/lib64/libnss_dns-2.17.so
7fa736cbd000-7fa736ebc000 ---p 00005000 fd:01 16787613                   /usr/lib64/libnss_dns-2.17.so
7fa736ebc000-7fa736ebd000 r--p 00004000 fd:01 16787613                   /usr/lib64/libnss_dns-2.17.so
7fa736ebd000-7fa736ebe000 rw-p 00005000 fd:01 16787613                   /usr/lib64/libnss_dns-2.17.so
7fa73d0ce000-7fa73d1b7000 r-xp 00000000 fd:01 16790814                   /usr/lib64/libstdc++.so.6.0.19
7fa73d1b7000-7fa73d3b6000 ---p 000e9000 fd:01 16790814                   /usr/lib64/libstdc++.so.6.0.19
7fa73d3b6000-7fa73d3be000 r--p 000e8000 fd:01 16790814                   /usr/lib64/libstdc++.so.6.0.19
7fa73d3be000-7fa73d3c0000 rw-p 000f0000 fd:01 16790814                   /usr/lib64/libstdc++.so.6.0.19
7fa73d3c0000-7fa73d3d5000 rw-p 00000000 00:00 0
7fa7400cd000-7fa7400e2000 r-xp 00000000 fd:01 17619692                   /usr/lib64/libgcc_s-4.8.2-20140120.so.1
7fa7400e2000-7fa7402e1000 ---p 00015000 fd:01 17619692                   /usr/lib64/libgcc_s-4.8.2-20140120.so.1
7fa7402e1000-7fa7402e2000 r--p 00014000 fd:01 17619692                   /usr/lib64/libgcc_s-4.8.2-20140120.so.1
7fa7402e2000-7fa7402e3000 rw-p 00015000 fd:01 17619692                   /usr/lib64/libgcc_s-4.8.2-20140120.so.1
7fa745045000-7fa74b56c000 r--p 00000000 fd:01 34260954                   /usr/lib/locale/locale-archive
7fa74b56c000-7fa74b56e000 r-xp 00000000 fd:01 16797722                   /usr/lib64/apr-util-1/apr_ldap-1.so
7fa74b56e000-7fa74b76d000 ---p 00002000 fd:01 16797722                   /usr/lib64/apr-util-1/apr_ldap-1.so
7fa74b76d000-7fa74b76e000 r--p 00001000 fd:01 16797722                   /usr/lib64/apr-util-1/apr_ldap-1.so
7fa74b76e000-7fa74b76f000 rw-p 00002000 fd:01 16797722                   /usr/lib64/apr-util-1/apr_ldap-1.so
7fa74b76f000-7fa74b77a000 r-xp 00000000 fd:01 16787615                   /usr/lib64/libnss_files-2.17.so
7fa74b77a000-7fa74b979000 ---p 0000b000 fd:01 16787615                   /usr/lib64/libnss_files-2.17.so
7fa74b979000-7fa74b97a000 r--p 0000a000 fd:01 16787615                   /usr/lib64/libnss_files-2.17.so
7fa74b97a000-7fa74b97b000 rw-p 0000b000 fd:01 16787615                   /usr/lib64/libnss_files-2.17.so
7fa74b97b000-7fa74b9a1000 r-xp 00000000 fd:01 61336                      /usr/lib64/pkcs11/p11-kit-trust.so
7fa74b9a1000-7fa74bba1000 ---p 00026000 fd:01 61336                      /usr/lib64/pkcs11/p11-kit-trust.so
7fa74bba1000-7fa74bba6000 r--p 00026000 fd:01 61336                      /usr/lib64/pkcs11/p11-kit-trust.so
7fa74bba6000-7fa74bba7000 rw-p 0002b000 fd:01 61336                      /usr/lib64/pkcs11/p11-kit-trust.so
7fa74bba7000-7fa74bbcd000 r-xp 00000000 fd:01 16968608                   /usr/lib64/libhogweed.so.2.5
7fa74bbcd000-7fa74bdcc000 ---p 00026000 fd:01 16968608                   /usr/lib64/libhogweed.so.2.5
7fa74bdcc000-7fa74bdcd000 r--p 00025000 fd:01 16968608                   /usr/lib64/libhogweed.so.2.5
7fa74bdcd000-7fa74bdce000 rw-p 00026000 fd:01 16968608                   /usr/lib64/libhogweed.so.2.5
7fa74bdce000-7fa74bdfd000 r-xp 00000000 fd:01 16968610                   /usr/lib64/libnettle.so.4.7
7fa74bdfd000-7fa74bffd000 ---p 0002f000 fd:01 16968610                   /usr/lib64/libnettle.so.4.7
7fa74bffd000-7fa74bffe000 r--p 0002f000 fd:01 16968610                   /usr/lib64/libnettle.so.4.7
7fa74bffe000-7fa74bfff000 rw-p 00030000 fd:01 16968610                   /usr/lib64/libnettle.so.4.7
7fa74bfff000-7fa74c011000 r-xp 00000000 fd:01 16820519                   /usr/lib64/libtasn1.so.6.1.1
7fa74c011000-7fa74c211000 ---p 00012000 fd:01 16820519                   /usr/lib64/libtasn1.so.6.1.1
7fa74c211000-7fa74c212000 r--p 00012000 fd:01 16820519                   /usr/lib64/libtasn1.so.6.1.1
7fa74c212000-7fa74c213000 rw-p 00013000 fd:01 16820519                   /usr/lib64/libtasn1.so.6.1.1
7fa74c213000-7fa74c232000 r-xp 00000000 fd:01 16820531                   /usr/lib64/libp11-kit.so.0.0.0
7fa74c232000-7fa74c432000 ---p 0001f000 fd:01 16820531                   /usr/lib64/libp11-kit.so.0.0.0
7fa74c432000-7fa74c434000 r--p 0001f000 fd:01 16820531                   /usr/lib64/libp11-kit.so.0.0.0
7fa74c434000-7fa74c435000 rw-p 00021000 fd:01 16820531                   /usr/lib64/libp11-kit.so.0.0.0
7fa74c435000-7fa74c53a000 r-xp 00000000 fd:01 16968622                   /usr/lib64/libgnutls.so.28.20.4
7fa74c53a000-7fa74c73a000 ---p 00105000 fd:01 16968622                   /usr/lib64/libgnutls.so.28.20.4
7fa74c73a000-7fa74c742000 r--p 00105000 fd:01 16968622                   /usr/lib64/libgnutls.so.28.20.4
7fa74c742000-7fa74c744000 rw-p 0010d000 fd:01 16968622                   /usr/lib64/libgnutls.so.28.20.4
7fa74c744000-7fa74c745000 rw-p 00000000 00:00 0
7fa74c745000-7fa74c753000 r-xp 00000000 fd:01 18507008                   /usr/lib64/httpd/modules/mod_gnutls.so
7fa74c753000-7fa74c953000 ---p 0000e000 fd:01 18507008                   /usr/lib64/httpd/modules/mod_gnutls.so
7fa74c953000-7fa74c954000 r--p 0000e000 fd:01 18507008                   /usr/lib64/httpd/modules/mod_gnutls.so
7fa74c954000-7fa74c955000 rw-p 0000f000 fd:01 18507008                   /usr/lib64/httpd/modules/mod_gnutls.so
7fa74c955000-7fa74c958000 r-xp 00000000 fd:01 16883017                   /usr/lib64/libkeyutils.so.1.5
7fa74c958000-7fa74cb57000 ---p 00003000 fd:01 16883017                   /usr/lib64/libkeyutils.so.1.5Segmentation fault (core dumped) 

Actual results:
crashing httpd after step "3c" reloading apache-config with "systemctl reload httpd.service":
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

Expected results:
not crashing! ;)

Comment 1 Django 2014-10-07 07:26:21 UTC
You'll find more informations in bugreport "Bug 1143908 - mod_gnutls crashes httpd if mod_php is loaded (CentOS 7)" here: https://bugzilla.redhat.com/show_bug.cgi?id=1143908

Comment 3 Joe Orton 2014-10-07 10:31:26 UTC
Thanks for the report.  If this is an issue for you on productions systems, please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization to assure a timely resolution.  http://www.redhat.com/en/services/support

Comment 4 Django 2014-10-07 13:58:12 UTC
It's a evaluationsystem, based on CentOS7 - I've no customer-id on http://www.redhat.com/en/services/support so I must hope someone will fix this bug in the php-packages on RHEL7.

Comment 5 Remi Collet 2014-10-10 08:58:25 UTC
Seems very probably related to https://bugs.php.net/63595

Comment 6 Django 2014-10-10 11:11:20 UTC
As I can say, yes it is. 
Nikos Mavrogiannopoulos said so:
https://bugzilla.redhat.com/show_bug.cgi?id=1143908#c8

Comment 7 Nikos Mavrogiannopoulos 2014-10-13 14:01:53 UTC
*** Bug 1152162 has been marked as a duplicate of this bug. ***

Comment 8 Remi Collet 2014-10-14 06:20:21 UTC
Notice, as libcurl doesnt use gnutls (but only nss), php-curl is not affected by this issue.

So only (kwown as) affected : mod_gnutls and freetds (php-odbc or php-mssql)

Possible mitigation: build gmp as shared, as in php 5.5+, so it can be disabled (not a very common extension)

A heavy patch have been attached to upstream bug report, waiting for php internals feedback.

Comment 9 Remi Collet 2014-10-14 06:30:55 UTC
php-mapi (zarafa in EPEL) is also affected, see #1075617

Comment 12 Joe Orton 2016-01-05 12:40:11 UTC
Should be fixed with:

https://rhn.redhat.com/errata/RHSA-2015-1135.html

Comment 13 Nikos Mavrogiannopoulos 2017-08-14 08:53:41 UTC
*** Bug 1143908 has been marked as a duplicate of this bug. ***