Bug 1659768 - httpd: endless crash loop
Summary: httpd: endless crash loop
Keywords:
Status: CLOSED DUPLICATE of bug 1680481
Alias: None
Product: Fedora
Classification: Fedora
Component: php
Version: 29
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Remi Collet
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-16 09:40 UTC by redhat
Modified: 2019-02-28 09:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-28 09:17:58 UTC


Attachments (Terms of Use)
output of httpd -M (2.45 KB, text/plain)
2019-02-19 09:03 UTC, redhat
no flags Details

Description redhat 2018-12-16 09:40:29 UTC
Description of problem:
Every week in the night from saturday to sunday httpd runs in an endless crash loop. It is restarted by systemd and then crashes again immediately. This happens every second.

Version-Release number of selected component (if applicable):
httpd-2.4.37-5.fc29.x86_64

How reproducible:
Every week.

Steps to Reproduce:
1. don't know, happens automatically
2.
3.

Actual results:
Sun 2018-12-16 03:34:05 CET    4945     0     0  11 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:07 CET    4944     0     0  11 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    4980     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    5158     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    4985     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    4992     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    4977     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    4998     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:09 CET    5155     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:11 CET    5413     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:11 CET    5535     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:11 CET    5538     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:11 CET    5415     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:11 CET    5699     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:12 CET    5698     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:13 CET    5949     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:13 CET    5828     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:13 CET    5827     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:14 CET    5950     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:14 CET    6084     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:15 CET    6086     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:15 CET    6085     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:17 CET    6420     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:19 CET    6275     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:19 CET    6418     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:19 CET    6425     0     0   4 missing   /usr/sbin/httpd
Sun 2018-12-16 03:34:20 CET    6798     0     0   4 missing   /usr/sbin/httpd
....

(gdb) bt
#0  0x00007fc9468286b2 in ?? () from /usr/lib64/php/modules/intl.so
#1  0x000055bfa4cdcca0 in ?? ()
#2  0x00007fc94935dd98 in apr_proc_fork () from /lib64/libapr-1.so.0
#3  0x00007fc947fe46b9 in ?? () from /etc/httpd/modules/mod_wsgi.so
#4  0x00007fc947fe7a34 in ?? () from /etc/httpd/modules/mod_wsgi.so
#5  0x00007fc949355d5f in apr_proc_other_child_alert () from /lib64/libapr-1.so.0
#6  0x00007fc9489ea816 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so
#7  0x000055bfa2d23bde in ap_run_mpm ()
#8  0x000055bfa2d1c2d3 in main ()

Expected results:
No crash at all

Additional info:
It loops until I run "systemctl restart httpd.service".

Comment 1 Joe Orton 2019-01-18 12:55:56 UTC
Can you double-check this system has updated to httpd-2.4.37-5?  There was a fix for an issue which would manifest like this in -5.

Comment 2 redhat 2019-01-18 18:19:40 UTC
Yes, it's httpd-2.4.37-5 indeed.

Comment 3 redhat 2019-01-20 12:27:17 UTC
The bug can be triggered by

    systemctl reload httpd.service

which is executed regularly by logrotate, for example.

Whereas an 

    systemctl restart httpd.service 

does not trigger the error. Hope this helps.

Comment 4 Luboš Uhliarik ✈ 2019-01-24 15:28:52 UTC
Hi Erik,

thanks for your report! Could you please share with us, what's the value of your SSLProtocol option, if you are using mod_ssl?

Comment 5 redhat 2019-01-25 19:18:46 UTC
mod_ssl is not used in my setup.

Comment 6 Joe Orton 2019-01-26 06:57:36 UTC
Do you mean it's not configured or the package is not installed?

Comment 7 redhat 2019-01-26 17:32:16 UTC
The package is not installed.

Comment 8 Joe Orton 2019-02-18 15:22:39 UTC
Please provide output from root running both:

httpd -M
rpm -qf /etc/httpd/modules/*.so | sort -u

wondering if this has the same root-cause as bug 1676842 which was a non-Fedora php package.

Comment 9 redhat 2019-02-19 09:03:02 UTC
Created attachment 1536268 [details]
output of httpd -M

The output requested.

Comment 10 redhat 2019-02-19 09:03:19 UTC
# rpm -qf /etc/httpd/modules/*.so | sort -u
httpd-2.4.38-2.fc29.x86_64
mod_http2-1.11.1-1.fc29.x86_64
php-7.2.14-1.fc29.x86_64
python2-mod_wsgi-4.6.4-2.fc29.x86_64

Comment 11 Joe Orton 2019-02-19 09:10:07 UTC
Have you modified /etc/httpd/conf.modules.d/15-php.conf or is it stock?  Please provide the config.

Remi, this looks like bug 1676842 but with prefork MPM & mod_php loaded.

Comment 12 redhat 2019-02-19 10:56:18 UTC
No, /etc/httpd/conf.modules.d/15-php.conf is stock.

# less 15-php.conf
#
# PHP is an HTML-embedded scripting language which attempts to make it
# easy for developers to write dynamically generated webpages.
#

# Cannot load both php5 and php7 modules
<IfModule !mod_php5.c>
  <IfModule prefork.c>
    LoadModule php7_module modules/libphp7.so
  </IfModule>
</IfModule>


<IfModule !mod_php5.c>
  <IfModule !prefork.c>
    # ZTS module is not supported, so FPM is preferred
    LoadModule php7_module modules/libphp7-zts.so
  </IfModule>
</IfModule>

Comment 13 Trevor Cordes 2019-02-26 06:09:56 UTC
Are you sure it's the parent apache that is crashing and systemd is restarting it... or... is it the apache children crashing and the parent apache restarting them?

What does journalctl say (assuming it's handling coredumps)?

What does apache error.log show when the endless crashes are occurring?

What does:
systemctl status httpd
say is the start time in the Active: line?  Is the start time static or increasing (i.e. parent really is dying)?

If it's the children dying, not the parent, you may have the same bug as me bz#1680481

If parent is dying, it's a different bug.

Comment 14 Joe Orton 2019-02-28 07:58:48 UTC
Cross-checking with the other bug - can you confirm whether you have php-imap installed?

Comment 15 redhat 2019-02-28 08:46:40 UTC
Yes, php-imap is installed.

You might want to have a closer look at the stack backtrace I provied. This helps more than guessing into blue.

Comment 16 Joe Orton 2019-02-28 09:16:22 UTC
Thanks.  Yeah, the backtraces all show the crash is in a bogus pointer in an pthread_atfork handler, the actual memory address may show up in some php .so but it's a stale pointer from an unloaded DSO, most likely.  I'm working on it.

Comment 17 Joe Orton 2019-02-28 09:17:58 UTC
Let's just track this once, it's clearly a common bug at this ponit.

*** This bug has been marked as a duplicate of bug 1680481 ***


Note You need to log in before you can comment on or make changes to this bug.