Bug 1676842 - httpd, php-fpm ---> cgid daemon process died, restarting
Summary: httpd, php-fpm ---> cgid daemon process died, restarting
Status: CLOSED DUPLICATE of bug 1680481
Alias: None
Product: Fedora
Classification: Fedora
Component: httpd   
(Show other bugs)
Version: 29
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Luboš Uhliarik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-13 11:48 UTC by Claudiu
Modified: 2019-02-28 16:33 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2019-02-28 16:33:54 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description Claudiu 2019-02-13 11:48:31 UTC
I'm using latest httpd packages with mpm-event and cgid_module, php-fpm

[root@serv coredump]# rpm -qa |grep httpd
httpd-devel-2.4.38-2.fc29.x86_64
httpd-debuginfo-2.4.38-2.fc29.x86_64
fedora-logos-httpd-28.0.3-2.fc29.noarch
httpd-filesystem-2.4.38-2.fc29.noarch
httpd-debugsource-2.4.34-3.fc28.x86_64
httpd-tools-2.4.38-2.fc29.x86_64
httpd-2.4.38-2.fc29.x86_64

This is from error_log, where I have many and many lines:
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:03.601852 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22194 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:03.601984 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22196 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:03.602117 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22198 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:03.602160 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:04.606586 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22204 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:04.606746 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22206 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:04.606873 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22208 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:04.606985 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22210 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:04.607100 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22212 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:04.607125 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:05.613154 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22216 exit signal Aborted (6), possible coredump in /etc/httpd
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:05.613241 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting
*** stack smashing detected ***: <unknown> terminated
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:06.619108 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22218 exit signal Aborted (6), possible coredump in /etc/httpd
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:06.619285 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22220 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:06.619327 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting
*** stack smashing detected ***: <unknown> terminated
*** stack smashing detected ***: <unknown> terminated
*** stack smashing detected ***: <unknown> terminated
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:07.629436 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22222 exit signal Aborted (6), possible coredump in /etc/httpd
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:07.629559 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22224 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:07.629693 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22226 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:07.629737 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting
*** stack smashing detected ***: <unknown> terminated
[Wed Feb 13 12:44:08.633775 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22228 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:08.633913 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22230 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:08.634031 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22232 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:08.634103 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22234 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:08.634245 2019] [core:notice] [pid 528:tid 140585053079808] AH00051: child pid 22236 exit signal Aborted (6), possible coredump in /etc/httpd
[Wed Feb 13 12:44:08.634261 2019] [cgid:error] [pid 528:tid 140585053079808] AH01239: cgid daemon process died, restarting



And a backtrace from a coredump

Core was generated by `/usr/sbin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007feaa115d6b0 in ?? ()

(gdb) bt
#0  0x00007feaa115d6b0 in ?? ()
#1  0x00007feaa4a7a358 in __run_fork_handlers () from /lib64/libc.so.6
#2  0x00007feaa4a3896e in fork () from /lib64/libc.so.6
#3  0x00007feaa3e6a27c in cgid_start (p=0x55e3b71b5a78, main_server=0x55e3b71ded20, procnew=0x55e3b71dba18) at mod_cgid.c:874
#4  0x00007feaa3e6a467 in cgid_init (p=0x55e3b71b5a78, plog=<optimized out>, ptemp=<optimized out>, main_server=0x55e3b71ded20) at mod_cgid.c:943
#5  0x000055e3b63ff453 in ap_run_post_config (pconf=0x55e3b71b5a78, plog=0x55e3b71e2c98, ptemp=0x55e3b7205b58, s=0x55e3b71ded20) at config.c:103
#6  0x000055e3b63da29f in main (argc=<optimized out>, argv=<optimized out>) at main.c:807

Comment 1 Claudiu 2019-02-18 08:18:37 UTC
UPDATE:
This is only happening on httpd reload, never on httpd restart. Any suggestion?

Comment 2 Claudiu 2019-02-18 08:27:56 UTC
I don't find the way to edit the post... So I have to make a new comment.
On every, on absolutely every httpd reload I have these lines in error_log:

[Mon Feb 18 09:46:04.095416 2019] [mpm_event:notice] [pid 34094:tid 140142004291840] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 18 09:46:05.793859 2019] [http2:info] [pid 34094:tid 140142004291840] AH03090: mod_http2 (v1.11.1-git, feats=CHPRIO+SHA256+INVHD+DWINS, nghttp2 1.34.0), initializing...
[Mon Feb 18 09:46:06.189158 2019] [mpm_event:notice] [pid 34094:tid 140142004291840] AH00489: Apache/2.4.38 (Fedora) OpenSSL/1.1.1a PHP/7.2.15 mod_perl/2.0.10 Perl/v5.28.1 configured -- resuming normal opera
[Mon Feb 18 09:46:06.189188 2019] [core:notice] [pid 34094:tid 140142004291840] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Feb 18 09:46:06.189270 2019] [core:notice] [pid 34094:tid 140142004291840] AH00051: child pid 38104 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Mon Feb 18 09:46:06.189289 2019] [cgid:error] [pid 34094:tid 140142004291840] AH01239: cgid daemon process died, restarting


I don't know, maybe it's a different bug, as  I don't have that
*** stack smashing detected ***: <unknown> terminated

on httpd reload.

Please someone take a look. I have to reload new configurations many times a day and this is very annoying.

Comment 3 Joe Orton 2019-02-18 08:35:29 UTC
Can you give us a full list of loaded modules?

httpd -M
rpm -qf /etc/httpd/modules/*.so | sort -u

Comment 4 Claudiu 2019-02-18 13:52:57 UTC
httpd -M
--------
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 brotli_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 lua_module (shared)
 mpm_event_module (shared)
 proxy_module (shared)
 proxy_fcgi_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgid_module (shared)
 perl_module (shared)
 http2_module (shared)
 php7_module (shared)


rpm -qf /etc/httpd/modules/*.so | sort -u
----
httpd-2.4.38-2.fc29.x86_64
mod_http2-1.11.1-1.fc29.x86_64
mod_perl-2.0.10-13.fc29.x86_64
mod_ssl-2.4.38-2.fc29.x86_64
php-7.2.15-1.fc29.remi.x86_64

I'm using multi-php 5.6,7.1,7.2,7.3 from Remi repo as Software Collection. 
After you asked for these modules and run rpm -qf /etc/httpd/modules/*.so | sort -u, I remembered that default php was as well upgraded to remi-repo version.
I have performed a downgrade and php changed from php-7.2.15-1.fc29.remi.x86_64 to php-7.2.15-1.fc29.x86_64. Now on httpd reload I no longer have core dump and error and php version is no longer included in server signature

[Mon Feb 18 14:02:04.737791 2019] [mpm_event:notice] [pid 37090:tid 140232511719680] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 18 14:02:06.390406 2019] [http2:info] [pid 37090:tid 140232511719680] AH03090: mod_http2 (v1.11.1-git, feats=CHPRIO+SHA256+INVHD+DWINS, nghttp2 1.34.0), initializing...
[Mon Feb 18 14:02:06.719067 2019] [mpm_event:notice] [pid 37090:tid 140232511719680] AH00489: Apache/2.4.38 (Fedora) OpenSSL/1.1.1a mod_perl/2.0.10 Perl/v5.28.1 configured -- resuming normal operations
[Mon Feb 18 14:02:06.719100 2019] [core:notice] [pid 37090:tid 140232511719680] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

That's weird as the package from Fedora repo was compiled by Remi too.
This was a big issue, and luckily now I found what is going on. Maybe Remi will be able to identify where the issue is. 

But, I don't know what is the cause for that first bug
*** stack smashing detected ***: <unknown> terminated

because these seems to be 2 different bugs. However From February 13 I didn't see other similar error in logs, so maybe it was just something isolated.

Comment 5 Remi Collet 2019-02-18 15:17:29 UTC
> I have performed a downgrade and php changed from php-7.2.15-1.fc29.remi.x86_64 to php-7.2.15-1.fc29.x86_64. Now on httpd reload I no longer have core dump and error and php version is no longer included in server signature

PHP version in server signature means mod_php is loaded, which is not expected, and indeed is not supported, as segfault are expected.

Comment 6 Claudiu 2019-02-18 15:35:05 UTC
Indeed, I have commented out all lines in /etc/httpd/conf.modules.d/15-php.conf to disable mod_php. I no longer have those errors.
Most likely my issues are coming from mod_php loaded by default and this must be disabled, as php-fpm is now used by default.

Comment 7 Claudiu 2019-02-18 17:18:50 UTC
I have upgraded gradually (not a clean reinstall) from earlier versions until Fedora 29.
I'm not using mod_php for years and at some point I have changed something inside
/etc/httpd/conf.modules.d/15-php.conf
then all possible updates have been renamed to 15-php.conf.rpmsave and deleted by me without checking. By mistake, I have not noticed that mod_php is still enabled and this was the cause of my recent issues.

Probably, the file must be dropped completely to avoid similar issues. 
15-php.conf file is completely useless these days.

Comment 8 Joe Orton 2019-02-28 16:33:54 UTC
This is almost certainly a dupe - see bug 1680481 comment 15 for a scratch build which you can use.

*** This bug has been marked as a duplicate of bug 1680481 ***


Note You need to log in before you can comment on or make changes to this bug.