httpd segfaults on mod_perl ---- Versions: Linux 3.0.1-3.fc16.i686.PAE Name : httpd Arch : i686 Version : 2.2.19 Release : 4.fc16 Name : mod_perl Arch : i686 Version : 2.0.5 Release : 5.fc16 ---- How reproducible: Clean webserver group install Steps to Reproduce: 1. service httpd start Actual messages log results: Aug 15 14:43:35 f16 systemd[1]: Reloading. Aug 15 14:44:14 f16 kernel: [ 1065.458572] httpd[1629]: segfault at d ip 0b829f5 sp bfcc7000 error 4 in mod_perl.so[b66000+33000] Aug 15 14:44:14 f16 abrt[1630]: saved core dump of pid 1629 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2011-08-15-14:44:14-1629.new/coredump (3457024 bytes) Aug 15 14:44:14 f16 abrtd: Directory 'ccpp-2011-08-15-14:44:14-1629' creation detected Aug 15 14:44:14 f16 systemd[1]: httpd.service: control process exited, code=dumped status=11 Aug 15 14:44:14 f16 systemd[1]: Unit httpd.service entered failed state. Aug 15 14:44:18 f16 abrtd: New dump directory /var/spool/abrt/ccpp-2011-08-15-14:44:14-1629, processing Expected results: httpd starts OK Additional info: abrt core dump files available upon request
Commenting out the mod_perl module in the /etc/httpd/conf.d/perl.conf #LoadModule perl_module modules/mod_perl.so allows httpd to start normally
Created attachment 518490 [details] httpd segfaults on mod_perl abrt files from segfault
Maybe similar to https://bugs.mageia.org/show_bug.cgi?id=2439
Cannot reproduce on x86_64 with httpd-2.2.21-1.fc16.x86_64 and mod_perl-2.0.5-5.fc16.x86_64. Maybe it's platform specific.
Can somebody confirm the diagnosis of Shlomi Fish in that referenced bug report: "" (gdb) print sizeof(my_perl[0]) $5 = 1752 (gdb) print sizeof(PL_curinterp[0]) $6 = 1768 (gdb) "" I know of one possible cause for this, the APR apr_ino_t definition can change size if Perl is exporting -D_FILE_OFFSET_BITS=64.
(that would affect only 32-bit builds)
Oh, no, we already work around that in apr. I have no specific idea then; we have seen things like this in the past from _FILE_OFFSET_BITS, anyway.
I don't have any i686 machine available to test it, but the archlinux patch looks like it should fix this issue: # Patch from openSUSE to fix segfaults on i686 (FS#25008): # avoid a conflict between apache's LARGEFILE64_SOURCE and perl's # FILE_OFFSET_BITS=64 because the conflict isn't real and causes the perl # structures to be invalid if only the apache flags are used http://projects.archlinux.org/svntogit/packages.git/plain/trunk/nolfs.patch?h=packages/mod_perl
Patch (better said "workaround") from previous comment fixed the crash for me. It crashed because of bad pointer dereference (bt attached). sizeof(my_perl[0]) was the same as sizeof(PL_curinterp[0]). I presume httpd is compiled with large files support, but definitions for that are not mentioned in apxs: # apxs -q CFLAGS -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables Perl is compiled with: -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 mod_perl then thinks APR is is not built with largefile support and removes "-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64". I'm not sure if this is right or not (depends on if apxs is right and APR is not compiled with large files support or not), I was just trying to debug it little bit.
Created attachment 530050 [details] bt
*** Bug 752178 has been marked as a duplicate of this bug. ***
I can't get mod_perl running on release of Fedora 16 on 686 platform with the same symptoms. The workaround from archlinux doesn't help me. any other suggestions? -- vlad
*** Bug 752294 has been marked as a duplicate of this bug. ***
Vlad, you have tried rebuilding mod_perl with the patch I sent in Comment 8 and it didn't fix the problem for you, right?
Jan, no, but I manually edited that file on the system with installed mod_perl /usr/lib/perl5/vendor_perl/Apache2/Build.pm does it influence on mod_perl building process as well? I'll give it a try later today then. -- vlad
ok, I rebuilt mod_perl via rpmbuild with patch archlinux applied. Can starting httpd now with mod_perl enabled. -- vlad
(In reply to comment #16) > ok, I rebuilt mod_perl via rpmbuild with patch archlinux applied. > Can starting httpd now with mod_perl enabled. > -- vlad Could you please give me some instruction/direction on how to rpmbuild a patched mod_perl and I'll test it on my system. Thanks!
try something like that following yumdownloader --source mod_perl rpm -ivh mod_perl-*.src.rpm rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec manually apply patch from above (comment 8) rpmbuild -bc ~/rpmbuild/SPECS/mod_perl.spec rpmbuild -bi ~/rpmbuild/SPECS/mod_perl.spec rpmbuild -bb ~/rpmbuild/SPECS/mod_perl.spec remove (if already installed mod_perl, mod_perl-devel) rpm -i ~/rpmbuild/RPMS/i386/mod_perl-*.rpm
(In reply to comment #18) > try something like that following > yumdownloader --source mod_perl > rpm -ivh mod_perl-*.src.rpm > rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec > manually apply patch from above (comment 8) > rpmbuild -bc ~/rpmbuild/SPECS/mod_perl.spec > rpmbuild -bi ~/rpmbuild/SPECS/mod_perl.spec > rpmbuild -bb ~/rpmbuild/SPECS/mod_perl.spec > remove (if already installed mod_perl, mod_perl-devel) > rpm -i ~/rpmbuild/RPMS/i386/mod_perl-*.rpm I get to step #3 and throw an error (see below). ---- screen output --- [root@f16 ~]# yumdownloader --source mod_perl Loaded plugins: langpacks, presto, refresh-packagekit Enabling updates-testing-source repository Enabling fedora-source repository mod_perl-2.0.5-5.fc16.src.rpm | 3.6 MB 00:12 [root@f16 ~]# rpm -ivh mod_perl-*.src.rpm 1:mod_perl warning: user mockbuild does not exist - using root warning: group mockbuild does not exist - using root warning: user mockbuild does not exist - using root warning: group mockbuild does not exist - using root warning: user mockbuild does not exist - using root warning: group mockbuild does not exist - using root ########################################### [100%] warning: user mockbuild does not exist - using root warning: group mockbuild does not exist - using root warning: user mockbuild does not exist - using root warning: group mockbuild does not exist - using root [root@f16 ~]# yumdownloader --source mod_perl^C [root@f16 ~]# rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec error: line 25: Unknown tag: %filter_from_provides /perl(Apache2::Connection)$/d; /perl(Apache2::RequestRec)$/d; /perl(warnings)$/d;
edit mod_perl.spec and comment-out everything in # RPM 4.8 style part
Thanks for testing. The owner of mod_perl is not online for some time, I will ask some provepackager to commit this fix into mod_perl.
*** Bug 718926 has been marked as a duplicate of this bug. ***
mod_perl-2.0.5-6.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/mod_perl-2.0.5-6.fc16
Please confirm the mod_perl-2.0.5-6.fc16 packages fix this: http://koji.fedoraproject.org/koji/buildinfo?buildID=273329
Confirmed working and fixed on 32-bit FC16. # systemctl restart httpd.service # systemctl status httpd.service httpd.service - The Apache HTTP Server (prefork MPM) Loaded: loaded (/lib/systemd/system/httpd.service; enabled) Active: active (running) since Wed, 09 Nov 2011 23:54:32 -0400; 1s ago Process: 18484 ExecStop=/usr/sbin/httpd $OPTIONS -k stop (code=exited, status=0/SUCCESS) Process: 18486 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/httpd.service â 18487 /usr/sbin/nss_pcache 622596 off /etc/httpd/alias â 18489 /usr/bin/crlhelper 655365 /etc/httpd/alias â 18490 /usr/sbin/httpd -k start â 18491 /usr/sbin/httpd -k start â 18493 /usr/sbin/httpd -k start â 18494 /usr/sbin/httpd -k start â 18495 /usr/sbin/httpd -k start â 18496 /usr/sbin/httpd -k start â 18497 /usr/sbin/httpd -k start â 18498 /usr/sbin/httpd -k start â 18499 /usr/sbin/httpd -k start â 18500 /usr/sbin/httpd -k start # rpm -q -a | grep mod_perl mod_perl-devel-2.0.5-6.fc16.i686 mod_perl-2.0.5-6.fc16.i686 mod_perl-debuginfo-2.0.5-6.fc16.i686
confirming mod_perl-2.0.5-6.fc16 RPMs from comment #24 do not cause sigfault on my 32bit platform as well.
Package mod_perl-2.0.5-6.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing mod_perl-2.0.5-6.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2011-15621 then log in and leave karma (feedback).
Updated RPM mod_perl-2.0.5-6 Fix Confirmed. Fresh webserver install. [root@f16 ~]# yum list mod_perl Loaded plugins: langpacks, presto, refresh-packagekit Installed Packages mod_perl.i686 2.0.5-5.fc16 @fedora [root@f16 ~]# systemctl start httpd.service Job failed. See system logs and 'systemctl status' for details. [root@f16 ~]# systemctl status httpd.service httpd.service - The Apache HTTP Server (prefork MPM) Loaded: loaded (/lib/systemd/system/httpd.service; disabled) Active: failed since Fri, 11 Nov 2011 08:09:55 -0600; 48s ago Process: 1388 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=dumped, signal=SEGV) CGroup: name=systemd:/system/httpd.service dmesg entry: [ 1593.086400] httpd[1388]: segfault at d ip 007ae9f5 sp bf9eeb00 error 4 in mod_perl.so[792000+33000] ----------- Updated mod_perl to mod_perl-2.0.5-6 [root@f16 ~]# rpm -Uvh mod_perl-2.0.5-6.fc16.i686.rpm mod_perl-devel-2.0.5-6.fc16.i686.rpm Preparing... ########################################### [100%] 1:mod_perl-devel ########################################### [ 50%] 2:mod_perl ########################################### [100%] [root@f16 ~]# systemctl start httpd.service [root@f16 ~]# systemctl status httpd.service httpd.service - The Apache HTTP Server (prefork MPM) Loaded: loaded (/lib/systemd/system/httpd.service; disabled) Active: active (running) since Fri, 11 Nov 2011 08:20:51 -0600; 10s ago Process: 1465 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS) Main PID: 1466 (/usr/sbin/httpd) CGroup: name=systemd:/system/httpd.service â 1466 /usr/sbin/httpd -k start â 1468 /usr/sbin/httpd -k start â 1469 /usr/sbin/httpd -k start â 1470 /usr/sbin/httpd -k start â 1471 /usr/sbin/httpd -k start â 1472 /usr/sbin/httpd -k start â 1473 /usr/sbin/httpd -k start â 1474 /usr/sbin/httpd -k start â 1475 /usr/sbin/httpd -k start Confirmed Working! Thanks to everyone :)
*** Bug 753522 has been marked as a duplicate of this bug. ***
Package: httpd-2.2.21-1.fc16 Architecture: i686 OS Release: Fedora release 16 (Verne) Comment ----- I did a fresh install of fedora16 webserver & tried to start httpd from a command prompt because httpd does not appear in service management even though it is installed. Httpd immediately crashed.
mod_perl-2.0.5-6.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
Updated to mod_perl-devel-2.0.5-6.fc16.i686, mod_perl-2.0.5-6.fc16.i686 but I am still seeing the same errors. This is a clean FC16 install. # systemctl status httpd.service httpd.service - The Apache HTTP Server (prefork MPM) Loaded: loaded (/lib/systemd/system/httpd.service; disabled) Active: failed since Tue, 13 Dec 2011 06:38:55 -0500; 14min ago Process: 2514 ExecStop=/usr/sbin/httpd $OPTIONS -k stop (code=exited, status=0/SUCCESS) Process: 2640 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=1/FAILURE) Main PID: 2461 (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/httpd.service Any thoughts?
It works for me properly with mentioned mod_perl. Does it really crash? Try to start httpd without systemd in debug mode: httpd -X If it crashes, try to get the backtrace: http://httpd.apache.org/dev/debugging.html#backtrace
*** Bug 753305 has been marked as a duplicate of this bug. ***
I meet the same problem but can not get the fix in comment 8 (the link is died) Can you help to repost the patch? My system Fedora 20 32bit httpd 2.2.24 mod_perl 2.0.8 I build httpd and mod_perl from source Thanks, Quang