Bug 730832 - httpd segfaults on mod_perl
Summary: httpd segfaults on mod_perl
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mod_perl
Version: rawhide
Hardware: i686
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 718926 752178 752294 753305 753522 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-15 21:13 UTC by Jim Knowler
Modified: 2015-03-09 07:00 UTC (History)
13 users (show)

Fixed In Version: mod_perl-2.0.5-6.fc16
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-11-14 22:26:24 UTC


Attachments (Terms of Use)
httpd segfaults on mod_perl (648.53 KB, application/zip)
2011-08-16 13:46 UTC, Jim Knowler
no flags Details
bt (1.56 KB, text/plain)
2011-10-25 10:30 UTC, Jan Kaluža
no flags Details

Description Jim Knowler 2011-08-15 21:13:28 UTC
httpd segfaults on mod_perl
----
Versions:
Linux 3.0.1-3.fc16.i686.PAE

Name        : httpd
Arch        : i686
Version     : 2.2.19
Release     : 4.fc16

Name        : mod_perl
Arch        : i686
Version     : 2.0.5
Release     : 5.fc16
----
How reproducible:
Clean webserver group install 

Steps to Reproduce:
1. service httpd start

Actual messages log results:
Aug 15 14:43:35 f16 systemd[1]: Reloading.
Aug 15 14:44:14 f16 kernel: [ 1065.458572] httpd[1629]: segfault at d ip  0b829f5 sp bfcc7000 error 4 in mod_perl.so[b66000+33000]
Aug 15 14:44:14 f16 abrt[1630]: saved core dump of pid 1629 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2011-08-15-14:44:14-1629.new/coredump (3457024 bytes)
Aug 15 14:44:14 f16 abrtd: Directory 'ccpp-2011-08-15-14:44:14-1629' creation detected
Aug 15 14:44:14 f16 systemd[1]: httpd.service: control process exited, code=dumped status=11
Aug 15 14:44:14 f16 systemd[1]: Unit httpd.service entered failed state.
Aug 15 14:44:18 f16 abrtd: New dump directory /var/spool/abrt/ccpp-2011-08-15-14:44:14-1629, processing

Expected results:
httpd starts OK

Additional info:
abrt core dump files available upon request

Comment 1 Jim Knowler 2011-08-16 13:43:34 UTC
Commenting out the mod_perl module in the /etc/httpd/conf.d/perl.conf

#LoadModule perl_module modules/mod_perl.so

allows httpd to start normally

Comment 2 Jim Knowler 2011-08-16 13:46:13 UTC
Created attachment 518490 [details]
httpd segfaults on mod_perl

abrt files from segfault

Comment 3 Frédéric Buclin 2011-10-10 19:36:12 UTC
Maybe similar to https://bugs.mageia.org/show_bug.cgi?id=2439

Comment 4 Petr Pisar 2011-10-24 11:13:16 UTC
Cannot reproduce on x86_64 with httpd-2.2.21-1.fc16.x86_64 and mod_perl-2.0.5-5.fc16.x86_64. Maybe it's platform specific.

Comment 5 Joe Orton 2011-10-24 11:17:56 UTC
Can somebody confirm the diagnosis of Shlomi Fish in that referenced bug report:

""
(gdb) print sizeof(my_perl[0])
$5 = 1752
(gdb) print sizeof(PL_curinterp[0])
$6 = 1768
(gdb) 
""

I know of one possible cause for this, the APR apr_ino_t definition can change size if Perl is exporting -D_FILE_OFFSET_BITS=64.

Comment 6 Joe Orton 2011-10-24 11:18:36 UTC
(that would affect only 32-bit builds)

Comment 7 Joe Orton 2011-10-24 11:25:08 UTC
Oh, no, we already work around that in apr.  I have no specific idea then; we have seen things like this in the past from _FILE_OFFSET_BITS, anyway.

Comment 8 Jan Kaluža 2011-10-24 12:32:15 UTC
I don't have any i686 machine available to test it, but the archlinux patch looks like it should fix this issue:

# Patch from openSUSE to fix segfaults on i686 (FS#25008):
# avoid a conflict between apache's LARGEFILE64_SOURCE and perl's
# FILE_OFFSET_BITS=64 because the conflict isn't real and causes the perl
# structures to be invalid if only the apache flags are used

http://projects.archlinux.org/svntogit/packages.git/plain/trunk/nolfs.patch?h=packages/mod_perl

Comment 9 Jan Kaluža 2011-10-25 10:29:47 UTC
Patch (better said "workaround") from previous comment fixed the crash for me. It crashed because of bad pointer dereference (bt attached). sizeof(my_perl[0]) was the same as sizeof(PL_curinterp[0]).

I presume httpd is compiled with large files support, but definitions for that are not mentioned in apxs:

# apxs -q CFLAGS
-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables

Perl is compiled with:

-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

mod_perl then thinks APR is is not built with largefile support and removes "-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64". I'm not sure if this is right or not (depends on if apxs is right and APR is not compiled with large files support or not), I was just trying to debug it little bit.

Comment 10 Jan Kaluža 2011-10-25 10:30:26 UTC
Created attachment 530050 [details]
bt

Comment 11 Vlad 2011-11-08 18:42:56 UTC
*** Bug 752178 has been marked as a duplicate of this bug. ***

Comment 12 Vlad 2011-11-08 18:44:50 UTC
I can't get mod_perl running on release of Fedora 16 on 686 platform with the same symptoms. 

The workaround from archlinux doesn't help me. any other suggestions?

-- vlad

Comment 13 Jan Kaluža 2011-11-09 06:44:13 UTC
*** Bug 752294 has been marked as a duplicate of this bug. ***

Comment 14 Jan Kaluža 2011-11-09 06:46:14 UTC
Vlad, you have tried rebuilding mod_perl with the patch I sent in Comment 8 and it didn't fix the problem for you, right?

Comment 15 Vlad 2011-11-09 13:10:31 UTC
Jan,

no, but I manually edited that file on the system with installed mod_perl
/usr/lib/perl5/vendor_perl/Apache2/Build.pm

does it influence on mod_perl  building process as well? I'll give it a try later today then.

-- vlad

Comment 16 Vlad 2011-11-09 16:22:45 UTC
ok, I rebuilt mod_perl via rpmbuild with patch archlinux applied. 
Can starting httpd now with mod_perl enabled.

-- vlad

Comment 17 Jim Knowler 2011-11-09 16:38:01 UTC
(In reply to comment #16)
> ok, I rebuilt mod_perl via rpmbuild with patch archlinux applied. 
> Can starting httpd now with mod_perl enabled.
> -- vlad
Could you please give me some instruction/direction on how to rpmbuild a patched mod_perl and I'll test it on my system. Thanks!

Comment 18 Vlad 2011-11-09 16:56:40 UTC
try something like that following

yumdownloader --source mod_perl
rpm -ivh  mod_perl-*.src.rpm
rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec
    manually apply patch from above (comment 8)
rpmbuild -bc ~/rpmbuild/SPECS/mod_perl.spec
rpmbuild -bi ~/rpmbuild/SPECS/mod_perl.spec
rpmbuild -bb ~/rpmbuild/SPECS/mod_perl.spec

remove (if already installed mod_perl, mod_perl-devel)
rpm -i ~/rpmbuild/RPMS/i386/mod_perl-*.rpm

Comment 19 Jim Knowler 2011-11-09 18:06:27 UTC
(In reply to comment #18)
> try something like that following
> yumdownloader --source mod_perl
> rpm -ivh  mod_perl-*.src.rpm
> rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec
>     manually apply patch from above (comment 8)
> rpmbuild -bc ~/rpmbuild/SPECS/mod_perl.spec
> rpmbuild -bi ~/rpmbuild/SPECS/mod_perl.spec
> rpmbuild -bb ~/rpmbuild/SPECS/mod_perl.spec
> remove (if already installed mod_perl, mod_perl-devel)
> rpm -i ~/rpmbuild/RPMS/i386/mod_perl-*.rpm

I get to step #3 and throw an error (see below). 

---- screen output ---
[root@f16 ~]# yumdownloader --source mod_perl
Loaded plugins: langpacks, presto, refresh-packagekit
Enabling updates-testing-source repository
Enabling fedora-source repository
mod_perl-2.0.5-5.fc16.src.rpm                                                                           | 3.6 MB     00:12
[root@f16 ~]# rpm -ivh  mod_perl-*.src.rpm
   1:mod_perl               warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
########################################### [100%]
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
[root@f16 ~]# yumdownloader --source mod_perl^C
[root@f16 ~]# rpmbuild -bp ~/rpmbuild/SPECS/mod_perl.spec
error: line 25: Unknown tag: %filter_from_provides /perl(Apache2::Connection)$/d; /perl(Apache2::RequestRec)$/d; /perl(warnings)$/d;

Comment 20 Vlad 2011-11-09 18:12:15 UTC
edit mod_perl.spec and comment-out everything in 
# RPM 4.8 style
part

Comment 21 Jan Kaluža 2011-11-09 20:11:08 UTC
Thanks for testing. The owner of mod_perl is not online for some time, I will ask some provepackager to commit this fix into mod_perl.

Comment 22 Jan Kaluža 2011-11-09 20:11:36 UTC
*** Bug 718926 has been marked as a duplicate of this bug. ***

Comment 23 Fedora Update System 2011-11-09 23:06:59 UTC
mod_perl-2.0.5-6.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/mod_perl-2.0.5-6.fc16

Comment 24 Joe Orton 2011-11-09 23:20:13 UTC
Please confirm the mod_perl-2.0.5-6.fc16 packages fix this:

http://koji.fedoraproject.org/koji/buildinfo?buildID=273329

Comment 25 Jason Burgess 2011-11-10 03:56:32 UTC
Confirmed working and fixed on 32-bit FC16.

# systemctl restart httpd.service

# systemctl status httpd.service
httpd.service - The Apache HTTP Server (prefork MPM)
          Loaded: loaded (/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Wed, 09 Nov 2011 23:54:32 -0400; 1s ago
         Process: 18484 ExecStop=/usr/sbin/httpd $OPTIONS -k stop (code=exited, status=0/SUCCESS)
         Process: 18486 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/httpd.service
                  â 18487 /usr/sbin/nss_pcache 622596 off /etc/httpd/alias
                  â 18489 /usr/bin/crlhelper 655365 /etc/httpd/alias
                  â 18490 /usr/sbin/httpd -k start
                  â 18491 /usr/sbin/httpd -k start
                  â 18493 /usr/sbin/httpd -k start
                  â 18494 /usr/sbin/httpd -k start
                  â 18495 /usr/sbin/httpd -k start
                  â 18496 /usr/sbin/httpd -k start
                  â 18497 /usr/sbin/httpd -k start
                  â 18498 /usr/sbin/httpd -k start
                  â 18499 /usr/sbin/httpd -k start
                  â 18500 /usr/sbin/httpd -k start


# rpm -q -a | grep mod_perl
mod_perl-devel-2.0.5-6.fc16.i686
mod_perl-2.0.5-6.fc16.i686
mod_perl-debuginfo-2.0.5-6.fc16.i686

Comment 26 Vlad 2011-11-10 16:01:42 UTC
confirming mod_perl-2.0.5-6.fc16 RPMs from comment #24 do not cause sigfault on my 32bit platform as well.

Comment 27 Fedora Update System 2011-11-10 17:36:40 UTC
Package mod_perl-2.0.5-6.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing mod_perl-2.0.5-6.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-15621
then log in and leave karma (feedback).

Comment 28 Jim Knowler 2011-11-11 14:30:34 UTC
Updated RPM mod_perl-2.0.5-6 Fix Confirmed.

Fresh webserver install. 

[root@f16 ~]# yum list mod_perl
Loaded plugins: langpacks, presto, refresh-packagekit
Installed Packages
mod_perl.i686 2.0.5-5.fc16 @fedora

[root@f16 ~]# systemctl start httpd.service
Job failed. See system logs and 'systemctl status' for details.

[root@f16 ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server (prefork MPM)
          Loaded: loaded (/lib/systemd/system/httpd.service; disabled)
          Active: failed since Fri, 11 Nov 2011 08:09:55 -0600; 48s ago
         Process: 1388 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=dumped, signal=SEGV)
          CGroup: name=systemd:/system/httpd.service

          
dmesg entry:
[ 1593.086400] httpd[1388]: segfault at d ip 007ae9f5 sp bf9eeb00 error 4 in mod_perl.so[792000+33000]

----------- Updated mod_perl to mod_perl-2.0.5-6
[root@f16 ~]# rpm -Uvh mod_perl-2.0.5-6.fc16.i686.rpm mod_perl-devel-2.0.5-6.fc16.i686.rpm
Preparing...                ########################################### [100%]
   1:mod_perl-devel         ########################################### [ 50%]
   2:mod_perl               ########################################### [100%]
[root@f16 ~]# systemctl start httpd.service
[root@f16 ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server (prefork MPM)
          Loaded: loaded (/lib/systemd/system/httpd.service; disabled)
          Active: active (running) since Fri, 11 Nov 2011 08:20:51 -0600; 10s ago
         Process: 1465 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
        Main PID: 1466 (/usr/sbin/httpd)
          CGroup: name=systemd:/system/httpd.service
                  â 1466 /usr/sbin/httpd -k start
                  â 1468 /usr/sbin/httpd -k start
                  â 1469 /usr/sbin/httpd -k start
                  â 1470 /usr/sbin/httpd -k start
                  â 1471 /usr/sbin/httpd -k start
                  â 1472 /usr/sbin/httpd -k start
                  â 1473 /usr/sbin/httpd -k start
                  â 1474 /usr/sbin/httpd -k start
                  â 1475 /usr/sbin/httpd -k start

Confirmed Working! Thanks to everyone :)

Comment 29 Jan Kaluža 2011-11-13 12:29:32 UTC
*** Bug 753522 has been marked as a duplicate of this bug. ***

Comment 30 adrian1h 2011-11-14 02:48:26 UTC
Package: httpd-2.2.21-1.fc16
Architecture: i686
OS Release: Fedora release 16 (Verne)

Comment
-----
I did a fresh install of fedora16 webserver & tried to start httpd from a command prompt because httpd does not appear in service management even though it is installed. Httpd immediately crashed.

Comment 31 Fedora Update System 2011-11-14 22:26:24 UTC
mod_perl-2.0.5-6.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 32 Andrew Long 2011-12-13 11:55:00 UTC
Updated to mod_perl-devel-2.0.5-6.fc16.i686, mod_perl-2.0.5-6.fc16.i686 but I am still seeing the same errors. This is a clean FC16 install.

# systemctl status httpd.service
httpd.service - The Apache HTTP Server (prefork MPM)
	  Loaded: loaded (/lib/systemd/system/httpd.service; disabled)
	  Active: failed since Tue, 13 Dec 2011 06:38:55 -0500; 14min ago
	 Process: 2514 ExecStop=/usr/sbin/httpd $OPTIONS -k stop (code=exited, status=0/SUCCESS)
	 Process: 2640 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=1/FAILURE)
	Main PID: 2461 (code=exited, status=0/SUCCESS)
	  CGroup: name=systemd:/system/httpd.service

Any thoughts?

Comment 33 Jan Kaluža 2011-12-14 08:39:40 UTC
It works for me properly with mentioned mod_perl. Does it really crash? Try to start httpd without systemd in debug mode: httpd -X

If it crashes, try to get the backtrace: http://httpd.apache.org/dev/debugging.html#backtrace

Comment 34 Jan Kaluža 2012-03-05 13:19:27 UTC
*** Bug 753305 has been marked as a duplicate of this bug. ***

Comment 35 Quang 2015-03-09 07:00:35 UTC
I meet the same problem but can not get the fix in comment 8 (the link is died)
Can you help to repost the patch?
My system
Fedora 20 32bit
httpd 2.2.24
mod_perl 2.0.8
I build httpd and mod_perl from source

Thanks,
Quang


Note You need to log in before you can comment on or make changes to this bug.