Bug 1298551

Summary: Segmentation fault in modperl_wbucket_flush
Product: Red Hat Enterprise Linux 6 Reporter: Martin Frodl <mfrodl>
Component: mod_perlAssignee: perl-maint-list
Status: CLOSED WONTFIX QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.7CC: jkaluza, jorton, ppisar
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-17 07:36:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martin Frodl 2016-01-14 12:20:33 UTC
Description of problem:

When running Apache upstream test toolkit with mod_perl and mod_auth_kerb installed, there are occasional segfaults in mod_perl.

Version-Release number of selected component (if applicable):
mod_perl-2.0.4-11.el6_5.x86_64

How reproducible:
occasionally

Steps to Reproduce:
# yum -y install httpd mod_perl mod_auth_kerb subversion perl-Test-Simple
# svn co https://svn.apache.org/repos/asf/perl/Apache-Test/trunk
# chown apache:apache trunk
# cd trunk
# sudo -u apache perl Makefile.PL -apxs /usr/bin/apxs
# make
# sudo -u apache make test

Actual results:
Segmentation fault.

Expected results:
All tests pass, no segmentation faults.

Comment 6 Petr Pisar 2016-11-18 13:12:33 UTC
I can reproduce it with:

httpd-2.2.15-53.el6.x86_64
mod_auth_kerb-5.4-14.el6.x86_64
mod_perl-2.0.4-11.el6_5.x86_64
perl-5.10.1-141.el6_7.1.x86_64

It cashes within t/more/04testmore.t test:

$ prove -b -v t/more/04testmore.t
t/more/04testmore.t .. request has failed (the response code was: 500)
see t/logs/error_log for more details
[  error] oh rats, server dumped core
[  error] for stacktrace, run: gdb /usr/sbin/httpd -core /tmp/trunk/t/core.1861
Dubious, test returned 111 (wstat 28416, 0x6f00)

Comment 7 Petr Pisar 2016-11-18 14:58:31 UTC
Actually it crashes randomly and sometimes sooner.

Smaller reproducer in the trunk tree cloned into /tmp/trunk as apache user:

(1) Build the test suite, especially the configuration for the httpd:
$ perl Makefile.PL
$ make

(2) Start the httpd, it will deamonize:
$ /usr/sbin/httpd  -d /tmp/trunk/t -f /tmp/trunk/t/conf/httpd.conf -D APACHE2 -D PERL_USEITHREADS

(3) Run some tests (from t/alltest/all.t to t/more/04testmore.t):
$ while (prove -b t/alltest/all.t t/alltest2/all.t t/bad_coding.t t/cookies.t t/import.t t/log_watch.t t/log_watch_for_broken_lines.t t/more/01testpm.t t/more/02testmore.t t/more/03testpm.t t/more/04testmore.t); do :;done

When the bug emerges, a test will report that the server crashed. If you created /tmp/trunk/logs directory, you can see in the /tmp/trunk/logs/error_log:

[Fri Nov 18 15:40:58 2016] [notice] child pid 22674 exit signal Segmentation fault (11), possible coredump in /tmp/trunk/t
[Fri Nov 18 15:41:11 2016] [debug] proxy_util.c(1909): proxy: grabbed scoreboard slot 0 in child 22754 for worker proxy:reverse
[Fri Nov 18 15:41:11 2016] [debug] proxy_util.c(1929): proxy: worker proxy:reverse already initialized
[Fri Nov 18 15:41:11 2016] [debug] proxy_util.c(2025): proxy: initialized single connection worker 0 in child 22754 for (*)

It really crashes randomly and very rarely, even with httpd-2.2.15-55.el6_8.2.x86_64.

Comment 8 Petr Pisar 2016-11-18 15:22:34 UTC
Back trace:

#0  0x00007f3fdebf81c3 in PerlIOApache_flush (my_perl=0x7f3feb6cbc30, f=0x7f3feb8db760) at modperl_io_apache.c:167
#1  0x00007f3fde989b15 in Perl_PerlIO_flush (my_perl=0x7f3feb6cbc30, f=<value optimized out>) at perlio.c:1669
#2  0x00007f3fde98a4da in PerlIOBase_close (my_perl=0x7f3feb6cbc30, f=0x7f3feb8db760) at perlio.c:2177
#3  0x00007f3fdebf8239 in PerlIOApache_close (my_perl=<value optimized out>, f=0x7f3feb8db760) at modperl_io_apache.c:189
#4  0x00007f3fde98a5f8 in PerlIO__close (my_perl=<value optimized out>, f=<value optimized out>) at perlio.c:1419
#5  0x00007f3fde98b88f in Perl_PerlIO_close (my_perl=0x7f3feb6cbc30, f=0x7f3feb8db760) at perlio.c:1432
#6  0x00007f3fde96caeb in Perl_do_openn (my_perl=0x7f3feb6cbc30, gv=0x7f3fecb0de30, oname=0x7f3fecaf0a80 ">&STDOUT", len=8, as_raw=0, rawmode=0, rawperm=0, supplied_fp=0x0, 
    svp=0x7f3feca69638, num_svs=0) at doio.c:125
#7  0x00007f3fde963aff in Perl_pp_open (my_perl=0x7f3feb6cbc30) at pp_sys.c:560
#8  0x00007f3fde914b06 in Perl_runops_standard (my_perl=0x7f3feb6cbc30) at run.c:40
#9  0x00007f3fde8bc5df in Perl_call_sv (my_perl=0x7f3feb6cbc30, sv=0x7f3fecb5caf8, flags=4) at perl.c:2721
#10 0x00007f3fdebf30be in modperl_callback (my_perl=0x7f3feb6cbc30, handler=0x7f3feb7d8718, p=0x7f3feb92f2a8, r=0x7f3feb92f328, s=0x7f3feb65f870, args=0x7f3fec9a9bc8)
    at modperl_callback.c:101
#11 0x00007f3fdebf380b in modperl_callback_run_handlers (idx=6, type=<value optimized out>, r=0x7f3feb92f328, c=<value optimized out>, s=0x7f3feb65f870, pconf=<value optimized out>, 
    plog=0x0, ptemp=0x0, run_mode=MP_HOOK_RUN_FIRST) at modperl_callback.c:262
#12 0x00007f3fdebf3e0f in modperl_callback_per_dir (idx=<value optimized out>, r=<value optimized out>, run_mode=<value optimized out>) at modperl_callback.c:369
#13 0x00007f3fdebed75f in modperl_response_handler_run (r=0x7f3feb92f328, finish=0) at mod_perl.c:1000
#14 0x00007f3fdebed913 in modperl_response_handler_cgi (r=0x7f3feb92f328) at mod_perl.c:1100
#15 0x00007f3fea4adfc0 in ap_run_handler (r=0x7f3feb92f328) at /usr/src/debug/httpd-2.2.15/server/config.c:158
#16 0x00007f3fea4b187e in ap_invoke_handler (r=0x7f3feb92f328) at /usr/src/debug/httpd-2.2.15/server/config.c:376
#17 0x00007f3fea4bcfd0 in ap_process_request (r=0x7f3feb92f328) at /usr/src/debug/httpd-2.2.15/modules/http/http_request.c:282
#18 0x00007f3fea4b9e18 in ap_process_http_connection (c=0x7f3feb91b478) at /usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
#19 0x00007f3fea4b5ae8 in ap_run_process_connection (c=0x7f3feb91b478) at /usr/src/debug/httpd-2.2.15/server/connection.c:43
#20 0x00007f3fea4c1d77 in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:670
#21 0x00007f3fea4c2099 in make_child (s=0x7f3feb65f870, slot=0) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:773
#22 0x00007f3fea4c23cb in startup_children (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
    at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:791
#23 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:1012
#24 0x00007f3fea499aa0 in main (argc=9, argv=0x7ffe91f1e8b8) at /usr/src/debug/httpd-2.2.15/server/main.c:763

It crashed in mod_perl's src/modules/perl/modperl_io_apache.c:167:

    rcfg = modperl_config_req_get(st->r);

→   MP_CHECK_WBUCKET_INIT("flush");

The MP_CHECK_WBUCKET_INIT macro does:

/* check whether the response phase has been initialized already */
#define MP_CHECK_WBUCKET_INIT(func) \
    if (!rcfg->wbucket) { \
        Perl_croak(aTHX_ "%s: " func " can't be called "  \
                   "before the response phase", MP_FUNC); \
    }

The rcfg is NULL as disassembly and CPU registers confirm.

Comment 9 Red Hat Bugzilla Rules Engine 2017-10-17 07:36:25 UTC
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.