Bug 912288
| Summary: | Random crashes when service httpd stop | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | GV <rhel> |
| Component: | httpd | Assignee: | Joe Orton <jorton> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 18 | CC: | jkaluza, jorton, pahan, Philippe.Vouters |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | httpd-2.4.4-3.fc18 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-04-25 13:54:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
GV
2013-02-18 10:13:35 UTC
# grep "glibc detected" /var/log/httpd/error_log *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fc8a997b960 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f9b90d57620 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f9b90d57620 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f7c650a35c0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x00007f62dc9bf3a0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fea22bcc570 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fea22bcc570 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f0eebde8520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f0eebde8520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f120db184b0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007ffeba52e520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f364b700500 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f41fe2d8430 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f41fe2d8430 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fc80ef723c0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f18e0ce63c0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f14467983f0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f327b28e450 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f422c4f1580 *** 7f421acb0000-7f421acb1000 rw-p 0000c000 08:02 33*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f422c4f1580 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007faa15158ea0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6b6071ced0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6b6071ced0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007ffb3f8ce8e0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007f4d229a6c90 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid size: 0x00007f4d229b0630 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007feb349c58a0 *** *** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x00007fc4de127780 *** *** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x00007f6a9545af60 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f2748cbfe00 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f359e265590 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007fd09b447db0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007fd09b447db0 *** Please check the systemd module. I was unable to duplicate this bug running httpd from command line ( httpd -DFOREGROUND ) or via gdb (run -X). Starting and stopping httpd via systemctl almost every time result in "glibc detected..." error messages or frozen httpd threads. Sorry! Not mod_systemd.so's fault. :-( Again I had some spare time so I'm back on this bug again. Also I enabled core dump files. # cat crash-test.php <?php phpinfo(); ?> # ab -n 50000 -c 10 http://localhost/~gabriel/crash-test.php # service httpd stop <most of the time apache crash or freeze here.> # find /tmp -name core\* ./systemd-private-MFoNAu/core.28874 # gdb httpd -c /tmp/systemd-private-MFoNAu/core.28874 GNU gdb (GDB) Fedora (7.5.1-37.fc18) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done. done. [New LWP 28874] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/httpd -DFOREGROUND'. Program terminated with signal 6, Aborted. #0 0x00007f5b2ef6bba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 63 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) where #0 0x00007f5b2ef6bba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 #1 0x00007f5b2ef6d358 in __GI_abort () at abort.c:90 #2 0x00007f5b2efab59b in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f5b2f0afba8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:197 #3 0x00007f5b2efb2a8e in malloc_printerr (ptr=0x7f5b2f2e7798 <main_arena+88>, str=0x7f5b2f0ada01 "free(): invalid pointer", action=3) at malloc.c:4969 #4 _int_free (av=0x7f5b2f2e7740 <main_arena>, p=0x7f5b2f2e7788 <main_arena+72>, have_lock=0) at malloc.c:3826 #5 0x00007f5b2f72785c in allocator_free (node=0x7f5b2f2e7798 <main_arena+88>, allocator=0x7f5b321572c0) at memory/unix/apr_pools.c:430 #6 apr_pool_destroy (pool=0x7f5b321573b8) at memory/unix/apr_pools.c:856 #7 0x00007f5b29def1be in clean_child_exit (code=code@entry=0) at prefork.c:218 #8 0x00007f5b29def9fb in just_die (sig=<optimized out>) at prefork.c:344 #9 <signal handler called> #10 0x00007f5b30806535 in _dl_fini () at dl-fini.c:194 #11 0x00007f5b2ef6edf1 in __run_exit_handlers (status=status@entry=0, listp=0x7f5b2f2e76a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #12 0x00007f5b2ef6ee75 in __GI_exit (status=status@entry=0) at exit.c:99 #13 0x00007f5b29def1db in clean_child_exit (code=code@entry=0) at prefork.c:227 #14 0x00007f5b29def60e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:725 #15 0x00007f5b29def97c in make_child (s=0x7f5b31de4368, slot=slot@entry=3) at prefork.c:800 #16 0x00007f5b29def9d6 in startup_children (number_to_start=2) at prefork.c:818 #17 0x00007f5b29df0886 in prefork_run (_pconf=<optimized out>, plog=0x7f5b31e164f8, s=0x7f5b31de4368) at prefork.c:976 #18 0x00007f5b30a3ed5e in ap_run_mpm (pconf=0x7f5b31db9158, plog=0x7f5b31e164f8, s=0x7f5b31de4368) at mpm_common.c:98 #19 0x00007f5b30a3840a in main (argc=2, argv=0x7fffdb816188) at main.c:777 This is from another computer. I don't even have to make a coonection to http server. start && stop is all I have to to. Almost every time I have at least one core dump file (usuallly 3 to 5). All have this kind of backtrace:
(gdb) where
#0 _dl_fini () at dl-fini.c:249
#1 0x00007f362c898df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f362cc116a8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#2 0x00007f362c898e75 in __GI_exit (status=status@entry=0) at exit.c:99
#3 0x00007f36277191db in clean_child_exit (code=code@entry=0) at prefork.c:227
#4 0x00007f36277199fb in just_die (sig=<optimized out>) at prefork.c:344
#5 <signal handler called>
#6 0x00007f362e138a37 in munmap () at ../sysdeps/unix/syscall-template.S:81
#7 0x00007f362e1381ad in _dl_unmap (map=map@entry=0x7f362f9e09f0) at ../sysdeps/x86_64/tlsdesc.c:139
#8 0x00007f362e13511f in _dl_close_worker (map=map@entry=0x7f362f9e09f0) at dl-close.c:634
#9 0x00007f362e13597e in _dl_close (_map=0x7f362f9e09f0) at dl-close.c:776
#10 0x00007f362e12fdc6 in _dl_catch_error (objname=0x7f362f7e95b0, errstring=0x7f362f7e95b8, mallocedp=0x7f362f7e95a8,
operate=0x7f362cc190e0 <dlclose_doit>, args=0x7f362f9e09f0) at dl-error.c:177
#11 0x00007f362cc1963c in _dlerror_run (operate=operate@entry=0x7f362cc190e0 <dlclose_doit>, args=0x7f362f9e09f0)
at dlerror.c:163
#12 0x00007f362cc1910f in __dlclose (handle=<optimized out>) at dlclose.c:47
#13 0x00007f3625325e14 in module_destructor (module=0x7f362f9e1c20) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2286
#14 0x00007f362532b085 in zend_hash_apply_deleter (ht=ht@entry=0x7f3625710220 <module_registry>, p=0x7f362f9e1bc0)
at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650
#15 0x00007f362532c878 in zend_hash_graceful_reverse_destroy (ht=0x7f3625710220 <module_registry>)
at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687
#16 0x00007f362532468c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797
#17 0x00007f362531ea0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820
#18 0x00007f36252c035b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365
#19 0x00007f36252c0419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>)
at /usr/src/debug/php-5.4.12/main/main.c:2333
#20 0x00007f36253c6fc1 in php_apache_child_shutdown (tmp=<optimized out>)
at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398
#21 0x00007f362d05173e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
#22 apr_pool_destroy (pool=0x7f362fb1b6d8) at memory/unix/apr_pools.c:814
#23 0x00007f36277191be in clean_child_exit (code=code@entry=0) at prefork.c:218
#24 0x00007f362771960e in child_main (child_num_arg=child_num_arg@entry=4) at prefork.c:725
#25 0x00007f362771997c in make_child (s=0x7f362f7ad368, slot=slot@entry=4) at prefork.c:800
#26 0x00007f36277199d6 in startup_children (number_to_start=1) at prefork.c:818
#27 0x00007f362771a886 in prefork_run (_pconf=<optimized out>, plog=0x7f362f7df4f8, s=0x7f362f7ad368) at prefork.c:976
#28 0x00007f362e368d5e in ap_run_mpm (pconf=0x7f362f782158, plog=0x7f362f7df4f8, s=0x7f362f7ad368) at mpm_common.c:98
#29 0x00007f362e36240a in main (argc=2, argv=0x7fff59920b28) at main.c:777
Uninstalling php does not help. I still get occasionally core dumps just restarting httpd. And the backtrace is that from comment #4.
I think there are 2 errors. One in httpd and one in php module.
I'm still not able to reproduce it here on my machine... :/ Hmm... I can reproduce this on two different machines (same httpd configuration). Maybe I should send you my /etc/httpd directory. It could be related to how httpd is configured. This is the bugreport on apache.org: https://issues.apache.org/bugzilla/show_bug.cgi?id=54785 Oh, I'm able now and even 'service httpd stop' freezes for me now. I will see what I can do. I have found similar bug from 2006 in the mean time: https://issues.apache.org/bugzilla/show_bug.cgi?id=23238 . > Oh, I'm able now and even 'service httpd stop' freezes for me now.
Welcome to the club! :-)
Really, very nasty bug! I get the same behaviour.
The bug has been reported in 2003! Seems to be related. And still not fixed? :-( Can you please try to reproduce it with following scratch-build? http://koji.fedoraproject.org/koji/taskinfo?taskID=5203004 It could fix/workaround the crash from Comment 3. Hm, I think this fixed only crash, but it still can freeze... > Hm, I think this fixed only crash, but it still can freeze... Works for me, at on the test computer. # watch -n 5 "service httpd restart" Every around 7 seconds I get this [Wed Apr 03 07:05:00.087609 2013] [core:notice] [pid 2198] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' in error_log file. It seems to work. Also, where is no php module loaded. 1. restart 20+ times 2. ab -n 50000 -c 10 http://localhost/index.html 1. restart 20+ times 3. find /tmp -name core* ==> nothing Sorry. It should be: at least on the test computer. [Wed Apr 03 07:15:19.549906 2013] [core:notice] [pid 3036] core dump file size limit raised to 18446744073709551615 bytes Really? 16777215 TB? I haven't changed the core dump file size limit in that scratch-build. I think this is what "infinite" mean in this context. Anyway, can you try also http://koji.fedoraproject.org/koji/taskinfo?taskID=5203133 ? This should fix also potential freezing with php installed. If it will work, I will try to propose the fix upstream. > I haven't changed the core dump file size limit in that scratch-build. I know... httpd-2.4.1-corelimit.patch > I think this is what "infinite" mean in this context. Ah... right. :-) Thank you. # ulimit -a | grep core core file size (blocks, -c) unlimited # rpm -qa | grep httpd | grep -v micro httpd-2.4.4-5.fc18.x86_64 httpd-debuginfo-2.4.4-5.fc18.x86_64 httpd-tools-2.4.4-5.fc18.x86_64 php module enabled. # service httpd start # service httpd stop # find /tmp -name core* /tmp/systemd-private-bwjc3G/core.6610 # gdb httpd -c /tmp/systemd-private-bwjc3G/core.6610 ... (gdb) where #0 _dl_fini () at dl-fini.c:249 #1 0x00007f199b284df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f199b5fd6a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #2 0x00007f199b284e75 in __GI_exit (status=status@entry=0) at exit.c:99 #3 0x00007f19961051f6 in clean_child_exit (code=code@entry=0) at prefork.c:232 #4 0x00007f1996105a33 in just_die (sig=<optimized out>) at prefork.c:351 #5 <signal handler called> #6 0x00007f199cb24a37 in munmap () at ../sysdeps/unix/syscall-template.S:81 #7 0x00007f199cb241ad in _dl_unmap (map=map@entry=0x7f199d54aed0) at ../sysdeps/x86_64/tlsdesc.c:139 #8 0x00007f199cb2111f in _dl_close_worker (map=map@entry=0x7f199d54a9e0) at dl-close.c:634 #9 0x00007f199cb2197e in _dl_close (_map=0x7f199d54a9e0) at dl-close.c:776 #10 0x00007f199cb1bdc6 in _dl_catch_error (objname=0x7f199d3535b0, errstring=0x7f199d3535b8, mallocedp=0x7f199d3535a8, operate=0x7f199b6050e0 <dlclose_doit>, args=0x7f199d54a9e0) at dl-error.c:177 #11 0x00007f199b60563c in _dlerror_run (operate=operate@entry=0x7f199b6050e0 <dlclose_doit>, args=0x7f199d54a9e0) at dlerror.c:163 #12 0x00007f199b60510f in __dlclose (handle=<optimized out>) at dlclose.c:47 #13 0x00007f1993d11e14 in module_destructor (module=0x7f199d54bc10) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2286 #14 0x00007f1993d17085 in zend_hash_apply_deleter (ht=ht@entry=0x7f19940fc220 <module_registry>, p=0x7f199d54bbb0) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650 #15 0x00007f1993d18878 in zend_hash_graceful_reverse_destroy (ht=0x7f19940fc220 <module_registry>) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687 #16 0x00007f1993d1068c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797 #17 0x00007f1993d0aa0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820 #18 0x00007f1993cac35b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365 #19 0x00007f1993cac419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.12/main/main.c:2333 #20 0x00007f1993db2fc1 in php_apache_child_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398 #21 0x00007f199ba3d73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 #22 apr_pool_destroy (pool=0x7f199d6856c8) at memory/unix/apr_pools.c:814 #23 0x00007f19961051be in clean_child_exit (code=code@entry=0) at prefork.c:218 #24 0x00007f199610562e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:732 #25 0x00007f199610599c in make_child (s=0x7f199d317368, slot=slot@entry=3) at prefork.c:807 #26 0x00007f19961059f6 in startup_children (number_to_start=2) at prefork.c:825 #27 0x00007f19961068c6 in prefork_run (_pconf=<optimized out>, plog=0x7f199d3494f8, s=0x7f199d317368) at prefork.c:983 #28 0x00007f199cd54d5e in ap_run_mpm (pconf=0x7f199d2ec158, plog=0x7f199d3494f8, s=0x7f199d317368) at mpm_common.c:98 #29 0x00007f199cd4e40a in main (argc=2, argv=0x7fffa5fdf198) at main.c:777 Does not work. Sorry. Right, happened to me after 30 minutes of testing too... I will test more before submitting next build. Ok, I had run the test over the night and I didn't see any freeze or crash with this one: http://koji.fedoraproject.org/koji/taskinfo?taskID=5209578 It would be great if you could try it too. I hope this is the last build :). > I hope this is the last build
I don't really care how many are. Do as many as necessary. All that matters is that the bug will is fixed.
Anyway, I think is not la last one. :-(
# rpm -qa | grep httpd | grep -v micro
httpd-2.4.4-7.fc18.x86_64
httpd-tools-2.4.4-7.fc18.x86_64
httpd-debuginfo-2.4.4-5.fc18.x86_64
# find /tmp -name core*
<nothing>
# service httpd start
Redirecting to /bin/systemctl start httpd.service
# service httpd stop
Redirecting to /bin/systemctl stop httpd.service
# find /tmp -name core*
/tmp/systemd-private-n4JXRH/core.722
/tmp/systemd-private-n4JXRH/core.721
/tmp/systemd-private-n4JXRH/core.723
/tmp/systemd-private-n4JXRH/core.719
/tmp/systemd-private-n4JXRH/core.720
# gdb httpd -c /tmp/systemd-private-n4JXRH/core.720
...
(gdb) where
#0 apr_palloc (pool=pool@entry=0x0, in_size=in_size@entry=32) at memory/unix/apr_pools.c:681
#1 0x00007f246160952b in ap_mutex_register (pconf=pconf@entry=0x0, type=type@entry=0x7f245a996ea1 "mpm-accept",
default_dir=default_dir@entry=0x0, default_mech=default_mech@entry=APR_LOCK_DEFAULT, options=options@entry=0)
at util_mutex.c:259
#2 0x00007f245a995aa4 in prefork_pre_config (p=0x0, plog=<optimized out>, ptemp=<optimized out>) at prefork.c:1302
#3 0x00007f2461fd5bc0 in ?? ()
#4 <signal handler called>
#5 __strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31
#6 0x00007f24585a0c45 in zend_unregister_functions (functions=<optimized out>, count=count@entry=-1,
function_table=function_table@entry=0x0) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2206
#7 0x00007f24585a1d9a in module_destructor (module=0x7f24621a2b60) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2280
#8 0x00007f24585a7085 in zend_hash_apply_deleter (ht=ht@entry=0x7f245898c220 <module_registry>, p=0x7f24621a2b00)
at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650
#9 0x00007f24585a8878 in zend_hash_graceful_reverse_destroy (ht=0x7f245898c220 <module_registry>)
at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687
#10 0x00007f24585a068c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797
#11 0x00007f245859aa0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820
#12 0x00007f245853c35b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365
#13 0x00007f245853c419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>)
at /usr/src/debug/php-5.4.12/main/main.c:2333
#14 0x00007f2458642fc1 in php_apache_child_shutdown (tmp=<optimized out>)
at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398
#15 0x00007f24602cd73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
#16 apr_pool_destroy (pool=0x7f24622f0658) at memory/unix/apr_pools.c:814
#17 0x00007f245a9951d4 in clean_child_exit (code=code@entry=0) at prefork.c:225
#18 0x00007f245a99563e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:739
#19 0x00007f245a9959ac in make_child (s=0x7f2461f82368, slot=slot@entry=1) at prefork.c:814
#20 0x00007f245a995a06 in startup_children (number_to_start=4) at prefork.c:832
#21 0x00007f245a9968d6 in prefork_run (_pconf=<optimized out>, plog=0x7f2461fb44f8, s=0x7f2461f82368) at prefork.c:990
#22 0x00007f24615e4d5e in ap_run_mpm (pconf=0x7f2461f57158, plog=0x7f2461fb44f8, s=0x7f2461f82368) at mpm_common.c:98
#23 0x00007f24615de40a in main (argc=2, argv=0x7fff20f5c7a8) at main.c:777
Sorry. :-(
Same result with php 5.4.13 (after yum update):
(gdb) where
#0 apr_palloc (pool=pool@entry=0x0, in_size=in_size@entry=32) at memory/unix/apr_pools.c:681
#1 0x00007f52a4bfa52b in ap_mutex_register (pconf=pconf@entry=0x0, type=type@entry=0x7f529df87ea1 "mpm-accept",
default_dir=default_dir@entry=0x0, default_mech=default_mech@entry=APR_LOCK_DEFAULT, options=options@entry=0)
at util_mutex.c:259
#2 0x00007f529df86aa4 in prefork_pre_config (p=0x0, plog=<optimized out>, ptemp=<optimized out>) at prefork.c:1302
#3 0x00007f52a5787190 in ?? ()
#4 <signal handler called>
#5 0x00007f529bb995f6 in zend_hash_del_key_or_index (ht=ht@entry=0x7f52a57a3b70,
arKey=arKey@entry=0x7f5296776289 "mcrypt_generic_end", nKeyLength=19, h=8096388789755616310, h@entry=0,
flag=flag@entry=0) at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:514
#6 0x00007f529bb91cd8 in zend_unregister_functions (functions=<optimized out>, count=count@entry=-1,
function_table=function_table@entry=0x0) at /usr/src/debug/php-5.4.13/Zend/zend_API.c:2206
#7 0x00007f529bb92e1a in module_destructor (module=0x7f52a582ac10) at /usr/src/debug/php-5.4.13/Zend/zend_API.c:2280
#8 0x00007f529bb98105 in zend_hash_apply_deleter (ht=ht@entry=0x7f529bf7d240 <module_registry>, p=0x7f52a582abb0)
at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:650
#9 0x00007f529bb998f8 in zend_hash_graceful_reverse_destroy (ht=0x7f529bf7d240 <module_registry>)
at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:687
#10 0x00007f529bb9170c in zend_destroy_modules () at /usr/src/debug/php-5.4.13/Zend/zend_API.c:1797
#11 0x00007f529bb8ba8e in zend_shutdown () at /usr/src/debug/php-5.4.13/Zend/zend.c:820
#12 0x00007f529bb2d3cb in php_module_shutdown () at /usr/src/debug/php-5.4.13/main/main.c:2365
#13 0x00007f529bb2d489 in php_module_shutdown_wrapper (sapi_globals=<optimized out>)
at /usr/src/debug/php-5.4.13/main/main.c:2333
#14 0x00007f529bc34041 in php_apache_child_shutdown (tmp=<optimized out>)
at /usr/src/debug/php-5.4.13/sapi/apache2handler/sapi_apache2.c:398
#15 0x00007f52a38be73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
#16 apr_pool_destroy (pool=0x7f52a59646c8) at memory/unix/apr_pools.c:814
#17 0x00007f529df861d4 in clean_child_exit (code=code@entry=0) at prefork.c:225
#18 0x00007f529df8663e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:739
#19 0x00007f529df869ac in make_child (s=0x7f52a55f6368, slot=slot@entry=3) at prefork.c:814
#20 0x00007f529df86a06 in startup_children (number_to_start=2) at prefork.c:832
#21 0x00007f529df878d6 in prefork_run (_pconf=<optimized out>, plog=0x7f52a56284f8, s=0x7f52a55f6368) at prefork.c:990
#22 0x00007f52a4bd5d5e in ap_run_mpm (pconf=0x7f52a55cb158, plog=0x7f52a56284f8, s=0x7f52a55f6368) at mpm_common.c:98
#23 0x00007f52a4bcf40a in main (argc=2, argv=0x7fff2c297fb8) at main.c:777
Can you paste error_log from this restart try, please? # find /tmp -name core* # >/var/log/httpd/error_log # service httpd start Redirecting to /bin/systemctl start httpd.service # service httpd stop Redirecting to /bin/systemctl stop httpd.service # find /tmp -name core* /tmp/systemd-private-JJkv1V/core.5306 /tmp/systemd-private-JJkv1V/core.5308 /tmp/systemd-private-JJkv1V/core.5307 /tmp/systemd-private-JJkv1V/core.5304 /tmp/systemd-private-JJkv1V/core.5305 # cat /var/log/httpd/error_log [Thu Apr 04 03:35:48.149713 2013] [core:notice] [pid 5303] core dump file size limit raised to 18446744073709551615 bytes [Thu Apr 04 03:35:48.150686 2013] [core:notice] [pid 5303] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Thu Apr 04 03:35:49.001561 2013] [ssl:notice] [pid 5303] AH01886: SSL FIPS mode disabled [Thu Apr 04 03:35:49.147944 2013] [auth_digest:notice] [pid 5303] AH01757: generating secret for digest authentication ... [Thu Apr 04 03:35:50.001376 2013] [ssl:notice] [pid 5303] AH01886: SSL FIPS mode disabled [Thu Apr 04 03:35:50.141728 2013] [mpm_prefork:notice] [pid 5303] AH00163: Apache/2.4.4 (Fedora) OpenSSL/1.0.1e-fips configured -- resuming normal operations [Thu Apr 04 03:35:50.148173 2013] [core:notice] [pid 5303] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu Apr 04 03:35:52.684792 2013] [mpm_prefork:notice] [pid 5303] AH00170: caught SIGWINCH, shutting down gracefully Sorry, timezone was misconfigured. Actually, is around 10:48 here. Hi, I have another candidate here: http://koji.fedoraproject.org/koji/taskinfo?taskID=5252585 It would be great to try this one and write here if it still freezes/crashes when stopping httpd using systemd. Seems to work fine. No more segfaults. Thank you. Well, does not work. I still have core dumps and it still freezes. # find /tmp -name core\* /tmp/core.11920 /tmp/core.12041 /tmp/core.12089 On first computer does not generate core dumps. On the second computer I have coredumps and sometimes restart freeze. (gdb) where
#0 0x00007f7b58bc2ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63
#1 0x00007f7b58bc4358 in __GI_abort () at abort.c:90
#2 0x00007f7b58c0259b in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f7b58d06ba8 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:197
#3 0x00007f7b58c09a8e in malloc_printerr (ptr=0x7f7b5c279180, str=0x7f7b58d04a01 "free(): invalid pointer", action=3)
at malloc.c:4969
#4 _int_free (av=0x7f7b58f3e740 <main_arena>, p=0x7f7b5c279170, have_lock=0) at malloc.c:3826
#5 0x00007f7b5937e85c in allocator_free (node=0x7f7b5c279180, allocator=0x7f7b5c43a2c0) at memory/unix/apr_pools.c:430
#6 apr_pool_destroy (pool=0x7f7b5c43a3b8) at memory/unix/apr_pools.c:856
#7 0x00007f7b53a461be in clean_child_exit (code=code@entry=0) at prefork.c:218
#8 0x00007f7b53a469fb in just_die (sig=<optimized out>) at prefork.c:344
#9 <signal handler called>
#10 cleanup_fscreatecon () at selinux.c:78
#11 0x00007f7b5a45d5f7 in _dl_fini () at dl-fini.c:253
#12 0x00007f7b58bc5df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f7b58f3e6a8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#13 0x00007f7b58bc5e75 in __GI_exit (status=status@entry=0) at exit.c:99
#14 0x00007f7b53a461db in clean_child_exit (code=code@entry=0) at prefork.c:227
#15 0x00007f7b53a4660e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:725
#16 0x00007f7b53a4697c in make_child (s=0x7f7b5c0c7368, slot=slot@entry=1) at prefork.c:800
#17 0x00007f7b53a469d6 in startup_children (number_to_start=4) at prefork.c:818
#18 0x00007f7b53a47886 in prefork_run (_pconf=<optimized out>, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at prefork.c:976
#19 0x00007f7b5a695d5e in ap_run_mpm (pconf=0x7f7b5c09c158, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at mpm_common.c:98
#20 0x00007f7b5a68f40a in main (argc=2, argv=0x7fff2f64eb88) at main.c:777
(gdb) bt full
#0 0x00007f7b58bc2ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63
resultvar = 0
pid = 11920
selftid = 11920
#1 0x00007f7b58bc4358 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7f7b58d04bc1, sa_sigaction = 0x7f7b58d04bc1}, sa_mask = {__val = {
3, 140733988525067, 5, 140167747747220, 1, 140167747753918, 3, 140733988525044, 12, 140167747753922, 2,
140167747753922, 2, 140733988525856, 19, 140733988527616}}, sa_flags = 88, sa_restorer = 0x7}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007f7b58c0259b in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f7b58d06ba8 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:197
ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fff2f64de10, reg_save_area = 0x7fff2f64dd20}}
ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7fff2f64de10,
reg_save_area = 0x7fff2f64dd20}}
fd = 2
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
#3 0x00007f7b58c09a8e in malloc_printerr (ptr=0x7f7b5c279180, str=0x7f7b58d04a01 "free(): invalid pointer", action=3)
at malloc.c:4969
buf = "00007f7b5c279180"
cp = <optimized out>
#4 _int_free (av=0x7f7b58f3e740 <main_arena>, p=0x7f7b5c279170, have_lock=0) at malloc.c:3826
size = <optimized out>
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
errstr = 0x7f7b58d04a01 "free(): invalid pointer"
locked = <optimized out>
#5 0x00007f7b5937e85c in allocator_free (node=0x7f7b5c279180, allocator=0x7f7b5c43a2c0) at memory/unix/apr_pools.c:430
freelist = 0x0
max_index = <optimized out>
max_free_index = <optimized out>
next = <optimized out>
index = <optimized out>
current_free_index = <optimized out>
#6 apr_pool_destroy (pool=0x7f7b5c43a3b8) at memory/unix/apr_pools.c:856
active = <optimized out>
allocator = 0x7f7b5c43a2c0
#7 0x00007f7b53a461be in clean_child_exit (code=code@entry=0) at prefork.c:218
No locals.
#8 0x00007f7b53a469fb in just_die (sig=<optimized out>) at prefork.c:344
No locals.
#9 <signal handler called>
No locals.
#10 cleanup_fscreatecon () at selinux.c:78
No locals.
#11 0x00007f7b5a45d5f7 in _dl_fini () at dl-fini.c:253
array = 0x7f7b52889cc8
i = 2
nmaps = 76
nloaded = <optimized out>
i = 53
l = 0x7f7b5c260450
ns = 0
maps = 0x7fff2f64e390
maps_size = 608
do_audit = 0
#12 0x00007f7b58bc5df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f7b58f3e6a8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:77
atfct = <optimized out>
onfct = <optimized out>
cxafct = <optimized out>
f = <optimized out>
#13 0x00007f7b58bc5e75 in __GI_exit (status=status@entry=0) at exit.c:99
No locals.
#14 0x00007f7b53a461db in clean_child_exit (code=code@entry=0) at prefork.c:227
No locals.
#15 0x00007f7b53a4660e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:725
thd = 0x7f7b5c43a430
osthd = 140167774263360
ptrans = 0x7f7b5c43c3c8
allocator = 0x7f7b5c43a2c0
status = <optimized out>
i = <optimized out>
lr = <optimized out>
pollset = 0x7f7b5c43a510
sbh = 0x7f7b5c43a508
bucket_alloc = 0x7f7b5c4403e8
last_poll_idx = <optimized out>
lockfile = <optimized out>
#16 0x00007f7b53a4697c in make_child (s=0x7f7b5c0c7368, slot=slot@entry=1) at prefork.c:800
pid = 0
#17 0x00007f7b53a469d6 in startup_children (number_to_start=4) at prefork.c:818
i = 1
#18 0x00007f7b53a47886 in prefork_run (_pconf=<optimized out>, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at prefork.c:976
index = <optimized out>
remaining_children_to_start = <optimized out>
rv = <optimized out>
#19 0x00007f7b5a695d5e in ap_run_mpm (pconf=0x7f7b5c09c158, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at mpm_common.c:98
pHook = 0x7f7b5c16dce0
n = 0
rv = 1384684992
#20 0x00007f7b5a68f40a in main (argc=2, argv=0x7fff2f64eb88) at main.c:777
c = 68 'D'
showcompile = 0
showdirectives = 0
confname = 0x7f7b5a6ca2af "conf/httpd.conf"
def_server_root = 0x7f7b5a6ca2a4 "/etc/httpd"
temp_error_log = <optimized out>
error = <optimized out>
process = 0x7f7b5c09a238
pconf = 0x7f7b5c09c158
plog = 0x7f7b5c0f94f8
ptemp = 0x7f7b5c0c5318
pcommands = 0x7f7b5c0be268
opt = 0x7f7b5c0be358
rv = <optimized out>
mod = 0x7f7b5a8e5098 <ap_prelinked_modules+24>
opt_arg = 0x7fff2f64ef33 "FOREGROUND"
signal_server = <optimized out>
Oh, I see... You added KillSignal=SIGWINCH to httpd.service On the machine where I still have coredumps I have a custom httpd.service (I temporarily disabled PrivateTmp). If you ask me, is an ugly workaround. Still, at least I do not have coredumps anymore: # watch -n 5 "service httpd start; sleep 5; /usr/sbin/httpd -k graceful-stop" So with KillSignal=SIGWINCH and KillMode=process it does work for you (with the updated httpd.service)? I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. It crashes/freezes, because SIGTERM signal interrupts SIGWINCH signal handler which can be in the middle of malloc/free and those functions are not reentrant. The best way would be to have httpd which can handle signals in any order or at any time, but unfortunately we don't have it because of httpd's design. I can try to open discussion about that on upstream mailing list, but before we make some consensus, I would like to fix it this way, so if you could try that changed httpd.service works for you, it would be great. I think this is better workaround, because if graceful-stop fails for any reason, then systemd will still SIGKILL it after 90 seconds: [Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target [Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # grafecul stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true [Install] WantedBy=multi-user.target > So with KillSignal=SIGWINCH and KillMode=process it does work for you (with the updated httpd.service)? Yes, the new httpd.serviceit works fine. > I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. Then this seems like a systemd bug. The unit from comment #33 seems is a lot better. I like-it. Thx. > > I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. > Then this seems like a systemd bug. You can follow up here if you want: https://bugzilla.redhat.com/show_bug.cgi?id=952634 Thank you. httpd-2.4.4-4.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/httpd-2.4.4-4.fc19 Thank you. Package httpd-2.4.4-4.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing httpd-2.4.4-4.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-5797/httpd-2.4.4-4.fc19 then log in and leave karma (feedback). httpd-2.4.4-4.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report. *** Bug 909703 has been marked as a duplicate of this bug. *** httpd-2.4.4-3.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/httpd-2.4.4-3.fc18 httpd-2.4.4-3.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report. |