Description of problem: httpd most of the time crash when stoped. *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f327b28e450 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x300087ca8e)[0x7f3277b87a8e] /lib64/libapr-1.so.0(apr_pool_destroy+0x19c)[0x7f32782fc85c] /etc/httpd/modules/mod_mpm_prefork.so(+0x31be)[0x7f326f1fd1be] /etc/httpd/modules/mod_mpm_prefork.so(+0x39fb)[0x7f326f1fd9fb] /lib64/libpthread.so.0(+0x300100f000)[0x7f32780d6000] /lib64/ld-linux-x86-64.so.2(+0x300000f541)[0x7f32793db541] /lib64/libc.so.6(+0x3000838df1)[0x7f3277b43df1] /lib64/libc.so.6(+0x3000838e75)[0x7f3277b43e75] /etc/httpd/modules/mod_mpm_prefork.so(+0x31db)[0x7f326f1fd1db] /etc/httpd/modules/mod_mpm_prefork.so(+0x360e)[0x7f326f1fd60e] /etc/httpd/modules/mod_mpm_prefork.so(+0x397c)[0x7f326f1fd97c] /etc/httpd/modules/mod_mpm_prefork.so(+0x47be)[0x7f326f1fe7be] /usr/sbin/httpd(ap_run_mpm+0x4e)[0x7f327961376e] /usr/sbin/httpd(main+0xa5a)[0x7f327960d23a] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3277b2ca05] /usr/sbin/httpd(+0x1e375)[0x7f327960d375] ======= Memory map: ======== 7f325c000000-7f325c021000 rw-p 00000000 00:00 0 7f325c021000-7f3260000000 ---p 00000000 00:00 0 7f32634e2000-7f32634e3000 ---p 00000000 00:00 0 7f32634e3000-7f3263ce3000 rw-p 00000000 00:00 0 7f326505a000-7f326513f000 r-xp 00000000 08:02 36559998 /usr/lib64/libstdc++.so.6.0.17 7f326513f000-7f326533e000 ---p 000e5000 08:02 36559998 /usr/lib64/libstdc++.so.6.0.17 7f326533e000-7f3265346000 r--p 000e4000 08:02 36559998 /usr/lib64/libstdc++.so.6.0.17 7f3265346000-7f3265348000 rw-p 000ec000 08:02 36559998 /usr/lib64/libstdc++.so.6.0.17 7f3265348000-7f326535d000 rw-p 00000000 00:00 0 7f3269f05000-7f3269f11000 r-xp 00000000 08:02 33789228 /usr/lib64/libnss_files-2.16.so 7f3269f11000-7f326a110000 ---p 0000c000 08:02 33789228 /usr/lib64/libnss_files-2.16.so 7f326a110000-7f326a111000 r--p 0000b000 08:02 33789228 /usr/lib64/libnss_files-2.16.so 7f326a111000-7f326a112000 rw-p 0000c000 08:02 33789228 /usr/lib64/libnss_files-2.16.so 7f326a112000-7f326a135000 r-xp 00000000 08:02 37327143 /usr/lib64/liblzma.so.5.0.99 7f326a135000-7f326a335000 ---p 00023000 08:02 37327143 /usr/lib64/liblzma.so.5.0.99 7f326a335000-7f326a336000 r--p 00023000 08:02 37327143 /usr/lib64/liblzma.so.5.0.99 7f326a336000-7f326a337000 rw-p 00024000 08:02 37327143 /usr/lib64/liblzma.so.5.0.99 7f326a337000-7f326a490000 r-xp 00000000 08:02 37327153 /usr/lib64/libxml2.so.2.9.0 7f326a490000-7f326a68f000 ---p 00159000 08:02 37327153 /usr/lib64/libxml2.so.2.9.0 7f326a68f000-7f326a697000 r--p 00158000 08:02 37327153 /usr/lib64/libxml2.so.2.9.0 7f326a697000-7f326a699000 rw-p 00160000 08:02 37327153 /usr/lib64/libxml2.so.2.9.0 7f326a699000-7f326a69b000 rw-p 00000000 00:00 0 7f326a69b000-7f326a6b1000 r-xp 00000000 08:02 37322526 /usr/lib64/libnsl-2.16.so 7f326a6b1000-7f326a8b0000 ---p 00016000 08:02 37322526 /usr/lib64/libnsl-2.16.so 7f326a8b0000-7f326a8b1000 r--p 00015000 08:02 37322526 /usr/lib64/libnsl-2.16.so 7f326a8b1000-7f326a8b2000 rw-p 00016000 08:02 37322526 /usr/lib64/libnsl-2.16.so 7f326a8b2000-7f326a8b4000 rw-p 00000000 00:00 0 7f326a8b4000-7f326a8c3000 r-xp 00000000 08:02 37327142 /usr/lib64/libbz2.so.1.0.6 7f326a8c3000-7f326aac2000 ---p 0000f000 08:02 37327142 /usr/lib64/libbz2.so.1.0.6 7f326aac2000-7f326aac3000 r--p 0000e000 08:02 37327142 /usr/lib64/libbz2.so.1.0.6 7f326aac3000-7f326aac4000 rw-p 0000f000 08:02 37327142 /usr/lib64/libbz2.so.1.0.6 7f326aac4000-7f326ab29000 r-xp 00000000 08:02 33789376 /usr/lib64/libgmp.so.10.0.5 7f326ab29000-7f326ad28000 ---p 00065000 08:02 33789376 /usr/lib64/libgmp.so.10.0.5 7f326ad28000-7f326ad29000 r--p 00064000 08:02 33789376 /usr/lib64/libgmp.so.10.0.5 7f326ad29000-7f326ad32000 rw-p 00065000 08:02 33789376 /usr/lib64/libgmp.so.10.0.5 7f326ad32000-7f326b0dd000 r-xp 00000000 08:02 1762089 /usr/lib64/httpd/modules/libphp5.so 7f326b0dd000-7f326b2dc000 ---p 003ab000 08:02 1762089 /usr/lib64/httpd/modules/libphp5.so 7f326b2dc000-7f326b368000 r--p 003aa000 08:02 1762089 /usr/lib64/httpd/modules/libphp5.so 7f326b368000-7f326b370000 rw-p 00436000 08:02 1762089 /usr/lib64/httpd/modules/libphp5.so 7f326b370000-7f326b38b000 rw-p 00000000 00:00 0 7f326b38b000-7f326b391000 r-xp 00000000 08:02 1779191 /usr/lib64/httpd/modules/mod_cgi.so 7f326b391000-7f326b590000 ---p 00006000 08:02 1779191 /usr/lib64/httpd/modules/mod_cgi.so 7f326b590000-7f326b591000 r--p 00005000 08:02 1779191 /usr/lib64/httpd/modules/mod_cgi.so 7f326b591000-7f326b592000 rw-p 00006000 08:02 1779191 /usr/lib64/httpd/modules/mod_cgi.so 7f326b592000-7f326b599000 r-xp 00000000 08:02 37152373 /usr/lib64/librt-2.16.so 7f326b599000-7f326b798000 ---p 00007000 08:02 37152373 /usr/lib64/librt-2.16.so 7f326b798000-7f326b799000 r--p 00006000 08:02 37152373 /usr/lib64/librt-2.16.so 7f326b799000-7f326b79a000 rw-p 00007000 08:02 37152373 /usr/lib64/librt-2.16.so 7f326b79a000-7f326b79c000 r-xp 00000000 08:02 36571443 /usr/lib64/libsystemd-daemon.so.0.0.7 7f326b79c000-7f326b99c000 ---p 00002000 08:02 36571443 /usr/lib64/libsystemd-daemon.so.0.0.7 7f326b99c000-7f326b99d000 r--p 00002000 08:02 36571443 /usr/lib64/libsystemd-daemon.so.0.0.7 7f326b99d000-7f326b99e000 rw-p 00000000 00:00 0 7f326b99e000-7f326b9a0000 r-xp 00000000 08:02 2086341 /usr/lib64/httpd/modules/mod_systemd.so 7f326b9a0000-7f326bb9f000 ---p 00002000 08:02 2086341 /usr/lib64/httpd/modules/mod_systemd.so 7f326bb9f000-7f326bba0000 r--p 00001000 08:02 2086341 /usr/lib64/httpd/modules/mod_systemd.so 7f326bba0000-7f326bba1000 rw-p 00002000 08:02 2086341 /usr/lib64/httpd/modules/mod_systemd.so 7f326bba1000-7f326bbb6000 r-xp 00000000 08:02 36559996 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 7f326bbb6000-7f326bdb5000 ---p 00015000 08:02 36559996 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 7f326bdb5000-7f326bdb6000 r--p 00014000 08:02 36559996 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 7f326bdb6000-7f326bdb7000 rw-p 00015000 08:02 36559996 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 7f326bdb7000-7f326bdcd000 r-xp 00000000 08:02 36640783 /usr/lib64/libresolv-2.16.so 7f326bdcd000-7f326bfcc000 ---p 00016000 08:02 36640783 /usr/lib64/libresolv-2.16.so 7f326bfcc000-7f326bfcd000 r--p 00015000 08:02 36640783 /usr/lib64/libresolv-2.16.so 7f326bfcd000-7f326bfce000 rw-p 00016000 08:02 36640783 /usr/lib64/libresolv-2.16.so 7f326bfce000-7f326bfd0000 rw-p 00000000 00:00 0 7f326bfd0000-7f326bfd3000 r-xp 00000000 08:02 36889888 /usr/lib64/libkeyutils.so.1.4 7f326bfd3000-7f326c1d2000 ---p 00003000 08:02 36889888 /usr/lib64/libkeyutils.so.1.4 7f326c1d2000-7f326c1d3000 r--p 00002000 08:02 36889888 /usr/lib64/libkeyutils.so.1.4 7f326c1d3000-7f326c1d4000 rw-p 00003000 08:02 36889888 /usr/lib64/libkeyutils.so.1.4 7f326c1d4000-7f326c1de000 r-xp 00000000 08:02 33797581 /usr/lib64/libkrb5support.so.0.1 7f326c1de000-7f326c3dd000 ---p 0000a000 08:02 33797581 /usr/lib64/libkrb5support.so.0.1 7f326c3dd000-7f326c3de000 r--p 00009000 08:02 33797581 /usr/lib64/libkrb5support.so.0.1 7f326c3de000-7f326c3df000 rw-p 0000a000 08:02 33797581 /usr/lib64/libkrb5support.so.0.1 7f326c3df000-7f326c407000 r-xp 00000000 08:02 33797583 /usr/lib64/libk5crypto.so.3.1 7f326c407000-7f326c607000 ---p 00028000 08:02 33797583 /usr/lib64/libk5crypto.so.3.1 7f326c607000-7f326c608000 r--p 00028000 08:02 33797583 /usr/lib64/libk5crypto.so.3.1 7f326c608000-7f326c609000 rw-p 00029000 08:02 33797583 /usr/lib64/libk5crypto.so.3.1 7f326c609000-7f326c60a000 rw-p 00000000 00:00 0 7f326c60a000-7f326c60d000 r-xp 00000000 08:02 37077867 /usr/lib64/libcom_err.so.2.1 7f326c60d000-7f326c80c000 ---p 00003000 08:02 37077867 /usr/lib64/libcom_err.so.2.1 7f326c80c000-7f326c80d000 r--p 00002000 08:02 37077867 /usr/lib64/libcom_err.so.2.1 7f326c80d000-7f326c80e000 rw-p 00003000 08:02 37077867 /usr/lib64/libcom_err.so.2.1 7f326c80e000-7f326c8e7000 r-xp 00000000 08:02 33797584 /usr/lib64/libkrb5.so.3.3 7f326c8e7000-7f326cae7000 ---p 000d9000 08:02 33797584 /usr/lib64/libkrb5.so.3.3 7f326cae7000-7f326caf1000 r--p 000d9000 08:02 33797584 /usr/lib64/libkrb5.so.3.3 7f326caf1000-7f326caf3000 rw-p 000e3000 08:02 33797584 /usr/lib64/libkrb5.so.3.3 7f326caf3000-7f326cb34000 r-xp 00000000 08:02 33797585 /usr/lib64/libgssapi_krb5.so.2.2 7f326cb34000-7f326cd33000 ---p 00041000 08:02 33797585 /usr/lib64/libgssapi_krb5.so.2.2 7f326cd33000-7f326cd34000 r--p 00040000 08:02 33797585 /usr/lib64/libgssapi_krb5.so.2.2 7f326cd34000-7f326cd36000 rw-p 00041000 08:02 33797585 /usr/lib64/libgssapi_krb5.so.2.2 7f326cd36000-7f326ced1000 r-xp 00000000 08:02 37327135 /usr/lib64/libcrypto.so.1.0.1c 7f326ced1000-7f326d0d0000 ---p 0019b000 08:02 37327135 /usr/lib64/libcrypto.so.1.0.1c 7f326d0d0000-7f326d0ea000 r--p 0019a000 08:02 37327135 /usr/lib64/libcrypto.so.1.0.1c 7f326d0ea000-7f326d0f5000 rw-p 001b4000 08:02 37327135 /usr/lib64/libcrypto.so.1.0.1c 7f326d0f5000-7f326d0fa000 rw-p 00000000 00:00 0 7f326d0fa000-7f326d155000 r-xp 00000000 08:02 33796536 /usr/lib64/libssl.so.1.0.1c 7f326d155000-7f326d354000 ---p 0005b000 08:02 33796536 /usr/lib64/libssl.so.1.0.1c 7f326d354000-7f326d358000 r--p 0005a000 08:02 33796536 /usr/lib64/libssl.so.1.0.1c 7f326d358000-7f326d35e000 rw-p 0005e000 08:02 33796536 /usr/lib64/libssl.so.1.0.1c 7f326d35e000-7f326d38f000 r-xp 00000000 08:02 1761313 /usr/lib64/httpd/modules/mod_ssl.so 7f326d38f000-7f326d58f000 ---p 00031000 08:02 1761313 /usr/lib64/httpd/modules/mod_ssl.so 7f326d58f000-7f326d591000 r--p 00031000 08:02 1761313 /usr/lib64/httpd/modules/mod_ssl.so 7f326d591000-7f326d592000 rw-p 00033000 08:02 1761313 /usr/lib64/httpd/modules/mod_ssl.so 7f326d592000-7f326d594000 rw-p 00000000 00:00 0 7f326d594000-7f326d598000 r-xp 00000000 08:02 1761366 /usr/lib64/httpd/modules/mod_proxy_scgi.so 7f326d598000-7f326d797000 ---p 00004000 08:02 1761366 /usr/lib64/httpd/modules/mod_proxy_scgi.so 7f326d797000-7f326d798000 r--p 00003000 08:02 1761366 /usr/lib64/httpd/modules/mod_proxy_scgi.so 7f326d798000-7f326d799000 rw-p 00004000 08:02 1761366 /usr/lib64/httpd/modules/mod_proxy_scgi.so 7f326d799000-7f326d7a2000 r-xp 00000000 08:02 1761363 /usr/lib64/httpd/modules/mod_proxy_http.so 7f326d7a2000-7f326d9a1000 ---p 00009000 08:02 1761363 /usr/lib64/httpd/modules/mod_proxy_http.so 7f326d9a1000-7f326d9a2000 r--p 00008000 08:02 1761363 /usr/lib64/httpd/modules/mod_proxy_http.so 7f326d9a2000-7f326d9a3000 rw-p 00009000 08:02 1761363 /usr/lib64/httpd/modules/mod_proxy_http.so 7f326d9a3000-7f326d9ad000 r-xp 00000000 08:02 1761356 /usr/lib64/httpd/modules/mod_proxy_ftp.so 7f326d9ad000-7f326dbac000 ---p 0000a000 08:02 1761356 /usr/lib64/httpd/modules/mod_proxy_ftp.so 7f326dbac000-7f326dbad000 r--p 00009000 08:02 1761356 /usr/lib64/httpd/modules/mod_proxy_ftp.so 7f326dbad000-7f326dbae000 rw-p 0000a000 08:02 1761356 /usr/lib64/httpd/modules/mod_proxy_ftp.so 7f326dbae000-7f326dbb0000 r-xp 00000000 08:02 1761349 /usr/lib64/httpd/modules/mod_proxy_fdpass.so 7f326dbb0000-7f326ddaf000 ---p 00002000 08:02 1761349 /usr/lib64/httpd/modules/mod_proxy_fdpass.so 7f326ddaf000-7f326ddb0000 r--p 00001000 08:02 1761349 /usr/lib64/httpd/modules/mod_proxy_fdpass.so 7f326ddb0000-7f326ddb1000 rw-p 00002000 08:02 1761349 /usr/lib64/httpd/modules/mod_proxy_fdpass.so 7f326ddb1000-7f326ddb5000 r-xp 00000000 08:02 1761348 /usr/lib64/httpd/modules/mod_proxy_fcgi.so 7f326ddb5000-7f326dfb4000 ---p 00004000 08:02 1761348 /usr/lib64/httpd/modules/mod_proxy_fcgi.so 7f326dfb4000-7f326dfb5000 r--p 00003000 08:02 1761348 /usr/lib64/httpd/modules/mod_proxy_fcgi.so 7f326dfb5000-7f326dfb6000 rw-p 00004000 08:02 1761348 /usr/lib64/httpd/modules/mod_proxy_fcgi.so 7f326dfb6000-7f326dfb8000 r-xp 00000000 08:02 1761344 /usr/lib64/httpd/modules/mod_proxy_express.so 7f326dfb8000-7f326e1b7000 ---p 00002000 08:02 1761344 /usr/lib64/httpd/modules/mod_proxy_express.so 7f326e1b7000-7f326e1b8000 r--p 00001000 08:02 1761344 /usr/lib64/httpd/modules/mod_proxy_express.so 7f326e1b8000-7f326e1b9000 rw-p 00002000 08:02 1761344 /usr/lib64/httpd/modules/mod_proxy_express.so 7f326e1b9000-7f326e1bd000 r-xp 00000000 08:02 1730079 /usr/lib64/httpd/modules/mod_proxy_connect.so 7f326e1bd000-7f326e3bc000 ---p 00004000 08:02 1730079 /usr/lib64/httpd/modules/mod_proxy_connect.so 7f326e3bc000-7f326e3bd000 r--p 00003000 08:02 1730079 /usr/lib64/httpd/modules/mod_proxy_connect.so 7f326e3bd000-7f326e3be000 rw-p 00004000 08:02 1730079 /usr/lib64/httpd/modules/mod_proxy_connect.so 7f326e3be000-7f326e3c7000 r-xp 00000000 08:02 1730078 /usr/lib64/httpd/modules/mod_proxy_balancer.so 7f326e3c7000-7f326e5c6000 ---p 00009000 08:02 1730078 /usr/lib64/httpd/modules/mod_proxy_balancer.so 7f326e5c6000-7f326e5c7000 r--p 00008000 08:02 1730078 /usr/lib64/httpd/modules/mod_proxy_balancer.so 7f326e5c7000-7f326e5c8000 rw-p 00009000 08:02 1730078 /usr/lib64/httpd/modules/mod_proxy_balancer.so 7f326e5c8000-7f326e5d4000 r-xp 00000000 08:02 1730077 /usr/lib64/httpd/modules/mod_proxy_ajp.so 7f326e5d4000-7f326e7d3000 ---p 0000c000 08:02 1730077 /usr/lib64/httpd/modules/mod_proxy_ajp.so 7f326e7d3000-7f326e7d4000 r--p 0000b000 08:02 1730077 /usr/lib64/httpd/modules/mod_proxy_ajp.so 7f326e7d4000-7f326e7d5000 rw-p 0000c000 08:02 1730077 /usr/lib64/httpd/modules/mod_proxy_ajp.so 7f326e7d5000-7f326e7d8000 r-xp 00000000 08:02 1730064 /usr/lib64/httpd/modules/mod_lbmethod_heartbeat.so 7f326e7d8000-7f326e9d7000 ---p 00003000 08:02 1730064 /usr/lib64/httpd/modules/mod_lbmethod_heartbeat.so 7f326e9d7000-7f326e9d8000 r--p 00002000 08:02 1730064 /usr/lib64/httpd/modules/mod_lbmethod_heartbeat.so 7f326e9d8000-7f326e9d9000 rw-p 00003000 08:02 1730064 /usr/lib64/httpd/modules/mod_lbmethod_heartbeat.so 7f326e9d9000-7f326e9da000 r-xp 00000000 08:02 1730063 /usr/lib64/httpd/modules/mod_lbmethod_bytraffic.so 7f326e9da000-7f326ebda000 ---p 00001000 08:02 1730063 /usr/lib64/httpd/modules/mod_lbmethod_bytraffic.so 7f326ebda000-7f326ebdb000 r--p 00001000 08:02 1730063 /usr/lib64/httpd/modules/mod_lbmethod_bytraffic.so 7f326ebdb000-7f326ebdc000 rw-p 00002000 08:02 1730063 /usr/lib64/httpd/modules/mod_lbmethod_bytraffic.so 7f326ebdc000-7f326ebdd000 r-xp 00000000 08:02 1730062 /usr/lib64/httpd/modules/mod_lbmethod_byrequests.so 7f326ebdd000-7f326eddd000 ---p 00001000 08:02 1730062 /usr/lib64/httpd/modules/mod_lbmethod_byrequests.so 7f326eddd000-7f326edde000 r--p 00001000 08:02 1730062 /usr/lib64/httpd/modules/mod_lbmethod_byrequests.so 7f326edde000-7f326eddf000 rw-p 00002000 08:02 1730062 /usr/lib64/httpd/modules/mod_lbmethod_byrequests.so 7f326eddf000-7f326ede0000 r-xp 00000000 08:02 1730061 /usr/lib64/httpd/modules/mod_lbmethod_bybusyness.so 7f326ede0000-7f326efe0000 ---p 00001000 08:02 1730061 /usr/lib64/httpd/modules/mod_lbmethod_bybusyness.so 7f326efe0000-7f326efe1000 r--p 00001000 08:02 1730061 /usr/lib64/httpd/modules/mod_lbmethod_bybusyness.so 7f326efe1000-7f326efe2000 rw-p 00002000 08:02 1730061 /usr/lib64/httpd/modules/mod_lbmethod_bybusyness.so 7f326efe2000-7f326eff9000 r-xp 00000000 08:02 1730076 /usr/lib64/httpd/modules/mod_proxy.so 7f326eff9000-7f326f1f8000 ---p 00017000 08:02 1730076 /usr/lib64/httpd/modules/mod_proxy.so 7f326f1f8000-7f326f1f9000 r--p 00016000 08:02 1730076 /usr/lib64/httpd/modules/mod_proxy.so 7f326f1f9000-7f326f1fa000 rw-p 00017000 08:02 1730076 /usr/lib64/httpd/modules/mod_proxy.so 7f326f1fa000-7f326f201000 r-xp 00000000 08:02 1730073 /usr/lib64/httpd/modules/mod_mpm_prefork.so 7f326f201000-7f326f400000 ---p 00007000 08:02 1730073 /usr/lib64/httpd/modules/mod_mpm_prefork.so 7f326f400000-7f326f401000 r--p 00006000 08:02 1730073 /usr/lib64/httpd/modules/mod_mpm_prefork.so 7f326f401000-7f326f402000 rw-p 00007000 08:02 1730073 /usr/lib64/httpd/modules/mod_mpm_prefork.so 7f326f402000-7f326f502000 r-xp 00000000 08:02 36088606 /usr/lib64/libm-2.16.so 7f326f502000-7f326f701000 ---p 00100000 08:02 36088606 /usr/lib64/libm-2.16.so 7f326f701000-7f326f702000 r--p 000ff000 08:02 36088606 /usr/lib64/libm-2.16.so 7f326f702000-7f326f703000 rw-p 00100000 08:02 36088606 /usr/lib64/libm-2.16.so 7f326f703000-7f326f72e000 r-xp 00000000 08:02 37541117 /usr/lib64/liblua-5.1.so 7f326f72e000-7f326f92d000 ---p 0002b000 08:02 37541117 /usr/lib64/liblua-5.1.so 7f326f92d000-7f326f92f000 r--p 0002a000 08:02 37541117 /usr/lib64/liblua-5.1.so 7f326f92f000-7f326f930000 rw-p 0002c000 08:02 37541117 /usr/lib64/liblua-5.1.so 7f326f930000-7f326f93e000 r-xp 00000000 08:02 1730069 /usr/lib64/httpd/modules/mod_lua.so 7f326f93e000-7f326fb3d000 ---p 0000e000 08:02 1730069 /usr/lib64/httpd/modules/mod_lua.so 7f326fb3d000-7f326fb3e000 r--p 0000d000 08:02 1730069 /usr/lib64/httpd/modules/mod_lua.so 7f326fb3e000-7f326fb3f000 rw-p 0000e000 08:02 1730069 /usr/lib64/httpd/modules/mod_lua.so 7f326fb3f000-7f326fb43000 r-xp 00000000 08:02 1779197 /usr/lib64/httpd/modules/mod_dav_lock.so 7f326fb43000-7f326fd42000 ---p 00004000 08:02 1779197 /usr/lib64/httpd/modules/mod_dav_lock.so 7f326fd42000-7f326fd43000 r--p 00003000 08:02 1779197 /usr/lib64/httpd/modules/mod_dav_lock.so 7f326fd43000-7f326fd44000 rw-p 00004000 08:02 1779197 /usr/lib64/httpd/modules/mod_dav_lock.so 7f326fd44000-7f326fd50000 r-xp 00000000 08:02 1779196 /usr/lib64/httpd/modules/mod_dav_fs.so 7f326fd50000-7f326ff4f000 ---p 0000c000 08:02 1779196 /usr/lib64/httpd/modules/mod_dav_fs.so 7f326ff4f000-7f326ff50000 r--p 0000b000 08:02 1779196 /usr/lib64/httpd/modules/mod_dav_fs.so 7f326ff50000-7f326ff51000 rw-p 0000c000 08:02 1779196 /usr/lib64/httpd/modules/mod_dav_fs.so 7f326ff51000-7f326ff67000 r-xp 00000000 08:02 1779195 /usr/lib64/httpd/modules/mod_dav.so 7f326ff67000-7f3270167000 ---p 00016000 08:02 1779195 /usr/lib64/httpd/modules/mod_dav.so 7f3270167000-7f3270168000 r--p 00016000 08:02 1779195 /usr/lib64/httpd/modules/mod_dav.so 7f3270168000-7f3270169000 rw-p 00017000 08:02 1779195 /usr/lib64/httpd/modules/mod_dav.so 7f3270169000-7f327016b000 r-xp 00000000 08:02 2090877 /usr/lib64/httpd/modules/mod_vhost_alias.so 7f327016b000-7f327036a000 ---p 00002000 08:02 2090877 /usr/lib64/httpd/modules/mod_vhost_alias.so 7f327036a000-7f327036b000 r--p 00001000 08:02 2090877 /usr/lib64/httpd/modules/mod_vhost_alias.so 7f327036b000-7f327036c000 rw-p 00002000 08:02 2090877 /usr/lib64/httpd/modules/mod_vhost_alias.so 7f327036c000-7f327036e000 r-xp 00000000 08:02 1761288 /usr/lib64/httpd/modules/mod_version.so 7f327036e000-7f327056d000 ---p 00002000 08:02 1761288 /usr/lib64/httpd/modules/mod_version.so 7f327056d000-7f327056e000 r--p 00001000 08:02 1761288 /usr/lib64/httpd/modules/mod_version.so 7f327056e000-7f327056f000 rw-p 00002000 08:02 1761288 /usr/lib64/httpd/modules/mod_version.so 7f327056f000-7f3270571000 r-xp 00000000 08:02 2090875 /usr/lib64/httpd/modules/mod_userdir.so 7f3270571000-7f3270770000 ---p 00002000 08:02 2090875 /usr/lib64/httpd/modules/mod_userdir.so 7f3270770000-7f3270771000 r--p 00001000 08:02 2090875 /usr/lib64/httpd/modules/mod_userdir.so 7f3270771000-7f3270772000 rw-p 00002000 08:02 2090875 /usr/lib64/httpd/modules/mod_userdir.so 7f3270772000-7f3270775000 r-xp 00000000 08:02 2086464 /usr/lib64/httpd/modules/mod_unixd.so 7f3270775000-7f3270974000 ---p 00003000 08:02 2086464 /usr/lib64/httpd/modules/mod_unixd.so 7f3270974000-7f3270975000 r--p 00002000 08:02 2086464 /usr/lib64/httpd/modules/mod_unixd.so 7f3270975000-7f3270976000 rw-p 00003000 08:02 2086464 /usr/lib64/httpd/modules/mod_unixd.so 7f3270976000-7f3270978000 r-xp 00000000 08:02 2086431 /usr/lib64/httpd/modules/mod_unique_id.so 7f3270978000-7f3270b77000 ---p 00002000 08:02 2086431 /usr/lib64/httpd/modules/mod_unique_id.so 7f3270b77000-7f3270b78000 r--p 00001000 08:02 2086431 /usr/lib64/httpd/modules/mod_unique_id.so 7f3270b78000-7f3270b79000 rw-p 00002000 08:02 2086431 /usr/lib64/httpd/modules/mod_unique_id.so 7f3270b79000-7f3270b7a000 r-xp 00000000 08:02 2067724 /usr/lib64/httpd/modules/mod_suexec.so 7f3270b7a000-7f3270d7a000 ---p 00001000 08:02 2067724 /usr/lib64/httpd/modules/mod_suexec.so 7f3270d7a000-7f3270d7b000 r--p 00001000 08:02 2067724 /usr/lib64/httpd/modules/mod_suexec.so 7f3270d7b000-7f3270d7c000 rw-p 00002000 08:02 2067724 /usr/lib64/httpd/modules/mod_suexec.so 7f3270d7c000-7f3270d7f000 r-xp 00000000 08:02 2067723 /usr/lib64/httpd/modules/mod_substitute.so 7f3270d7f000-7f3270f7e000 ---p 00003000 08:02 2067723 /usr/lib64/httpd/modules/mod_substitute.so 7f3270f7e000-7f3270f7f000 r--p 00002000 08:02 2067723 /usr/lib64/httpd/modules/mod_substitute.so 7f3270f7f000-7f3270f80000 rw-p 00003000 08:02 2067723 /usr/lib64/httpd/modules/mod_substitute.so 7f3270f80000-7f3270f85000 r-xp 00000000 08:02 1761280 /usr/lib64/httpd/modules/mod_status.so 7f3270f85000-7f3271184000 ---p 00005000 08:02 1761280 /usr/lib64/httpd/modules/mod_status.so 7f3271184000-7f3271185000 r--p 00004000 08:02 1761280 /usr/lib64/httpd/modules/mod_status.so 7f3271185000-7f3271186000 rw-p 00005000 08:02 1761280 /usr/lib64/httpd/modules/mod_status.so 7f3271186000-7f327118b000 r-xp 00000000 08:02 2067719 /usr/lib64/httpd/modules/mod_socache_shmcb.so 7f327118b000-7f327138a000 ---p 00005000 08:02 2067719 /usr/lib64/httpd/modules/mod_socache_shmcb.so 7f327138a000-7f327138b000 r--p 00004000 08:02 2067719 /usr/lib64/httpd/modules/mod_socache_shmcb.so 7f327138b000-7f327138c000 rw-p 00005000 08:02 2067719 /usr/lib64/httpd/modules/mod_socache_shmcb.so 7f327138c000-7f327138e000 r-xp 00000000 08:02 2067718 /usr/lib64/httpd/modules/mod_socache_memcache.so 7f327138e000-7f327158d000 ---p 00002000 08:02 2067718 /usr/lib64/httpd/modules/mod_socache_memcache.so 7f327158d000-7f327158e000 r--p 00001000 08:02 2067718 /usr/lib64/httpd/modules/mod_socache_memcache.so 7f327158e000-7f327158f000 rw-p 00002000 08:02 2067718 /usr/lib64/httpd/modules/mod_socache_memcache.so 7f327158f000-7f3271592000 r-xp 00000000 08:02 2025585 /usr/lib64/httpd/modules/mod_socache_dbm.so 7f3271592000-7f3271791000 ---p 00003000 08:02 2025585 /usr/lib64/httpd/modules/mod_socache_dbm.so 7f3271791000-7f3271792000 r--p 00002000 08:02 2025585 /usr/lib64/httpd/modules/mod_socache_dbm.so 7f3271792000-7f3271793000 rw-p 00003000 08:02 2025585 /usr/lib64/httpd/modules/mod_socache_dbm.so 7f3271793000-7f3271796000 r-xp 00000000 08:02 2025582 /usr/lib64/httpd/modules/mod_slotmem_shm.so 7f3271796000-7f3271995000 ---p 00003000 08:02 2025582 /usr/lib64/httpd/modules/mod_slotmem_shm.so 7f3271995000-7f3271996000 r--p 00002000 08:02 2025582 /usr/lib64/httpd/modules/mod_slotmem_shm.so 7f3271996000-7f3271997000 rw-p 00003000 08:02 2025582 /usr/lib64/httpd/modules/mod_slotmem_shm.so 7f3271997000-7f3271999000 r-xp 00000000 08:02 1761401 /usr/lib64/httpd/modules/mod_slotmem_plain.so 7f3271999000-7f3271b98000 ---p 00002000 08:02 1761401 /usr/lib64/httpd/modules/mod_slotmem_plain.so 7f3271b98000-7f3271b99000 r--p 00001000 08:02 1761401 /usr/lib64/httpd/modules/mod_slotmem_plain.so 7f3271b99000-7f3271b9a000 rw-p 00002000 08:02 1761401 /usr/lib64/httpd/modules/mod_slotmem_plain.so 7f3271b9a000-7f3271b9d000 r-xp 00000000 08:02 1761399 /usr/lib64/httpd/modules/mod_setenvif.so 7f3271b9d000-7f3271d9c000 ---p 00003000 08:02 1761399 /usr/lib64/httpd/modules/mod_setenvif.so 7f3271d9c000-7f3271d9d000 r--p 00002000 08:02 1761399 /usr/lib64/httpd/modules/mod_setenvif.so 7f3271d9d000-7f3271d9e000 rw-p 00003000 08:02 1761399 /usr/lib64/httpd/modules/mod_setenvif.so 7f3271d9e000-7f3271dad000 r-xp 00000000 08:02 1761395 /usr/lib64/httpd/modules/mod_rewrite.so 7f3271dad000-7f3271fad000 ---p 0000f000 08:02 1761395 /usr/lib64/httpd/modules/mod_rewrite.so 7f3271fad000-7f3271fae000 r--p 0000f000 08:02 1761395 /usr/lib64/httpd/modules/mod_rewrite.so 7f3271fae000-7f3271faf000 rw-p 00010000 08:02 1761395 /usr/lib64/httpd/modules/mod_rewrite.so 7f3271faf000-7f3271fb2000 r-xp 00000000 08:02 1761381 /usr/lib64/httpd/modules/mod_reqtimeout.so 7f3271fb2000-7f32721b1000 ---p 00003000 08:02 1761381 /usr/lib64/httpd/modules/mod_reqtimeout.so 7f32721b1000-7f32721b2000 r--p 00002000 08:02 1761381 /usr/lib64/httpd/modules/mod_reqtimeout.so 7f32721b2000-7f32721b3000 rw-p 00003000 08:02 1761381 /usr/lib64/httpd/modules/mod_reqtimeout.so 7f32721b3000-7f32721b6000 r-xp 00000000 08:02 1761376 /usr/lib64/httpd/modules/mod_remoteip.so 7f32721b6000-7f32723b5000 ---p 00003000 08:02 1761376 /usr/lib64/httpd/modules/mod_remoteip.so 7f32723b5000-7f32723b6000 r--p 00002000 08:02 1761376 /usr/lib64/httpd/modules/mod_remoteip.so 7f32723b6000-7f32723b7000 rw-p 00003000 08:02 1761376 /usr/lib64/httpd/modules/mod_remoteip.so 7f32723b7000-7f32723be000 r-xp 00000000 08:02 1730075 /usr/lib64/httpd/modules/mod_negotiation.so 7f32723be000-7f32725be000 ---p 00007000 08:02 1730075 /usr/lib64/httpd/modules/mod_negotiation.so 7f32725be000-7f32725bf000 r--p 00007000 08:02 1730075 /usr/lib64/httpd/modules/mod_negotiation.so 7f32725bf000-7f32725c0000 rw-p 00008000 08:02 1730075 /usr/lib64/httpd/modules/mod_negotiation.so 7f32725c0000-7f32725c4000 r-xp 00000000 08:02 1730070 /usr/lib64/httpd/modules/mod_mime.so 7f32725c4000-7f32727c4000 ---p 00004000 08:02 1730070 /usr/lib64/httpd/modules/mod_mime.so 7f32727c4000-7f32727c5000 r--p 00004000 08:02 1730070 /usr/lib64/httpd/modules/mod_mime.so 7f32727c5000-7f32727c6000 rw-p 00005000 08:02 1730070 /usr/lib64/httpd/modules/mod_mime.so 7f32727c6000-7f32727cc000 r-xp 00000000 08:02 1730071 /usr/lib64/httpd/modules/mod_mime_magic.so 7f32727cc000-7f32729cb000 ---p 00006000 08:02 1730071 /usr/lib64/httpd/modules/mod_mime_magic.so 7f32729cb000-7f32729cc000 r--p 00005000 08:02 1730071 /usr/lib64/httpd/modules/mod_mime_magic.so 7f32729cc000-7f32729cd000 rw-p 00006000 08:02 1730071 /usr/lib64/httpd/modules/mod_mime_magic.so 7f32729cd000-7f32729ce000 r-xp 00000000 08:02 1730068 /usr/lib64/httpd/modules/mod_logio.so 7f32729ce000-7f3272bce000 ---p 00001000 08:02 1730068 /usr/lib64/httpd/modules/mod_logio.so 7f3272bce000-7f3272bcf000 r--p 00001000 08:02 1730068 /usr/lib64/httpd/modules/mod_logio.so 7f3272bcf000-7f3272bd0000 rw-p 00002000 08:02 1730068 /usr/lib64/httpd/modules/mod_logio.so 7f3272bd0000-7f3272bd6000 r-xp 00000000 08:02 1730065 /usr/lib64/httpd/modules/mod_log_config.so 7f3272bd6000-7f3272dd5000 ---p 00006000 08:02 1730065 /usr/lib64/httpd/modules/mod_log_config.so 7f3272dd5000-7f3272dd6000 r--p 00005000 08:02 1730065 /usr/lib64/httpd/modules/mod_log_config.so 7f3272dd6000-7f3272dd7000 rw-p 00006000 08:02 1730065 /usr/lib64/httpd/modules/mod_log_config.so 7f3272dd7000-7f3272ddd000 r-xp 00000000 08:02 1730060 /usr/lib64/httpd/modules/mod_info.so 7f3272ddd000-7f3272fdc000 ---p 00006000 08:02 1730060 /usr/lib64/httpd/modules/mod_info.so 7f3272fdc000-7f3272fdd000 r--p 00005000 08:02 1730060 /usr/lib64/httpd/modules/mod_info.so 7f3272fdd000-7f3272fde000 rw-p 00006000 08:02 1730060 /usr/lib64/httpd/modules/mod_info.so 7f3272fde000-7f3272fe9000 r-xp 00000000 08:02 1730059 /usr/lib64/httpd/modules/mod_include.so 7f3272fe9000-7f32731e8000 ---p 0000b000 08:02 1730059 /usr/lib64/httpd/modules/mod_include.so 7f32731e8000-7f32731e9000 r--p 0000a000 08:02 1730059 /usr/lib64/httpd/modules/mod_include.so 7f32731e9000-7f32731ea000 rw-p 0000b000 08:02 1730059 /usr/lib64/httpd/modules/mod_include.so 7f32731ea000-7f32731ee000 r-xp 00000000 08:02 1738953 /usr/lib64/httpd/modules/mod_headers.so 7f32731ee000-7f32733ed000 ---p 00004000 08:02 1738953 /usr/lib64/httpd/modules/mod_headers.so 7f32733ed000-7f32733ee000 r--p 00003000 08:02 1738953 /usr/lib64/httpd/modules/mod_headers.so 7f32733ee000-7f32733ef000 rw-p 00004000 08:02 1738953 /usr/lib64/httpd/modules/mod_headers.so 7f32733ef000-7f32733f3000 r-xp 00000000 08:02 1779221 /usr/lib64/httpd/modules/mod_filter.so 7f32733f3000-7f32735f2000 ---p 00004000 08:02 1779221 /usr/lib64/httpd/modules/mod_filter.so 7f32735f2000-7f32735f3000 r--p 00003000 08:02 1779221 /usr/lib64/httpd/modules/mod_filter.so 7f32735f3000-7f32735f4000 rw-p 00004000 08:02 1779221 /usr/lib64/httpd/modules/mod_filter.so 7f32735f4000-7f32735f9000 r-xp 00000000 08:02 1779209 /usr/lib64/httpd/modules/mod_ext_filter.so 7f32735f9000-7f32737f8000 ---p 00005000 08:02 1779209 /usr/lib64/httpd/modules/mod_ext_filter.so 7f32737f8000-7f32737f9000 r--p 00004000 08:02 1779209 /usr/lib64/httpd/modules/mod_ext_filter.so 7f32737f9000-7f32737fa000 rw-p 00005000 08:02 1779209 /usr/lib64/httpd/modules/mod_ext_filter.so 7f32737fa000-7f32737fc000 r-xp 00000000 08:02 1779205 /usr/lib64/httpd/modules/mod_expires.so 7f32737fc000-7f32739fc000 ---p 00002000 08:02 1779205 /usr/lib64/httpd/modules/mod_expires.so 7f32739fc000-7f32739fd000 r--p 00002000 08:02 1779205 /usr/lib64/httpd/modules/mod_expires.so 7f32739fd000-7f32739fe000 rw-p 00003000 08:02 1779205 /usr/lib64/httpd/modules/mod_expires.so 7f32739fe000-7f3273a00000 r-xp 00000000 08:02 1779204 /usr/lib64/httpd/modules/mod_env.so 7f3273a00000-7f3273bff000 ---p 00002000 08:02 1779204 /usr/lib64/httpd/modules/mod_env.so 7f3273bff000-7f3273c00000 r--p 00001000 08:02 1779204 /usr/lib64/httpd/modules/mod_env.so 7f3273c00000-7f3273c01000 rw-p 00002000 08:02 1779204 /usr/lib64/httpd/modules/mod_env.so 7f3273c01000-7f3273c03000 r-xp 00000000 08:02 1779203 /usr/lib64/httpd/modules/mod_echo.so 7f3273c03000-7f3273e02000 ---p 00002000 08:02 1779203 /usr/lib64/httpd/modules/mod_echo.so 7f3273e02000-7f3273e03000 r--p 00001000 08:02 1779203 /usr/lib64/httpd/modules/mod_echo.so 7f3273e03000-7f3273e04000 rw-p 00002000 08:02 1779203 /usr/lib64/httpd/modules/mod_echo.so 7f3273e04000-7f3273e06000 r-xp 00000000 08:02 1779202 /usr/lib64/httpd/modules/mod_dumpio.so 7f3273e06000-7f3274005000 ---p 00002000 08:02 1779202 /usr/lib64/httpd/modules/mod_dumpio.so 7f3274005000-7f3274006000 r--p 00001000 08:02 1779202 /usr/lib64/httpd/modules/mod_dumpio.so 7f3274006000-7f3274007000 rw-p 00002000 08:02 1779202 /usr/lib64/httpd/modules/mod_dumpio.so 7f3274007000-7f3274009000 r-xp 00000000 08:02 1779201 /usr/lib64/httpd/modules/mod_dir.so 7f3274009000-7f3274208000 ---p 00002000 08:02 1779201 /usr/lib64/httpd/modules/mod_dir.so 7f3274208000-7f3274209000 r--p 00001000 08:02 1779201 /usr/lib64/httpd/modules/mod_dir.so 7f3274209000-7f327420a000 rw-p 00002000 08:02 1779201 /usr/lib64/httpd/modules/mod_dir.so 7f327420a000-7f327421f000 r-xp 00000000 08:02 36088583 /usr/lib64/libz.so.1.2.7 7f327421f000-7f327441e000 ---p 00015000 08:02 36088583 /usr/lib64/libz.so.1.2.7 7f327441e000-7f327441f000 r--p 00014000 08:02 36088583 /usr/lib64/libz.so.1.2.7 7f327441f000-7f3274420000 rw-p 00015000 08:02 36088583 /usr/lib64/libz.so.1.2.7 7f3274420000-7f3274427000 r-xp 00000000 08:02 1779199 /usr/lib64/httpd/modules/mod_deflate.so 7f3274427000-7f3274626000 ---p 00007000 08:02 1779199 /usr/lib64/httpd/modules/mod_deflate.so 7f3274626000-7f3274627000 r--p 00006000 08:02 1779199 /usr/lib64/httpd/modules/mod_deflate.so 7f3274627000-7f3274628000 rw-p 00007000 08:02 1779199 /usr/lib64/httpd/modules/mod_deflate.so 7f3274628000-7f327462c000 r-xp 00000000 08:02 1779198 /usr/lib64/httpd/modules/mod_dbd.so 7f327462c000-7f327482c000 ---p 00004000 08:02 1779198 /usr/lib64/httpd/modules/mod_dbd.so 7f327482c000-7f327482d000 r--p 00004000 08:02 1779198 /usr/lib64/httpd/modules/mod_dbd.so 7f327482d000-7f327482e000 rw-p 00005000 08:02 1779198 /usr/lib64/httpd/modules/mod_dbd.so 7f327482e000-7f3274830000 r-xp 00000000 08:02 1779194 /usr/lib64/httpd/modules/mod_data.so 7f3274830000-7f3274a2f000 ---p 00002000 08:02 1779194 /usr/lib64/httpd/modules/mod_data.so 7f3274a2f000-7f3274a30000 r--p 00001000 08:02 1779194 /usr/lib64/httpd/modules/mod_data.so 7f3274a30000-7f3274a31000 rw-p 00002000 08:02 1779194 /usr/lib64/httpd/modules/mod_data.so 7f3274a31000-7f3274a38000 r-xp 00000000 08:02 1779190 /usr/lib64/httpd/modules/mod_cache_disk.so 7f3274a38000-7f3274c38000 ---p 00007000 08:02 1779190 /usr/lib64/httpd/modules/mod_cache_disk.so 7f3274c38000-7f3274c39000 r--p 00007000 08:02 1779190 /usr/lib64/httpd/modules/mod_cache_disk.so 7f3274c39000-7f3274c3a000 rw-p 00008000 08:02 1779190 /usr/lib64/httpd/modules/mod_cache_disk.so 7f3274c3a000-7f3274c48000 r-xp 00000000 08:02 1779189 /usr/lib64/httpd/modules/mod_cache.so 7f3274c48000-7f3274e48000 ---p 0000e000 08:02 1779189 /usr/lib64/httpd/modules/mod_cache.so 7f3274e48000-7f3274e49000 r--p 0000e000 08:02 1779189 /usr/lib64/httpd/modules/mod_cache.so 7f3274e49000-7f3274e4a000 rw-p 0000f000 08:02 1779189 /usr/lib64/httpd/modules/mod_cache.so 7f3274e4a000-7f3274e53000 r-xp 00000000 08:02 1779187 /usr/lib64/httpd/modules/mod_autoindex.so 7f3274e53000-7f3275052000 ---p 00009000 08:02 1779187 /usr/lib64/httpd/modules/mod_autoindex.so 7f3275052000-7f3275053000 r--p 00008000 08:02 1779187 /usr/lib64/httpd/modules/mod_autoindex.so 7f3275053000-7f3275054000 rw-p 00009000 08:02 1779187 /usr/lib64/httpd/modules/mod_autoindex.so 7f3275054000-7f3275055000 r-xp 00000000 08:02 1779186 /usr/lib64/httpd/modules/mod_authz_user.so 7f3275055000-7f3275254000 ---p 00001000 08:02 1779186 /usr/lib64/httpd/modules/mod_authz_user.so 7f3275254000-7f3275255000 r--p 00000000 08:02 1779186 /usr/lib64/httpd/modules/mod_authz_user.so 7f3275255000-7f3275256000 rw-p 00001000 08:02 1779186 /usr/lib64/httpd/modules/mod_authz_user.so 7f3275256000-7f3275258000 r-xp 00000000 08:02 1779185 /usr/lib64/httpd/modules/mod_authz_owner.so 7f3275258000-7f3275457000 ---p 00002000 08:02 1779185 /usr/lib64/httpd/modules/mod_authz_owner.so 7f3275457000-7f3275458000 r--p 00001000 08:02 1779185 /usr/lib64/httpd/modules/mod_authz_owner.so 7f3275458000-7f3275459000 rw-p 00002000 08:02 1779185 /usr/lib64/httpd/modules/mod_authz_owner.so 7f3275459000-7f327545b000 r-xp 00000000 08:02 1779184 /usr/lib64/httpd/modules/mod_authz_host.so 7f327545b000-7f327565a000 ---p 00002000 08:02 1779184 /usr/lib64/httpd/modules/mod_authz_host.so 7f327565a000-7f327565b000 r--p 00001000 08:02 1779184 /usr/lib64/httpd/modules/mod_authz_host.so 7f327565b000-7f327565c000 rw-p 00002000 08:02 1779184 /usr/lib64/httpd/modules/mod_authz_host.so 7f327565c000-7f327565e000 r-xp 00000000 08:02 1779183 /usr/lib64/httpd/modules/mod_authz_groupfile.so 7f327565e000-7f327585d000 ---p 00002000 08:02 1779183 /usr/lib64/httpd/modules/mod_authz_groupfile.so 7f327585d000-7f327585e000 r--p 00001000 08:02 1779183 /usr/lib64/httpd/modules/mod_authz_groupfile.so 7f327585e000-7f327585f000 rw-p 00002000 08:02 1779183 /usr/lib64/httpd/modules/mod_authz_groupfile.so 7f327585f000-7f3275861000 r-xp 00000000 08:02 1779182 /usr/lib64/httpd/modules/mod_authz_dbm.so 7f3275861000-7f3275a60000 ---p 00002000 08:02 1779182 /usr/lib64/httpd/modules/mod_authz_dbm.so 7f3275a60000-7f3275a61000 r--p 00001000 08:02 1779182 /usr/lib64/httpd/modules/mod_authz_dbm.so 7f3275a61000-7f3275a62000 rw-p 00002000 08:02 1779182 /usr/lib64/httpd/modules/mod_authz_dbm.so 7f3275a62000-7f3275a64000 r-xp 00000000 08:02 1779181 /usr/lib64/httpd/modules/mod_authz_dbd.so 7f3275a64000-7f3275c64000 ---p 00002000 08:02 1779181 /usr/lib64/httpd/modules/mod_authz_dbd.so 7f3275c64000-7f3275c65000 r--p 00002000 08:02 1779181 /usr/lib64/httpd/modules/mod_authz_dbd.so 7f3275c65000-7f3275c66000 rw-p 00003000 08:02 1779181 /usr/lib64/httpd/modules/mod_authz_dbd.so 7f3275c66000-7f3275c6b000 r-xp 00000000 08:02 1779180 /usr/lib64/httpd/modules/mod_authz_core.so 7f3275c6b000-7f3275e6a000 ---p 00005000 08:02 1779180 /usr/lib64/httpd/modules/mod_authz_core.so 7f3275e6a000-7f3275e6b000 r--p 00004000 08:02 1779180 /usr/lib64/httpd/modules/mod_authz_core.so 7f3275e6b000-7f3275e6c000 rw-p 00005000 08:02 1779180 /usr/lib64/httpd/modules/mod_authz_core.so 7f3275e6c000-7f3275e70000 r-xp 00000000 08:02 1779179 /usr/lib64/httpd/modules/mod_authn_socache.so 7f3275e70000-7f327606f000 ---p 00004000 08:02 1779179 /usr/lib64/httpd/modules/mod_authn_socache.so 7f327606f000-7f3276070000 r--p 00003000 08:02 1779179 /usr/lib64/httpd/modules/mod_authn_socache.so 7f3276070000-7f3276071000 rw-p 00004000 08:02 1779179 /usr/lib64/httpd/modules/mod_authn_socache.so 7f3276071000-7f3276073000 r-xp 00000000 08:02 1779178 /usr/lib64/httpd/modules/mod_authn_file.so 7f3276073000-7f3276272000 ---p 00002000 08:02 1779178 /usr/lib64/httpd/modules/mod_authn_file.so 7f3276272000-7f3276273000 r--p 00001000 08:02 1779178 /usr/lib64/httpd/modules/mod_authn_file.so 7f3276273000-7f3276274000 rw-p 00002000 08:02 1779178 /usr/lib64/httpd/modules/mod_authn_file.so 7f3276274000-7f3276276000 r-xp 00000000 08:02 1779177 /usr/lib64/httpd/modules/mod_authn_dbm.so 7f3276276000-7f3276475000 ---p 00002000 08:02 1779177 /usr/lib64/httpd/modules/mod_authn_dbm.so 7f3276475000-7f3276476000 r--p 00001000 08:02 1779177 /usr/lib64/httpd/modules/mod_authn_dbm.so 7f3276476000-7f3276477000 rw-p 00002000 08:02 1779177 /usr/lib64/httpd/modules/mod_authn_dbm.so 7f3276477000-7f3276479000 r-xp 00000000 08:02 1779176 /usr/lib64/httpd/modules/mod_authn_dbd.so 7f3276479000-7f3276679000 ---p 00002000 08:02 1779176 /usr/lib64/httpd/modules/mod_authn_dbd.so 7f3276679000-7f327667a000 r--p 00002000 08:02 1779176 /usr/lib64/httpd/modules/mod_authn_dbd.so 7f327667a000-7f327667b000 rw-p 00003000 08:02 1779176 /usr/lib64/httpd/modules/mod_authn_dbd.so 7f327667b000-7f327667d000 r-xp 00000000 08:02 1779175 /usr/lib64/httpd/modules/mod_authn_core.so 7f327667d000-7f327687d000 ---p 00002000 08:02 1779175 /usr/lib64/httpd/modules/mod_authn_core.so 7f327687d000-7f327687e000 r--p 00002000 08:02 1779175 /usr/lib64/httpd/modules/mod_authn_core.so 7f327687e000-7f327687f000 rw-p 00003000 08:02 1779175 /usr/lib64/httpd/modules/mod_authn_core.so 7f327687f000-7f3276881000 r-xp 00000000 08:02 1761489 /usr/lib64/httpd/modules/mod_authn_anon.so 7f3276881000-7f3276a80000 ---p 00002000 08:02 1761489 /usr/lib64/httpd/modules/mod_authn_anon.so 7f3276a80000-7f3276a81000 r--p 00001000 08:02 1761489 /usr/lib64/httpd/modules/mod_authn_anon.so 7f3276a81000-7f3276a82000 rw-p 00002000 08:02 1761489 /usr/lib64/httpd/modules/mod_authn_anon.so 7f3276a82000-7f3276a8a000 r-xp 00000000 08:02 1779173 /usr/lib64/httpd/modules/mod_auth_digest.so 7f3276a8a000-7f3276c89000 ---p 00008000 08:02 1779173 /usr/lib64/httpd/modules/mod_auth_digest.so 7f3276c89000-7f3276c8a000 r--p 00007000 08:02 1779173 /usr/lib64/httpd/modules/mod_auth_digest.so 7f3276c8a000-7f3276c8b000 rw-p 00008000 08:02 1779173 /usr/lib64/httpd/modules/mod_auth_digest.so 7f3276c8b000-7f3276c8d000 r-xp 00000000 08:02 1761486 /usr/lib64/httpd/modules/mod_auth_basic.so 7f3276c8d000-7f3276e8c000 ---p 00002000 08:02 1761486 /usr/lib64/httpd/modules/mod_auth_basic.so 7f3276e8c000-7f3276e8d000 r--p 00001000 08:02 1761486 /usr/lib64/httpd/modules/mod_auth_basic.so 7f3276e8d000-7f3276e8e000 rw-p 00002000 08:02 1761486 /usr/lib64/httpd/modules/mod_auth_basic.so 7f3276e8e000-7f3276e8f000 r-xp 00000000 08:02 1779171 /usr/lib64/httpd/modules/mod_allowmethods.so 7f3276e8f000-7f327708f000 ---p 00001000 08:02 1779171 /usr/lib64/httpd/modules/mod_allowmethods.so 7f327708f000-7f3277090000 r--p 00001000 08:02 1779171 /usr/lib64/httpd/modules/mod_allowmethods.so 7f3277090000-7f3277091000 rw-p 00002000 08:02 1779171 /usr/lib64/httpd/modules/mod_allowmethods.so 7f3277091000-7f3277094000 r-xp 00000000 08:02 552852 /usr/lib64/httpd/modules/mod_alias.so 7f3277094000-7f3277293000 ---p 00003000 08:02 552852 /usr/lib64/httpd/modules/mod_alias.so 7f3277293000-7f3277294000 r--p 00002000 08:02 552852 /usr/lib64/httpd/modules/mod_alias.so 7f3277294000-7f3277295000 rw-p 00003000 08:02 552852 /usr/lib64/httpd/modules/mod_alias.so 7f3277295000-7f3277297000 r-xp 00000000 08:02 1761480 /usr/lib64/httpd/modules/mod_actions.so 7f3277297000-7f3277496000 ---p 00002000 08:02 1761480 /usr/lib64/httpd/modules/mod_actions.so 7f3277496000-7f3277497000 r--p 00001000 08:02 1761480 /usr/lib64/httpd/modules/mod_actions.so 7f3277497000-7f3277498000 rw-p 00002000 08:02 1761480 /usr/lib64/httpd/modules/mod_actions.so 7f3277498000-7f327749a000 r-xp 00000000 08:02 1761479 /usr/lib64/httpd/modules/mod_access_compat.so 7f327749a000-7f3277699000 ---p 00002000 08:02 1761479 /usr/lib64/httpd/modules/mod_access_compat.so 7f3277699000-7f327769a000 r--p 00001000 08:02 1761479 /usr/lib64/httpd/modules/mod_access_compat.so 7f327769a000-7f327769b000 rw-p 00002000 08:02 1761479 /usr/lib64/httpd/modules/mod_access_compat.so 7f327769b000-7f3277700000 r-xp 00000000 08:02 33733541 /usr/lib64/libfreebl3.so 7f3277700000-7f3277900000 ---p 00065000 08:02 33733541 /usr/lib64/libfreebl3.so 7f3277900000-7f3277901000 r--p 00065000 08:02 33733541 /usr/lib64/libfreebl3.so 7f3277901000-7f3277902000 rw-p 00066000 08:02 33733541 /usr/lib64/libfreebl3.so 7f3277902000-7f3277906000 rw-p 00000000 00:00 0 7f3277906000-7f327790a000 r-xp 00000000 08:02 35923595 /usr/lib64/libuuid.so.1.3.0 7f327790a000-7f3277b09000 ---p 00004000 08:02 35923595 /usr/lib64/libuuid.so.1.3.0 7f3277b09000-7f3277b0a000 r--p 00003000 08:02 35923595 /usr/lib64/libuuid.so.1.3.0 7f3277b0a000-7f3277b0b000 rw-p 00004000 08:02 35923595 /usr/lib64/libuuid.so.1.3.0 7f3277b0b000-7f3277cb8000 r-xp 00000000 08:02 36088568 /usr/lib64/libc-2.16.so 7f3277cb8000-7f3277eb8000 ---p 001ad000 08:02 36088568 /usr/lib64/libc-2.16.so 7f3277eb8000-7f3277ebc000 r--p 001ad000 08:02 36088568 /usr/lib64/libc-2.16.so 7f3277ebc000-7f3277ebe000 rw-p 001b1000 08:02 36088568 /usr/lib64/libc-2.16.so 7f3277ebe000-7f3277ec3000 rw-p 00000000 00:00 0 7f3277ec3000-7f3277ec6000 r-xp 00000000 08:02 36542944 /usr/lib64/libdl-2.16.so 7f3277ec6000-7f32780c5000 ---p 00003000 08:02 36542944 /usr/lib64/libdl-2.16.so 7f32780c5000-7f32780c6000 r--p 00002000 08:02 36542944 /usr/lib64/libdl-2.16.so 7f32780c6000-7f32780c7000 rw-p 00003000 08:02 36542944 /usr/lib64/libdl-2.16.so 7f32780c7000-7f32780dd000 r-xp 00000000 08:02 36088569 /usr/lib64/libpthread-2.16.so 7f32780dd000-7f32782dd000 ---p 00016000 08:02 36088569 /usr/lib64/libpthread-2.16.so 7f32782dd000-7f32782de000 r--p 00016000 08:02 36088569 /usr/lib64/libpthread-2.16.so 7f32782de000-7f32782df000 rw-p 00017000 08:02 36088569 /usr/lib64/libpthread-2.16.so 7f32782df000-7f32782e3000 rw-p 00000000 00:00 0 7f32782e3000-7f3278311000 r-xp 00000000 08:02 36558022 /usr/lib64/libapr-1.so.0.4.6 7f3278311000-7f3278511000 ---p 0002e000 08:02 36558022 /usr/lib64/libapr-1.so.0.4.6 7f3278511000-7f3278512000 r--p 0002e000 08:02 36558022 /usr/lib64/libapr-1.so.0.4.6 7f3278512000-7f3278513000 rw-p 0002f000 08:02 36558022 /usr/lib64/libapr-1.so.0.4.6 7f3278513000-7f32786bd000 r-xp 00000000 08:02 36088582 /usr/lib64/libdb-5.3.so 7f32786bd000-7f32788bd000 ---p 001aa000 08:02 36088582 /usr/lib64/libdb-5.3.so 7f32788bd000-7f32788c4000 r--p 001aa000 08:02 36088582 /usr/lib64/libdb-5.3.so 7f32788c4000-7f32788c7000 rw-p 001b1000 08:02 36088582 /usr/lib64/libdb-5.3.so 7f32788c7000-7f32788ed000 r-xp 00000000 08:02 36543062 /usr/lib64/libexpat.so.1.6.0 7f32788ed000-7f3278aed000 ---p 00026000 08:02 36543062 /usr/lib64/libexpat.so.1.6.0 7f3278aed000-7f3278aef000 r--p 00026000 08:02 36543062 /usr/lib64/libexpat.so.1.6.0 7f3278aef000-7f3278af0000 rw-p 00028000 08:02 36543062 /usr/lib64/libexpat.so.1.6.0 7f3278af0000-7f3278af8000 r-xp 00000000 08:02 33796604 /usr/lib64/libcrypt-2.16.so 7f3278af8000-7f3278cf7000 ---p 00008000 08:02 33796604 /usr/lib64/libcrypt-2.16.so 7f3278cf7000-7f3278cf8000 r--p 00007000 08:02 33796604 /usr/lib64/libcrypt-2.16.so 7f3278cf8000-7f3278cf9000 rw-p 00008000 08:02 33796604 /usr/lib64/libcrypt-2.16.so 7f3278cf9000-7f3278d27000 rw-p 00000000 00:00 0 7f3278d27000-7f3278d4b000 r-xp 00000000 08:02 36589642 /usr/lib64/libaprutil-1.so.0.4.1 7f3278d4b000-7f3278f4a000 ---p 00024000 08:02 36589642 /usr/lib64/libaprutil-1.so.0.4.1 7f3278f4a000-7f3278f4b000 r--p 00023000 08:02 36589642 /usr/lib64/libaprutil-1.so.0.4.1 7f3278f4b000-7f3278f4c000 rw-p 00024000 08:02 36589642 /usr/lib64/libaprutil-1.so.0.4.1 7f3278f4c000-7f3278f6b000 r-xp 00000000 08:02 33797580 /usr/lib64/libselinux.so.1 7f3278f6b000-7f327916a000 ---p 0001f000 08:02 33797580 /usr/lib64/libselinux.so.1 7f327916a000-7f327916b000 r--p 0001e000 08:02 33797580 /usr/lib64/libselinux.so.1 7f327916b000-7f327916c000 rw-p 0001f000 08:02 33797580 /usr/lib64/libselinux.so.1 7f327916c000-7f327916e000 rw-p 00000000 00:00 0 7f327916e000-7f32791ca000 r-xp 00000000 08:02 33797578 /usr/lib64/libpcre.so.1.0.1 7f32791ca000-7f32793ca000 ---p 0005c000 08:02 33797578 /usr/lib64/libpcre.so.1.0.1 7f32793ca000-7f32793cb000 r--p 0005c000 08:02 33797578 /usr/lib64/libpcre.so.1.0.1 7f32793cb000-7f32793cc000 rw-p 0005d000 08:02 33797578 /usr/lib64/libpcre.so.1.0.1 7f32793cc000-7f32793ec000 r-xp 00000000 08:02 33941994 /usr/lib64/ld-2.16.so 7f327954d000-7f32795cb000 rw-s 00000000 00:04 2880029 /dev/zero (deleted) 7f32795cb000-7f32795d2000 rw-p 00000000 00:00 0 7f32795d7000-7f32795d8000 rw-p 00000000 00:00 0 7f32795d8000-7f32795ea000 rw-s 00000000 00:04 2880033 /dev/zero (deleted) 7f32795ea000-7f32795eb000 rw-s 00000000 00:04 232259584 /SYSV0111f1fe (deleted) 7f32795eb000-7f32795ec000 rw-p 00000000 00:00 0 7f32795ec000-7f32795ed000 r--p 00020000 08:02 33941994 /usr/lib64/ld-2.16.so 7f32795ed000-7f32795ee000 rw-p 00021000 08:02 33941994 /usr/lib64/ld-2.16.so 7f32795ee000-7f32795ef000 rw-p 00000000 00:00 0 7f32795ef000-7f3279660000 r-xp 00000000 08:02 67764284 /usr/sbin/httpd 7f3279860000-7f3279863000 r--p 00071000 08:02 67764284 /usr/sbin/httpd 7f3279863000-7f3279865000 rw-p 00074000 08:02 67764284 /usr/sbin/httpd 7f3279865000-7f3279868000 rw-p 00000000 00:00 0 7f327b05b000-7f327b45f000 rw-p 00000000 00:00 0 [heap] 7fff29bea000-7fff29c0b000 rw-p 00000000 00:00 0 [stack] 7fff29d3a000-7fff29d3b000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Version-Release number of selected component (if applicable): httpd-2.4.3-15.fc18.x86_64 How reproducible: I cannot reproduce this bug everytime I stop the service. Usually I start the service load a page and stop the service. httpd crash in 90% of times. Sometimes all I do is to start and stop the service. also backtrace is not all the time the same: # grep -c Backtrace: error_log 18 Sometimes I do not get an backtrace but this: [Mon Feb 18 12:08:24.281744 2013] [mpm_prefork:notice] [pid 9427] AH00170: caught SIGWINCH, shutting down gracefully [Mon Feb 18 12:08:25.282341 2013] [core:notice] [pid 9427] AH00052: child pid 9433 exit signal Segmentation fault (11) or [Mon Feb 18 12:12:00.903014 2013] [mpm_prefork:notice] [pid 9623] AH00170: caught SIGWINCH, shutting down gracefully [Mon Feb 18 12:12:00.956260 2013] [core:notice] [pid 9623] AH00052: child pid 9628 exit signal Segmentation fault (11) [Mon Feb 18 12:12:00.958272 2013] [core:notice] [pid 9623] AH00052: child pid 9645 exit signal Segmentation fault (11) After this point sometimes I have to kill the remaining httpd processes. Sometimes not. abrt does not report anything.
# grep "glibc detected" /var/log/httpd/error_log *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fc8a997b960 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f9b90d57620 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f9b90d57620 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f7c650a35c0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x00007f62dc9bf3a0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fea22bcc570 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fea22bcc570 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f0eebde8520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f0eebde8520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f120db184b0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007ffeba52e520 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f364b700500 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f41fe2d8430 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f41fe2d8430 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007fc80ef723c0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f18e0ce63c0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f14467983f0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f327b28e450 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f422c4f1580 *** 7f421acb0000-7f421acb1000 rw-p 0000c000 08:02 33*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f422c4f1580 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007faa15158ea0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6b6071ced0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6b6071ced0 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007ffb3f8ce8e0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007f4d229a6c90 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid size: 0x00007f4d229b0630 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007feb349c58a0 *** *** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x00007fc4de127780 *** *** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x00007f6a9545af60 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f2748cbfe00 *** *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f359e265590 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007fd09b447db0 *** *** glibc detected *** /usr/sbin/httpd: double free or corruption (out): 0x00007fd09b447db0 *** Please check the systemd module. I was unable to duplicate this bug running httpd from command line ( httpd -DFOREGROUND ) or via gdb (run -X). Starting and stopping httpd via systemctl almost every time result in "glibc detected..." error messages or frozen httpd threads.
Sorry! Not mod_systemd.so's fault. :-(
Again I had some spare time so I'm back on this bug again. Also I enabled core dump files. # cat crash-test.php <?php phpinfo(); ?> # ab -n 50000 -c 10 http://localhost/~gabriel/crash-test.php # service httpd stop <most of the time apache crash or freeze here.> # find /tmp -name core\* ./systemd-private-MFoNAu/core.28874 # gdb httpd -c /tmp/systemd-private-MFoNAu/core.28874 GNU gdb (GDB) Fedora (7.5.1-37.fc18) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done. done. [New LWP 28874] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/httpd -DFOREGROUND'. Program terminated with signal 6, Aborted. #0 0x00007f5b2ef6bba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 63 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) where #0 0x00007f5b2ef6bba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 #1 0x00007f5b2ef6d358 in __GI_abort () at abort.c:90 #2 0x00007f5b2efab59b in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f5b2f0afba8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:197 #3 0x00007f5b2efb2a8e in malloc_printerr (ptr=0x7f5b2f2e7798 <main_arena+88>, str=0x7f5b2f0ada01 "free(): invalid pointer", action=3) at malloc.c:4969 #4 _int_free (av=0x7f5b2f2e7740 <main_arena>, p=0x7f5b2f2e7788 <main_arena+72>, have_lock=0) at malloc.c:3826 #5 0x00007f5b2f72785c in allocator_free (node=0x7f5b2f2e7798 <main_arena+88>, allocator=0x7f5b321572c0) at memory/unix/apr_pools.c:430 #6 apr_pool_destroy (pool=0x7f5b321573b8) at memory/unix/apr_pools.c:856 #7 0x00007f5b29def1be in clean_child_exit (code=code@entry=0) at prefork.c:218 #8 0x00007f5b29def9fb in just_die (sig=<optimized out>) at prefork.c:344 #9 <signal handler called> #10 0x00007f5b30806535 in _dl_fini () at dl-fini.c:194 #11 0x00007f5b2ef6edf1 in __run_exit_handlers (status=status@entry=0, listp=0x7f5b2f2e76a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #12 0x00007f5b2ef6ee75 in __GI_exit (status=status@entry=0) at exit.c:99 #13 0x00007f5b29def1db in clean_child_exit (code=code@entry=0) at prefork.c:227 #14 0x00007f5b29def60e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:725 #15 0x00007f5b29def97c in make_child (s=0x7f5b31de4368, slot=slot@entry=3) at prefork.c:800 #16 0x00007f5b29def9d6 in startup_children (number_to_start=2) at prefork.c:818 #17 0x00007f5b29df0886 in prefork_run (_pconf=<optimized out>, plog=0x7f5b31e164f8, s=0x7f5b31de4368) at prefork.c:976 #18 0x00007f5b30a3ed5e in ap_run_mpm (pconf=0x7f5b31db9158, plog=0x7f5b31e164f8, s=0x7f5b31de4368) at mpm_common.c:98 #19 0x00007f5b30a3840a in main (argc=2, argv=0x7fffdb816188) at main.c:777
This is from another computer. I don't even have to make a coonection to http server. start && stop is all I have to to. Almost every time I have at least one core dump file (usuallly 3 to 5). All have this kind of backtrace: (gdb) where #0 _dl_fini () at dl-fini.c:249 #1 0x00007f362c898df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f362cc116a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #2 0x00007f362c898e75 in __GI_exit (status=status@entry=0) at exit.c:99 #3 0x00007f36277191db in clean_child_exit (code=code@entry=0) at prefork.c:227 #4 0x00007f36277199fb in just_die (sig=<optimized out>) at prefork.c:344 #5 <signal handler called> #6 0x00007f362e138a37 in munmap () at ../sysdeps/unix/syscall-template.S:81 #7 0x00007f362e1381ad in _dl_unmap (map=map@entry=0x7f362f9e09f0) at ../sysdeps/x86_64/tlsdesc.c:139 #8 0x00007f362e13511f in _dl_close_worker (map=map@entry=0x7f362f9e09f0) at dl-close.c:634 #9 0x00007f362e13597e in _dl_close (_map=0x7f362f9e09f0) at dl-close.c:776 #10 0x00007f362e12fdc6 in _dl_catch_error (objname=0x7f362f7e95b0, errstring=0x7f362f7e95b8, mallocedp=0x7f362f7e95a8, operate=0x7f362cc190e0 <dlclose_doit>, args=0x7f362f9e09f0) at dl-error.c:177 #11 0x00007f362cc1963c in _dlerror_run (operate=operate@entry=0x7f362cc190e0 <dlclose_doit>, args=0x7f362f9e09f0) at dlerror.c:163 #12 0x00007f362cc1910f in __dlclose (handle=<optimized out>) at dlclose.c:47 #13 0x00007f3625325e14 in module_destructor (module=0x7f362f9e1c20) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2286 #14 0x00007f362532b085 in zend_hash_apply_deleter (ht=ht@entry=0x7f3625710220 <module_registry>, p=0x7f362f9e1bc0) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650 #15 0x00007f362532c878 in zend_hash_graceful_reverse_destroy (ht=0x7f3625710220 <module_registry>) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687 #16 0x00007f362532468c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797 #17 0x00007f362531ea0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820 #18 0x00007f36252c035b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365 #19 0x00007f36252c0419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.12/main/main.c:2333 #20 0x00007f36253c6fc1 in php_apache_child_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398 #21 0x00007f362d05173e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 #22 apr_pool_destroy (pool=0x7f362fb1b6d8) at memory/unix/apr_pools.c:814 #23 0x00007f36277191be in clean_child_exit (code=code@entry=0) at prefork.c:218 #24 0x00007f362771960e in child_main (child_num_arg=child_num_arg@entry=4) at prefork.c:725 #25 0x00007f362771997c in make_child (s=0x7f362f7ad368, slot=slot@entry=4) at prefork.c:800 #26 0x00007f36277199d6 in startup_children (number_to_start=1) at prefork.c:818 #27 0x00007f362771a886 in prefork_run (_pconf=<optimized out>, plog=0x7f362f7df4f8, s=0x7f362f7ad368) at prefork.c:976 #28 0x00007f362e368d5e in ap_run_mpm (pconf=0x7f362f782158, plog=0x7f362f7df4f8, s=0x7f362f7ad368) at mpm_common.c:98 #29 0x00007f362e36240a in main (argc=2, argv=0x7fff59920b28) at main.c:777 Uninstalling php does not help. I still get occasionally core dumps just restarting httpd. And the backtrace is that from comment #4. I think there are 2 errors. One in httpd and one in php module.
I'm still not able to reproduce it here on my machine... :/
Hmm... I can reproduce this on two different machines (same httpd configuration). Maybe I should send you my /etc/httpd directory. It could be related to how httpd is configured.
This is the bugreport on apache.org: https://issues.apache.org/bugzilla/show_bug.cgi?id=54785
Oh, I'm able now and even 'service httpd stop' freezes for me now. I will see what I can do.
I have found similar bug from 2006 in the mean time: https://issues.apache.org/bugzilla/show_bug.cgi?id=23238 .
> Oh, I'm able now and even 'service httpd stop' freezes for me now. Welcome to the club! :-) Really, very nasty bug! I get the same behaviour.
The bug has been reported in 2003! Seems to be related. And still not fixed? :-(
Can you please try to reproduce it with following scratch-build? http://koji.fedoraproject.org/koji/taskinfo?taskID=5203004 It could fix/workaround the crash from Comment 3.
Hm, I think this fixed only crash, but it still can freeze...
> Hm, I think this fixed only crash, but it still can freeze... Works for me, at on the test computer. # watch -n 5 "service httpd restart" Every around 7 seconds I get this [Wed Apr 03 07:05:00.087609 2013] [core:notice] [pid 2198] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' in error_log file. It seems to work. Also, where is no php module loaded. 1. restart 20+ times 2. ab -n 50000 -c 10 http://localhost/index.html 1. restart 20+ times 3. find /tmp -name core* ==> nothing
Sorry. It should be: at least on the test computer.
[Wed Apr 03 07:15:19.549906 2013] [core:notice] [pid 3036] core dump file size limit raised to 18446744073709551615 bytes Really? 16777215 TB?
I haven't changed the core dump file size limit in that scratch-build. I think this is what "infinite" mean in this context. Anyway, can you try also http://koji.fedoraproject.org/koji/taskinfo?taskID=5203133 ? This should fix also potential freezing with php installed. If it will work, I will try to propose the fix upstream.
> I haven't changed the core dump file size limit in that scratch-build. I know... httpd-2.4.1-corelimit.patch > I think this is what "infinite" mean in this context. Ah... right. :-) Thank you. # ulimit -a | grep core core file size (blocks, -c) unlimited # rpm -qa | grep httpd | grep -v micro httpd-2.4.4-5.fc18.x86_64 httpd-debuginfo-2.4.4-5.fc18.x86_64 httpd-tools-2.4.4-5.fc18.x86_64 php module enabled. # service httpd start # service httpd stop # find /tmp -name core* /tmp/systemd-private-bwjc3G/core.6610 # gdb httpd -c /tmp/systemd-private-bwjc3G/core.6610 ... (gdb) where #0 _dl_fini () at dl-fini.c:249 #1 0x00007f199b284df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f199b5fd6a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #2 0x00007f199b284e75 in __GI_exit (status=status@entry=0) at exit.c:99 #3 0x00007f19961051f6 in clean_child_exit (code=code@entry=0) at prefork.c:232 #4 0x00007f1996105a33 in just_die (sig=<optimized out>) at prefork.c:351 #5 <signal handler called> #6 0x00007f199cb24a37 in munmap () at ../sysdeps/unix/syscall-template.S:81 #7 0x00007f199cb241ad in _dl_unmap (map=map@entry=0x7f199d54aed0) at ../sysdeps/x86_64/tlsdesc.c:139 #8 0x00007f199cb2111f in _dl_close_worker (map=map@entry=0x7f199d54a9e0) at dl-close.c:634 #9 0x00007f199cb2197e in _dl_close (_map=0x7f199d54a9e0) at dl-close.c:776 #10 0x00007f199cb1bdc6 in _dl_catch_error (objname=0x7f199d3535b0, errstring=0x7f199d3535b8, mallocedp=0x7f199d3535a8, operate=0x7f199b6050e0 <dlclose_doit>, args=0x7f199d54a9e0) at dl-error.c:177 #11 0x00007f199b60563c in _dlerror_run (operate=operate@entry=0x7f199b6050e0 <dlclose_doit>, args=0x7f199d54a9e0) at dlerror.c:163 #12 0x00007f199b60510f in __dlclose (handle=<optimized out>) at dlclose.c:47 #13 0x00007f1993d11e14 in module_destructor (module=0x7f199d54bc10) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2286 #14 0x00007f1993d17085 in zend_hash_apply_deleter (ht=ht@entry=0x7f19940fc220 <module_registry>, p=0x7f199d54bbb0) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650 #15 0x00007f1993d18878 in zend_hash_graceful_reverse_destroy (ht=0x7f19940fc220 <module_registry>) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687 #16 0x00007f1993d1068c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797 #17 0x00007f1993d0aa0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820 #18 0x00007f1993cac35b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365 #19 0x00007f1993cac419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.12/main/main.c:2333 #20 0x00007f1993db2fc1 in php_apache_child_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398 #21 0x00007f199ba3d73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 #22 apr_pool_destroy (pool=0x7f199d6856c8) at memory/unix/apr_pools.c:814 #23 0x00007f19961051be in clean_child_exit (code=code@entry=0) at prefork.c:218 #24 0x00007f199610562e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:732 #25 0x00007f199610599c in make_child (s=0x7f199d317368, slot=slot@entry=3) at prefork.c:807 #26 0x00007f19961059f6 in startup_children (number_to_start=2) at prefork.c:825 #27 0x00007f19961068c6 in prefork_run (_pconf=<optimized out>, plog=0x7f199d3494f8, s=0x7f199d317368) at prefork.c:983 #28 0x00007f199cd54d5e in ap_run_mpm (pconf=0x7f199d2ec158, plog=0x7f199d3494f8, s=0x7f199d317368) at mpm_common.c:98 #29 0x00007f199cd4e40a in main (argc=2, argv=0x7fffa5fdf198) at main.c:777 Does not work. Sorry.
Right, happened to me after 30 minutes of testing too... I will test more before submitting next build.
Ok, I had run the test over the night and I didn't see any freeze or crash with this one: http://koji.fedoraproject.org/koji/taskinfo?taskID=5209578 It would be great if you could try it too. I hope this is the last build :).
> I hope this is the last build I don't really care how many are. Do as many as necessary. All that matters is that the bug will is fixed. Anyway, I think is not la last one. :-( # rpm -qa | grep httpd | grep -v micro httpd-2.4.4-7.fc18.x86_64 httpd-tools-2.4.4-7.fc18.x86_64 httpd-debuginfo-2.4.4-5.fc18.x86_64 # find /tmp -name core* <nothing> # service httpd start Redirecting to /bin/systemctl start httpd.service # service httpd stop Redirecting to /bin/systemctl stop httpd.service # find /tmp -name core* /tmp/systemd-private-n4JXRH/core.722 /tmp/systemd-private-n4JXRH/core.721 /tmp/systemd-private-n4JXRH/core.723 /tmp/systemd-private-n4JXRH/core.719 /tmp/systemd-private-n4JXRH/core.720 # gdb httpd -c /tmp/systemd-private-n4JXRH/core.720 ... (gdb) where #0 apr_palloc (pool=pool@entry=0x0, in_size=in_size@entry=32) at memory/unix/apr_pools.c:681 #1 0x00007f246160952b in ap_mutex_register (pconf=pconf@entry=0x0, type=type@entry=0x7f245a996ea1 "mpm-accept", default_dir=default_dir@entry=0x0, default_mech=default_mech@entry=APR_LOCK_DEFAULT, options=options@entry=0) at util_mutex.c:259 #2 0x00007f245a995aa4 in prefork_pre_config (p=0x0, plog=<optimized out>, ptemp=<optimized out>) at prefork.c:1302 #3 0x00007f2461fd5bc0 in ?? () #4 <signal handler called> #5 __strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31 #6 0x00007f24585a0c45 in zend_unregister_functions (functions=<optimized out>, count=count@entry=-1, function_table=function_table@entry=0x0) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2206 #7 0x00007f24585a1d9a in module_destructor (module=0x7f24621a2b60) at /usr/src/debug/php-5.4.12/Zend/zend_API.c:2280 #8 0x00007f24585a7085 in zend_hash_apply_deleter (ht=ht@entry=0x7f245898c220 <module_registry>, p=0x7f24621a2b00) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:650 #9 0x00007f24585a8878 in zend_hash_graceful_reverse_destroy (ht=0x7f245898c220 <module_registry>) at /usr/src/debug/php-5.4.12/Zend/zend_hash.c:687 #10 0x00007f24585a068c in zend_destroy_modules () at /usr/src/debug/php-5.4.12/Zend/zend_API.c:1797 #11 0x00007f245859aa0e in zend_shutdown () at /usr/src/debug/php-5.4.12/Zend/zend.c:820 #12 0x00007f245853c35b in php_module_shutdown () at /usr/src/debug/php-5.4.12/main/main.c:2365 #13 0x00007f245853c419 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.12/main/main.c:2333 #14 0x00007f2458642fc1 in php_apache_child_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.12/sapi/apache2handler/sapi_apache2.c:398 #15 0x00007f24602cd73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 #16 apr_pool_destroy (pool=0x7f24622f0658) at memory/unix/apr_pools.c:814 #17 0x00007f245a9951d4 in clean_child_exit (code=code@entry=0) at prefork.c:225 #18 0x00007f245a99563e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:739 #19 0x00007f245a9959ac in make_child (s=0x7f2461f82368, slot=slot@entry=1) at prefork.c:814 #20 0x00007f245a995a06 in startup_children (number_to_start=4) at prefork.c:832 #21 0x00007f245a9968d6 in prefork_run (_pconf=<optimized out>, plog=0x7f2461fb44f8, s=0x7f2461f82368) at prefork.c:990 #22 0x00007f24615e4d5e in ap_run_mpm (pconf=0x7f2461f57158, plog=0x7f2461fb44f8, s=0x7f2461f82368) at mpm_common.c:98 #23 0x00007f24615de40a in main (argc=2, argv=0x7fff20f5c7a8) at main.c:777 Sorry. :-(
Same result with php 5.4.13 (after yum update): (gdb) where #0 apr_palloc (pool=pool@entry=0x0, in_size=in_size@entry=32) at memory/unix/apr_pools.c:681 #1 0x00007f52a4bfa52b in ap_mutex_register (pconf=pconf@entry=0x0, type=type@entry=0x7f529df87ea1 "mpm-accept", default_dir=default_dir@entry=0x0, default_mech=default_mech@entry=APR_LOCK_DEFAULT, options=options@entry=0) at util_mutex.c:259 #2 0x00007f529df86aa4 in prefork_pre_config (p=0x0, plog=<optimized out>, ptemp=<optimized out>) at prefork.c:1302 #3 0x00007f52a5787190 in ?? () #4 <signal handler called> #5 0x00007f529bb995f6 in zend_hash_del_key_or_index (ht=ht@entry=0x7f52a57a3b70, arKey=arKey@entry=0x7f5296776289 "mcrypt_generic_end", nKeyLength=19, h=8096388789755616310, h@entry=0, flag=flag@entry=0) at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:514 #6 0x00007f529bb91cd8 in zend_unregister_functions (functions=<optimized out>, count=count@entry=-1, function_table=function_table@entry=0x0) at /usr/src/debug/php-5.4.13/Zend/zend_API.c:2206 #7 0x00007f529bb92e1a in module_destructor (module=0x7f52a582ac10) at /usr/src/debug/php-5.4.13/Zend/zend_API.c:2280 #8 0x00007f529bb98105 in zend_hash_apply_deleter (ht=ht@entry=0x7f529bf7d240 <module_registry>, p=0x7f52a582abb0) at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:650 #9 0x00007f529bb998f8 in zend_hash_graceful_reverse_destroy (ht=0x7f529bf7d240 <module_registry>) at /usr/src/debug/php-5.4.13/Zend/zend_hash.c:687 #10 0x00007f529bb9170c in zend_destroy_modules () at /usr/src/debug/php-5.4.13/Zend/zend_API.c:1797 #11 0x00007f529bb8ba8e in zend_shutdown () at /usr/src/debug/php-5.4.13/Zend/zend.c:820 #12 0x00007f529bb2d3cb in php_module_shutdown () at /usr/src/debug/php-5.4.13/main/main.c:2365 #13 0x00007f529bb2d489 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.13/main/main.c:2333 #14 0x00007f529bc34041 in php_apache_child_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.13/sapi/apache2handler/sapi_apache2.c:398 #15 0x00007f52a38be73e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 #16 apr_pool_destroy (pool=0x7f52a59646c8) at memory/unix/apr_pools.c:814 #17 0x00007f529df861d4 in clean_child_exit (code=code@entry=0) at prefork.c:225 #18 0x00007f529df8663e in child_main (child_num_arg=child_num_arg@entry=3) at prefork.c:739 #19 0x00007f529df869ac in make_child (s=0x7f52a55f6368, slot=slot@entry=3) at prefork.c:814 #20 0x00007f529df86a06 in startup_children (number_to_start=2) at prefork.c:832 #21 0x00007f529df878d6 in prefork_run (_pconf=<optimized out>, plog=0x7f52a56284f8, s=0x7f52a55f6368) at prefork.c:990 #22 0x00007f52a4bd5d5e in ap_run_mpm (pconf=0x7f52a55cb158, plog=0x7f52a56284f8, s=0x7f52a55f6368) at mpm_common.c:98 #23 0x00007f52a4bcf40a in main (argc=2, argv=0x7fff2c297fb8) at main.c:777
Can you paste error_log from this restart try, please?
# find /tmp -name core* # >/var/log/httpd/error_log # service httpd start Redirecting to /bin/systemctl start httpd.service # service httpd stop Redirecting to /bin/systemctl stop httpd.service # find /tmp -name core* /tmp/systemd-private-JJkv1V/core.5306 /tmp/systemd-private-JJkv1V/core.5308 /tmp/systemd-private-JJkv1V/core.5307 /tmp/systemd-private-JJkv1V/core.5304 /tmp/systemd-private-JJkv1V/core.5305 # cat /var/log/httpd/error_log [Thu Apr 04 03:35:48.149713 2013] [core:notice] [pid 5303] core dump file size limit raised to 18446744073709551615 bytes [Thu Apr 04 03:35:48.150686 2013] [core:notice] [pid 5303] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Thu Apr 04 03:35:49.001561 2013] [ssl:notice] [pid 5303] AH01886: SSL FIPS mode disabled [Thu Apr 04 03:35:49.147944 2013] [auth_digest:notice] [pid 5303] AH01757: generating secret for digest authentication ... [Thu Apr 04 03:35:50.001376 2013] [ssl:notice] [pid 5303] AH01886: SSL FIPS mode disabled [Thu Apr 04 03:35:50.141728 2013] [mpm_prefork:notice] [pid 5303] AH00163: Apache/2.4.4 (Fedora) OpenSSL/1.0.1e-fips configured -- resuming normal operations [Thu Apr 04 03:35:50.148173 2013] [core:notice] [pid 5303] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu Apr 04 03:35:52.684792 2013] [mpm_prefork:notice] [pid 5303] AH00170: caught SIGWINCH, shutting down gracefully
Sorry, timezone was misconfigured. Actually, is around 10:48 here.
Hi, I have another candidate here: http://koji.fedoraproject.org/koji/taskinfo?taskID=5252585 It would be great to try this one and write here if it still freezes/crashes when stopping httpd using systemd.
Seems to work fine. No more segfaults. Thank you.
Well, does not work. I still have core dumps and it still freezes. # find /tmp -name core\* /tmp/core.11920 /tmp/core.12041 /tmp/core.12089 On first computer does not generate core dumps. On the second computer I have coredumps and sometimes restart freeze.
(gdb) where #0 0x00007f7b58bc2ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 #1 0x00007f7b58bc4358 in __GI_abort () at abort.c:90 #2 0x00007f7b58c0259b in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f7b58d06ba8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:197 #3 0x00007f7b58c09a8e in malloc_printerr (ptr=0x7f7b5c279180, str=0x7f7b58d04a01 "free(): invalid pointer", action=3) at malloc.c:4969 #4 _int_free (av=0x7f7b58f3e740 <main_arena>, p=0x7f7b5c279170, have_lock=0) at malloc.c:3826 #5 0x00007f7b5937e85c in allocator_free (node=0x7f7b5c279180, allocator=0x7f7b5c43a2c0) at memory/unix/apr_pools.c:430 #6 apr_pool_destroy (pool=0x7f7b5c43a3b8) at memory/unix/apr_pools.c:856 #7 0x00007f7b53a461be in clean_child_exit (code=code@entry=0) at prefork.c:218 #8 0x00007f7b53a469fb in just_die (sig=<optimized out>) at prefork.c:344 #9 <signal handler called> #10 cleanup_fscreatecon () at selinux.c:78 #11 0x00007f7b5a45d5f7 in _dl_fini () at dl-fini.c:253 #12 0x00007f7b58bc5df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f7b58f3e6a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 #13 0x00007f7b58bc5e75 in __GI_exit (status=status@entry=0) at exit.c:99 #14 0x00007f7b53a461db in clean_child_exit (code=code@entry=0) at prefork.c:227 #15 0x00007f7b53a4660e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:725 #16 0x00007f7b53a4697c in make_child (s=0x7f7b5c0c7368, slot=slot@entry=1) at prefork.c:800 #17 0x00007f7b53a469d6 in startup_children (number_to_start=4) at prefork.c:818 #18 0x00007f7b53a47886 in prefork_run (_pconf=<optimized out>, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at prefork.c:976 #19 0x00007f7b5a695d5e in ap_run_mpm (pconf=0x7f7b5c09c158, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at mpm_common.c:98 #20 0x00007f7b5a68f40a in main (argc=2, argv=0x7fff2f64eb88) at main.c:777
(gdb) bt full #0 0x00007f7b58bc2ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63 resultvar = 0 pid = 11920 selftid = 11920 #1 0x00007f7b58bc4358 in __GI_abort () at abort.c:90 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7f7b58d04bc1, sa_sigaction = 0x7f7b58d04bc1}, sa_mask = {__val = { 3, 140733988525067, 5, 140167747747220, 1, 140167747753918, 3, 140733988525044, 12, 140167747753922, 2, 140167747753922, 2, 140733988525856, 19, 140733988527616}}, sa_flags = 88, sa_restorer = 0x7} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f7b58c0259b in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f7b58d06ba8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:197 ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fff2f64de10, reg_save_area = 0x7fff2f64dd20}} ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7fff2f64de10, reg_save_area = 0x7fff2f64dd20}} fd = 2 on_2 = <optimized out> list = <optimized out> nlist = <optimized out> cp = <optimized out> written = <optimized out> #3 0x00007f7b58c09a8e in malloc_printerr (ptr=0x7f7b5c279180, str=0x7f7b58d04a01 "free(): invalid pointer", action=3) at malloc.c:4969 buf = "00007f7b5c279180" cp = <optimized out> #4 _int_free (av=0x7f7b58f3e740 <main_arena>, p=0x7f7b5c279170, have_lock=0) at malloc.c:3826 size = <optimized out> fb = <optimized out> nextchunk = <optimized out> nextsize = <optimized out> nextinuse = <optimized out> prevsize = <optimized out> bck = <optimized out> fwd = <optimized out> errstr = 0x7f7b58d04a01 "free(): invalid pointer" locked = <optimized out> #5 0x00007f7b5937e85c in allocator_free (node=0x7f7b5c279180, allocator=0x7f7b5c43a2c0) at memory/unix/apr_pools.c:430 freelist = 0x0 max_index = <optimized out> max_free_index = <optimized out> next = <optimized out> index = <optimized out> current_free_index = <optimized out> #6 apr_pool_destroy (pool=0x7f7b5c43a3b8) at memory/unix/apr_pools.c:856 active = <optimized out> allocator = 0x7f7b5c43a2c0 #7 0x00007f7b53a461be in clean_child_exit (code=code@entry=0) at prefork.c:218 No locals. #8 0x00007f7b53a469fb in just_die (sig=<optimized out>) at prefork.c:344 No locals. #9 <signal handler called> No locals. #10 cleanup_fscreatecon () at selinux.c:78 No locals. #11 0x00007f7b5a45d5f7 in _dl_fini () at dl-fini.c:253 array = 0x7f7b52889cc8 i = 2 nmaps = 76 nloaded = <optimized out> i = 53 l = 0x7f7b5c260450 ns = 0 maps = 0x7fff2f64e390 maps_size = 608 do_audit = 0 #12 0x00007f7b58bc5df1 in __run_exit_handlers (status=status@entry=0, listp=0x7f7b58f3e6a8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77 atfct = <optimized out> onfct = <optimized out> cxafct = <optimized out> f = <optimized out> #13 0x00007f7b58bc5e75 in __GI_exit (status=status@entry=0) at exit.c:99 No locals. #14 0x00007f7b53a461db in clean_child_exit (code=code@entry=0) at prefork.c:227 No locals. #15 0x00007f7b53a4660e in child_main (child_num_arg=child_num_arg@entry=1) at prefork.c:725 thd = 0x7f7b5c43a430 osthd = 140167774263360 ptrans = 0x7f7b5c43c3c8 allocator = 0x7f7b5c43a2c0 status = <optimized out> i = <optimized out> lr = <optimized out> pollset = 0x7f7b5c43a510 sbh = 0x7f7b5c43a508 bucket_alloc = 0x7f7b5c4403e8 last_poll_idx = <optimized out> lockfile = <optimized out> #16 0x00007f7b53a4697c in make_child (s=0x7f7b5c0c7368, slot=slot@entry=1) at prefork.c:800 pid = 0 #17 0x00007f7b53a469d6 in startup_children (number_to_start=4) at prefork.c:818 i = 1 #18 0x00007f7b53a47886 in prefork_run (_pconf=<optimized out>, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at prefork.c:976 index = <optimized out> remaining_children_to_start = <optimized out> rv = <optimized out> #19 0x00007f7b5a695d5e in ap_run_mpm (pconf=0x7f7b5c09c158, plog=0x7f7b5c0f94f8, s=0x7f7b5c0c7368) at mpm_common.c:98 pHook = 0x7f7b5c16dce0 n = 0 rv = 1384684992 #20 0x00007f7b5a68f40a in main (argc=2, argv=0x7fff2f64eb88) at main.c:777 c = 68 'D' showcompile = 0 showdirectives = 0 confname = 0x7f7b5a6ca2af "conf/httpd.conf" def_server_root = 0x7f7b5a6ca2a4 "/etc/httpd" temp_error_log = <optimized out> error = <optimized out> process = 0x7f7b5c09a238 pconf = 0x7f7b5c09c158 plog = 0x7f7b5c0f94f8 ptemp = 0x7f7b5c0c5318 pcommands = 0x7f7b5c0be268 opt = 0x7f7b5c0be358 rv = <optimized out> mod = 0x7f7b5a8e5098 <ap_prelinked_modules+24> opt_arg = 0x7fff2f64ef33 "FOREGROUND" signal_server = <optimized out>
Oh, I see... You added KillSignal=SIGWINCH to httpd.service On the machine where I still have coredumps I have a custom httpd.service (I temporarily disabled PrivateTmp). If you ask me, is an ugly workaround. Still, at least I do not have coredumps anymore: # watch -n 5 "service httpd start; sleep 5; /usr/sbin/httpd -k graceful-stop"
So with KillSignal=SIGWINCH and KillMode=process it does work for you (with the updated httpd.service)? I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. It crashes/freezes, because SIGTERM signal interrupts SIGWINCH signal handler which can be in the middle of malloc/free and those functions are not reentrant. The best way would be to have httpd which can handle signals in any order or at any time, but unfortunately we don't have it because of httpd's design. I can try to open discussion about that on upstream mailing list, but before we make some consensus, I would like to fix it this way, so if you could try that changed httpd.service works for you, it would be great.
I think this is better workaround, because if graceful-stop fails for any reason, then systemd will still SIGKILL it after 90 seconds: [Unit] Description=The Apache HTTP Server After=network.target remote-fs.target nss-lookup.target [Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop # We want systemd to give httpd some time to finish gracefully, but still want # it to kill httpd after TimeoutStopSec if something went wrong during the # grafecul stop. Normally, Systemd sends SIGTERM signal right after the # ExecStop, which would kill httpd. We are sending useless SIGCONT here to give # httpd time to finish. KillSignal=SIGCONT PrivateTmp=true [Install] WantedBy=multi-user.target
> So with KillSignal=SIGWINCH and KillMode=process it does work for you (with the updated httpd.service)? Yes, the new httpd.serviceit works fine. > I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. Then this seems like a systemd bug. The unit from comment #33 seems is a lot better. I like-it. Thx.
> > I think there is no better way to fix this. Systemd executes ExecStop, which results in sending SIGWINCH to httpd, but right after that, systemd sends SIGTERM, so httpd does not have any time to stop gracefully and exits right away, which is not what is intended. > Then this seems like a systemd bug. You can follow up here if you want: https://bugzilla.redhat.com/show_bug.cgi?id=952634
Thank you.
httpd-2.4.4-4.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/httpd-2.4.4-4.fc19
Package httpd-2.4.4-4.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing httpd-2.4.4-4.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-5797/httpd-2.4.4-4.fc19 then log in and leave karma (feedback).
httpd-2.4.4-4.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 909703 has been marked as a duplicate of this bug. ***
httpd-2.4.4-3.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/httpd-2.4.4-3.fc18
httpd-2.4.4-3.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.