Fedora Account System
Red Hat Associate
Red Hat Customer
I attach a logfile with the complete coredump and relevant log entries. Dez 01 09:10:10 SERVERNAME systemd[1]: Stopped httpd.service - The Apache HTTP Server (prefork MPM). Dez 01 09:10:10 SERVERNAME systemd[1]: httpd.service: Consumed 10h 17min 46.910s CPU time, 9.5G memory peak, 416K memory swap peak. Dez 01 09:38:08 SERVERNAME systemd[1]: Reloaded httpd.service - The Apache HTTP Server (prefork MPM). Dez 01 09:38:19 SERVERNAME httpd[3148244]: Server configured, listening on: xxxxxxxxxxxxxxxx port 443... Dez 01 09:41:46 SERVERNAME systemd[1]: /etc/systemd/system/httpd.service:7: PIDFile= references a path below legacy directory /var/run/, updating /var/run/httpd/httpd.pid → /run/httpd/httpd.pid; please update the unit file accordingly. Dez 01 09:42:18 SERVERNAME systemd-coredump[3323760]: Process 3317975 (httpd) of user 48 dumped core. Module mod_proxy_http2.so from rpm mod_http2-2.0.35-1.fc42.x86_64 Module libnghttp2.so.14 from rpm nghttp2-1.64.0-3.fc42.x86_64 Module mod_http2.so from rpm mod_http2-2.0.35-1.fc42.x86_64 Module libselinux.so.1 from rpm libselinux-3.8-3.fc42.x86_64 Module libbrotlicommon.so.1 from rpm brotli-1.1.0-6.fc42.x86_64 Module libbrotlienc.so.1 from rpm brotli-1.1.0-6.fc42.x86_64 Module libnss_myhostname.so.2 from rpm systemd-257.10-1.fc42.x86_64 Module libnss_systemd.so.2 from rpm systemd-257.10-1.fc42.x86_64 Module libnss_sss.so.2 from rpm sssd-2.11.1-2.fc42.x86_64 Module libssl.so.3 from rpm openssl-3.2.6-2.fc42.x86_64 Module libcrypto.so.3 from rpm openssl-3.2.6-2.fc42.x86_64 Module libcap.so.2 from rpm libcap-2.73-2.fc42.x86_64 Module libsystemd.so.0 from rpm systemd-257.10-1.fc42.x86_64 Module libz.so.1 from rpm zlib-ng-2.2.5-2.fc42.x86_64 Module libcrypt.so.2 from rpm libxcrypt-4.5.2-1.fc42.x86_64 Module libexpat.so.1 from rpm expat-2.7.2-1.fc42.x86_64 Module libuuid.so.1 from rpm util-linux-2.40.4-7.fc42.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.46-1.fc42.x86_64 Stack trace of thread 3318048: #0 0x00007faf87ed1e9c __pthread_kill_implementation (libc.so.6 + 0x73e9c) #1 0x00007faf87e77f3e raise (libc.so.6 + 0x19f3e) #2 0x00007faf87e5f6d0 abort (libc.so.6 + 0x16d0) #3 0x00007faf87e606f3 __libc_message_impl.cold (libc.so.6 + 0x26f3) #4 0x00007faf87edc035 malloc_printerr (libc.so.6 + 0x7e035) #5 0x00007faf87ee13cc free (libc.so.6 + 0x833cc) #6 0x00007faf8806fcac apr_pool_destroy (libapr-1.so.0 + 0x1fcac) #7 0x00007faf8788872c c1_purge_streams.lto_priv.0 (mod_http2.so + 0x1672c) #8 0x00007faf87888ef8 h2_mplx_c1_poll.constprop.0 (mod_http2.so + 0x16ef8) #9 0x00007faf8787b396 h2_c1_run (mod_http2.so + 0x9396) #10 0x000055a0c920f1da ap_run_process_connection (/usr/bin/httpd + 0x161da) #11 0x00007faf87a8ac23 process_socket (mod_mpm_event.so + 0x3c23) #12 0x00007faf87a8ba3b worker_thread (mod_mpm_event.so + 0x4a3b) #13 0x00007faf87ecff54 start_thread (libc.so.6 + 0x71f54) #14 0x00007faf87f5332c __clone3 (libc.so.6 + 0xf532c) Reproducible: Always Actual Results: httpd fail on startup, but starting it from bash afterwards works. Is it possible that the ipv6 network is not up, so he can't bind on the ipv6 addr? Additional Information: of course httpd worked flawless in fedora 41. After the upgrade to f42, it started to fail, but it's possible that we just did not notice it, due to a lack of reboots.
Created attachment 2116931 [details] httpd coredump and logs
I can now say clearly, that IPv6 is the issue. It's not rdy when httpd is started. [ipv6] addr-gen-mode=default address1=**REMOVED** address2=**REMOVED** method=manual I checked against sshd, and it bound to 0.0.0.0 on ipv4 and ivp6, which is always possible, even if the ipv6 has interface yet. The Difference is, apache wants a specific ipv6 addr, and that is not available at that time. Suggest to change: After=syslog.target network.target remote-fs.target nss-lookup.target to someting more reliable or involve the NetworkManager Team to fix the underlaying NetworkManager issue with hardcoded ipv6 addresses. (maybe they can explain why ipv6 became the preferred protocol after switching to fedora 42 from 41.. thats my next issue on the list of bugs)
Another red flag here: you have a custom httpd.service which claims to be "The Apache HTTP Server (prefork MPM)" - but the backtrace is for event. Please provide the actual core dump or install all the debuginfo and provide the output from "coredumpctl gdb" running "t a a bt".
the service from /usr/l/s/s/httpd.service just got copied to /etc/s/s/ and than edited, thats it. There is no core dump, because it gets killed with SIGTERM and not crashing.
sorry for the misunderstanding, i mixed the tickets..
Actual backtrace is as below. Hard to see how this is related to the IPv6 config. What's your Listen configuration, and can you reproduce with a standard httpd.service? #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007faf87ed1f63 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:89 #2 0x00007faf87e77f3e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007faf87e5f6d0 in __GI_abort () at abort.c:77 #4 0x00007faf87e606f3 in __libc_message_impl (fmt=fmt@entry=0x7faf880124ec "%s\n") at ../sysdeps/posix/libc_fatal.c:134 #5 0x00007faf87edc035 in malloc_printerr (str=str@entry=0x7faf880102c0 "free(): invalid pointer") at malloc.c:5829 #6 0x00007faf87ee13cc in _int_free_check (av=<optimized out>, p=<optimized out>, size=<optimized out>) at malloc.c:4560 #7 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:4692 #8 __GI___libc_free (mem=<optimized out>) at malloc.c:3476 #9 0x00007faf8806fcac in allocator_free (allocator=0x7faeb0013a20, node=<optimized out>) at memory/unix/apr_pools.c:492 #10 apr_pool_destroy (pool=0x7faf1c000b98) at memory/unix/apr_pools.c:1026 #11 0x00007faf8788872c in h2_stream_destroy (stream=<optimized out>) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_stream.c:635 #12 c1_purge_streams (m=m@entry=0x7faf3c01bc78) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_mplx.c:626 #13 0x00007faf87888ef8 in h2_mplx_c1_poll (on_stream_input=<optimized out>, on_stream_output=<optimized out>, m=0x7faf3c01bc78, timeout=0, on_ctx=0x7faf3c01b5e0) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_mplx.c:660 #14 h2_mplx_c1_poll.constprop.0 (m=0x7faf3c01bc78, timeout=0, on_ctx=0x7faf3c01b5e0, on_stream_output=<optimized out>, on_stream_input=<optimized out>) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_mplx.c:640 #15 0x00007faf8787b396 in h2_session_process (session=<optimized out>, async=<optimized out>, pkeepalive=<synthetic pointer>) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_session.c:2013 #16 h2_c1_run (c=0x7faef8015860) at /usr/src/debug/mod_http2-2.0.35-1.fc42.x86_64/mod_http2/h2_c1.c:135 #17 0x000055a0c920f1da in ap_run_process_connection (c=c@entry=0x7faef8015860) at server/connection.c:42 #18 0x00007faf87a8ac23 in process_socket (thd=thd@entry=0x55a0dddef700, p=<optimized out>, sock=<optimized out>, cs=<optimized out>, my_child_num=my_child_num@entry=0, my_thread_num=my_thread_num@entry=19) at /usr/src/debug/httpd-2.4.65-3.fc42.x86_64/server/mpm/event/event.c:1098 #19 0x00007faf87a8ba3b in worker_thread (thd=0x55a0dddef700, dummy=<optimized out>) at /usr/src/debug/httpd-2.4.65-3.fc42.x86_64/server/mpm/event/event.c:2252 #20 0x00007faf87ecff54 in start_thread (arg=<optimized out>) at pthread_create.c:448 #21 0x00007faf87f5332c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Hi, may I quickly ask a related question? Sorry in case this is not the right place to ask. I've already commented on the same bug in Apache's bugzilla (see https://bz.apache.org/bugzilla/show_bug.cgi?id=69899). However, I was unable to generate a usable backtrace from a coredump. The following is the only output I get from systemd-coredump: Stack trace of thread 15290: #0 0x00007f58f0a0ce9c n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 coredumpctl gdb does not help either. Is there anything I can do to generate a meaningful backtrace? I've already installed the debuginfo packages for httpd, httpd-core and mod_http2, but to no avail. Do I need more debuginfo packages? In the end, I attached gdb to httpd running in debug mode, but not having a sensible backtrace has kept me from reporting this bug for nearly two months now. If you have an idea, I'll try it on my system. Maybe this can confirm that I'm experiencing the same bug.
"coredumpctl gdb" generally works for me on Fedora systems and will pull needed debuginfo via debuginfod. Sometimes with gdb and httpd you won't get a backtrace of the right thread, so you have to use "thread apply all bt" to get the relevant backtrace. It used to be necessary to raise the coredump ulimit by configuring httpd with "CoreDumpDirectory /var/tmp" or similar.
I've actually raised the coredump ulimit by creating a drop-in file for httpd's systemd service with LimitCORE=unlimited. Also, I had applied CoreDumpDirectory /tmp (without changing the kernel.core_pattern though, so that the coredump would still be piped into systemd-coredump). But still, I did not get any meaningful backtraces. I'll try this again before installing 2.0.36 to see if I had done anything wrong.
FEDORA-2025-653f273fd0 (mod_http2-2.0.36-1.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-653f273fd0
FEDORA-2025-ae3ef77ad9 (mod_http2-2.0.36-1.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2025-ae3ef77ad9
2.0.36 updates available above - please test.
(In reply to Joshua Noeske from comment #10) > I've actually raised the coredump ulimit by creating a drop-in file for > httpd's systemd service with LimitCORE=unlimited. Also, I had applied > CoreDumpDirectory /tmp (without changing the kernel.core_pattern though, so > that the coredump would still be piped into systemd-coredump). But still, I > did not get any meaningful backtraces. > > I'll try this again before installing 2.0.36 to see if I had done anything > wrong. hmm.. i get coredumps all the time without altering anything related # systemctl show httpd |grep -i core CoredumpReceive=no LimitCORE=infinity LimitCORESoft=infinity OOMScoreAdjust=0 CoredumpFilter=0x33 the generated coredumps /var/lib/systemd/coredump/ (mostly from php) eating my hdspace without alerting anyone that they happend, except for the rare case of an issue as in this ticket. TBH systemd needs a housekeeper cronjob for those. Regarding your question: > Actual backtrace is as below. Hard to see how this is related to the IPv6 config. What's your Listen configuration, and can you > reproduce with a standard httpd.service? I unfortunatly can't do tests on the servers as they are production machines /etc/httpd/conf/httpd.conf:Listen 83.246.90.XXX:80 /etc/httpd/conf/httpd.conf:Listen 127.0.0.1:80 /etc/httpd/conf/httpd.conf:Listen [2a02:XXXX:1:20::XXX]:80
Created attachment 2118279 [details] unsuccessful backtrace generation attempt from coredump I've again tried to get a meaningful backtrace with having set ulimit -c unlimited and CoreDumpDirectory /tmp, however, I was unsuccessful. I've attached the logs from the gdb session, maybe they are helpful with figuring out why I can't get any logs.
(In reply to Joe Orton from comment #13) > 2.0.36 updates available above - please test. Thanks! I'll install the update immediately.
Updated the bodhi updates with 2.0.37 which has a likely fix for this now.
FEDORA-2025-ae3ef77ad9 has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ae3ef77ad9` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ae3ef77ad9 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-653f273fd0 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-653f273fd0` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-653f273fd0 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-ae3ef77ad9 (mod_http2-2.0.37-1.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-653f273fd0 (mod_http2-2.0.37-1.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.