2010166 – httpd sometimes core dumps when it is started

Bug 2010166 - httpd sometimes core dumps when it is started

Summary: httpd sometimes core dumps when it is started

Keywords:
Status:	NEW
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	httpd
Sub Component:
Version:	35
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Luboš Uhliarik
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-10-04 05:26 UTC by Patrick Dung
Modified:	2021-10-11 07:29 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type:	Bug
Dependent Products:

Attachments	(Terms of Use)
output of "thread apply all bt full" (41.38 KB, text/plain) 2021-10-04 05:36 UTC, Patrick Dung	no flags	Details
output for httpd 2.4.51.txt (31.88 KB, text/plain) 2021-10-11 07:29 UTC, Patrick Dung	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Patrick Dung 2021-10-04 05:26:34 UTC

Description of problem:
The problem exists since FC34.
Sometimes when httpd starts, it core dumps several files but it is able to start up without intervention.

-rw-------  1 root          apache        271429632 Oct  3 14:45 core.httpd-16891-11-1633243502
-rw-------  1 root          apache        179515392 Oct  3 14:45 core.httpd-17782-11-1633243503
-rw-------  1 root          apache        356098048 Oct  3 14:45 core.httpd-18384-11-1633243503
-rw-------  1 root          apache        398430208 Oct  3 14:45 core.httpd-18999-11-1633243504
-rw-------  1 root          apache        440766464 Oct  3 14:45 core.httpd-19383-11-1633243507
-rw-------  1 root          apache        483098624 Oct  3 14:45 core.httpd-19737-11-1633243508
-rw-------  1 root          apache        525430784 Oct  3 14:45 core.httpd-20154-11-1633243508
-rw-------  1 root          apache        567767040 Oct  3 14:45 core.httpd-20633-11-1633243510
-rw-------  1 root          apache        610099200 Oct  3 14:45 core.httpd-21085-11-1633243511
-rw-------  1 root          apache        652435456 Oct  3 14:45 core.httpd-21519-11-1633243512
-rw-------  1 root          apache        694767616 Oct  3 14:45 core.httpd-22906-11-1633243515
-rw-------  1 root          apache        743391232 Oct  3 14:45 core.httpd-24520-11-1633243518
-rw-------  1 root          apache        785727488 Oct  3 14:45 core.httpd-26026-11-1633243520
-rw-------  1 root          apache        828059648 Oct  3 14:45 core.httpd-29431-11-1633243525

Backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f0fb6c7d659 in apr_atomic_dec32 () from /lib64/libapr-1.so.0
[Current thread is 1 (Thread 0x7f0fb66e9140 (LWP 16891))]
(gdb) bt
#0  0x00007f0fb6c7d659 in apr_atomic_dec32 () at /lib64/libapr-1.so.0
#1  0x00007f0fb6c90219 in proc_pthread_mutex_unref () at /lib64/libapr-1.so.0
#2  0x00007f0fb6c90cbe in apr_pool_destroy () at /lib64/libapr-1.so.0
#3  0x00007f0fb5b8c52f in clean_child_exit (code=0) at /usr/src/debug/httpd-2.4.49-1.fc35.x86_64/server/mpm/event/event.c:730
#4  0x00007f0fb5b8e99a in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0)
    at /usr/src/debug/httpd-2.4.49-1.fc35.x86_64/server/mpm/event/event.c:2556
#5  0x00007f0fb5b8f2cf in make_child (s=0x7f0fb6247110, slot=slot@entry=0, bucket=0)
    at /usr/src/debug/httpd-2.4.49-1.fc35.x86_64/server/mpm/event/event.c:2754
#6  0x00007f0fb5b8f371 in startup_children (number_to_start=number_to_start@entry=1)
    at /usr/src/debug/httpd-2.4.49-1.fc35.x86_64/server/mpm/event/event.c:2777
#7  0x00007f0fb5b9013f in event_run (_pconf=<optimized out>, plog=0x7f0fb624b028, s=0x7f0fb6247110)
    at /usr/src/debug/httpd-2.4.49-1.fc35.x86_64/server/mpm/event/event.c:3150
#8  0x00005618346493f8 in ap_run_mpm (pconf=0x7f0fb621f028, plog=0x7f0fb624b028, s=0x7f0fb6247110) at server/mpm_common.c:95
#9  0x0000561834637484 in main (argc=<optimized out>, argv=<optimized out>) at server/main.c:819

Version-Release number of selected component (if applicable):
httpd-2.4.49-1.fc35.x86_64

How reproducible:
It is not like this on every time apache starts.
But it is occurring about two to three times every week.

Steps to Reproduce:
System reboot and systemd starts apache

Comment 1 Patrick Dung 2021-10-04 05:36:51 UTC

Created attachment 1828813 [details]
output of "thread apply all bt full"

Comment 2 Joe Orton 2021-10-08 07:24:28 UTC

Please try httpd 2.4.51 and see if this still occurs.

https://bodhi.fedoraproject.org/updates/FEDORA-2021-2a10bc68a4

Comment 3 Patrick Dung 2021-10-08 16:04:00 UTC

When I upgrade to httpd 2.4.51, FC35 binary has not yet been released/built.
So I use httpd 2.4.51 FC34 rpm on FC35.

It still have core dumps. Here's one of it:

(gdb) thread apply all bt full

Thread 18 (Thread 0x7f77a61fc640 (LWP 1012186)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 17 (Thread 0x7f77a09f1640 (LWP 1012197)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 16 (Thread 0x7f77a39f7640 (LWP 1012191)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 15 (Thread 0x7f77a31f6640 (LWP 1012192)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
--Type <RET> for more, q to quit, c to continue without paging--c
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 14 (Thread 0x7f77a19f3640 (LWP 1012195)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 13 (Thread 0x7f77a41f8640 (LWP 1012190)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 12 (Thread 0x7f77a51fa640 (LWP 1012188)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 11 (Thread 0x7f77a21f4640 (LWP 1012194)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 10 (Thread 0x7f77a29f5640 (LWP 1012193)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 9 (Thread 0x7f77a59fb640 (LWP 1012187)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 8 (Thread 0x7f77a8615640 (LWP 1012182)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 7 (Thread 0x7f77a69fd640 (LWP 1012185)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 6 (Thread 0x7f77a79ff640 (LWP 1012183)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 5 (Thread 0x7f77a49f9640 (LWP 1012189)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 4 (Thread 0x7f77a11f2640 (LWP 1012196)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 3 (Thread 0x7f77a71fe640 (LWP 1012184)):
#0  0x00007f77b573d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007f77b573fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007f77b598cc0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6

Thread 2 (Thread 0x7f77a8e16640 (LWP 1012181)):
#0  0x00007f77b57bb02d in select () at /lib64/libc.so.6
#1  0x00007f77b58f4639 in apr_sleep () at /lib64/libapr-1.so.0
#2  0x00007f77b491ad9e in wd_worker (thread=<optimized out>, data=0x7f77b2db2770) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/modules/core/mod_watchdog.c:192
        curr = <optimized out>
        wl = 0x7f77b2db2900
        w = 0x7f77b2db2770
        rv = <optimized out>
        locked = 1
        probed = <optimized out>
        inited = 0
        mpmq_s = 1
        temp_pool = 0x7f77b4fa5028
#3  0x00007f77b5740b17 in start_thread () at /lib64/libc.so.6
#4  0x00007f77b57c56a0 in clone3 () at /lib64/libc.so.6


Thread 1 (Thread 0x7f77b533f140 (LWP 1012180)):
#0  0x00007f77b58dd659 in apr_atomic_dec32 () at /lib64/libapr-1.so.0
#1  0x00007f77b58f0219 in proc_pthread_mutex_unref () at /lib64/libapr-1.so.0
#2  0x00007f77b58f0cbe in apr_pool_destroy () at /lib64/libapr-1.so.0
#3  0x00007f77b47e152f in clean_child_exit (code=0) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/server/mpm/event/event.c:730
#4  0x00007f77b47e396a in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/server/mpm/event/event.c:2555
        threads = <optimized out>
        rv = <optimized out>
        ts = <optimized out>
        thread_attr = 0x7f77b2e465e8
        start_thread_id = 0x7f77b2e46648
        i = <optimized out>
#5  0x00007f77b47e429f in make_child (s=0x7f77b4e47110, slot=slot@entry=0, bucket=bucket@entry=0) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/server/mpm/event/event.c:2753
        pid = <optimized out>
#6  0x00007f77b47e5269 in server_main_loop (num_buckets=1, remaining_children_to_start=1) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/server/mpm/event/event.c:3052
        ps = <optimized out>
        status = 11
        pid = {pid = 1006218, in = 0x7f77b2d4ab00, out = 0x2e3836312e323931, err = 0x38392e32}
        i = <optimized out>
        child_slot = 0
        exitwhy = (APR_PROC_SIGNAL | APR_PROC_SIGNAL_CORE)
        processed_status = 0
        num_buckets = 1
        remaining_children_to_start = <optimized out>
        i = <optimized out>
#7  event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at /usr/src/debug/httpd-2.4.51-1.fc34.x86_64/server/mpm/event/event.c:3171
        num_buckets = 1
        remaining_children_to_start = <optimized out>
        i = <optimized out>
#8  0x000055f66ebe2428 in ap_run_mpm (pconf=0x7f77b4e1f028, plog=0x7f77b4e4b028, s=0x7f77b4e47110) at server/mpm_common.c:95
        pHook = <optimized out>
        n = 0
        rv = -1
#9  0x000055f66ebd0484 in main (argc=<optimized out>, argv=<optimized out>) at server/main.c:819
        c = 102 'f'
        showcompile = <optimized out>
        showdirectives = 0
        confname = 0x7fff37458ea2 "/etc/httpd-web1/conf/httpd.conf"
        def_server_root = 0x55f66ec1d761 "/etc/httpd"
        temp_error_log = <optimized out>
        error = <optimized out>
        process = 0x7f77b4e1d118
        pconf = <optimized out>
        plog = 0x7f77b4e4b028
        ptemp = 0x7f77b4e76028
        pcommands = 0x7f77b4e41028
        opt = 0x7f77b4e41118
        rv = <optimized out>
        mod = <optimized out>
        opt_arg = 0x7fff37458ea2 "/etc/httpd-web1/conf/httpd.conf"
        signal_server = <optimized out>
        rc = <optimized out>


Now I see httpd-2.4.51 FC35 rpm is out in koji. I would install it.

Comment 4 Patrick Dung 2021-10-10 06:24:56 UTC

It is still crashing with the httpd 2.4.51 FC35 rpm:

Core was generated by `/usr/sbin/httpd -DFOREGROUND -f /etc/httpd-web1/conf/httpd.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ff05da33659 in apr_atomic_dec32 () from /lib64/libapr-1.so.0
[Current thread is 1 (Thread 0x7ff05d49f140 (LWP 19523))]
(gdb)
(gdb)
(gdb) thread apply all bt full

Thread 18 (Thread 0x7ff0568d6640 (LWP 19527)):
#0  0x00007ff05d91b02d in select () at /lib64/libc.so.6
#1  0x00007ff05da4a639 in apr_sleep () at /lib64/libapr-1.so.0
#2  0x00007ff05c98dd9e in wd_worker (thread=<optimized out>, data=0x7ff05af56770) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/modules/core/mod_watchdog.c:192
        curr = <optimized out>
        wl = 0x7ff05af56900
        w = 0x7ff05af56770
        rv = <optimized out>
        locked = 1
        probed = <optimized out>
        inited = 0
        mpmq_s = 1
        temp_pool = 0x7ff05d17a028
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 17 (Thread 0x7ff0533fb640 (LWP 19533)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 16 (Thread 0x7ff051bf8640 (LWP 19536)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 15 (Thread 0x7ff04ebf2640 (LWP 19542)):
--Type <RET> for more, q to quit, c to continue without paging--c
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 14 (Thread 0x7ff04f3f3640 (LWP 19541)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 13 (Thread 0x7ff04fbf4640 (LWP 19540)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 12 (Thread 0x7ff0503f5640 (LWP 19539)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 11 (Thread 0x7ff050bf6640 (LWP 19538)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 10 (Thread 0x7ff0513f7640 (LWP 19537)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 9 (Thread 0x7ff0523f9640 (LWP 19535)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 8 (Thread 0x7ff052bfa640 (LWP 19534)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 7 (Thread 0x7ff053bfc640 (LWP 19532)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 6 (Thread 0x7ff0543fd640 (LWP 19531)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 5 (Thread 0x7ff04e3f1640 (LWP 19543)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 4 (Thread 0x7ff054bfe640 (LWP 19530)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 3 (Thread 0x7ff0553ff640 (LWP 19529)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6

Thread 2 (Thread 0x7ff0560d5640 (LWP 19528)):
#0  0x00007ff05d89d77a in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007ff05d89fef0 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libc.so.6
#2  0x00007ff05dae2c0f in thread_pool_func () at /lib64/libaprutil-1.so.0
#3  0x00007ff05d8a0b17 in start_thread () at /lib64/libc.so.6
#4  0x00007ff05d9256a0 in clone3 () at /lib64/libc.so.6


Thread 1 (Thread 0x7ff05d49f140 (LWP 19523)):
#0  0x00007ff05da33659 in apr_atomic_dec32 () at /lib64/libapr-1.so.0
#1  0x00007ff05da46219 in proc_pthread_mutex_unref () at /lib64/libapr-1.so.0
#2  0x00007ff05da46cbe in apr_pool_destroy () at /lib64/libapr-1.so.0
#3  0x00007ff05c94a52f in clean_child_exit (code=0) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/server/mpm/event/event.c:730
#4  0x00007ff05c94c96a in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/server/mpm/event/event.c:2555
        threads = <optimized out>
        rv = <optimized out>
        ts = <optimized out>
        thread_attr = 0x7ff05afc85e8
        start_thread_id = 0x7ff05afc8648
        i = <optimized out>
#5  0x00007ff05c94d29f in make_child (s=0x7ff05d047110, slot=slot@entry=0, bucket=0) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/server/mpm/event/event.c:2753
        pid = <optimized out>
#6  0x00007ff05c94d341 in startup_children (number_to_start=number_to_start@entry=1) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/server/mpm/event/event.c:2776
        i = 0
#7  0x00007ff05c94e14f in event_run (_pconf=<optimized out>, plog=0x7ff05d04b028, s=0x7ff05d047110) at /usr/src/debug/httpd-2.4.51-1.fc35.x86_64/server/mpm/event/event.c:3152
        num_buckets = 1
        remaining_children_to_start = 1
        i = <optimized out>
#8  0x0000560e5a05a428 in ap_run_mpm (pconf=0x7ff05d01f028, plog=0x7ff05d04b028, s=0x7ff05d047110) at server/mpm_common.c:95
        pHook = <optimized out>
        n = 0
        rv = -1
#9  0x00007ff05d01f028 in  ()
#10 0x00007ff05d01d118 in  ()
#11 0x0000560e5a0ba4a0 in ap_conftree ()
#12 0x00007ffecf2916c0 in  ()
#13 0x0000560e5a0957b3 in  ()
#14 0x0000560e5a048484 in main (argc=<optimized out>, argv=<optimized out>) at server/main.c:819
        c = 93 ']'
        showcompile = <optimized out>
        showdirectives = 1552159360
        confname = 0x71f476b4 <error: Cannot access memory at address 0x71f476b4>
        def_server_root = 0x0
        temp_error_log = <optimized out>
        error = <optimized out>
        process = 0x7ff05d01f028
        pconf = <optimized out>
        plog = 0x7
        ptemp = 0xf4908b9c7cd56700
        pcommands = 0x7ff05c841280 <cgi_handle_exec>
        opt = 0x14
        rv = <optimized out>
        mod = <optimized out>
        opt_arg = 0x0
        signal_server = <optimized out>
        rc = <optimized out>

Comment 5 Joe Orton 2021-10-11 07:13:31 UTC

Thanks for testing further.  Could you do

# debuginfo-install apr glibc

and regenerate the backtraces?

Comment 6 Patrick Dung 2021-10-11 07:29:23 UTC

Created attachment 1831718 [details]
output for httpd 2.4.51.txt

For https://bugzilla.redhat.com/show_bug.cgi?id=2010166#c5

Note You need to log in before you can comment on or make changes to this bug.