Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
I have a customer who is seeing segfaults at:
(gdb) bt
#0 0x00007f960ed94041 in ap_proxy_set_scoreboard_lb (worker=0x7f961a602948, balancer=<optimized out>, server=<optimized out>)
at /usr/include/bits/string3.h:52
#1 0x00007f960ed940c4 in ap_proxy_initialize_worker_share (conf=0x7f961a602770, worker=0x7f961a602948, s=0x7f961a692548)
at /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1903
#2 0x00007f960ed8fb0e in child_init (p=0x7f961a549418, s=0x7f961a692548) at /usr/src/debug/httpd-2.2.15/modules/proxy/mod_proxy.c:2308
#3 0x00007f9617e5a89c in ap_run_child_init (pchild=0x7f961a549418, s=0x7f961a039880)
at /usr/src/debug/httpd-2.2.15/server/config.c:155
#4 0x00007f9617e6f420 in child_main (child_num_arg=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1161
#5 0x00007f9617e6f81f in make_child (s=0x7f961a039880, slot=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1341
#6 0x00007f9617e701f8 in startup_children (number_to_start=<optimized out>)
at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1375
#7 server_main_loop (remaining_children_to_start=<optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1654
#8 ap_mpm_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>)
at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1765
#9 0x00007f9617e46420 in main (argc=1, argv=0x7fff97f90bd8) at /usr/src/debug/httpd-2.2.15/server/main.c:763
Segfaults occur at proxy_util.c:1874 accessing free_slot->digest:
(gdb) l /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1874
1869 }
1870 }
1871
1872 /* We failed to find out shared memory, so just use a free slot (if any) */
1873 if (free_slot) {
1874 memcpy(free_slot->digest, digest, APR_MD5_DIGESTSIZE);
1875 worker->s = free_slot;
1876 }
1877 }
1878 return worker->s;
(gdb)
Memory is not assigned in the midst of the free_slot:
(gdb) print free_slot
$1 = (proxy_worker_stat *) 0x7f96091f3f48
(gdb) x/32gx 0x7f96091f3f48
0x7f96091f3f48: 0x0000000000000000 0x0000000000000000
0x7f96091f3f58: 0x0000000000000000 0x0000000000000000
0x7f96091f3f68: 0x0000000000000000 0x0000000000000000
0x7f96091f3f78: 0x0000000000000000 0x0000000000000000
0x7f96091f3f88: 0x0000000000000000 0x0000000000000000
0x7f96091f3f98: 0x0000000000000000 0x0000000000000000
0x7f96091f3fa8: 0x0000000000000000 0x0000000000000000
0x7f96091f3fb8: 0x0000000000000000 0x0000000000000000
0x7f96091f3fc8: 0x0000000000000000 0x0000000000000000
0x7f96091f3fd8: 0x0000000000000000 0x0000000000000000
0x7f96091f3fe8: 0x0000000000000000 0x0000000000000000
0x7f96091f3ff8: 0x0000000000000000 Cannot access memory at address 0x7f96091f4000
(gdb) print free_slot.digest
Cannot access memory at address 0x7f96091f4014
In the core, lb_limit == 78,
((gdb) print lb_limit
$53 = 78
The free_slot == the 78th element of ap_scoreboard_image.balancers, and it is stored at:
gdb) print &(proxy_worker_stat)(ap_scoreboard_image.balancers[78])
$54 = (struct {...} *) 0x7f96091f3f48
According to ./usr/src/debug/httpd-2.2.15/server/scoreboard.c, it is assumed that ap_scoreboard_image->balancers has lb_limit+1 elements (0 ~ lb_limit):
543 AP_DECLARE(lb_score *) ap_get_scoreboard_lb(int lb_num)
544 {
545 if (((lb_num < 0) || (lb_limit < lb_num))) {
546 return(NULL); /* Out of range */
547 }
548 return &ap_scoreboard_image->balancers[lb_num];
549 }
But, in fact, calloc'ed memory was smaller than it needs:
95 AP_DECLARE(int) ap_calc_scoreboard_size(void)
96 {
97 ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, &thread_limit);
98 ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, &server_limit);
99
100 if (!proxy_lb_workers)
101 proxy_lb_workers = APR_RETRIEVE_OPTIONAL_FN(ap_proxy_lb_workers);
102 if (proxy_lb_workers)
103 lb_limit = proxy_lb_workers();
104 else
105 lb_limit = 0;
106
107 scoreboard_size = sizeof(global_score);
108 scoreboard_size += sizeof(process_score) * server_limit;
109 scoreboard_size += sizeof(worker_score) * server_limit * thread_limit;
110 if (lb_limit)
111 scoreboard_size += sizeof(lb_score) * lb_limit;
112
113 return scoreboard_size;
114 }
It should be:
111 scoreboard_size += sizeof(lb_score) * (lb_limit+1);
and there needs some more minor changes.
Version-Release number of selected component (if applicable):
httpd-2.2.15-53.el6.x86_64
How reproducible:
Segfault may rarely occurs, because it needs that ap_scoreboard_image.balancers is allocated at the near end of the main_arena or one of mmap'ed arena, and ap_scoreboard_image.balancers[78] is on the border of the end of the arena. In this case, 0x7f96091f4000 is the border. The allocation is not controllable for the user space process, and it is hard to create a reproducer.
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:1891
Description of problem: I have a customer who is seeing segfaults at: (gdb) bt #0 0x00007f960ed94041 in ap_proxy_set_scoreboard_lb (worker=0x7f961a602948, balancer=<optimized out>, server=<optimized out>) at /usr/include/bits/string3.h:52 #1 0x00007f960ed940c4 in ap_proxy_initialize_worker_share (conf=0x7f961a602770, worker=0x7f961a602948, s=0x7f961a692548) at /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1903 #2 0x00007f960ed8fb0e in child_init (p=0x7f961a549418, s=0x7f961a692548) at /usr/src/debug/httpd-2.2.15/modules/proxy/mod_proxy.c:2308 #3 0x00007f9617e5a89c in ap_run_child_init (pchild=0x7f961a549418, s=0x7f961a039880) at /usr/src/debug/httpd-2.2.15/server/config.c:155 #4 0x00007f9617e6f420 in child_main (child_num_arg=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1161 #5 0x00007f9617e6f81f in make_child (s=0x7f961a039880, slot=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1341 #6 0x00007f9617e701f8 in startup_children (number_to_start=<optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1375 #7 server_main_loop (remaining_children_to_start=<optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1654 #8 ap_mpm_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1765 #9 0x00007f9617e46420 in main (argc=1, argv=0x7fff97f90bd8) at /usr/src/debug/httpd-2.2.15/server/main.c:763 Segfaults occur at proxy_util.c:1874 accessing free_slot->digest: (gdb) l /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1874 1869 } 1870 } 1871 1872 /* We failed to find out shared memory, so just use a free slot (if any) */ 1873 if (free_slot) { 1874 memcpy(free_slot->digest, digest, APR_MD5_DIGESTSIZE); 1875 worker->s = free_slot; 1876 } 1877 } 1878 return worker->s; (gdb) Memory is not assigned in the midst of the free_slot: (gdb) print free_slot $1 = (proxy_worker_stat *) 0x7f96091f3f48 (gdb) x/32gx 0x7f96091f3f48 0x7f96091f3f48: 0x0000000000000000 0x0000000000000000 0x7f96091f3f58: 0x0000000000000000 0x0000000000000000 0x7f96091f3f68: 0x0000000000000000 0x0000000000000000 0x7f96091f3f78: 0x0000000000000000 0x0000000000000000 0x7f96091f3f88: 0x0000000000000000 0x0000000000000000 0x7f96091f3f98: 0x0000000000000000 0x0000000000000000 0x7f96091f3fa8: 0x0000000000000000 0x0000000000000000 0x7f96091f3fb8: 0x0000000000000000 0x0000000000000000 0x7f96091f3fc8: 0x0000000000000000 0x0000000000000000 0x7f96091f3fd8: 0x0000000000000000 0x0000000000000000 0x7f96091f3fe8: 0x0000000000000000 0x0000000000000000 0x7f96091f3ff8: 0x0000000000000000 Cannot access memory at address 0x7f96091f4000 (gdb) print free_slot.digest Cannot access memory at address 0x7f96091f4014 In the core, lb_limit == 78, ((gdb) print lb_limit $53 = 78 The free_slot == the 78th element of ap_scoreboard_image.balancers, and it is stored at: gdb) print &(proxy_worker_stat)(ap_scoreboard_image.balancers[78]) $54 = (struct {...} *) 0x7f96091f3f48 According to ./usr/src/debug/httpd-2.2.15/server/scoreboard.c, it is assumed that ap_scoreboard_image->balancers has lb_limit+1 elements (0 ~ lb_limit): 543 AP_DECLARE(lb_score *) ap_get_scoreboard_lb(int lb_num) 544 { 545 if (((lb_num < 0) || (lb_limit < lb_num))) { 546 return(NULL); /* Out of range */ 547 } 548 return &ap_scoreboard_image->balancers[lb_num]; 549 } But, in fact, calloc'ed memory was smaller than it needs: 95 AP_DECLARE(int) ap_calc_scoreboard_size(void) 96 { 97 ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, &thread_limit); 98 ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, &server_limit); 99 100 if (!proxy_lb_workers) 101 proxy_lb_workers = APR_RETRIEVE_OPTIONAL_FN(ap_proxy_lb_workers); 102 if (proxy_lb_workers) 103 lb_limit = proxy_lb_workers(); 104 else 105 lb_limit = 0; 106 107 scoreboard_size = sizeof(global_score); 108 scoreboard_size += sizeof(process_score) * server_limit; 109 scoreboard_size += sizeof(worker_score) * server_limit * thread_limit; 110 if (lb_limit) 111 scoreboard_size += sizeof(lb_score) * lb_limit; 112 113 return scoreboard_size; 114 } It should be: 111 scoreboard_size += sizeof(lb_score) * (lb_limit+1); and there needs some more minor changes. Version-Release number of selected component (if applicable): httpd-2.2.15-53.el6.x86_64 How reproducible: Segfault may rarely occurs, because it needs that ap_scoreboard_image.balancers is allocated at the near end of the main_arena or one of mmap'ed arena, and ap_scoreboard_image.balancers[78] is on the border of the end of the arena. In this case, 0x7f96091f4000 is the border. The allocation is not controllable for the user space process, and it is hard to create a reproducer. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: