Description of problem:
I have a customer who is seeing segfaults at:


(gdb) bt
#0  0x00007f960ed94041 in ap_proxy_set_scoreboard_lb (worker=0x7f961a602948, balancer=<optimized out>, server=<optimized out>)
    at /usr/include/bits/string3.h:52
#1  0x00007f960ed940c4 in ap_proxy_initialize_worker_share (conf=0x7f961a602770, worker=0x7f961a602948, s=0x7f961a692548)
    at /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1903
#2  0x00007f960ed8fb0e in child_init (p=0x7f961a549418, s=0x7f961a692548) at /usr/src/debug/httpd-2.2.15/modules/proxy/mod_proxy.c:2308
#3  0x00007f9617e5a89c in ap_run_child_init (pchild=0x7f961a549418, s=0x7f961a039880)
    at /usr/src/debug/httpd-2.2.15/server/config.c:155
#4  0x00007f9617e6f420 in child_main (child_num_arg=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1161
#5  0x00007f9617e6f81f in make_child (s=0x7f961a039880, slot=1) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1341
#6  0x00007f9617e701f8 in startup_children (number_to_start=<optimized out>)
    at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1375
#7  server_main_loop (remaining_children_to_start=<optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1654
#8  ap_mpm_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>)
    at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:1765
#9  0x00007f9617e46420 in main (argc=1, argv=0x7fff97f90bd8) at /usr/src/debug/httpd-2.2.15/server/main.c:763


Segfaults occur at proxy_util.c:1874 accessing free_slot->digest:


(gdb) l /usr/src/debug/httpd-2.2.15/modules/proxy/proxy_util.c:1874
1869                }
1870            }
1871
1872            /* We failed to find out shared memory, so just use a free slot (if any) */
1873            if (free_slot) {
1874                memcpy(free_slot->digest, digest, APR_MD5_DIGESTSIZE);
1875                worker->s = free_slot;
1876            }
1877        }
1878        return worker->s;
(gdb) 


Memory is not assigned in the midst of the free_slot:


(gdb) print free_slot
$1 = (proxy_worker_stat *) 0x7f96091f3f48
(gdb) x/32gx 0x7f96091f3f48
0x7f96091f3f48: 0x0000000000000000      0x0000000000000000
0x7f96091f3f58: 0x0000000000000000      0x0000000000000000
0x7f96091f3f68: 0x0000000000000000      0x0000000000000000
0x7f96091f3f78: 0x0000000000000000      0x0000000000000000
0x7f96091f3f88: 0x0000000000000000      0x0000000000000000
0x7f96091f3f98: 0x0000000000000000      0x0000000000000000
0x7f96091f3fa8: 0x0000000000000000      0x0000000000000000
0x7f96091f3fb8: 0x0000000000000000      0x0000000000000000
0x7f96091f3fc8: 0x0000000000000000      0x0000000000000000
0x7f96091f3fd8: 0x0000000000000000      0x0000000000000000
0x7f96091f3fe8: 0x0000000000000000      0x0000000000000000
0x7f96091f3ff8: 0x0000000000000000      Cannot access memory at address 0x7f96091f4000
(gdb) print free_slot.digest
Cannot access memory at address 0x7f96091f4014


In the core, lb_limit == 78, 


((gdb) print lb_limit
$53 = 78


The free_slot == the 78th element of ap_scoreboard_image.balancers, and it is stored at:


gdb) print &(proxy_worker_stat)(ap_scoreboard_image.balancers[78])
$54 = (struct {...} *) 0x7f96091f3f48


According to ./usr/src/debug/httpd-2.2.15/server/scoreboard.c, it is assumed that ap_scoreboard_image->balancers has lb_limit+1 elements (0 ~ lb_limit):


543 AP_DECLARE(lb_score *) ap_get_scoreboard_lb(int lb_num)
544 {   
545     if (((lb_num < 0) || (lb_limit < lb_num))) {
546         return(NULL); /* Out of range */
547     }
548     return &ap_scoreboard_image->balancers[lb_num];
549 }


But, in fact, calloc'ed memory was smaller than it needs:


 95 AP_DECLARE(int) ap_calc_scoreboard_size(void)
 96 {
 97     ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, &thread_limit);
 98     ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, &server_limit);
 99 
100     if (!proxy_lb_workers)
101         proxy_lb_workers = APR_RETRIEVE_OPTIONAL_FN(ap_proxy_lb_workers);
102     if (proxy_lb_workers)
103         lb_limit = proxy_lb_workers();
104     else
105         lb_limit = 0;
106 
107     scoreboard_size = sizeof(global_score);
108     scoreboard_size += sizeof(process_score) * server_limit;
109     scoreboard_size += sizeof(worker_score) * server_limit * thread_limit;
110     if (lb_limit)
111         scoreboard_size += sizeof(lb_score) * lb_limit;
112 
113     return scoreboard_size;
114 }

It should be:

111         scoreboard_size += sizeof(lb_score) * (lb_limit+1);

and there needs some more minor changes.



Version-Release number of selected component (if applicable):

httpd-2.2.15-53.el6.x86_64


How reproducible:

Segfault may rarely occurs, because it needs that ap_scoreboard_image.balancers is allocated at the near end of the main_arena or one of mmap'ed arena, and ap_scoreboard_image.balancers[78] is on the border of the end of the arena. In this case, 0x7f96091f4000 is the border. The allocation is not controllable for the user space process, and it is hard to create a reproducer.



Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: