Bug 1288395

What other httpd packages do you have installed? It does not happen to me with only "httpd" and "mod_ssl" installed on my system with clean configuration. What's the httpd configuration you use?

Can you get a coredump from crash? The description how to do that is here: https://access.redhat.com/solutions/1467

Reviewing my costomer's core file, I found the function handler pfn_ap_logio_get_last_bytes has a wrong value. The backtrace is as follows:-


(gdb) bt
#0  0x00007f633cd0ab60 in ?? ()
#1  0x00007f6343b377d2 in ap_increment_counts (sb=0x7f6344a004d0, r=r@entry=0x7f6344a06310) at scoreboard.c:369
...


The function ap_increment_counts is :-


359: AP_DECLARE(void) ap_increment_counts(ap_sb_handle_t *sb, request_rec *r)
360: {
361:     worker_score *ws;
362:     apr_off_t bytes;
363: 
364:     if (!sb)
365:         return;
366: 
367:     ws = &ap_scoreboard_image->servers[sb->child_num][sb->thread_num];
368:     if (pfn_ap_logio_get_last_bytes != NULL) {
369:        bytes = pfn_ap_logio_get_last_bytes(r->connection);


And the handler pfn_ap_logio_get_last_bytes is :-


(gdb) print pfn_ap_logio_get_last_bytes
$9 = (apr_OFN_ap_logio_get_last_bytes_t *) 0x7f633cd0ab60


It looks pfn_ap_logio_get_last_bytes points a head of a valid function, but in fact, it is a wrong value.


(gdb) x 0x7f633cd0ab60
0x7f633cd0ab60: Cannot access memory at address 0x7f633cd0ab60


Calling non-existent memory address killed the process with segfault. This is likely caused by unloading a library which should have a valid function where pfn_ap_logio_get_last_bytes points. The problem is "what is unloaded?".

I attached a debugger to my httpd process using the configs and noticed that pfn_ap_logio_get_last_bytes points the function <ap_logio_get_last_bytes> in mod_logio.so. 
Then I reviewed again the core file, and found pfn_ap_logio_get_last_bytes in the core also points somewhere in mod_logio.so as well:-


(gdb) print pfn_ap_logio_get_last_bytes
$1 = (apr_OFN_ap_logio_get_last_bytes_t *) 0x7f633cd0ab60

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
...
0x00007f633cb14a30  0x00007f633cb14de4  Yes         ./etc/httpd/modules/mod_logio.so <===where pfn_ap_logio_get_last_bytes points


The customer is using logrotate command to rotate logs. It invokes `systemctl reload httpd.service`, and `systemctl reload httpd.service` invokes `/usr/sbin/httpd $OPTIONS -k graceful`. Searching for "httpd logrotate segfault", you can find some reports that say the "graceful" command is problematic. In fact, changing `systemctl reload httpd.service` to `systemctl restart httpd.service`, the issue is gone on the customer's setup.

Jan, honestly to say, I could not reproduce the issue, but theoretically it can happen in the race condition that mod_logio is unloaded during an access is processed. It is caused by unloading mod_logio, you can (theoretically) reproduce the issue without any additional packages.

Thanks for the additional information. I've found out this can be fixed by following upstream patch:

https://svn.apache.org/viewvc?view=revision&revision=1668532

There's also follow-up commit:

https://svn.apache.org/viewvc?view=revision&revision=1668553

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0826