Bug 1339412

Summary: mod_gnutls causing apach to hang at 100 % CPU load
Product: [Fedora] Fedora Reporter: Marcel Ziswiler <marcel>
Component: mod_gnutlsAssignee: Jiri Kastner <cz172638>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 23CC: cz172638, e
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mod_gnutls-0.7.5-1.fc23 mod_gnutls-0.7.5-1.fc24 mod_gnutls-0.7.5-1.fc22 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-09 23:54:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marcel Ziswiler 2016-05-25 00:05:08 UTC
Description of problem:

Upgrading my home server to Fedora 23 causing it to consume 100 % CPU after a while just running apache with php, gallery3 and an owncloud instance. Debugging into it shows all apache processes to hang at some mod_gnutls operation.

Version-Release number of selected component (if applicable):

Fedora 23 fully upt-to-date:
kernel-4.4.9-300.fc23.x86_64
httpd-2.4.18-1.fc23.x86_64
mod_gnutls-0.7-2.fc23.x86_64
gnutls-3.4.11-1.fc23.x86_64
gallery3-3.0.9-5.fc23.noarch
owncloud-8.2.4-1.fc23.noarch
php-5.6.21-1.fc23.x86_64

How reproducible:

restart httpd, wait a few minutes, do some https accesses and the first few apache processes will consume 100 % CPU

Steps to Reproduce:
1. sudo systemctl restart httpd
2. wget --no-check-certificate https://localhost
3. top

Actual results:
Tasks: 379 total,   4 running, 375 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.2 us,  0.1 sy,  0.0 ni, 95.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 98994728 total, 82720880 free,  1201508 used, 15072340 buff/cache
KiB Swap: 24780796 total, 24780796 free,        0 used. 97535424 avail Mem
 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10109 apache    20   0  804756  25956  13136 R 100.0  0.0   0:09.43 httpd

Expected results:
apache should not consume 100 % CPU

Additional info:
[user@host ~]$ sudo pstack 10109
Thread 2 (Thread 0x7efef88c2700 (LWP 10110)):
#0  0x00007eff20abfb10 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libpthread.so.0
#1  0x00007eff19cbf07a in get_mplx_next () from target:/etc/httpd/modules/mod_http2.so
#2  0x00007eff19cbe8ac in execute () from target:/etc/httpd/modules/mod_http2.so
#3  0x00007eff20aba60a in start_thread () from target:/lib64/libpthread.so.0
#4  0x00007eff205f059d in clone () from target:/lib64/libc.so.6
Thread 1 (Thread 0x7eff21f508c0 (LWP 10109)):
#0  0x00007eff1093c0bb in _gnutls_recv_in_buffers () from target:/lib64/libgnutls.so.30
#1  0x00007eff1093da38 in _gnutls_recv_int () from target:/lib64/libgnutls.so.30
#2  0x00007eff1093df64 in gnutls_record_recv () from target:/lib64/libgnutls.so.30
#3  0x00007eff10c4af2a in gnutls_io_input_read () from target:/etc/httpd/modules/mod_gnutls.so
#4  0x00007eff10c4bb40 in mgs_filter_input () from target:/etc/httpd/modules/mod_gnutls.so
#5  0x0000559be5e5efc7 in ap_process_request_after_handler ()
#6  0x0000559be5e604b0 in ap_process_request ()
#7  0x0000559be5e5c59e in ap_process_http_connection ()
#8  0x0000559be5e53950 in ap_run_process_connection ()
#9  0x00007eff15964b21 in child_main () from target:/etc/httpd/modules/mod_mpm_prefork.so
#10 0x00007eff15964db3 in make_child () from target:/etc/httpd/modules/mod_mpm_prefork.so
#11 0x00007eff15964e2b in startup_children () from target:/etc/httpd/modules/mod_mpm_prefork.so
#12 0x00007eff15965e07 in prefork_run () from target:/etc/httpd/modules/mod_mpm_prefork.so
#13 0x0000559be5e2ddae in ap_run_mpm ()
#14 0x0000559be5e27437 in main ()

I found a report of a similar problem upstream unfortunately at the time of this writing their OpenID Login did not work for me:
https://mod.gnutls.org/ticket/136

Comment 1 Marcel Ziswiler 2016-05-25 00:42:35 UTC
OK, looking at upstream changelog shows this to be a known issue fixed in version 0.7.2 (2015-11-21):
https://mod.gnutls.org/browser/mod_gnutls/CHANGELOG

I rebuilt 0.7.3 from FC24 beta on FC23 and can confirm that this fixes the issue. Finally!

I would strongly suggest for at least 0.7.2 with this fix to be officially backported to FC23.

Comment 2 Fedora Update System 2016-06-13 10:36:07 UTC
mod_gnutls-0.7.5-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-47d118a74b

Comment 3 Fedora Update System 2016-06-13 10:36:48 UTC
mod_gnutls-0.7.5-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dcfa6b6e51

Comment 4 Fedora Update System 2016-06-13 10:37:19 UTC
mod_gnutls-0.7.5-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e7f8326a78

Comment 5 Fedora Update System 2016-06-13 15:57:49 UTC
mod_gnutls-0.7.5-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-47d118a74b

Comment 6 Fedora Update System 2016-06-13 22:26:00 UTC
mod_gnutls-0.7.5-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dcfa6b6e51

Comment 7 Fedora Update System 2016-06-13 22:27:32 UTC
mod_gnutls-0.7.5-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e7f8326a78

Comment 8 Fedora Update System 2016-07-09 23:54:35 UTC
mod_gnutls-0.7.5-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2016-07-10 06:00:20 UTC
mod_gnutls-0.7.5-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2016-07-12 02:23:38 UTC
mod_gnutls-0.7.5-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.