Description of problem: I'm running mod_ssl on a server with about a dozen vhosts and the event MPM, although the issue reproduces with the prefork MPM as well. When someone on the outside runs a Qualys scan from https://www.ssllabs.com/ssltest/analyze.html on my server, one of the child forks goes to 100% CPU and stops serving requests, and is unresponsive until kill -9'ed. The same symptoms are discussed in this forum discussion:https://community.qualys.com/thread/19027-apache-threads-stuck-at-100-after-scan. in the logs can be seen: ./alembic.sqlalchemy.org/logs/error_log:[Thu Dec 06 09:51:29.574388 2018] [ssl:error] [pid 32339:tid 140324684879616] [client 64.41.200.104:43236] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Thu Dec 06 09:51:39.791714 2018] [ssl:error] [pid 18719:tid 140324793984768] [client 64.41.200.104:46964] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Mon Dec 17 17:46:26.713047 2018] [ssl:error] [pid 10740:tid 139985525073664] [client 149.202.170.250:40663] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Tue Dec 18 22:56:15.748275 2018] [ssl:error] [pid 10740:tid 139985625786112] [client 64.41.200.104:55826] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Tue Dec 18 22:56:25.981016 2018] [ssl:error] [pid 10741:tid 139985617393408] [client 64.41.200.104:33424] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Wed Dec 19 10:53:22.513141 2018] [ssl:error] [pid 1105:tid 140393639253760] [client 64.41.200.102:41958] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Wed Dec 19 10:53:32.737070 2018] [ssl:error] [pid 727:tid 140393748358912] [client 64.41.200.102:43920] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Tue Jan 01 13:24:32.033175 2019] [ssl:error] [pid 315:tid 140333199312640] [client 64.41.200.103:58752] AH02042: rejecting client initiated renegotiation ./alembic.sqlalchemy.org/logs/error_log:[Tue Jan 01 13:24:42.270493 2019] [ssl:error] [pid 9095:tid 140333437437696] [client 64.41.200.103:33332] AH02042: rejecting client initiated renegotiation I've done some pstack with debugging symbols, so a few stack traces against the process while it's spinning are below. Version-Release number of selected component (if applicable): openssl-1.1.1-3.fc29.x86_64 openssl-pkcs11-0.4.8-2.fc29.x86_64 compat-openssl10-1.0.2o-3.fc29.x86_64 openssl-libs-1.1.1-3.fc29.x86_64 httpd-tools-2.4.37-5.fc29.x86_64 httpd-2.4.37-5.fc29.x86_64 httpd-filesystem-2.4.37-5.fc29.noarch How reproducible: always Steps to Reproduce: 1. run apache website with mod_ssl 2. hit the site with https://www.ssllabs.com/ssltest/analyze.html Actual results: one webserver process goes to 90-100% CPU permanently and stops serving requests until kill -9'ed Expected results: server continues to run normally Additional info: Issue was originally posted on the apache bugzilla linked in the external bug. ptrace excerpts: Thread 57 (Thread 0x7f3e6aff5700 (LWP 14740)): #0 0x00007f3e8d8fd1ba in ?? () from target:/lib64/libssl.so.1.1 #1 0x00007f3e8d8fab7e in ?? () from target:/lib64/libssl.so.1.1 #2 0x00007f3e8d901f75 in ?? () from target:/lib64/libssl.so.1.1 #3 0x00007f3e8d90ce4e in ?? () from target:/lib64/libssl.so.1.1 #4 0x00007f3e8d90cf57 in SSL_read () from target:/lib64/libssl.so.1.1 #5 0x00007f3e8d98a41a in ssl_io_input_read (inctx=inctx@entry=0x7f3e700a8c98, buf=buf@entry=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., len=len@entry=0x7f3e6aff49c0) at ssl_engine_io.c:673 #6 0x00007f3e8d98c6fc in ssl_io_input_getline (len=0x7f3e6aff49b8, buf=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., inctx=0x7f3e700a8c98) at ssl_engine_io.c:802 #7 ssl_io_filter_input (f=0x7f3e700aace8, bb=0x7f3e700b6bc8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at ssl_engine_io.c:1565 #8 0x0000557194fd7f95 in ap_rgetline_core (s=s@entry=0x7f3e6aff4b08, n=8192, read=read@entry=0x7f3e6aff4b10, r=r@entry=0x7f3e700b53f0, flags=flags@entry=2, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:246 #9 0x0000557194fd873d in ap_get_mime_headers_core (r=r@entry=0x7f3e700b53f0, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:1013 #10 0x0000557194fdb5a4 in ap_read_request (conn=conn@entry=0x7f3e700a3748) at protocol.c:1372 #11 0x0000557195005355 in ap_process_http_async_connection (c=0x7f3e700a3748) at http_core.c:146 #12 ap_process_http_connection (c=0x7f3e700a3748) at http_core.c:248 #13 0x0000557194ffb7f8 in ap_run_process_connection (c=c@entry=0x7f3e700a3748) at connection.c:42 #14 0x00007f3e8da61f97 in process_socket (thd=0x55719699ed88, p=<optimized out>, sock=<optimized out>, cs=0x7f3e700a36a0, my_child_num=<optimized out>, my_thread_num=<optimized out>) at event.c:1049 #15 0x00007f3e8da6293c in worker_thread (thd=0x55719699ed88, dummy=<optimized out>) at event.c:2082 #16 0x00007f3e8e3e458e in start_thread () from target:/lib64/libpthread.so.0 #17 0x00007f3e8e30d6a3 in clone () from target:/lib64/libc.so.6 Thread 57 (Thread 0x7f3e6aff5700 (LWP 14740)): #0 0x00007f3e8e2939f4 in _int_free () from target:/lib64/libc.so.6 #1 0x00007f3e8d8fb980 in ?? () from target:/lib64/libssl.so.1.1 #2 0x00007f3e8d8f90de in ?? () from target:/lib64/libssl.so.1.1 #3 0x00007f3e8d8fd1ce in ?? () from target:/lib64/libssl.so.1.1 #4 0x00007f3e8d8fab7e in ?? () from target:/lib64/libssl.so.1.1 #5 0x00007f3e8d901f75 in ?? () from target:/lib64/libssl.so.1.1 #6 0x00007f3e8d90ce4e in ?? () from target:/lib64/libssl.so.1.1 #7 0x00007f3e8d90cf57 in SSL_read () from target:/lib64/libssl.so.1.1 #8 0x00007f3e8d98a41a in ssl_io_input_read (inctx=inctx@entry=0x7f3e700a8c98, buf=buf@entry=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., len=len@entry=0x7f3e6aff49c0) at ssl_engine_io.c:673 #9 0x00007f3e8d98c6fc in ssl_io_input_getline (len=0x7f3e6aff49b8, buf=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., inctx=0x7f3e700a8c98) at ssl_engine_io.c:802 #10 ssl_io_filter_input (f=0x7f3e700aace8, bb=0x7f3e700b6bc8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at ssl_engine_io.c:1565 #11 0x0000557194fd7f95 in ap_rgetline_core (s=s@entry=0x7f3e6aff4b08, n=8192, read=read@entry=0x7f3e6aff4b10, r=r@entry=0x7f3e700b53f0, flags=flags@entry=2, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:246 #12 0x0000557194fd873d in ap_get_mime_headers_core (r=r@entry=0x7f3e700b53f0, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:1013 #13 0x0000557194fdb5a4 in ap_read_request (conn=conn@entry=0x7f3e700a3748) at protocol.c:1372 #14 0x0000557195005355 in ap_process_http_async_connection (c=0x7f3e700a3748) at http_core.c:146 #15 ap_process_http_connection (c=0x7f3e700a3748) at http_core.c:248 #16 0x0000557194ffb7f8 in ap_run_process_connection (c=c@entry=0x7f3e700a3748) at connection.c:42 #17 0x00007f3e8da61f97 in process_socket (thd=0x55719699ed88, p=<optimized out>, sock=<optimized out>, cs=0x7f3e700a36a0, my_child_num=<optimized out>, my_thread_num=<optimized out>) at event.c:1049 #18 0x00007f3e8da6293c in worker_thread (thd=0x55719699ed88, dummy=<optimized out>) at event.c:2082 #19 0x00007f3e8e3e458e in start_thread () from target:/lib64/libpthread.so.0 #20 0x00007f3e8e30d6a3 in clone () from target:/lib64/libc.so.6 Thread 57 (Thread 0x7f3e6aff5700 (LWP 14740)): #0 0x00007f3e8d98a468 in ssl_io_input_read (inctx=inctx@entry=0x7f3e700a8c98, buf=buf@entry=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., len=len@entry=0x7f3e6aff49c0) at ssl_engine_io.c:712 #1 0x00007f3e8d98c6fc in ssl_io_input_getline (len=0x7f3e6aff49b8, buf=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., inctx=0x7f3e700a8c98) at ssl_engine_io.c:802 #2 ssl_io_filter_input (f=0x7f3e700aace8, bb=0x7f3e700b6bc8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at ssl_engine_io.c:1565 #3 0x0000557194fd7f95 in ap_rgetline_core (s=s@entry=0x7f3e6aff4b08, n=8192, read=read@entry=0x7f3e6aff4b10, r=r@entry=0x7f3e700b53f0, flags=flags@entry=2, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:246 #4 0x0000557194fd873d in ap_get_mime_headers_core (r=r@entry=0x7f3e700b53f0, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:1013 #5 0x0000557194fdb5a4 in ap_read_request (conn=conn@entry=0x7f3e700a3748) at protocol.c:1372 #6 0x0000557195005355 in ap_process_http_async_connection (c=0x7f3e700a3748) at http_core.c:146 #7 ap_process_http_connection (c=0x7f3e700a3748) at http_core.c:248 #8 0x0000557194ffb7f8 in ap_run_process_connection (c=c@entry=0x7f3e700a3748) at connection.c:42 #9 0x00007f3e8da61f97 in process_socket (thd=0x55719699ed88, p=<optimized out>, sock=<optimized out>, cs=0x7f3e700a36a0, my_child_num=<optimized out>, my_thread_num=<optimized out>) at event.c:1049 #10 0x00007f3e8da6293c in worker_thread (thd=0x55719699ed88, dummy=<optimized out>) at event.c:2082 #11 0x00007f3e8e3e458e in start_thread () from target:/lib64/libpthread.so.0 #12 0x00007f3e8e30d6a3 in clone () from target:/lib64/libc.so.6 Thread 57 (Thread 0x7f3e6aff5700 (LWP 14740)): #0 0x00007f3e8d77b584 in OPENSSL_init_crypto () from target:/lib64/libcrypto.so.1.1 #1 0x00007f3e8d753506 in ERR_get_state () from target:/lib64/libcrypto.so.1.1 #2 0x00007f3e8d75370d in ERR_clear_error () from target:/lib64/libcrypto.so.1.1 #3 0x00007f3e8d98a403 in ssl_io_input_read (inctx=inctx@entry=0x7f3e700a8c98, buf=buf@entry=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., len=len@entry=0x7f3e6aff49c0) at ssl_engine_io.c:667 #4 0x00007f3e8d98c6fc in ssl_io_input_getline (len=0x7f3e6aff49b8, buf=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., inctx=0x7f3e700a8c98) at ssl_engine_io.c:802 #5 ssl_io_filter_input (f=0x7f3e700aace8, bb=0x7f3e700b6bc8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at ssl_engine_io.c:1565 #6 0x0000557194fd7f95 in ap_rgetline_core (s=s@entry=0x7f3e6aff4b08, n=8192, read=read@entry=0x7f3e6aff4b10, r=r@entry=0x7f3e700b53f0, flags=flags@entry=2, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:246 #7 0x0000557194fd873d in ap_get_mime_headers_core (r=r@entry=0x7f3e700b53f0, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:1013 #8 0x0000557194fdb5a4 in ap_read_request (conn=conn@entry=0x7f3e700a3748) at protocol.c:1372 #9 0x0000557195005355 in ap_process_http_async_connection (c=0x7f3e700a3748) at http_core.c:146 #10 ap_process_http_connection (c=0x7f3e700a3748) at http_core.c:248 #11 0x0000557194ffb7f8 in ap_run_process_connection (c=c@entry=0x7f3e700a3748) at connection.c:42 #12 0x00007f3e8da61f97 in process_socket (thd=0x55719699ed88, p=<optimized out>, sock=<optimized out>, cs=0x7f3e700a36a0, my_child_num=<optimized out>, my_thread_num=<optimized out>) at event.c:1049 #13 0x00007f3e8da6293c in worker_thread (thd=0x55719699ed88, dummy=<optimized out>) at event.c:2082 #14 0x00007f3e8e3e458e in start_thread () from target:/lib64/libpthread.so.0 #15 0x00007f3e8e30d6a3 in clone () from target:/lib64/libc.so.6 Thread 57 (Thread 0x7f3e6aff5700 (LWP 14740)): #0 0x00007f3e8d7534f0 in ERR_get_state () from target:/lib64/libcrypto.so.1.1 #1 0x00007f3e8d7537d5 in ?? () from target:/lib64/libcrypto.so.1.1 #2 0x00007f3e8d90ff3e in SSL_get_error () from target:/lib64/libssl.so.1.1 #3 0x00007f3e8d98a45a in ssl_io_input_read (inctx=inctx@entry=0x7f3e700a8c98, buf=buf@entry=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., len=len@entry=0x7f3e6aff49c0) at ssl_engine_io.c:711 #4 0x00007f3e8d98c6fc in ssl_io_input_getline (len=0x7f3e6aff49b8, buf=0x7f3e700a8ce0 "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0\r\nUser-Agent: SSL Labs (https://www.ssllabs.com/about/assessment.html)\r\n AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Sa"..., inctx=0x7f3e700a8c98) at ssl_engine_io.c:802 #5 ssl_io_filter_input (f=0x7f3e700aace8, bb=0x7f3e700b6bc8, mode=<optimized out>, block=<optimized out>, readbytes=<optimized out>) at ssl_engine_io.c:1565 #6 0x0000557194fd7f95 in ap_rgetline_core (s=s@entry=0x7f3e6aff4b08, n=8192, read=read@entry=0x7f3e6aff4b10, r=r@entry=0x7f3e700b53f0, flags=flags@entry=2, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:246 #7 0x0000557194fd873d in ap_get_mime_headers_core (r=r@entry=0x7f3e700b53f0, bb=bb@entry=0x7f3e700b6bc8) at protocol.c:1013 #8 0x0000557194fdb5a4 in ap_read_request (conn=conn@entry=0x7f3e700a3748) at protocol.c:1372 #9 0x0000557195005355 in ap_process_http_async_connection (c=0x7f3e700a3748) at http_core.c:146 #10 ap_process_http_connection (c=0x7f3e700a3748) at http_core.c:248 #11 0x0000557194ffb7f8 in ap_run_process_connection (c=c@entry=0x7f3e700a3748) at connection.c:42 #12 0x00007f3e8da61f97 in process_socket (thd=0x55719699ed88, p=<optimized out>, sock=<optimized out>, cs=0x7f3e700a36a0, my_child_num=<optimized out>, my_thread_num=<optimized out>) at event.c:1049 #13 0x00007f3e8da6293c in worker_thread (thd=0x55719699ed88, dummy=<optimized out>) at event.c:2082 #14 0x00007f3e8e3e458e in start_thread () from target:/lib64/libpthread.so.0 #15 0x00007f3e8e30d6a3 in clone () from target:/lib64/libc.so.6
the server has many vhosts running, the one that gets SSL requests by default without a cert is alembic.sqlalchemy.org. The vhost config for this site is below: <VirtualHost *:80 *:443> ServerAdmin mike_mp ServerName alembic.sqlalchemy.org ServerAlias alembic.zzzcomputing.com DocumentRoot /www/hosts/alembic.sqlalchemy.org/static Errorlog /www/hosts/alembic.sqlalchemy.org/logs/error_log CustomLog "|/usr/sbin/rotatelogs /www/hosts/alembic.sqlalchemy.org/logs/access_log.%Y%m%d 86400" combined DirectoryIndex index.html RewriteEngine On # https-related behaviors SSLEngine On SSLCertificateFile /etc/letsencrypt/live/alembic.sqlalchemy.org/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/alembic.sqlalchemy.org/privkey.pem # used by letsencrypt <Directory "/www/hosts/alembic.sqlalchemy.org/static/.well-known"> AllowOverride None RewriteEngine off Require all granted </Directory> # allows use of redirects of the form: # %{ENV:ps}://www.foo.com # RewriteCond %{HTTPS} =on RewriteRule ^(.*)$ - [env=ps:https] RewriteCond %{HTTPS} !=on RewriteRule ^(.*)$ - [env=ps:http] RewriteCond %{HTTPS} !=on # rewrite all but the letsencrypt URI to https RewriteRule ^/(?!.well-known)(.*) https://%{SERVER_NAME}/$1 [R,L] # end https-related behaviors <Directory "//www/hosts/alembic.sqlalchemy.org/static"> AllowOverride None Require all granted RewriteEngine On RewriteRule ^/* %{ENV:ps}://alembic.sqlalchemy.org/en/latest/ ErrorDocument 404 /en/latest/search.html </Directory> # rewrite all occurrences of alternate domains with primary domain RewriteCond %{HTTP_HOST} !^alembic\.sqlalchemy\.org [NC] RewriteCond %{HTTP_HOST} !^$ RewriteRule ^/(.*) %{ENV:ps}://alembic.sqlalchemy.org/$1 [L,R] # handle letsencrypt RewriteRule ^/\.well-known/.* - [END] <Directory "//www/hosts/alembic.sqlalchemy.org/static/en/"> RewriteEngine Off </Directory> </VirtualHost>
the settings in /etc/httpd/conf.d/ssl.conf are kept at their defaults, however I also tried the configuration at https://httpd.apache.org/docs/trunk/ssl/ssl_howto.html which didn't change the results.
Reviewing the backtrace, this may be based on a misunderstanding by our developers of the SSL_clear_error() function, as first identified here; https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
Reproducer for the issue: https://github.com/tomato42/tlsfuzzer/pull/492 (scripts test-legacy-renegotiation.py and test-renegotiation-disabled.py in tlsfuzzer) the script is much stricter than is necessary to detect this issue; the bug manifests itself as a test failing with a timeout error
*** Bug 1665692 has been marked as a duplicate of this bug. ***
*** This bug has been marked as a duplicate of bug 1668489 ***