Description of problem: Intermittently route fails on SSL handshake when connecting to route. Version-Release number of selected component (if applicable): openshift3/ose-haproxy-router:v3.5.5.31 How reproducible: 100% with Apache bench mark. Actual results: SSL connection errors Expected results: No errors Additional info: Haproxy logs: 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34150 [16/Nov/2017:21:29:19.762] fe_no_sni~ be_edge_http_appmgm-dev-int-rtp_kk-dev-int-rtp/81e86b702d9ae38791d78a2c4b1602ea 612/0/1/1/614 200 157 - - --NI 76/11/0/0/0 0/0 "GET /probe.html HTTP/1.0" 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34346 [16/Nov/2017:21:29:20.444] public_ssl be_no_sni/<NOSRV> -1/-1/95 0 -- 88/29/27/0/3 0/0 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34308 [16/Nov/2017:21:29:20.379] public_ssl be_no_sni/<NOSRV> -1/-1/160 0 -- 87/28/27/0/3 0/0 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34222 [16/Nov/2017:21:29:20.355] public_ssl be_no_sni/fe_no_sni 45/1/184 2155 CD 86/27/26/26/0 0/0 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34236 [16/Nov/2017:21:29:20.402] fe_no_sni/1: Connection error during SSL handshake 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34206 [16/Nov/2017:21:29:20.401] fe_no_sni/1: Connection error during SSL handshake 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34280 [16/Nov/2017:21:29:20.453] fe_no_sni/1: Connection closed during SSL handshake 2017-11-16T21:29:20+00:00 localhost haproxy[3630]: 72.163.48.19:34208 [16/Nov/2017:21:29:20.401] fe_no_sni/1: Connection error during SSL handshake AB results: 100118 SSL/TLS Handshake [Start] before/connect initialization 99488 SSL/TLS Handshake [Done] SSL negotiation finished successfully # grep 'fail' data. | sort -nr | uniq -c 3 SSL read failed (5) - closing connection 1 SSL read failed (1) - closing connection 516 SSL handshake failed (5). # grep 'SSL/TLS' data -i | sort -nr | uniq -c | sort -nr 100118 SSL/TLS State [connect] before/connect initialization 100118 SSL/TLS Handshake [Start] before/connect initialization 100112 SSL/TLS State [connect] SSLv2/v3 write client hello A 99515 SSL/TLS State [connect] SSLv3 write finished A 99515 SSL/TLS State [connect] SSLv3 write client key exchange A 99515 SSL/TLS State [connect] SSLv3 write change cipher spec A 99515 SSL/TLS State [connect] SSLv3 read server key exchange A 99515 SSL/TLS State [connect] SSLv3 read server hello A 99515 SSL/TLS State [connect] SSLv3 read server done A 99515 SSL/TLS State [connect] SSLv3 read server certificate A 99515 SSL/TLS State [connect] SSLv3 flush data 99488 SSL/TLS State [connect] SSLv3 read server session ticket A 99488 SSL/TLS State [connect] SSLv3 read finished A 99488 SSL/TLS Handshake [Done] SSL negotiation finished successfully 99483 SSL/TLS Alert [write] warning:close notify 99483 SSL/TLS Alert [read] warning:close notify 3 SSL/TLS State [connect] SSLv2/v3 write client hello B
Weibin, can you attempt to reproduce please? I have a hunch that ab is hitting haproxy too fast, but we'll need to see it to confirm and then attempt to mitigate.
We can duplicate the problem when using the high number of multiple requests in ab command such as -c 1000. No SSL failure when using -c 100. What is number of multiple requests you used in ab testing? [root@dhcp-41-193 ~]# ab -n 50000 -c 1000 https://hello-openshift-default.apps.0117-y-r.qe.rhcloud.com/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking hello-openshift-default.apps.0117-y-r.qe.rhcloud.com (be patient) SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). SSL handshake failed (5). [root@host-172-16-120-4 ~]# oc version oc v3.9.0-0.20.0
It looks to me like we are just overloading haproxy. If you can provide the number of connections you are testing with, and it seems low for the hardware, please feel free to re-open the bug.