Description of problem: keystone vip is randomly failing when connecting using ssl. To reproduce the command openssl s_client -connect 172…23:5000 -msg -debug -ssl2 can be executed. This will connect successfully most of the time with random failures: CONNECTED(00000003) >>> SSL 2.0 [length 002e], CLIENT-HELLO 01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03 00 80 01 00 80 06 00 40 04 00 80 02 00 80 06 d8 2e a8 0a 7a 48 34 a6 75 fc 0e 50 89 b0 4a 140546248341320:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 350 bytes and written 48 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : SSLv2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1443028553 Timeout : 300 (sec) Verify return code: 0 (ok) Version-Release number of selected component (if applicable): How reproducible: unknown Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I see this on osp7 HA too, but not frequently. And I never hit it in a single controller setup. Syslog has a corresponding message, but nothing in keystone.log $ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug Wed Sep 30 14:10:28 EDT 2015 CONNECTED(00000003) write to 0xc2fd00 [0xc7adb1] (48 bytes => 48 (0x30)) 0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00 ................ 0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80 .........@...... 0020 - b9 6c 6f 62 54 7e 75 da-7b 6a af 67 8c f6 1e 4b .lobT~u.{j.g...K >>> SSL 2.0 [length 002e], CLIENT-HELLO 01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03 00 80 01 00 80 06 00 40 04 00 80 02 00 80 b9 6c 6f 62 54 7e 75 da 7b 6a af 67 8c f6 1e 4b ^C Sep 30 14:10:31 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:10:31] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80\xb9lobT~u\xda{j\xafg\x8c\xf6\x1eK') Sep 30 14:10:31 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:10:31] "\ufffd. $ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug Wed Sep 30 14:16:15 EDT 2015 CONNECTED(00000003) write to 0x17b1d00 [0x17fcdb1] (48 bytes => 48 (0x30)) 0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00 ................ 0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80 .........@...... 0020 - 2c 70 60 88 d7 de fb 0a-c4 44 fe 5e b1 42 44 ,p`......D.^.BD 0030 - <SPACES/NULS> >>> SSL 2.0 [length 002e], CLIENT-HELLO 01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03 00 80 01 00 80 06 00 40 04 00 80 02 00 80 2c 70 60 88 d7 de fb 0a c4 44 fe 5e b1 42 44 00 read from 0x17b1d00 [0x17f4da0] (2 bytes => 2 (0x2)) 0000 - 3c 68 <h read from 0x17b1d00 [0x17f4da2] (15465 bytes => 354 (0x162)) 0000 - 65 61 64 3e 0a 3c 74 69-74 6c 65 3e 45 72 72 6f ead>.<title>Erro 0010 - 72 20 72 65 73 70 6f 6e-73 65 3c 2f 74 69 74 6c r response</titl 0020 - 65 3e 0a 3c 2f 68 65 61-64 3e 0a 3c 62 6f 64 79 e>.</head>.<body 0030 - 3e 0a 3c 68 31 3e 45 72-72 6f 72 20 72 65 73 70 >.<h1>Error resp 0040 - 6f 6e 73 65 3c 2f 68 31-3e 0a 3c 70 3e 45 72 72 onse</h1>.<p>Err 0050 - 6f 72 20 63 6f 64 65 20-34 30 30 2e 0a 3c 70 3e or code 400..<p> 0060 - 4d 65 73 73 61 67 65 3a-20 42 61 64 20 72 65 71 Message: Bad req 0070 - 75 65 73 74 20 73 79 6e-74 61 78 20 28 27 5c 78 uest syntax ('\x 0080 - 38 30 2e 5c 78 30 31 5c-78 30 30 5c 78 30 32 5c 80.\x01\x00\x02\ 0090 - 78 30 30 5c 78 31 35 5c-78 30 30 5c 78 30 30 5c x00\x15\x00\x00\ 00a0 - 78 30 30 5c 78 31 30 5c-78 30 37 5c 78 30 30 5c x00\x10\x07\x00\ 00b0 - 78 63 30 5c 78 30 35 5c-78 30 30 5c 78 38 30 5c xc0\x05\x00\x80\ 00c0 - 78 30 33 5c 78 30 30 5c-78 38 30 5c 78 30 31 5c x03\x00\x80\x01\ 00d0 - 78 30 30 5c 78 38 30 5c-78 30 36 5c 78 30 30 40 x00\x80\x06\x00@ 00e0 - 5c 78 30 34 5c 78 30 30-5c 78 38 30 5c 78 30 32 \x04\x00\x80\x02 00f0 - 5c 78 30 30 5c 78 38 30-2c 70 60 5c 78 38 38 5c \x00\x80,p`\x88\ 0100 - 78 64 37 5c 78 64 65 5c-78 66 62 27 29 2e 0a 3c xd7\xde\xfb')..< 0110 - 70 3e 45 72 72 6f 72 20-63 6f 64 65 20 65 78 70 p>Error code exp 0120 - 6c 61 6e 61 74 69 6f 6e-3a 20 34 30 30 20 3d 20 lanation: 400 = 0130 - 42 61 64 20 72 65 71 75-65 73 74 20 73 79 6e 74 Bad request synt 0140 - 61 78 20 6f 72 20 75 6e-73 75 70 70 6f 72 74 65 ax or unsupporte 0150 - 64 20 6d 65 74 68 6f 64-2e 0a 3c 2f 62 6f 64 79 d method..</body 0160 - 3e 0a >. read from 0x17b1d00 [0x17f4f04] (15111 bytes => 0 (0x0)) 140447135475616:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 356 bytes and written 48 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : SSLv2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1443636975 Timeout : 300 (sec) Verify return code: 0 (ok) --- Sep 30 14:16:15 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:16:15] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80W\x02V\x80\x004\xedG\xc4\x1b\xe8\xd1\xdfR\x06\xd3') Sep 30 14:16:15 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:16:15] "\ufffd.
Here is another example. This time with a stack. It appears that either keystone ignores (or doesn't get) the request so the client hits the timeout. I would expect this to be the correct functionality. In the second case, keystone-all replies to the client telling it that the message has bad syntax. The client doesn't know what to do with the reply so it also shows a handshake failure. In this case, keystone-all may or may not produce a stack. This would seem to be incorrect behavior. $ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug Wed Sep 30 14:36:53 EDT 2015 CONNECTED(00000003) write to 0x1b9dd00 [0x1be8db1] (48 bytes => 48 (0x30)) 0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00 ................ 0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80 .........@...... 0020 - d6 84 39 e6 4e 47 5b 6c-38 12 22 3d e2 85 8b 80 ..9.NG[l8."=.... >>> SSL 2.0 [length 002e], CLIENT-HELLO 01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03 00 80 01 00 80 06 00 40 04 00 80 02 00 80 d6 84 39 e6 4e 47 5b 6c 38 12 22 3d e2 85 8b 80 read from 0x1b9dd00 [0x1be0da0] (2 bytes => 0 (0x0)) 139935920900000:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 48 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE SSL-Session: Protocol : SSLv2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1443638213 Timeout : 300 (sec) Verify return code: 0 (ok) --- Sep 30 14:37:53 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:37:53] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80\xd6\x849\xe6NG[l8\x12"=\xe2\x85\x8b\x80') Sep 30 14:37:53 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:37:53] "\ufffd. Sep 30 14:37:53 overcloud-controller-0 keystone-all: Traceback (most recent call last): Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl Sep 30 14:37:53 overcloud-controller-0 keystone-all: func(*args, **kwargs) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 686, in process_request Sep 30 14:37:53 overcloud-controller-0 keystone-all: proto.__init__(sock, address, self) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__ Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.handle() Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.handle_one_request() Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 325, in handle_one_request Sep 30 14:37:53 overcloud-controller-0 keystone-all: if not self.parse_request(): Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 286, in parse_request Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.send_error(400, "Bad request syntax (%r)" % requestline) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 373, in send_error Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.wfile.write(content) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/socket.py", line 324, in write Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.flush() Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/socket.py", line 303, in flush Sep 30 14:37:53 overcloud-controller-0 keystone-all: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall Sep 30 14:37:53 overcloud-controller-0 keystone-all: tail = self.send(data, flags) Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send Sep 30 14:37:53 overcloud-controller-0 keystone-all: total_sent += fd.send(data[total_sent:], flags) Sep 30 14:37:53 overcloud-controller-0 keystone-all: error: [Errno 32] Broken pipe