Bug 1265806 - SSL Connect returning failure randomly
SSL Connect returning failure randomly
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: pyOpenSSL (Show other bugs)
5.0 (RHEL 7)
Unspecified Unspecified
high Severity high
: ---
: 5.0 (RHEL 7)
Assigned To: RHOS Maint
Shai Revivo
: Unconfirmed, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-23 15:42 EDT by Jeremy
Modified: 2017-01-13 12:18 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-01-13 12:18:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeremy 2015-09-23 15:42:56 EDT
Description of problem:
keystone vip is randomly failing when connecting using ssl. To reproduce the command openssl s_client -connect 172…23:5000 -msg -debug -ssl2 can be executed. This will connect successfully most of the time with random failures: 
CONNECTED(00000003)
>>> SSL 2.0 [length 002e], CLIENT-HELLO
    01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03
    00 80 01 00 80 06 00 40 04 00 80 02 00 80 06 d8
    2e a8 0a 7a 48 34 a6 75 fc 0e 50 89 b0 4a
140546248341320:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 350 bytes and written 48 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1443028553
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)


Version-Release number of selected component (if applicable):


How reproducible: unknown


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Jon Thomas 2015-09-30 14:29:56 EDT
I see this on osp7 HA too, but not frequently. And I never hit it in a single controller setup. Syslog has a corresponding message, but nothing in keystone.log

$ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug
Wed Sep 30 14:10:28 EDT 2015
CONNECTED(00000003)
write to 0xc2fd00 [0xc7adb1] (48 bytes => 48 (0x30))
0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00   ................
0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80   .........@......
0020 - b9 6c 6f 62 54 7e 75 da-7b 6a af 67 8c f6 1e 4b   .lobT~u.{j.g...K
>>> SSL 2.0 [length 002e], CLIENT-HELLO
    01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03
    00 80 01 00 80 06 00 40 04 00 80 02 00 80 b9 6c
    6f 62 54 7e 75 da 7b 6a af 67 8c f6 1e 4b
^C


Sep 30 14:10:31 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:10:31] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80\xb9lobT~u\xda{j\xafg\x8c\xf6\x1eK')
Sep 30 14:10:31 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:10:31] "\ufffd.



$ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug
Wed Sep 30 14:16:15 EDT 2015
CONNECTED(00000003)
write to 0x17b1d00 [0x17fcdb1] (48 bytes => 48 (0x30))
0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00   ................
0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80   .........@......
0020 - 2c 70 60 88 d7 de fb 0a-c4 44 fe 5e b1 42 44      ,p`......D.^.BD
0030 - <SPACES/NULS>
>>> SSL 2.0 [length 002e], CLIENT-HELLO
    01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03
    00 80 01 00 80 06 00 40 04 00 80 02 00 80 2c 70
    60 88 d7 de fb 0a c4 44 fe 5e b1 42 44 00
read from 0x17b1d00 [0x17f4da0] (2 bytes => 2 (0x2))
0000 - 3c 68                                             <h
read from 0x17b1d00 [0x17f4da2] (15465 bytes => 354 (0x162))
0000 - 65 61 64 3e 0a 3c 74 69-74 6c 65 3e 45 72 72 6f   ead>.<title>Erro
0010 - 72 20 72 65 73 70 6f 6e-73 65 3c 2f 74 69 74 6c   r response</titl
0020 - 65 3e 0a 3c 2f 68 65 61-64 3e 0a 3c 62 6f 64 79   e>.</head>.<body
0030 - 3e 0a 3c 68 31 3e 45 72-72 6f 72 20 72 65 73 70   >.<h1>Error resp
0040 - 6f 6e 73 65 3c 2f 68 31-3e 0a 3c 70 3e 45 72 72   onse</h1>.<p>Err
0050 - 6f 72 20 63 6f 64 65 20-34 30 30 2e 0a 3c 70 3e   or code 400..<p>
0060 - 4d 65 73 73 61 67 65 3a-20 42 61 64 20 72 65 71   Message: Bad req
0070 - 75 65 73 74 20 73 79 6e-74 61 78 20 28 27 5c 78   uest syntax ('\x
0080 - 38 30 2e 5c 78 30 31 5c-78 30 30 5c 78 30 32 5c   80.\x01\x00\x02\
0090 - 78 30 30 5c 78 31 35 5c-78 30 30 5c 78 30 30 5c   x00\x15\x00\x00\
00a0 - 78 30 30 5c 78 31 30 5c-78 30 37 5c 78 30 30 5c   x00\x10\x07\x00\
00b0 - 78 63 30 5c 78 30 35 5c-78 30 30 5c 78 38 30 5c   xc0\x05\x00\x80\
00c0 - 78 30 33 5c 78 30 30 5c-78 38 30 5c 78 30 31 5c   x03\x00\x80\x01\
00d0 - 78 30 30 5c 78 38 30 5c-78 30 36 5c 78 30 30 40   x00\x80\x06\x00@
00e0 - 5c 78 30 34 5c 78 30 30-5c 78 38 30 5c 78 30 32   \x04\x00\x80\x02
00f0 - 5c 78 30 30 5c 78 38 30-2c 70 60 5c 78 38 38 5c   \x00\x80,p`\x88\
0100 - 78 64 37 5c 78 64 65 5c-78 66 62 27 29 2e 0a 3c   xd7\xde\xfb')..<
0110 - 70 3e 45 72 72 6f 72 20-63 6f 64 65 20 65 78 70   p>Error code exp
0120 - 6c 61 6e 61 74 69 6f 6e-3a 20 34 30 30 20 3d 20   lanation: 400 = 
0130 - 42 61 64 20 72 65 71 75-65 73 74 20 73 79 6e 74   Bad request synt
0140 - 61 78 20 6f 72 20 75 6e-73 75 70 70 6f 72 74 65   ax or unsupporte
0150 - 64 20 6d 65 74 68 6f 64-2e 0a 3c 2f 62 6f 64 79   d method..</body
0160 - 3e 0a                                             >.
read from 0x17b1d00 [0x17f4f04] (15111 bytes => 0 (0x0))
140447135475616:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 356 bytes and written 48 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1443636975
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Sep 30 14:16:15 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:16:15] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80W\x02V\x80\x004\xedG\xc4\x1b\xe8\xd1\xdfR\x06\xd3')
Sep 30 14:16:15 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:16:15] "\ufffd.
Comment 2 Jon Thomas 2015-09-30 14:51:47 EDT
Here is another example. This time with a stack. 

It appears that either keystone ignores (or doesn't get) the request so the client hits the timeout. I would expect this to be the correct functionality.

In the second case, keystone-all replies to the client telling it that the message has bad syntax. The client doesn't know what to do with the reply so it also shows a handshake failure. In this case, keystone-all may or may not produce a stack. This would seem to be incorrect behavior.


$ date; openssl s_client -connect 172.16.0.68:5000 -ssl2 -msg -debug
Wed Sep 30 14:36:53 EDT 2015
CONNECTED(00000003)
write to 0x1b9dd00 [0x1be8db1] (48 bytes => 48 (0x30))
0000 - 80 2e 01 00 02 00 15 00-00 00 10 07 00 c0 05 00   ................
0010 - 80 03 00 80 01 00 80 06-00 40 04 00 80 02 00 80   .........@......
0020 - d6 84 39 e6 4e 47 5b 6c-38 12 22 3d e2 85 8b 80   ..9.NG[l8."=....
>>> SSL 2.0 [length 002e], CLIENT-HELLO
    01 00 02 00 15 00 00 00 10 07 00 c0 05 00 80 03
    00 80 01 00 80 06 00 40 04 00 80 02 00 80 d6 84
    39 e6 4e 47 5b 6c 38 12 22 3d e2 85 8b 80
read from 0x1b9dd00 [0x1be0da0] (2 bytes => 0 (0x0))
139935920900000:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:429:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 48 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1443638213
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---


Sep 30 14:37:53 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:37:53] code 400, message Bad request syntax ('\x80.\x01\x00\x02\x00\x15\x00\x00\x00\x10\x07\x00\xc0\x05\x00\x80\x03\x00\x80\x01\x00\x80\x06\x00@\x04\x00\x80\x02\x00\x80\xd6\x849\xe6NG[l8\x12"=\xe2\x85\x8b\x80')
Sep 30 14:37:53 overcloud-controller-0 keystone-all: 172.16.0.72 - - [30/Sep/2015 14:37:53] "\ufffd.
Sep 30 14:37:53 overcloud-controller-0 keystone-all: Traceback (most recent call last):
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Sep 30 14:37:53 overcloud-controller-0 keystone-all: func(*args, **kwargs)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 686, in process_request
Sep 30 14:37:53 overcloud-controller-0 keystone-all: proto.__init__(sock, address, self)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.handle()
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.handle_one_request()
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 325, in handle_one_request
Sep 30 14:37:53 overcloud-controller-0 keystone-all: if not self.parse_request():
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 286, in parse_request
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.send_error(400, "Bad request syntax (%r)" % requestline)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 373, in send_error
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.wfile.write(content)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/socket.py", line 324, in write
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self.flush()
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Sep 30 14:37:53 overcloud-controller-0 keystone-all: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall
Sep 30 14:37:53 overcloud-controller-0 keystone-all: tail = self.send(data, flags)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Sep 30 14:37:53 overcloud-controller-0 keystone-all: total_sent += fd.send(data[total_sent:], flags)
Sep 30 14:37:53 overcloud-controller-0 keystone-all: error: [Errno 32] Broken pipe

Note You need to log in before you can comment on or make changes to this bug.