Bug 1244358

Summary:

rhel-osp-director: unable to register the nodes with SSL undercloud

Product:

Red Hat OpenStack

Reporter:

Alexander Chuzhoy <sasha>

Component:

openstack-puppet-modules

Assignee:

Ivan Chavero <ichavero>

Status:

CLOSED ERRATA

QA Contact:

Alexander Chuzhoy <sasha>

Severity:

urgent

Docs Contact:

Priority:

high

Version:

unspecified

CC:

ddomingo, jschluet, jslagle, lnatapov, mburns, ohochman, rhel-osp-director-maint, rrosa, yeylon

Target Milestone:

Target Release:

7.0 (Kilo)

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-puppet-modules-2015.1.8-8.el7ost

Doc Type:

Known Issue

Doc Text:

The Director uses misconfigured HAProxy settings when deploying the Bare Metal and Telemetry services with SSL enabled in the undercloud. This prevents some nodes from registering. To work around this, comment out 'option ssl-hello-chk' under the Bare Metal and Telemetry sections in /etc/haproxy/haproxy.cfg after installing the undercloud.

Story Points:

---

Clone Of:

Environment:

Last Closed:

2015-08-05 13:29:44 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
/var/log/messages	none

Description Alexander Chuzhoy 2015-07-17 22:01:55 UTC

rhel-osp-director: unable to register the nodes with "openstack baremetal import --json instackenv.json"

Environment:
instack-undercloud-2.1.2-21.el7ost.noarch

Steps to reproduce:
1. Install the undercloud with SSL.
2. Attempt to register the nodes using the (created in advance) instackenv.json file.

Result:
[stack@rhos-compute-node-13 ~]$ openstack baremetal import --json instackenv.json
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
ERROR: openstack ''



Expected result:
The hosts should be registered.

Comment 3 Alexander Chuzhoy 2015-07-17 22:09:30 UTC

Created attachment 1053251 [details]
/var/log/messages

Comment 4 Alexander Chuzhoy 2015-07-17 22:11:02 UTC

There are repeating glance-registry errors:
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: 192.0.2.1 - - [17/Jul/2015 18:10:06] code 400, message Bad request syntax ('\x16\x03\x00\x00y\x01\x00\x00u\x03\x00U\xa9}>HAPROXYSSLCHK')                       
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: 192.0.2.1 - - [17/Jul/2015 18:10:06] "                                                                                                                         
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: Traceback (most recent call last):                                                                                                                             
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl                                                                       
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: func(*args, **kwargs)                                                                                                                                          
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 686, in process_request                                                                         
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: proto.__init__(sock, address, self)                                                                                                                            
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__                                                                                             
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.handle()                                                                                                                                                  
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle                                                                                             
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.handle_one_request()                                                                                                                                      
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 325, in handle_one_request
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: if not self.parse_request():
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 286, in parse_request
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.send_error(400, "Bad request syntax (%r)" % requestline)
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 368, in send_error
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.send_response(code, message)
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 396, in send_response
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.send_header('Date', self.date_time_string())
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/BaseHTTPServer.py", line 401, in send_header
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.wfile.write("%s: %s\r\n" % (keyword, value))
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/socket.py", line 324, in write
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self.flush()
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib64/python2.7/socket.py", line 303, in flush
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 376, in sendall
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: tail = self.send(data, flags)
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: total_sent += fd.send(data[total_sent:], flags)
Jul 17 18:10:06 rhos-compute-node-13 glance-registry: error: [Errno 104] Connection reset by peer
Jul 17 18:10:06 rhos-compute-node-13 ironic-api: 192.0.2.1 - - [17/Jul/2015 18:10:06] code 400, message Bad request syntax ('\x16\x03\x00\x00y\x01\x00\x00u\x03\x00U\xa9}>HAPROXYSSLCHK')
Jul 17 18:10:06 rhos-compute-node-13 ironic-api: 192.0.2.1 - - [17/Jul/2015 18:10:06] "

Comment 5 James Slagle 2015-07-17 23:38:05 UTC

This is a regression in the undercloud ssl support caused by:
https://review.openstack.org/#/c/199507/

You can either:
- not use ssl in the undercloud ssl (the default)
- or, after the installation is done, edit /etc/haproxy/haproxy.cfg and comment out the "option ssl-hello-chk" under the "listen ironic section". so after the edit the listen ironic section looks like:

listen ironic
  bind 192.0.2.2:13385 ssl crt /etc/haproxy/test.pem
  bind 192.0.2.3:6385
  balance roundrobin
  option tcplog
  # option ssl-hello-chk
  server 192.0.2.1 192.0.2.1:6385 check fall 5 inter 2000 rise 2

Note that ceilometer api services are also probably not working in the undercloud due to this issue as well.

Comment 6 James Slagle 2015-07-18 01:53:39 UTC

proposed upstream fix https://review.openstack.org/#/c/203298

Comment 7 Alexander Chuzhoy 2015-07-18 17:27:01 UTC

After applying the suggestion in comment #5 and restarting the haproxy - I was able to register the hosts.

Comment 8 Mike Burns 2015-07-20 14:10:10 UTC

*** Bug 1244806 has been marked as a duplicate of this bug. ***

Comment 9 Mike Burns 2015-07-21 11:49:01 UTC

*** Bug 1244995 has been marked as a duplicate of this bug. ***

Comment 10 Mike Burns 2015-07-21 13:28:46 UTC

The patch was backported as part of bug 1236057 so this is resolved.

Comment 11 Omri Hochman 2015-07-21 14:13:37 UTC

Just important note ": 

This Errors "HAPROXYSSLCHK"  flood in /var/log/messages is not happen only with SSL  - Actually it happened to me on *Non-SSL* environment : 

('\x16\x03\x00\x00y\x01\x00\x00u\x03\x00U\xa9}>HAPROXYSSLCHK')    

this Bz should not be handle as it happens only with SSL=true.

Comment 12 Jon Schlueter 2015-07-22 12:35:14 UTC

could use pm and qe ack for this bug.  The fix for this bug made it into errata 20511 OSP7 ga puddle already would like to be able to verify this bug otherwise the bug will push out to a1

Comment 14 Alexander Chuzhoy 2015-07-23 20:38:19 UTC

Verified:

Environment:
instack-undercloud-2.1.2-22.el7ost.noarch


The reported issue is resolved.

Comment 15 Alexander Chuzhoy 2015-07-23 20:38:56 UTC

Environment:
openstack-puppet-modules-2015.1.8-8.el7ost.noarch
instack-undercloud-2.1.2-22.el7ost.noarch

Comment 17 errata-xmlrpc 2015-08-05 13:29:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1548