Bug 1234637 - HAProxy health check for glance_registry fails for all backend nodes
Summary: HAProxy health check for glance_registry fails for all backend nodes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ga
: Director
Assignee: Giulio Fidente
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-22 21:52 UTC by Marius Cornea
Modified: 2015-08-05 13:55 UTC (History)
7 users (show)

Fixed In Version: openstack-puppet-modules-2015.1.7-5.el7ost
Doc Type: Bug Fix
Doc Text:
An incorrect configuration of the HAProxy backend caused HAProxy to forward requests for the glance-registry service to offline nodes. This fix monitors the glance-registry service for updates to ensure offline nodes are detected.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:55:19 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 195370 None None None Never
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Marius Cornea 2015-06-22 21:52:12 UTC
Description of problem:
HAProxy health check for glance_registry fails for all backend nodes since the HTTP request return a 401 code. 

[root@overcloud-controller-0 ~]# grep -A6 glance_registry /etc/haproxy/haproxy.cfg 
listen glance_registry
  bind 192.0.2.12:9191 
  bind 192.0.2.13:9191 
  option httpchk GET /
  server overcloud-controller-0 192.0.2.15:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 192.0.2.18:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 192.0.2.17:9191 check fall 5 inter 2000 rise 2

[root@overcloud-controller-0 ~]# curl -I http://192.0.2.15:9191
HTTP/1.1 401 Unauthorized
Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/'
Content-Type: text/plain
Content-Length: 0
Date: Mon, 22 Jun 2015 21:46:25 GMT

[root@overcloud-controller-0 ~]# curl -I http://192.0.2.18:9191
HTTP/1.1 401 Unauthorized
Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/'
Content-Type: text/plain
Content-Length: 0
Date: Mon, 22 Jun 2015 21:46:30 GMT

[root@overcloud-controller-0 ~]# curl -I http://192.0.2.17:9191
HTTP/1.1 401 Unauthorized
Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/'
Content-Type: text/plain
Content-Length: 0
Date: Mon, 22 Jun 2015 21:46:35 GMT

Comment 3 chris alfonso 2015-06-23 17:59:06 UTC
Can you please provide the haproxy logs?

Comment 4 Marius Cornea 2015-06-23 18:14:05 UTC
Log shows Connection refused but the backends are actually accessible, just returning Authentication required. 

[root@overcloud-controller-0 heat-admin]# journalctl -l -o cat -u haproxy| grep regist
Proxy glance_registry started.
Server glance_registry/overcloud-controller-0 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server glance_registry/overcloud-controller-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server glance_registry/overcloud-controller-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
proxy glance_registry has no server available!

[root@overcloud-controller-0 heat-admin]# grep -A6 glance_registry /etc/haproxy/haproxy.cfg
listen glance_registry
  bind 172.16.2.4:9191 
  option httpchk GET /
  server overcloud-controller-0 172.16.2.7:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.2.6:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.2.8:9191 check fall 5 inter 2000 rise 2

[root@overcloud-controller-0 heat-admin]# curl 172.16.2.7:9191
Authentication required
[root@overcloud-controller-0 heat-admin]# curl 172.16.2.6:9191
Authentication required
[root@overcloud-controller-0 heat-admin]# curl 172.16.2.8:9191 
Authentication required

[root@overcloud-controller-0 heat-admin]# curl 172.16.2.4:9191
curl: (52) Empty reply from server

Comment 5 Giulio Fidente 2015-06-25 15:09:07 UTC
Fix should be included in openstack-puppet-modules-2015.1.7-5.el7ost

Comment 8 errata-xmlrpc 2015-08-05 13:55:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.