Description of problem: HAProxy health check for glance_registry fails for all backend nodes since the HTTP request return a 401 code. [root@overcloud-controller-0 ~]# grep -A6 glance_registry /etc/haproxy/haproxy.cfg listen glance_registry bind 192.0.2.12:9191 bind 192.0.2.13:9191 option httpchk GET / server overcloud-controller-0 192.0.2.15:9191 check fall 5 inter 2000 rise 2 server overcloud-controller-1 192.0.2.18:9191 check fall 5 inter 2000 rise 2 server overcloud-controller-2 192.0.2.17:9191 check fall 5 inter 2000 rise 2 [root@overcloud-controller-0 ~]# curl -I http://192.0.2.15:9191 HTTP/1.1 401 Unauthorized Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/' Content-Type: text/plain Content-Length: 0 Date: Mon, 22 Jun 2015 21:46:25 GMT [root@overcloud-controller-0 ~]# curl -I http://192.0.2.18:9191 HTTP/1.1 401 Unauthorized Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/' Content-Type: text/plain Content-Length: 0 Date: Mon, 22 Jun 2015 21:46:30 GMT [root@overcloud-controller-0 ~]# curl -I http://192.0.2.17:9191 HTTP/1.1 401 Unauthorized Www-Authenticate: Keystone uri='http://192.0.2.12:5000/v2.0/' Content-Type: text/plain Content-Length: 0 Date: Mon, 22 Jun 2015 21:46:35 GMT
Can you please provide the haproxy logs?
Log shows Connection refused but the backends are actually accessible, just returning Authentication required. [root@overcloud-controller-0 heat-admin]# journalctl -l -o cat -u haproxy| grep regist Proxy glance_registry started. Server glance_registry/overcloud-controller-0 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Server glance_registry/overcloud-controller-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Server glance_registry/overcloud-controller-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. proxy glance_registry has no server available! [root@overcloud-controller-0 heat-admin]# grep -A6 glance_registry /etc/haproxy/haproxy.cfg listen glance_registry bind 172.16.2.4:9191 option httpchk GET / server overcloud-controller-0 172.16.2.7:9191 check fall 5 inter 2000 rise 2 server overcloud-controller-1 172.16.2.6:9191 check fall 5 inter 2000 rise 2 server overcloud-controller-2 172.16.2.8:9191 check fall 5 inter 2000 rise 2 [root@overcloud-controller-0 heat-admin]# curl 172.16.2.7:9191 Authentication required [root@overcloud-controller-0 heat-admin]# curl 172.16.2.6:9191 Authentication required [root@overcloud-controller-0 heat-admin]# curl 172.16.2.8:9191 Authentication required [root@overcloud-controller-0 heat-admin]# curl 172.16.2.4:9191 curl: (52) Empty reply from server
Fix should be included in openstack-puppet-modules-2015.1.7-5.el7ost
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1549