Bug 1381841

Summary: keystone_public_api_node_ips and keystone_admin_api_node_ips are locked on controller nodes
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: achernet, dbecker, jschluet, kbasil, mburns, morazi, nkinder, rhel-osp-director-maint
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:08:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1337782    

Description Marius Cornea 2016-10-05 08:03:27 UTC
Description of problem:
keystone_public_api_node_ips and keystone_admin_api_node_ips are locked on controller nodes which leads to a broken overcloud deployment when keystone service is deployed on any other role than the controller.

Deployment with the following services allocation:

- name: Controller
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: ServiceApi
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaMetadata
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncProxy
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::Tripleo::Services::ManilaBackendGeneric
    - OS::Tripleo::Services::ManilaBackendNetapp
    - OS::Tripleo::Services::ManilaBackendCephFs
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::OpenDaylight
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

results in the following haproxy.cfg on the controller nodes:

listen keystone_admin
  bind 172.16.18.30:35357 transparent
  bind 192.168.0.20:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server cloudy-controller-0 192.168.0.22:35357 check fall 5 inter 2000 rise 2
  server cloudy-controller-1 192.168.0.12:35357 check fall 5 inter 2000 rise 2
  server cloudy-controller-2 192.168.0.21:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 10.0.0.13:5000 transparent
  bind 172.16.18.30:5000 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server cloudy-controller-0 10.0.0.11:5000 check fall 5 inter 2000 rise 2
  server cloudy-controller-1 10.0.0.19:5000 check fall 5 inter 2000 rise 2
  server cloudy-controller-2 10.0.0.24:5000 check fall 5 inter 2000 rise 2

hiera keystone_admin_api_node_ips
["192.168.0.22", "192.168.0.12", "192.168.0.21"]
hiera keystone_public_api_node_ips
["10.0.0.11", "10.0.0.19", "10.0.0.24"]

These are ip addresses set on the controller nodes where the Keystone service is not listening.

[root@cloudy-controller-0 heat-admin]# ip a | grep 10.0.0.11
    inet 10.0.0.11/25 brd 10.0.0.127 scope global vlan200
[root@cloudy-controller-0 heat-admin]# lsof -i :5000 -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 953 haproxy 27u IPv4 107227 0t0 TCP 10.0.0.13:5000 (LISTEN)
haproxy 953 haproxy 28u IPv4 107228 0t0 TCP 172.16.18.30:5000 (LISTEN)

[root@cloudy-controller-0 heat-admin]# ip a | grep 192.168.0.22
    inet 192.168.0.22/25 brd 192.168.0.127 scope global eth0
[root@cloudy-controller-0 heat-admin]# lsof -i :35357 -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 953 haproxy 25u IPv4 107225 0t0 TCP 172.16.18.30:35357 (LISTEN)
haproxy 953 haproxy 26u IPv4 107226 0t0 TCP 192.168.0.20:35357 (LISTEN)

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost.noarch

Comment 9 errata-xmlrpc 2016-12-14 16:08:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html