Bug 1381841 - keystone_public_api_node_ips and keystone_admin_api_node_ips are locked on controller nodes
Summary: keystone_public_api_node_ips and keystone_admin_api_node_ips are locked on co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: 10.0 (Newton)
Assignee: Jiri Stransky
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks: 1337782
TreeView+ depends on / blocked
 
Reported: 2016-10-05 08:03 UTC by Marius Cornea
Modified: 2016-12-14 16:08 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:08:24 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 380386 None None None 2016-10-05 08:04:25 UTC
Launchpad 1629096 None None None 2016-10-05 08:03:53 UTC

Description Marius Cornea 2016-10-05 08:03:27 UTC
Description of problem:
keystone_public_api_node_ips and keystone_admin_api_node_ips are locked on controller nodes which leads to a broken overcloud deployment when keystone service is deployed on any other role than the controller.

Deployment with the following services allocation:

- name: Controller
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: ServiceApi
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaMetadata
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncProxy
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::Tripleo::Services::ManilaBackendGeneric
    - OS::Tripleo::Services::ManilaBackendNetapp
    - OS::Tripleo::Services::ManilaBackendCephFs
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::OpenDaylight
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

results in the following haproxy.cfg on the controller nodes:

listen keystone_admin
  bind 172.16.18.30:35357 transparent
  bind 192.168.0.20:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server cloudy-controller-0 192.168.0.22:35357 check fall 5 inter 2000 rise 2
  server cloudy-controller-1 192.168.0.12:35357 check fall 5 inter 2000 rise 2
  server cloudy-controller-2 192.168.0.21:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 10.0.0.13:5000 transparent
  bind 172.16.18.30:5000 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server cloudy-controller-0 10.0.0.11:5000 check fall 5 inter 2000 rise 2
  server cloudy-controller-1 10.0.0.19:5000 check fall 5 inter 2000 rise 2
  server cloudy-controller-2 10.0.0.24:5000 check fall 5 inter 2000 rise 2

hiera keystone_admin_api_node_ips
["192.168.0.22", "192.168.0.12", "192.168.0.21"]
hiera keystone_public_api_node_ips
["10.0.0.11", "10.0.0.19", "10.0.0.24"]

These are ip addresses set on the controller nodes where the Keystone service is not listening.

[root@cloudy-controller-0 heat-admin]# ip a | grep 10.0.0.11
    inet 10.0.0.11/25 brd 10.0.0.127 scope global vlan200
[root@cloudy-controller-0 heat-admin]# lsof -i :5000 -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 953 haproxy 27u IPv4 107227 0t0 TCP 10.0.0.13:5000 (LISTEN)
haproxy 953 haproxy 28u IPv4 107228 0t0 TCP 172.16.18.30:5000 (LISTEN)

[root@cloudy-controller-0 heat-admin]# ip a | grep 192.168.0.22
    inet 192.168.0.22/25 brd 192.168.0.127 scope global eth0
[root@cloudy-controller-0 heat-admin]# lsof -i :35357 -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 953 haproxy 25u IPv4 107225 0t0 TCP 172.16.18.30:35357 (LISTEN)
haproxy 953 haproxy 26u IPv4 107226 0t0 TCP 192.168.0.20:35357 (LISTEN)

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost.noarch

Comment 9 errata-xmlrpc 2016-12-14 16:08:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.