Bug 1306623

Summary: Missing backend URL in ceilometer.conf file when deploying an external lb with IPv4
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: athomas, dbecker, dmacpher, jslagle, kbasil, mburns, morazi, rhel-osp-director-maint, srevivo
Target Milestone: y3Keywords: Regression
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-120.el7ost Doc Type: Bug Fix
Doc Text:
When deploying an Overcloud with an external load balancer, the 'RedisVipPort' parameter resolved to the 'from_service.yaml' template in the director's Heat template collection. However, an issue with the 'ip_address_uri' output parameter in 'from_service.yaml' template provided the wrong value. This fix corrects the 'ip_address_uri' output parameter in 'from_service.yaml'. Now the 'from_service.yaml' template returns the correct value to 'RedisVipPort'.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-18 16:53:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-02-11 13:16:46 UTC
Description of problem:
This bug is similar to BZ#1306040 but it shows up with IPv4 deployment

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-119.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
openstack overcloud deploy --templates ~/templates/my-overcloud  \
-e ~/templates/my-overcloud/environments/network-isolation.yaml \
-e ~/templates/network-environment.yaml \
-e ~/templates/enable-tls-external-lb.yaml \
-e ~/templates/inject-trust-anchor.yaml  \
-e ~/templates/my-overcloud/environments/external-loadbalancer-vip.yaml \
-e ~/templates/external-lb.yaml \
-e /home/stack/templates/firstboot-environment.yaml \
--control-scale 3 \
--compute-scale 1 \
--ntp-server 10.5.26.10 \
--libvirt-type qemu 

stack@instack:~>>> cat templates/network-environment.yaml 
resource_registry:
  OS::TripleO::BlockStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/cinder-storage.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::ObjectStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/swift-storage.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml

parameter_defaults:
  InternalApiNetCidr: 172.16.20.0/24
  StorageNetCidr: 172.16.21.0/24
  StorageMgmtNetCidr: 172.16.19.0/24
  TenantNetCidr: 172.16.22.0/24
  ExternalNetCidr: 172.16.23.0/24
  InternalApiAllocationPools: [{'start': '172.16.20.10', 'end': '172.16.20.100'}]
  StorageAllocationPools: [{'start': '172.16.21.10', 'end': '172.16.21.100'}]
  StorageMgmtAllocationPools: [{'start': '172.16.19.10', 'end': '172.16.19.100'}]
  TenantAllocationPools: [{'start': '172.16.22.10', 'end': '172.16.22.100'}]
  ExternalAllocationPools: [{'start': '172.16.23.10', 'end': '172.16.23.100'}]
  ExternalInterfaceDefaultRoute: 172.16.23.251
  NeutronExternalNetworkBridge: "''"
  ControlPlaneSubnetCidr: "24"
  ControlPlaneDefaultRoute: 192.0.2.1
  EC2MetadataIp: 192.0.2.1
  DnsServers: ["10.16.36.29","10.11.5.19"]  
  CloudName: rxtx.ro
stack@instack:~>>> cat templates/external-lb.yaml 
parameters:
  ServiceNetMap:
    NeutronTenantNetwork: tenant
    CeilometerApiNetwork: internal_api
    MongoDbNetwork: internal_api
    CinderApiNetwork: internal_api
    CinderIscsiNetwork: storage
    GlanceApiNetwork: storage
    GlanceRegistryNetwork: internal_api
    KeystoneAdminApiNetwork: internal_api
    KeystonePublicApiNetwork: internal_api
    NeutronApiNetwork: internal_api
    HeatApiNetwork: internal_api
    NovaApiNetwork: internal_api
    NovaMetadataNetwork: internal_api
    NovaVncProxyNetwork: internal_api
    SwiftMgmtNetwork: storage_mgmt
    SwiftProxyNetwork: storage
    HorizonNetwork: internal_api
    MemcachedNetwork: internal_api
    RabbitMqNetwork: internal_api
    RedisNetwork: internal_api
    MysqlNetwork: internal_api
    CephClusterNetwork: storage_mgmt
    CephPublicNetwork: storage
    ControllerHostnameResolveNetwork: internal_api
    ComputeHostnameResolveNetwork: internal_api
    BlockStorageHostnameResolveNetwork: internal_api
    ObjectStorageHostnameResolveNetwork: internal_api
    CephStorageHostnameResolveNetwork: storage

parameter_defaults:
  ControlPlaneIP: 192.0.2.250
  ExternalNetworkVip: 172.16.23.250
  InternalApiNetworkVip: 172.16.20.250
  StorageNetworkVip: 172.16.21.250
  StorageMgmtNetworkVip: 172.16.19.250
  ServiceVips:
    redis: 172.16.20.249
  ControllerIPs:
    external_cidr: "24"
    internal_api_cidr: "24"
    storage_cidr: "24"
    storage_mgmt_cidr: "24"
    tenant_cidr: "24"
    external:
    - 172.16.23.150
    - 172.16.23.151
    - 172.16.23.152
    internal_api:
    - 172.16.20.150
    - 172.16.20.151
    - 172.16.20.152
    storage:
    - 172.16.21.150
    - 172.16.21.151
    - 172.16.21.152
    storage_mgmt:
    - 172.16.19.150
    - 172.16.19.151
    - 172.16.19.152
    tenant:
    - 172.16.22.150
    - 172.16.22.151
    - 172.16.22.152

Actual results:
[root@overcloud-controller-0 ~]# grep backend_url /etc/ceilometer/ceilometer.conf  
#backend_url=<None>
backend_url=redis://:6379

Comment 1 Zane Bitter 2016-02-11 14:38:31 UTC
*** Bug 1306633 has been marked as a duplicate of this bug. ***

Comment 2 Marios Andreou 2016-02-11 15:07:31 UTC
the ip_address_uri defined for the network/ports/from_service.yaml is wrong it seems. This file is used for the RedisVip when the external loadbalancer environment is like like  environments/external-loadbalancer-vip.yaml 

The fixup yesterday for bug 1306040 introduced a new from_service_v6.yaml to correctly set the ip_address_uri for the v6 case. Testing an equivalent fix here, incoming.

Comment 3 Marios Andreou 2016-02-11 16:29:22 UTC
looks to me that this is set correctly in latest version of upstream at https://review.openstack.org/#/c/235423/38/network/ports/from_service.yaml so have set that as the upstream review for this.

Comment 4 Marios Andreou 2016-02-11 16:31:25 UTC
tested the downstream review using the same environment the error was reported on - the ceilometer backend/coordination url is set correctly in /etc/ceilometer/ceilometer.conf on the controllers and all pacemaker managed services are running with the deploy completing successfully:


*=* 17:22:39 *=*=*= 
openstack overcloud deploy --templates ~/templates/my-overcloud  \
-e ~/templates/my-overcloud/environments/network-isolation.yaml \
-e ~/templates/network-environment.yaml \
-e ~/templates/enable-tls-external-lb.yaml \
-e ~/templates/inject-trust-anchor.yaml  \
-e ~/templates/my-overcloud/environments/external-loadbalancer-vip.yaml \
-e ~/templates/external-lb.yaml \
-e /home/stack/templates/firstboot-environment.yaml \
--control-scale 3 \
--compute-scale 1 \
--ntp-server 10.5.26.10 \
--libvirt-type qemu 


*=* 17:43:50 *=*=*= 
stack@instack:~>>> for i in $(nova list|grep ctlplane|awk -F' ' '{ print $12 }'|awk -F'=' '{ print $2 }'); do nova list | grep $i; ssh heat-admin@$i sudo grep 'backend_url' /etc/ceilometer/ceilometer.conf; done
| 01b67756-0514-43e2-a14d-08014ef5b0e3 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
#backend_url=<None>
| 7bfeb13e-cf72-4397-a9c3-664afbb8638a | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.6  |
#backend_url=<None>
backend_url=redis://172.16.20.249:6379
| 4c6433e4-1323-40b0-b793-efc775dbbb63 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.24 |
#backend_url=<None>
backend_url=redis://172.16.20.249:6379
| 30aa4efe-0072-4c27-9247-251d0fd9ea2e | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
#backend_url=<None>
backend_url=redis://172.16.20.249:6379


*=* 17:48:29 *=*=*= 


[root@overcloud-controller-0 heat-admin]# pcs status | grep "ceil\|heat" -A 2
 Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
--
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
--
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-keystone-clone [openstack-keystone]
--
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
--
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-0


[root@overcloud-controller-0 heat-admin]# pcs status | grep -ni stop
[root@overcloud-controller-0 heat-admin]# 

Connection to 172.16.20.250 closed.
Overcloud Endpoint: https://rxtx.ro:13000/v2.0
Overcloud Deployed

stack@instack:~>>> heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 5f391be0-5505-41bc-b93f-9989bbdf0a0b | overcloud  | CREATE_COMPLETE | 2016-02-11T15:22:48Z |
+--------------------------------------+------------+-----------------+----------------------+

Comment 8 errata-xmlrpc 2016-02-18 16:53:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html