Bug 1281460

Summary: ospd HA w/ templates deployment only brings up one controller instead of 3
Product: Red Hat OpenStack Reporter: wes hayutin <whayutin>
Component: python-rdomanager-oscpluginAssignee: Brad P. Crochet <brad>
Status: CLOSED ERRATA QA Contact: Udi Shkalim <ushkalim>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: brad, calfonso, jcoufal, jslagle, mburns, rhel-osp-director-maint, sasha, whayutin
Target Milestone: y2   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-rdomanager-oscplugin-0.0.10-13.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-21 16:52:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description wes hayutin 2015-11-12 15:03:10 UTC 
Description of problem:

virt-deployment of HA. Only one controller is deployed and running.

Command used for deployment.

source /home/stack/stackrc; openstack overcloud deploy --debug --log-file overcloud_deployment_11.log --templates --libvirt-type=qemu --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server 10.5.26.10 --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e ~/network-environment.yaml &> overcloud_deployment_console.log

Deployment succeeds.

+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 240f888c-a148-4af9-a8f6-73075c46818a | overcloud  | CREATE_COMPLETE | 2015-11-11T22:37:37Z |
+--------------------------------------+------------+-----------------+----------------------+
+--------------------------------------+------------------------+--------+------------+-------------+--------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks           |
+--------------------------------------+------------------------+--------+------------+-------------+--------------------+
| ce3300ff-bfd4-47be-8dc4-7dc72ac986a2 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.7 |
| 3f922a2c-c896-4cd5-bef1-9ff469738644 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.8 |
+--------------------------------------+------------------------+--------+------------+-------------+--------------------+
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| ad06e23d-02c7-4b8e-ace2-9185cb0a8974 | None | None                                 | power off   | available       | False       |
| 98aa0a22-8e4d-4466-be92-909cbaf2bf24 | None | ce3300ff-bfd4-47be-8dc4-7dc72ac986a2 | power on    | active          | False       |
| 6ee0d878-1c0c-4a07-992c-9100e30b33a4 | None | 3f922a2c-c896-4cd5-bef1-9ff469738644 | power on    | active          | False       |
| 8c99d4d0-be39-4211-8975-a66df104a47c | None | None                                 | power off   | available       | False       |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
+--------------------------------------+------------------------+--------+--------------------+
| ID                                   | Name                   | Status | Networks           |
+--------------------------------------+------------------------+--------+--------------------+
| 3f922a2c-c896-4cd5-bef1-9ff469738644 | overcloud-controller-0 | ACTIVE | ctlplane=192.0.2.8 |
| ce3300ff-bfd4-47be-8dc4-7dc72ac986a2 | overcloud-compute-0    | ACTIVE | ctlplane=192.0.2.7 |
+--------------------------------------+------------------------+--------+--------------------+


Notice that 2/3 controllers are powered off.

This issue is repeatable.
Comment 4 James Slagle 2015-11-12 20:05:37 UTC 
pretty sure this was caused by this backport:
https://code.engineering.redhat.com/gerrit/#/c/60470/

this also likely needs to be backported:
https://review.openstack.org/#/c/227057
Comment 8 Udi Shkalim 2015-12-06 15:24:08 UTC 
Hi Wes,

Is there a way that i can verify this one quickly, like a job?

Thanks
Comment 9 Udi Shkalim 2015-12-06 16:29:57 UTC 
Verified on ospd 7.2 
python-rdomanager-oscplugin-0.0.10-19.el7ost.noarch

Deploy command: 
source /home/stack/stackrc; openstack overcloud deploy --debug --log-file overcloud_deployment_11.log --templates --libvirt-type=qemu --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server 10.5.26.10 --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml


[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 9c1c776c-2f64-401a-b4d4-968e5c75f27a | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 6ea5ec8c-4337-4394-bcad-324d3f9fe1c7 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| 533b9124-2602-4af4-9753-62b01668b978 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| b2c91324-9e68-424a-93d9-7f0602dc0d13 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| ee9043ad-5952-4362-aaf6-40e1c8eb8bfe | None | b2c91324-9e68-424a-93d9-7f0602dc0d13 | power on    | active          | False       |
| fc5d6c8f-6ad0-4b64-b00e-b08a208defd2 | None | 6ea5ec8c-4337-4394-bcad-324d3f9fe1c7 | power on    | active          | False       |
| ff0444b9-1eae-4efe-8868-f12425fb7061 | None | 533b9124-2602-4af4-9753-62b01668b978 | power on    | active          | False       |
| c1a712c7-f8a7-4b4f-876a-6bd35964ee5a | None | None                                 | power on    | available       | False       |
| ac9de543-26dc-4b2e-a51d-090052f9bc17 | None | 9c1c776c-2f64-401a-b4d4-968e5c75f27a | power on    | active          | False       |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
Comment 11 errata-xmlrpc 2015-12-21 16:52:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2650