Bug 1409851 - Servers get rebuilt during a stack update following new image rollout
Summary: Servers get rebuilt during a stack update following new image rollout
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Dougal Matthews
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On: 1354627
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-03 16:02 UTC by Zane Bitter
Modified: 2017-03-01 13:34 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-common-0.3.1-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1354627
Environment:
Last Closed: 2017-03-01 13:34:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1609020 0 None None None 2017-01-03 16:02:14 UTC
Red Hat Product Errata RHBA-2017:0362 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 director Bug Fix Advisory 2017-03-01 18:31:37 UTC

Description Zane Bitter 2017-01-03 16:02:14 UTC
+++ This bug was initially created as a clone of Bug #1354627 +++

This change in Mitaka:

https://review.openstack.org/#/c/287834/11

added a translation rule that causes the 'image' property passed to OS::Nova::Server to be automatically translated to a UUID prior to the properties being assembled. The result is that TripleO's previous trick of uploading a new image and keeping the same name (but getting a new UUID) no longer works to prevent Heat from rebuilding the server when the image changes.

Comment 3 Alexander Chuzhoy 2017-02-02 19:47:26 UTC
Verified:
Environment:
openstack-tripleo-common-0.3.1-3.el7ost.noarch

1) Deployed overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 3 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log



2)
Updated the overcloud-full image by setting the root password.
3)
Updated all images in glance with:
openstack overcloud image upload --update-existing

4)scaled with additional compute using:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 3 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log



Checked the nova list:
[stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 496db736-d925-45cd-bd96-4b4197000aca | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| de698f42-1346-41eb-b6cb-da5506e5833d | ceph-1       | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 2784c430-3763-4d0a-974b-438d93bc32a0 | ceph-2       | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| efe4b393-9ba3-4225-ae3e-254f4fc496cb | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 2b785fc9-f967-4104-9ed2-62a0e9fc4bde | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 6d792b44-fce8-4079-a408-d20e950bb351 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| dec1a59a-85a3-426d-953a-e34857504f20 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+





[stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 496db736-d925-45cd-bd96-4b4197000aca | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| de698f42-1346-41eb-b6cb-da5506e5833d | ceph-1       | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 2784c430-3763-4d0a-974b-438d93bc32a0 | ceph-2       | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| efe4b393-9ba3-4225-ae3e-254f4fc496cb | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c7e77b08-6484-42f4-95f1-ea19f1385b78 | compute-1    | BUILD  | spawning   | NOSTATE     | ctlplane=192.0.2.14 |
| 2b785fc9-f967-4104-9ed2-62a0e9fc4bde | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 6d792b44-fce8-4079-a408-d20e950bb351 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| dec1a59a-85a3-426d-953a-e34857504f20 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
[stack@undercloud-0 ~]$ 




[stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 496db736-d925-45cd-bd96-4b4197000aca | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| de698f42-1346-41eb-b6cb-da5506e5833d | ceph-1       | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| 2784c430-3763-4d0a-974b-438d93bc32a0 | ceph-2       | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| efe4b393-9ba3-4225-ae3e-254f4fc496cb | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c7e77b08-6484-42f4-95f1-ea19f1385b78 | compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 2b785fc9-f967-4104-9ed2-62a0e9fc4bde | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 6d792b44-fce8-4079-a408-d20e950bb351 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| dec1a59a-85a3-426d-953a-e34857504f20 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+


Once the stack was successfully scaled:
[stack@undercloud-0 ~]$ . overcloudrc 
[stack@undercloud-0 ~]$ nova hypervisor-list
+----+-----------------------+-------+---------+
| ID | Hypervisor hostname   | State | Status  |
+----+-----------------------+-------+---------+
| 3  | compute-0.localdomain | up    | enabled |
| 6  | compute-1.localdomain | up    | enabled |
+----+-----------------------+-------+---------+


[stack@undercloud-0 ~]$ . stackrc 
[stack@undercloud-0 ~]$ heat stack-list
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 3ae60b0c-d723-4f8b-a6ff-4bd52c4e345a | overcloud  | UPDATE_COMPLETE | 2017-02-02T18:44:49 | 2017-02-02T19:20:16 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+




Checked the uptime of all nodes (192.0.2.14  is the new compute):
[stack@undercloud-0 ~]$ for i in `nova list|awk '/ACTIVE/ {print $(NF-1)}'|awk -F"=" '{print $NF}'`; do echo $i; ssh -o StrictHostKeyChecking=no heat-admin@$i "uptime"; done
192.0.2.9
 19:41:18 up 52 min,  0 users,  load average: 0.06, 0.09, 0.07
192.0.2.7
 19:41:18 up 53 min,  0 users,  load average: 0.05, 0.08, 0.07
192.0.2.8
 19:41:18 up 53 min,  0 users,  load average: 0.05, 0.09, 0.07
192.0.2.13
 19:41:19 up 52 min,  0 users,  load average: 0.15, 0.34, 0.26
192.0.2.14
 19:41:19 up 17 min,  0 users,  load average: 0.06, 0.07, 0.11
192.0.2.11
 19:41:19 up 52 min,  0 users,  load average: 1.03, 1.76, 1.70
192.0.2.10
 19:41:20 up 53 min,  0 users,  load average: 1.72, 1.47, 1.26
192.0.2.12
 19:41:20 up 52 min,  0 users,  load average: 1.29, 1.33, 1.25


looks good - verifying.

Comment 5 errata-xmlrpc 2017-03-01 13:34:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0362.html


Note You need to log in before you can comment on or make changes to this bug.