Bug 1290574 - Scaling out an update overcloud (7.1 -> 7.2) doesn't provision a new node
Summary: Scaling out an update overcloud (7.1 -> 7.2) doesn't provision a new node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: y2
: 7.0 (Kilo)
Assignee: Giulio Fidente
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-10 20:35 UTC by Marius Cornea
Modified: 2015-12-21 16:54 UTC (History)
8 users (show)

Fixed In Version: python-rdomanager-oscplugin-0.0.10-22.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-21 16:54:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
deploy debug (2.45 MB, text/plain)
2015-12-11 13:47 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 256670 0 None None None Never
Red Hat Product Errata RHBA-2015:2651 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory 2015-12-21 21:50:26 UTC

Description Marius Cornea 2015-12-10 20:35:53 UTC
Description of problem:
I'm trying to add an additional compute node to an updated overcloud but no new node gets provisioned. After running the deploy command with +1 compute nodes for the 2nd time(BZ#1290572) the deploy finishes successfully but there's still only one compute node provisioned.  

Version-Release number of selected component (if applicable):
python-rdomanager-oscplugin-0.0.10-21.el7ost.noarch
penstack-heat-api-2015.1.2-4.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.2-4.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-91.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.2-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.2-4.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-engine-2015.1.2-4.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch

1. Deploy 7.1 by using 7.1 templates:
openstack overcloud deploy \
    --templates ~/templates/my-overcloud \
    --control-scale 3 --compute-scale 1 --ceph-storage-scale 3 \
    --ntp-server clock.redhat.com \
    --libvirt-type qemu \
    -e ~/templates/my-overcloud/environments/network-isolation.yaml \
    -e ~/templates/network-environment.yaml \
    -e ~/templates/firstboot-environment.yaml \
    -e ~/templates/ceph.yaml 

2. Update the undercloud to 7.2 and run the update procedure to 7.2 with 7.2 templates:
/usr/bin/yes '' | openstack overcloud update stack overcloud -i \
         --templates ~/templates/my-overcloud \
         -e ~/templates/my-overcloud/overcloud-resource-registry-puppet.yaml \
         -e ~/templates/my-overcloud/environments/network-isolation.yaml \
         -e ~/templates/network-environment.yaml \
         -e ~/templates/firstboot-environment.yaml \
         -e ~/templates/ceph.yaml \
         -e ~/templates/my-overcloud/environments/updates/update-from-vip.yaml \
         -e ~/templates/ctrlport.yaml

Wait for the update to complete

3. Try to scale out with an additional node:

openstack overcloud deploy \
    --templates ~/templates/my-overcloud \
    --control-scale 3 --compute-scale 2 --ceph-storage-scale 3 \
    --ntp-server clock.redhat.com \
    --libvirt-type qemu \
    -e ~/templates/my-overcloud/overcloud-resource-registry-puppet.yaml \
    -e ~/templates/my-overcloud/environments/network-isolation.yaml \
    -e ~/templates/network-environment.yaml \
    -e ~/templates/firstboot-environment.yaml \
    -e ~/templates/ceph.yaml \
    -e ~/templates/my-overcloud/environments/updates/update-from-vip.yaml \
    -e ~/templates/ctrlport.yaml

Actual results:
stack@instack:~>>> nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 4254a40d-22e7-4166-b6ec-9704d6683e7e | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| 8f3f8394-e31d-4460-9ac0-6aea16eb9cd1 | overcloud-cephstorage-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 57c61042-a99a-40f7-83b1-61e9deab5d4d | overcloud-cephstorage-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| c271fd10-dded-4726-8ec4-03cbb0ba9a6e | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 9f9fbcc1-7fd7-4e1a-afa5-a76909e989da | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| d94e597e-b01a-4804-ac63-9a0ac07cedfd | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 98bf473a-d440-4caa-9978-cee51f1882ea | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

stack@instack:~>>> heat stack-show overcloud  | grep ComputeCount
|                       |   "ComputeCount": "1",                                   

Expected results:


Additional info:
A number of 2 compute nodes get provisioned.

Comment 1 James Slagle 2015-12-10 21:43:57 UTC
can you attach /var/log/heat/heat-api.log and /var/log/heat/heat-engine.log here?

Comment 3 Marius Cornea 2015-12-11 13:47:24 UTC
Created attachment 1104693 [details]
deploy debug

Attaching the deploy command debug log. Command that I ran:

openstack overcloud deploy \
    --templates ~/templates/my-overcloud \
    --control-scale 3 --compute-scale 2 --ceph-storage-scale 3 \
    --ntp-server clock.redhat.com \
    --libvirt-type qemu \
    -e ~/templates/my-overcloud/overcloud-resource-registry-puppet.yaml \
    -e ~/templates/my-overcloud/environments/network-isolation.yaml \
    -e ~/templates/network-environment.yaml \
    -e ~/templates/firstboot-environment.yaml \
    -e ~/templates/ceph.yaml \
    -e ~/templates/my-overcloud/environments/updates/update-from-vip.yaml \
    -e ~/templates/ctrlport.yaml \
    --debug \
    --log-file scaleout-debug.log

Comment 4 Marius Cornea 2015-12-11 14:50:54 UTC
FWIW, I tested scale out with a fresh deployed 7.2 and it completed fine. The difference here is that I didn't pass the update environment files so maybe that's causing the failure during the update flow.

Comment 11 errata-xmlrpc 2015-12-21 16:54:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651


Note You need to log in before you can comment on or make changes to this bug.