Bug 1472142

Summary: Cannot scale up compute node
Product: Red Hat OpenStack Reporter: Gurenko Alex <agurenko>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Amit Ugol <augol>
Severity: high Docs Contact:
Priority: high    
Version: 11.0 (Ocata)CC: agurenko, aschultz, dbecker, mburns, morazi, rhel-osp-director-maint, slinaber
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-19 11:42:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1447859    

Description Gurenko Alex 2017-07-18 07:21:52 UTC 
Description of problem:



Version-Release number of selected component (if applicable):

RHOS 11
Build: 2017-06-30.3

How reproducible: 100%


Steps to Reproduce:
1. Deploy 1 controller, 1 compute node
2. Add additional virt node
3. Add --compute-scale 2 to overcloud_deploy.sh and execute

Actual results:

Update_Complete, but only 1 compute node is still deployed

Expected results:

2nd compute node added to the stack

Additional info:

overcloud_deploy.sh:

openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--compute-scale 2 \
--control-scale 1 \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/sattelite-5.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
--log-file overcloud_deployment_89.log

[stack@undercloud-0 ~]$ openstack server list
+-----------------------------+--------------+--------+-----------------------+----------------+
| ID                          | Name         | Status | Networks              | Image Name     |
+-----------------------------+--------------+--------+-----------------------+----------------+
| 8b06dbe8-fbe9-4a0d-         | controller-0 | ACTIVE | ctlplane=192.168.24.8 | overcloud-full |
| 8cf6-69ac9983d74b           |              |        |                       |                |
| 71d81bbe-3812-402e-8508-9e6 | compute-0    | ACTIVE | ctlplane=192.168.24.6 | overcloud-full |
| f4116af47                   |              |        |                       |                |
+-----------------------------+--------------+--------+-----------------------+----------------+

[stack@undercloud-0 ~]$ openstack baremetal node list
+------------------+--------------+------------------+-------------+--------------------+-------------+
| UUID             | Name         | Instance UUID    | Power State | Provisioning State | Maintenance |
+------------------+--------------+------------------+-------------+--------------------+-------------+
| 64a03d7a-c884-40 | controller-0 | 8b06dbe8-fbe9    | power on    | active             | False       |
| 62-acf9-1c93fa20 |              | -4a0d-8cf6-69ac9 |             |                    |             |
| 6721             |              | 983d74b          |             |                    |             |
| e0258248-6c90-48 | compute-0    | None             | power off   | available          | False       |
| 2d-9048-ca66d527 |              |                  |             |                    |             |
| 4d52             |              |                  |             |                    |             |
| cc0737d8-205e-46 | compute-1    | 71d81bbe-3812-40 | power on    | active             | False       |
| 29-8975-02536fb8 |              | 2e-8508-9e6f4116 |             |                    |             |
| 3cfd             |              | af47             |             |                    |             |
+------------------+--------------+------------------+-------------+--------------------+-------------+

[stack@undercloud-0 ~]$ openstack stack list
+----------------------+------------+-----------------+----------------------+----------------------+
| ID                   | Stack Name | Stack Status    | Creation Time        | Updated Time         |
+----------------------+------------+-----------------+----------------------+----------------------+
| c413e07e-fa63-4913-9 | overcloud  | UPDATE_COMPLETE | 2017-07-17T16:51:15Z | 2017-07-18T06:48:52Z |
| 407-60a73618e0d0     |            |                 |                      |                      |
+----------------------+------------+-----------------+----------------------+----------------------+
Comment 1 Alex Schultz 2017-07-18 18:49:14 UTC 
I am unable to reproduce this. I did notice a warning if you don't tag the node and are using --compute-flavor when deploying.  That being said, once i tagged the new compute node and updated the deployment to do --compute-scale 2, it deployed and I have all 3 nodes. Can you provide logs or an environment?  


[stack@undercloud-0 ~]$ openstack baremetal node list
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| 631819ee-2929-413e-8964-6d29846c4e7f | controller-0 | ac030df7-e972-4be6-acda-09b250786aef | power on    | active             | False       |
| 311fc0a9-df7c-49b1-97f5-61b7d5cb4ef9 | compute-0    | ceb8a71e-119a-4450-9d0e-fc2fccca1e87 | power on    | active             | False       |
| 2942d9ec-635e-4fcf-834a-a105f0a2ef60 | compute-1    | None                                 | power off   | available          | False       |
| cd9c64b6-e10b-4881-ae6b-4dd0a438e506 | compute-2    | None                                 | power off   | available          | False       |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+

[stack@undercloud-0 ~]$ cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
--templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--control-scale 1 \
--control-flavor controller \
--compute-scale 2 \
--compute-flavor compute \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
--log-file overcloud_deployment_62.log


...snip...
2017-07-18 17:50:46Z [overcloud.ControllerAllNodesDeployment]: UPDATE_COMPLETE  state changed
2017-07-18 17:50:46Z [overcloud.ComputeAllNodesValidationDeployment]: UPDATE_IN_PROGRESS  state changed
2017-07-18 17:50:46Z [overcloud.ControllerAllNodesValidationDeployment]: UPDATE_IN_PROGRESS  state changed
2017-07-18 17:50:47Z [overcloud.ControllerAllNodesValidationDeployment]: UPDATE_COMPLETE  state changed
2017-07-18 17:51:04Z [overcloud.ComputeAllNodesValidationDeployment]: UPDATE_COMPLETE  state changed
2017-07-18 17:51:04Z [overcloud.AllNodesExtraConfig]: UPDATE_IN_PROGRESS  state changed
2017-07-18 17:51:46Z [overcloud.AllNodesExtraConfig]: UPDATE_COMPLETE  state changed
2017-07-18 17:51:46Z [overcloud.AllNodesDeploySteps]: UPDATE_IN_PROGRESS  state changed
2017-07-18 18:07:43Z [overcloud.AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2017-07-18 18:07:54Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully
^C
(openstack) server list
+--------------------------------------+--------------+--------+------------------------+----------------+
| ID                                   | Name         | Status | Networks               | Image Name     |
+--------------------------------------+--------------+--------+------------------------+----------------+
| 1ae5dd78-ffc9-47ea-80cc-6911acba9864 | compute-1    | ACTIVE | ctlplane=192.168.24.7  | overcloud-full |
| ceb8a71e-119a-4450-9d0e-fc2fccca1e87 | compute-0    | ACTIVE | ctlplane=192.168.24.13 | overcloud-full |
| ac030df7-e972-4be6-acda-09b250786aef | controller-0 | ACTIVE | ctlplane=192.168.24.16 | overcloud-full |
+--------------------------------------+--------------+--------+------------------------+----------------+
Comment 2 Gurenko Alex 2017-07-19 07:49:12 UTC 
(In reply to Alex Schultz from comment #1)
> I am unable to reproduce this. I did notice a warning if you don't tag the
> node and are using --compute-flavor when deploying.  That being said, once i
> tagged the new compute node and updated the deployment to do --compute-scale
> 2, it deployed and I have all 3 nodes. Can you provide logs or an
> environment?  
> 
> 
> [stack@undercloud-0 ~]$ openstack baremetal node list
> +--------------------------------------+--------------+----------------------
> ----------------+-------------+--------------------+-------------+
> | UUID                                 | Name         | Instance UUID       
> | Power State | Provisioning State | Maintenance |
> +--------------------------------------+--------------+----------------------
> ----------------+-------------+--------------------+-------------+
> | 631819ee-2929-413e-8964-6d29846c4e7f | controller-0 |
> ac030df7-e972-4be6-acda-09b250786aef | power on    | active             |
> False       |
> | 311fc0a9-df7c-49b1-97f5-61b7d5cb4ef9 | compute-0    |
> ceb8a71e-119a-4450-9d0e-fc2fccca1e87 | power on    | active             |
> False       |
> | 2942d9ec-635e-4fcf-834a-a105f0a2ef60 | compute-1    | None                
> | power off   | available          | False       |
> | cd9c64b6-e10b-4881-ae6b-4dd0a438e506 | compute-2    | None                
> | power off   | available          | False       |
> +--------------------------------------+--------------+----------------------
> ----------------+-------------+--------------------+-------------+
> 
> [stack@undercloud-0 ~]$ cat overcloud_deploy.sh 
> #!/bin/bash
> 
> openstack overcloud deploy \
> --templates \
> --libvirt-type kvm \
> --ntp-server clock.redhat.com \
> --control-scale 1 \
> --control-flavor controller \
> --compute-scale 2 \
> --compute-flavor compute \
> --environment-file
> /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml \
> -e /home/stack/virt/network/network-environment.yaml \
> -e
> /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.
> yaml \
> -e /home/stack/virt/hostnames.yml \
> -e /home/stack/virt/debug.yaml \
> --log-file overcloud_deployment_62.log
> 
> 
> ...snip...
> 2017-07-18 17:50:46Z [overcloud.ControllerAllNodesDeployment]:
> UPDATE_COMPLETE  state changed
> 2017-07-18 17:50:46Z [overcloud.ComputeAllNodesValidationDeployment]:
> UPDATE_IN_PROGRESS  state changed
> 2017-07-18 17:50:46Z [overcloud.ControllerAllNodesValidationDeployment]:
> UPDATE_IN_PROGRESS  state changed
> 2017-07-18 17:50:47Z [overcloud.ControllerAllNodesValidationDeployment]:
> UPDATE_COMPLETE  state changed
> 2017-07-18 17:51:04Z [overcloud.ComputeAllNodesValidationDeployment]:
> UPDATE_COMPLETE  state changed
> 2017-07-18 17:51:04Z [overcloud.AllNodesExtraConfig]: UPDATE_IN_PROGRESS 
> state changed
> 2017-07-18 17:51:46Z [overcloud.AllNodesExtraConfig]: UPDATE_COMPLETE  state
> changed
> 2017-07-18 17:51:46Z [overcloud.AllNodesDeploySteps]: UPDATE_IN_PROGRESS 
> state changed
> 2017-07-18 18:07:43Z [overcloud.AllNodesDeploySteps]: UPDATE_COMPLETE  state
> changed
> 2017-07-18 18:07:54Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed
> successfully
> ^C
> (openstack) server list
> +--------------------------------------+--------------+--------+-------------
> -----------+----------------+
> | ID                                   | Name         | Status | Networks   
> | Image Name     |
> +--------------------------------------+--------------+--------+-------------
> -----------+----------------+
> | 1ae5dd78-ffc9-47ea-80cc-6911acba9864 | compute-1    | ACTIVE |
> ctlplane=192.168.24.7  | overcloud-full |
> | ceb8a71e-119a-4450-9d0e-fc2fccca1e87 | compute-0    | ACTIVE |
> ctlplane=192.168.24.13 | overcloud-full |
> | ac030df7-e972-4be6-acda-09b250786aef | controller-0 | ACTIVE |
> ctlplane=192.168.24.16 | overcloud-full |
> +--------------------------------------+--------------+--------+-------------
> -----------+----------------+

I have seal06.qa.lab.tlv with HA environment right now, trying to scale from 3 + 2 to 3 + 3 and having same issue. I have nodes tagged, so it should work, but I'm gonna try and specify flavor now, lets see whether it helps.
Comment 3 Gurenko Alex 2017-07-19 11:42:09 UTC 
 So yeah, the problem is that --compute/control-scale was overwritten with ControllerCount/ComputeCount in one of the yamls. I've missed this change.