Bug 1684868
| Summary: | when deploying with skip deploy-identifier tempest test test_server_basic_ops.TestServerBasicOps.test_server_basic_ops fails | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Victor Voronkov <vvoronko> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Rajesh Tailor <ratailor> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Joe H. Rahme <jhakimra> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 14.0 (Rocky) | CC: | mbooth, mburns, mschuppe, owalsh, ramishra, ratailor, rheslop, slinaber, vvoronko | ||||
| Target Milestone: | z3 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 14.0 (Rocky) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-9.3.1-0.20190314162752.d0a6cb1.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-07-02 20:08:27 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1562082, 1693563 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
Verified by running a scale out and running the tempest test:
1. Disable a compute service
----------------------------
(overcloud) [stack@undercloud-0 ~]$ openstack compute service list
+----+------------------+--------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+--------------------------+----------+---------+-------+----------------------------+
| 2 | nova-conductor | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:20:21.000000 |
| 11 | nova-conductor | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:20:24.000000 |
| 17 | nova-conductor | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:20:22.000000 |
| 23 | nova-consoleauth | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:20:25.000000 |
| 26 | nova-consoleauth | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:20:20.000000 |
| 29 | nova-consoleauth | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:20:24.000000 |
| 32 | nova-scheduler | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:20:25.000000 |
| 44 | nova-scheduler | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:20:22.000000 |
| 47 | nova-scheduler | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:20:20.000000 |
| 68 | nova-compute | compute-0.localdomain | nova | enabled | up | 2019-06-20T10:20:21.000000 |
| 80 | nova-compute | compute-1.localdomain | nova | enabled | up | 2019-06-20T10:20:24.000000 |
+----+------------------+--------------------------+----------+---------+-------+----------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack compute service set compute-1.localdomain nova-compute --disable
(overcloud) [stack@undercloud-0 ~]$ openstack compute service list
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
| 2 | nova-conductor | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:22:01.000000 |
| 11 | nova-conductor | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:22:04.000000 |
| 17 | nova-conductor | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:22:02.000000 |
| 23 | nova-consoleauth | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:22:05.000000 |
| 26 | nova-consoleauth | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:22:00.000000 |
| 29 | nova-consoleauth | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:22:04.000000 |
| 32 | nova-scheduler | controller-1.localdomain | internal | enabled | up | 2019-06-20T10:22:05.000000 |
| 44 | nova-scheduler | controller-0.localdomain | internal | enabled | up | 2019-06-20T10:22:02.000000 |
| 47 | nova-scheduler | controller-2.localdomain | internal | enabled | up | 2019-06-20T10:22:00.000000 |
| 68 | nova-compute | compute-0.localdomain | nova | enabled | up | 2019-06-20T10:22:01.000000 |
| 80 | nova-compute | compute-1.localdomain | nova | disabled | up | 2019-06-20T10:22:04.000000 |
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+
| d3c72b3c-e625-468d-9d52-91c55850f9cb | overcloud | 23eb5c82d4f24460a7b297aef27c35dc | UPDATE_COMPLETE | 2019-06-17T12:12:44Z | 2019-06-18T17:41:37Z |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| c60458ce-b589-4f31-bd0b-5b74efcc2871 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
| bbf1bc82-249b-46c7-8f69-03e83f65b91c | compute-0 | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | compute |
| 524fa118-0adb-4976-a821-a879cf0d4b44 | controller-2 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | controller |
| fb245279-7bc7-4e42-add5-d8ca602cda1e | compute-1 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | compute |
| 14dbce2f-fb48-4977-814b-efa4cea8cdc6 | controller-1 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
3. Delete node
--------------
(undercloud) [stack@undercloud-0 ~]$ cat overcloud_node_delete.sh
#!/bin/bash
openstack overcloud node delete --stack d3c72b3c-e625-468d-9d52-91c55850f9cb \
--templates -e /home/stack/virt/config_lvm.yaml \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/extra_templates.yaml \
fb245279-7bc7-4e42-add5-d8ca602cda1e
(undercloud) [stack@undercloud-0 ~]$ time bash overcloud_node_delete.sh
Deleting the following nodes from stack overcloud:
- fb245279-7bc7-4e42-add5-d8ca602cda1e
Waiting for messages on queue 'tripleo' with no timeout.
real 21m40.710s
user 0m0.913s
sys 0m0.218s
(undercloud) [stack@undercloud-0 ~]$ openstack server list # note that compute-1 is removed.
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| c60458ce-b589-4f31-bd0b-5b74efcc2871 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
| bbf1bc82-249b-46c7-8f69-03e83f65b91c | compute-0 | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | compute |
| 524fa118-0adb-4976-a821-a879cf0d4b44 | controller-2 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | controller |
| 14dbce2f-fb48-4977-814b-efa4cea8cdc6 | controller-1 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(overcloud) [stack@undercloud-0 ~]$ openstack compute service list
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
| 2 | nova-conductor | controller-1.localdomain | internal | enabled | up | 2019-06-20T14:54:35.000000 |
| 11 | nova-conductor | controller-2.localdomain | internal | enabled | up | 2019-06-20T14:54:38.000000 |
| 17 | nova-conductor | controller-0.localdomain | internal | enabled | up | 2019-06-20T14:54:36.000000 |
| 23 | nova-consoleauth | controller-1.localdomain | internal | enabled | up | 2019-06-20T14:54:38.000000 |
| 26 | nova-consoleauth | controller-0.localdomain | internal | enabled | up | 2019-06-20T14:54:35.000000 |
| 29 | nova-consoleauth | controller-2.localdomain | internal | enabled | up | 2019-06-20T14:54:38.000000 |
| 32 | nova-scheduler | controller-1.localdomain | internal | enabled | up | 2019-06-20T14:54:30.000000 |
| 44 | nova-scheduler | controller-0.localdomain | internal | enabled | up | 2019-06-20T14:54:39.000000 |
| 47 | nova-scheduler | controller-2.localdomain | internal | enabled | up | 2019-06-20T14:54:37.000000 |
| 68 | nova-compute | compute-0.localdomain | nova | enabled | up | 2019-06-20T14:54:33.000000 |
| 80 | nova-compute | compute-1.localdomain | nova | disabled | down | 2019-06-20T11:05:44.000000 |
+----+------------------+--------------------------+----------+----------+-------+----------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack compute service delete 80
4. re-add the node
------------------
(overcloud) [stack@undercloud-0 ~]$ cat /home/stack/virt/nodes_data.yaml
parameter_defaults:
ControllerCount: 3
OvercloudControlFlavor: controller
ComputeCount: 2
OvercloudComputeFlavor: compute
(overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy_blacklist.sh
#!/bin/bash
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--skip-deployer-identifier \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/extra_templates.yaml \
-e /home/stack/blacklist.yaml \
--log-file overcloud_deployment_47.log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672 |
Created attachment 1540273 [details] Full tempest test failure log Description of problem: Deployment with skip-deploy-identifier flag and performing stack update or scale out, causing tempest tests failures due fact, that nova discovery is not triggered Version-Release number of selected component (if applicable): How reproducible: Deployment with skip-deploy-identifier flag Steps to Reproduce: 1. 2. 3. Actual results: tempest test fails with error: tempest.exceptions.BuildErrorException: Server de052b96-fcfd-4a2e-b743-d565bc571c49 failed to build and is in ERROR status Details: {u'message': u"Host 'compute-3.localdomain' is not mapped to any cell", u'code': 400, u'created': u'2019-02-27T07:31:03Z'} Expected results: Additional info: