Bug 1622578
| Summary: | [OSP14] overcloud stack update attempts to redeploy servers which have already been deployed | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Emilien Macchi <emacchi> |
| Status: | CLOSED ERRATA | QA Contact: | Gurenko Alex <agurenko> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 14.0 (Rocky) | CC: | agurenko, johfulto, mburns, mcornea, michele, mkrcmari, ohochman |
| Target Milestone: | beta | Keywords: | AutomationBlocker, Regression, Triaged |
| Target Release: | 14.0 (Rocky) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | DFG:DF | ||
| Fixed In Version: | openstack-tripleo-heat-templates-9.0.0-0.20180823010420.792c26e.el7ost.noarch | Doc Type: | No Doc Update |
| Doc Text: |
undefined
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-01-11 11:51:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Also catch the same issue with re-deploy of 3ctr+3comp+3ceph env raising BZ priority/severity: as we're blocked with all stack update operation such: scale up/down compute, change stack configuration, replace controller, etc.. The error is in Nova Placement: 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager [req-bae88972-509d-4527-bd90-8e91ae9b1a11 - - - - -] Error updating resources for node f1c3ef9c-cfeb-4623-be5d-392717fb5f97.: ResourceProviderSyncFailed: Failed to synchronize the placement service with resource provider information supplied by the compute host. 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager Traceback (most recent call last): 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7717, in _update_available_resource_for_node 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 703, in update_available_resource 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager self._update_available_resource(context, resources) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager return f(*args, **kwargs) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 726, in _update_available_resource 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager self._init_compute_node(context, resources) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _init_compute_node 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager self._update(context, cn) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 938, in _update 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager self._update_to_placement(context, compute_node) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 907, in _update_to_placement 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager reportclient.update_from_provider_tree(context, prov_tree) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1531, in update_from_provider_tree 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager raise exception.ResourceProviderSyncFailed() 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager ResourceProviderSyncFailed: Failed to synchronize the placement service with resource provider information supplied by the compute host. 2018-08-27 05:35:48.839 1 ERROR nova.compute.manager I hit this upstream too. To reproduce use tripleo quickstart and then run overcloud-deploy.sh twice. The deploy will try to create new nodes rather than reuse them. This may result in the deployment failing the a "no valid host found" but it seems that's just a side effect and I conjecture that if I had more available ironic nodes it would try to use them to scale up. It doesn't try to create another heat stack however. An idempotence test where you just just reassert the configurations by re-running 'openstack overcloud deploy' would pick this up but for that test it cannot be reproduced using already-deployed servers [1] which might explain why upstream CI didn't reproduce this bug. [1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/deployed_server.html As discussed with Emilien/John this is likely due to missing: https://review.openstack.org/#/c/588529 in the tht version used (which was openstack-tripleo-heat-templates-9.0.0-0.20180804083746.el7ost.noarch) Should be fixed when using: openstack-tripleo-heat-templates-9.0.0-0.20180823010420 Artem will verify and update here with results. openstack-tripleo-heat-templates-9.0.0-0.20180823010420.792c26e.el7ost.noarch
2018-08-29 16:54:38Z [overcloud]: UPDATE_FAILED Resource UPDATE failed: resources[0]: resources.Compute.Resource CREATE failed: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
Stack overcloud/d3c313e1-497d-404e-8578-17812a4e9b5c UPDATE_FAILED
overcloud.Controller.0.Controller:
resource_type: OS::TripleO::ControllerServer
physical_resource_id: 795587c2-d400-4e2d-a08c-b57ae9b5cc80
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.Compute.0.NovaCompute:
resource_type: OS::TripleO::ComputeServer
physical_resource_id: 79bc0357-b72b-4c5d-ba7d-952082739d57
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
Heat Stack update failed.
Heat Stack update failed.
(undercloud) [stack@undercloud-0 ~]$ nova list --all
/usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SubjectAltNameWarning
/usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SubjectAltNameWarning
+--------------------------------------+--------------+----------------------------------+--------+------------+-------------+----------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+----------------------------------+--------+------------+-------------+----------+
| 79bc0357-b72b-4c5d-ba7d-952082739d57 | compute-0 | 94cc77af8fdd46a0ad6c4cc6e9d85d34 | ERROR | - | NOSTATE | |
| 84dfb60c-762b-41c8-a81b-662ce1733f08 | compute-0 | 94cc77af8fdd46a0ad6c4cc6e9d85d34 | ACTIVE | - | Running | |
| 795587c2-d400-4e2d-a08c-b57ae9b5cc80 | controller-0 | 94cc77af8fdd46a0ad6c4cc6e9d85d34 | ERROR | - | NOSTATE | |
| 89da428b-665f-4adf-9773-d30e891bd1e6 | controller-0 | 94cc77af8fdd46a0ad6c4cc6e9d85d34 | ACTIVE | - | Running | |
(In reply to Artem Hrechanychenko from comment #11) > openstack-tripleo-heat-templates-9.0.0-0.20180823010420.792c26e.el7ost.noarch > > > 2018-08-29 16:54:38Z [overcloud]: UPDATE_FAILED Resource UPDATE failed: > resources[0]: resources.Compute.Resource CREATE failed: ResourceInError: > resources.NovaCompute: Went to status ERROR due to "Message: No valid host > was found. , Code: 500" > It is the containerized undercloud which uses THT so you need to make sure that the undercloud is reinstalled. You basically did not apply the patch: (undercloud) [stack@undercloud-0 ~]$ docker inspect --format '{{json .Mounts}}' heat_engine |grep usr (undercloud) [stack@undercloud-0 ~]$ Unless heat_engine returns the following, you did not apply things correctly: "/usr/lib/heat","Destination":"/usr/lib/heat","Mode":"ro" In short: the heat_engine container needs some plugins from the undercloud host otherwise it will spawn up new nova hosts. I just tested it on the OSP14 env marian gave me and it does not try to spawn multiple nodes: +--------------------------------------+--------------+--------+------------+-------------+------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+------------------------+ | 8aec3477-26d5-4e99-95fd-bb5e8cd6dbb8 | ceph-0 | ACTIVE | - | Running | ctlplane=192.168.24.16 | | 31ee01e7-87c3-4e76-90dd-626c3434fa17 | compute-0 | ACTIVE | - | Running | ctlplane=192.168.24.7 | | 70630ff6-2ad1-4448-a68f-98ce1aaadb62 | compute-1 | ACTIVE | - | Running | ctlplane=192.168.24.20 | | 327b8639-6595-48df-904a-3090432cfeea | controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.15 | | da072218-1cb5-4d2c-a3fc-c5e809ab522b | controller-1 | ACTIVE | - | Running | ctlplane=192.168.24.6 | | 9acc5840-8214-4dc0-9d0a-bda85b47ca56 | controller-2 | ACTIVE | - | Running | ctlplane=192.168.24.10 | +--------------------------------------+--------------+--------+------------+-------------+------------------------+ (We are hitting a separate issue now, but it is unrelated to this one) (In reply to Michele Baldessari from comment #12) > (In reply to Artem Hrechanychenko from comment #11) > > openstack-tripleo-heat-templates-9.0.0-0.20180823010420.792c26e.el7ost.noarch > > > > > > 2018-08-29 16:54:38Z [overcloud]: UPDATE_FAILED Resource UPDATE failed: > > resources[0]: resources.Compute.Resource CREATE failed: ResourceInError: > > resources.NovaCompute: Went to status ERROR due to "Message: No valid host > > was found. , Code: 500" > > > > It is the containerized undercloud which uses THT so you need to make sure > that the undercloud is reinstalled. You basically did not apply the patch: > (undercloud) [stack@undercloud-0 ~]$ docker inspect --format '{{json > .Mounts}}' heat_engine |grep usr > (undercloud) [stack@undercloud-0 ~]$ > > Unless heat_engine returns the following, you did not apply things correctly: > "/usr/lib/heat","Destination":"/usr/lib/heat","Mode":"ro" > > In short: the heat_engine container needs some plugins from the undercloud > host otherwise it will spawn up new nova hosts. > > I just tested it on the OSP14 env marian gave me and it does not try to > spawn multiple nodes: > +--------------------------------------+--------------+--------+------------ > +-------------+------------------------+ > | ID | Name | Status | Task State > | Power State | Networks | > +--------------------------------------+--------------+--------+------------ > +-------------+------------------------+ > | 8aec3477-26d5-4e99-95fd-bb5e8cd6dbb8 | ceph-0 | ACTIVE | - > | Running | ctlplane=192.168.24.16 | > | 31ee01e7-87c3-4e76-90dd-626c3434fa17 | compute-0 | ACTIVE | - > | Running | ctlplane=192.168.24.7 | > | 70630ff6-2ad1-4448-a68f-98ce1aaadb62 | compute-1 | ACTIVE | - > | Running | ctlplane=192.168.24.20 | > | 327b8639-6595-48df-904a-3090432cfeea | controller-0 | ACTIVE | - > | Running | ctlplane=192.168.24.15 | > | da072218-1cb5-4d2c-a3fc-c5e809ab522b | controller-1 | ACTIVE | - > | Running | ctlplane=192.168.24.6 | > | 9acc5840-8214-4dc0-9d0a-bda85b47ca56 | controller-2 | ACTIVE | - > | Running | ctlplane=192.168.24.10 | > +--------------------------------------+--------------+--------+------------ > +-------------+------------------------+ > > > (We are hitting a separate issue now, but it is unrelated to this one) helped applied tht patch, re-install undercloud and re-deploy without problems (undercloud) [stack@undercloud-0 ~]$ heat stack-list WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ | id | stack_name | stack_status | creation_time | updated_time | project | +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ | 1b63eb20-e75a-4403-a2ac-ae4dee29527e | overcloud | UPDATE_COMPLETE | 2018-08-29T17:58:47Z | 2018-08-29T19:12:44Z | 94cc77af8fdd46a0ad6c4cc6e9d85d34 | +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ (undercloud) [stack@undercloud-0 ~]$ rpm -qa "*templates*" openstack-tripleo-heat-templates-9.0.0-0.20180823010420.792c26e.el7ost.noarch VERIFIED openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ cat core_puddle_version 2018-09-06.1 (undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /home/stack/templates/post_config.yaml \ --log-file overcloud_deployment_71.log undercloud) [stack@undercloud-0 ~]$ cat templates/post_config.yaml resource_registry: OS::TripleO::NodeExtraConfigPost: /home/stack/templates/nameserver.yaml parameter_defaults: nameserver_ip1: 10.11.5.19 nameserver_ip2: 10.10.160.2 nameserver_ip3: 10.5.30.160 [heat-admin@controller-1 ~]$ cat /etc/resolv.conf # Generated by NetworkManager nameserver 10.0.0.1 nameserver 10.11.5.19 nameserver 10.10.160.2 nameserver 10.5.30.160 (undercloud) [stack@undercloud-0 ~]$ heat stack-list WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ | id | stack_name | stack_status | creation_time | updated_time | project | +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ | 55dbc203-5bb4-471a-a9f7-2b966319acc7 | overcloud | UPDATE_COMPLETE | 2018-09-13T11:42:16Z | 2018-09-13T13:11:28Z | 0e96526e20fb4e03a7c6861589725cd7 | +--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+ (undercloud) [stack@undercloud-0 ~]$ openstack overcloud status +-----------+---------------------+---------------------+-------------------+ | Plan Name | Created | Updated | Deployment Status | +-----------+---------------------+---------------------+-------------------+ | overcloud | 2018-09-13 13:28:58 | 2018-09-13 14:04:30 | DEPLOY_SUCCESS | +-----------+---------------------+---------------------+-------------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Description of problem: OSP14 installation with 3controller, 3 computes and 3 ceph is completed without errors. After that I decided to re-deploy with small changes (undercloud) [stack@undercloud-0 ~]$ cat templates/post_config.yaml resource_registry: OS::TripleO::NodeExtraConfigPost: /home/stack/templates/nameserver.yaml parameter_defaults: nameserver_ip1: 10.11.5.19 nameserver_ip2: 10.10.160.2 nameserver_ip3: 10.5.30.160 (undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /home/stack/templates/post_config.yaml \ --log-file overcloud_deployment_85.log re-deploy failed for me 1874a1fb-3eee-49d6-b23d-bfed44c7dec8 | overcloud | UPDATE_FAILED | 2018-08-27T10:01:00Z | 2018-08-27T11:24:32Z | 46fc34ddbd0a4412875034a1127f7e93 | +--------------------------------------+------------+---------------+----------------------+----------------------+----------------------------------+ (undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud overcloud.Controller.1.Controller: resource_type: OS::TripleO::ControllerServer physical_resource_id: 2c4dfd18-f80d-4644-8466-e2090c6e6db6 status: CREATE_FAILED status_reason: | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.Controller.0.Controller: resource_type: OS::TripleO::ControllerServer physical_resource_id: 08305d4e-a21f-4bb9-892a-e975f92b1776 status: CREATE_FAILED status_reason: | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.Controller.2.Controller: resource_type: OS::TripleO::ControllerServer physical_resource_id: e99e46f3-dcfc-4426-8de4-489e0941dc04 status: CREATE_FAILED status_reason: | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.Compute.1.NovaCompute: resource_type: OS::TripleO::ComputeServer physical_resource_id: 26a8ce43-5755-4538-bb4a-3a93ca8b0a6d status: CREATE_FAILED status_reason: | ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.Compute.0.NovaCompute: resource_type: OS::TripleO::ComputeServer physical_resource_id: ec17b4c8-6b2f-445b-b4f7-aed18edf1887 status: CREATE_FAILED status_reason: | ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.CephStorage.1.CephStorage: resource_type: OS::TripleO::CephStorageServer physical_resource_id: 7935e6b2-c18f-41c9-86d9-fe1676472943 status: CREATE_FAILED status_reason: | ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.CephStorage.0.CephStorage: resource_type: OS::TripleO::CephStorageServer physical_resource_id: df0d2c51-1b10-4db9-97b2-a3657db1a15b status: CREATE_FAILED status_reason: | ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. , Code: 500" overcloud.CephStorage.2.CephStorage: resource_type: OS::TripleO::CephStorageServer physical_resource_id: 9e225298-96e6-46d1-8e88-57cffea7ea9f status: CREATE_FAILED status_reason: | ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. , Code: 500" by some reason it tried to duplicate nodes count +--------------------------------------+--------------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+----------+ | b2b41241-e97a-4622-9bb4-625bf9f4bceb | ceph-0 | ACTIVE | - | Running | | | df0d2c51-1b10-4db9-97b2-a3657db1a15b | ceph-0 | ERROR | - | NOSTATE | | | 17ad607d-cc4c-49aa-a441-bb465265d7c4 | ceph-1 | ACTIVE | - | Running | | | 7935e6b2-c18f-41c9-86d9-fe1676472943 | ceph-1 | ERROR | - | NOSTATE | | | 44377608-cac0-4065-9cb6-cc11e9a77699 | ceph-2 | ACTIVE | - | Running | | | 9e225298-96e6-46d1-8e88-57cffea7ea9f | ceph-2 | ERROR | - | NOSTATE | | | 9d89cc33-c824-45c6-85b8-7564fc4207bb | compute-0 | ACTIVE | - | Running | | | ec17b4c8-6b2f-445b-b4f7-aed18edf1887 | compute-0 | ERROR | - | NOSTATE | | | 266438e5-8edc-49bf-94f4-27ec8628e105 | compute-1 | ACTIVE | - | Running | | | 26a8ce43-5755-4538-bb4a-3a93ca8b0a6d | compute-1 | ERROR | - | NOSTATE | | | 08305d4e-a21f-4bb9-892a-e975f92b1776 | controller-0 | ERROR | - | NOSTATE | | | d3db59cd-b4af-486c-abbd-ff0479de6404 | controller-0 | ACTIVE | - | Running | | | 2c4dfd18-f80d-4644-8466-e2090c6e6db6 | controller-1 | ERROR | - | NOSTATE | | | 9cb8d777-4d6b-441f-b00a-86a072e83bf4 | controller-1 | ACTIVE | - | Running | | | 6bf9e15f-a3d0-4dee-abeb-136610fce9f6 | controller-2 | ACTIVE | - | Running | | | e99e46f3-dcfc-4426-8de4-489e0941dc04 | controller-2 | ERROR | - | NOSTATE | | +--------------------------------------+--------------+--------+------------+-------------+----------+ Version-Release number of selected component (if applicable): (undercloud) [stack@undercloud-0 ~]$ cat core_puddle_version 2018-08-23.3 rpm -qa "*openstack*" openstack-heat-common-12.0.0-0.20180604085325.7d878a8.el7ost.noarch python2-openstacksdk-0.17.1-0.20180801104359.f28a1b0.el7ost.noarch openstack-tripleo-puppet-elements-9.0.0-0.20180801001359.d9df3a3.el7ost.noarch openstack-tripleo-image-elements-9.0.0-0.20180801003129.39e37fe.el7ost.noarch openstack-heat-monolith-12.0.0-0.20180604085325.7d878a8.el7ost.noarch openstack-tripleo-validations-9.2.1-0.20180726214014.7627d15.el7ost.noarch openstack-tripleo-common-9.2.1-0.20180803214330.el7ost.noarch openstack-heat-api-12.0.0-0.20180604085325.7d878a8.el7ost.noarch openstack-tripleo-common-containers-9.2.1-0.20180803214330.el7ost.noarch python-openstackclient-lang-3.16.0-0.20180801103525.f77ca68.el7ost.noarch openstack-tripleo-heat-templates-9.0.0-0.20180804083746.el7ost.noarch puppet-openstacklib-13.2.0-0.20180724214016.a47c20e.el7ost.noarch openstack-heat-agents-1.6.1-0.20180709100740.fdd6a5f.el7ost.noarch openstack-heat-engine-12.0.0-0.20180604085325.7d878a8.el7ost.noarch python2-openstackclient-3.16.0-0.20180801103525.f77ca68.el7ost.noarch openstack-selinux-0.8.15-0.20180524134826.b63283a.el7ost.noarch puppet-openstack_extras-13.2.0-0.20180728124012.4540a49.el7ost.noarch How reproducible: Steps to Reproduce: 1.Deploy OSP13 with 3controller+3compute+3ceph 2.Re-deploy with config changes Actual results: failed with no-valid-host, number of instances *2 Expected results: passed Additional info: