Description of problem: The Director node is already upgraded to RHOS-10. The Overcloud nodes are running the latest RHOS-9 packages. The Overcloud upgrade fails with no error on: ~~~ * major-upgrade-ceilometer-wsgi-mitaka-newton.yaml - Converts OpenStack Telemetry (‘Ceilometer’) to a WSGI service. [1] ~~~ [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/upgrading_red_hat_openstack_platform/chap-upgrading_the_environment#sect-Major-Upgrading_the_Overcloud [stack@<director node> ~]$ time openstack overcloud deploy --templates < custom templates > Removing the current plan files Uploading new plan files Started Mistral Workflow. Execution ID: 7a017fe6-0c1b-474d-8829-de1fc76fe959 Plan updated Deploying templates in the directory /tmp/tripleoclient-CwR2nC/tripleo-heat-templates Started Mistral Workflow. Execution ID: 4df350bf-27b2-4d20-8f32-da5006ef82f0 Stack overcloud UPDATE_FAILED Heat Stack update failed. real 1m34.169s user 0m3.046s sys 0m0.606s [stack@<director node> ~]$ [stack@<director node> ~]$ heat stack-list --show-nested |grep -vi complete WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+ | id | stack_name | stack_status | creation_time | updated_time | parent | +--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+ | 5b093501-60aa-4b32-91a6-cd9003787b92 | overcloud | UPDATE_FAILED | 2016-11-30T11:27:13Z | 2017-04-18T14:05:16Z | None | | 6b2cb14b-92e1-4745-bcb9-b078a13493b2 | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | CREATE_FAILED | 2017-04-06T15:42:10Z | None | 5b093501-60aa-4b32-91a6-cd9003787b92 | | 7680285e-6884-45e2-9a53-735b1a2a987b | overcloud-ControllerHostsDeployment-e2oo5f3bo4we | CREATE_FAILED | 2017-04-06T15:42:11Z | None | 5b093501-60aa-4b32-91a6-cd9003787b92 | +--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+ [stack@<director node> ~]$ heat resource-list overcloud -n5 |grep -vi complete WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+ | 0 | ab442d42-bf3c-428a-992b-ffdfd336aefd | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | | 1 | 7b30b509-f178-4111-92e0-ee56a223feb1 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | | 2 | cc63971c-2bec-4b9e-a0a7-144187af9810 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | | 3 | 7dcfb52d-ed65-4b73-b412-f210333a1474 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | | 4 | 707715f9-a5ef-408d-b744-652a70b8733d | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow | | ComputeHostsDeployment | 6b2cb14b-92e1-4745-bcb9-b078a13493b2 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud | | ControllerHostsDeployment| 7680285e-6884-45e2-9a53-735b1a2a987b | OS::Heat::StructuredDeployments | CREATE_FAILED | 2017-04-06T15:42:10Z | overcloud | | 0 | 738fc79b-0f13-4bfa-8869-b283d9bf8ca2 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we | | 1 | 8884f752-c2d6-4da9-a42f-07dd4b8d52da | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we | | 2 | cc2147a4-8ac0-49d2-8923-cf6e8e0d605d | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we | +--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+ [stack@<director node> ~]$ [stack@<director node> ~]$ heat resource-show 6b2cb14b-92e1-4745-bcb9-b078a13493b2 0 WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "deploy_stdout": null, | | | "deploy_stderr": null, | | | "deploy_status_code": null | | | } | | creation_time | 2017-04-06T15:42:10Z | | description | | | links | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ComputeHostsDeployment-27c7sp7lw7ow/6b2cb14b-92e1-4745-bcb9-b078a13493b2/resources/0 (self) | | | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ComputeHostsDeployment-27c7sp7lw7ow/6b2cb14b-92e1-4745-bcb9-b078a13493b2 (stack) | | logical_resource_id | 0 | | parent_resource | ComputeHostsDeployment | | physical_resource_id | ab442d42-bf3c-428a-992b-ffdfd336aefd | | required_by | | | resource_name | 0 | | resource_status | CREATE_FAILED | | resource_status_reason | CREATE aborted | | resource_type | OS::Heat::StructuredDeployment | | updated_time | 2017-04-06T15:42:10Z | +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@<director node> ~]$ heat resource-show 7680285e-6884-45e2-9a53-735b1a2a987b 0 WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "deploy_stdout": null, | | | "deploy_stderr": null, | | | "deploy_status_code": null | | | } | | creation_time | 2017-04-06T15:42:11Z | | description | | | links | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ControllerHostsDeployment-e2oo5f3bo4we/7680285e-6884-45e2-9a53-735b1a2a987b/resources/0 (self) | | | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ControllerHostsDeployment-e2oo5f3bo4we/7680285e-6884-45e2-9a53-735b1a2a987b (stack) | | logical_resource_id | 0 | | parent_resource | ControllerHostsDeployment | | physical_resource_id | 738fc79b-0f13-4bfa-8869-b283d9bf8ca2 | | required_by | | | resource_name | 0 | | resource_status | CREATE_FAILED | | resource_status_reason | CREATE aborted | | resource_type | OS::Heat::StructuredDeployment | | updated_time | 2017-04-06T15:42:11Z | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@<director node> ~]$ [stack@<director node> ~]$ sudo mysql -D heat -e "select * from stack where status!='COMPLETE'\G;" *************************** 1. row *************************** id: 5b093501-60aa-4b32-91a6-cd9003787b92 created_at: 2016-11-30 11:27:13 updated_at: 2017-04-18 14:05:16 deleted_at: NULL name: overcloud raw_template_id: 33412 user_creds_id: 2 username: admin owner_id: NULL action: UPDATE status: FAILED status_reason: resources.allNodesConfig.properties.enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]] timeout: 240 tenant: 60220f2d4a4540fd912d1fb59172e634 disable_rollback: 1 stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb backup: 0 nested_depth: 0 convergence: 0 prev_raw_template_id: 34007 current_traversal: NULL current_deps: null parent_resource_name: NULL *************************** 2. row *************************** id: 6b2cb14b-92e1-4745-bcb9-b078a13493b2 created_at: 2017-04-06 15:42:10 updated_at: NULL deleted_at: NULL name: overcloud-ComputeHostsDeployment-27c7sp7lw7ow raw_template_id: 33981 user_creds_id: 2 username: admin owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92 action: CREATE status: FAILED status_reason: Resource CREATE failed: Operation cancelled timeout: 14400 tenant: 60220f2d4a4540fd912d1fb59172e634 disable_rollback: 1 stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb backup: 0 nested_depth: 1 convergence: 0 prev_raw_template_id: NULL current_traversal: NULL current_deps: null parent_resource_name: ComputeHostsDeployment *************************** 3. row *************************** id: 7680285e-6884-45e2-9a53-735b1a2a987b created_at: 2017-04-06 15:42:11 updated_at: NULL deleted_at: NULL name: overcloud-ControllerHostsDeployment-e2oo5f3bo4we raw_template_id: 33985 user_creds_id: 2 username: admin owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92 action: CREATE status: FAILED status_reason: Resource CREATE failed: Operation cancelled timeout: 14400 tenant: 60220f2d4a4540fd912d1fb59172e634 disable_rollback: 1 stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb backup: 0 nested_depth: 1 convergence: 0 prev_raw_template_id: NULL current_traversal: NULL current_deps: null parent_resource_name: ControllerHostsDeployment *************************** 4. row *************************** id: 8b89f408-01dd-426c-8775-8e80a1f61fd6 created_at: 2017-04-06 15:37:25 updated_at: NULL deleted_at: NULL name: overcloud* raw_template_id: 33414 user_creds_id: 2 username: admin owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92 action: CREATE status: FAILED status_reason: timeout: 240 tenant: 60220f2d4a4540fd912d1fb59172e634 disable_rollback: 1 stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb backup: 1 nested_depth: 0 convergence: 0 prev_raw_template_id: NULL current_traversal: NULL current_deps: null parent_resource_name: NULL [stack@<director node> ~]$ [stack@<director node> ~]$ openstack stack failures list overcloud overcloud.ComputeHostsDeployment.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 7b30b509-f178-4111-92e0-ee56a223feb1 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment 9cf9432a-bad9-43fc-acc8-54b3ce963a59 completed deploy_stderr: | None overcloud.ComputeHostsDeployment.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: ab442d42-bf3c-428a-992b-ffdfd336aefd status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment f7f52fb8-704f-405f-b0c4-3079bb241c96 completed deploy_stderr: | None overcloud.ComputeHostsDeployment.3: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 7dcfb52d-ed65-4b73-b412-f210333a1474 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment a3d3f6b2-d472-49e2-9291-b29d2ea525bd completed deploy_stderr: | None overcloud.ComputeHostsDeployment.2: resource_type: OS::Heat::StructuredDeployment physical_resource_id: cc63971c-2bec-4b9e-a0a7-144187af9810 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment ac2b6c2b-ed6d-4a81-8da8-6a1102c853ef completed deploy_stderr: | None overcloud.ComputeHostsDeployment.4: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 707715f9-a5ef-408d-b744-652a70b8733d status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment 83b08742-7054-48c3-b6ae-ddcdbc651d09 completed deploy_stderr: | None overcloud.ControllerHostsDeployment.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 8884f752-c2d6-4da9-a42f-07dd4b8d52da status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment 8c39594a-b3e3-4bcd-8d30-0398fece696b completed deploy_stderr: | None overcloud.ControllerHostsDeployment.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 738fc79b-0f13-4bfa-8869-b283d9bf8ca2 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment 98c5208a-de68-4ccc-a243-01bbf0f0f871 completed deploy_stderr: | None overcloud.ControllerHostsDeployment.2: resource_type: OS::Heat::StructuredDeployment physical_resource_id: cc2147a4-8ac0-49d2-8923-cf6e8e0d605d status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | os-apply-config deployment e828e50d-58d9-4b2f-84e5-c94c5dc03654 completed deploy_stderr: | None [stack@<director node> ~]$
Closing the BZ now. For resolution: check out the attached Red Hat Knowledge Base (Solution) [to be updated].
Hi, The issue is fixed in heat with: https://review.openstack.org/458497 and cherry picked into ocata and newton: https://review.openstack.org/#/c/464585/ https://review.openstack.org/464584 And for OSP 10: https://code.engineering.redhat.com/gerrit/#/c/106170/ So this fix, once release into the next 10 puddle, should prevent from this failure in the future.
*** Bug 1466725 has been marked as a duplicate of this bug. ***
ran tempest on latest Z stream: {0} heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_with_new_version [19.477346s] ... ok {0} heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_with_old_version [15.206239s] ... ok
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2655