Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1443186

Summary: Overcloud Upgrade [ RHOS-9 to RHOS-10 ] fails with no error
Product: Red Hat OpenStack Reporter: Irina Petrova <ipetrova>
Component: openstack-heatAssignee: mathieu bultel <mbultel>
Status: CLOSED ERRATA QA Contact: Ronnie Rasouli <rrasouli>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: augol, ccamacho, mbultel, mburns, mcornea, morazi, mschuppe, nlevinki, pablo.iranzo, rhel-osp-director-maint, samccann, sathlang, sbaker, shardy, srevivo, therve, yroblamo, zbitter
Target Milestone: z4Keywords: Reopened, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-heat-7.0.3-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1450965 (view as bug list) Environment:
Last Closed: 2017-09-06 17:13:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1450965    

Description Irina Petrova 2017-04-18 17:47:39 UTC 
Description of problem:

The Director node is already upgraded to RHOS-10. The Overcloud nodes are running the latest RHOS-9 packages.

The Overcloud upgrade fails with no error on:
~~~
* major-upgrade-ceilometer-wsgi-mitaka-newton.yaml - Converts OpenStack Telemetry (‘Ceilometer’) to a WSGI service. [1]
~~~
[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/upgrading_red_hat_openstack_platform/chap-upgrading_the_environment#sect-Major-Upgrading_the_Overcloud


[stack@<director node> ~]$ time openstack overcloud deploy --templates < custom templates >
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: 7a017fe6-0c1b-474d-8829-de1fc76fe959
Plan updated
Deploying templates in the directory /tmp/tripleoclient-CwR2nC/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 4df350bf-27b2-4d20-8f32-da5006ef82f0

 Stack overcloud UPDATE_FAILED

Heat Stack update failed.

real    1m34.169s
user    0m3.046s
sys     0m0.606s
[stack@<director node> ~]$


[stack@<director node> ~]$ heat stack-list --show-nested |grep -vi complete
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+
| id                                   | stack_name                                                      | stack_status    | creation_time        | updated_time         | parent                               |
+--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+
| 5b093501-60aa-4b32-91a6-cd9003787b92 | overcloud                                                       | UPDATE_FAILED   | 2016-11-30T11:27:13Z | 2017-04-18T14:05:16Z | None                                 |
| 6b2cb14b-92e1-4745-bcb9-b078a13493b2 | overcloud-ComputeHostsDeployment-27c7sp7lw7ow                   | CREATE_FAILED   | 2017-04-06T15:42:10Z | None                 | 5b093501-60aa-4b32-91a6-cd9003787b92 |
| 7680285e-6884-45e2-9a53-735b1a2a987b | overcloud-ControllerHostsDeployment-e2oo5f3bo4we                | CREATE_FAILED   | 2017-04-06T15:42:11Z | None                 | 5b093501-60aa-4b32-91a6-cd9003787b92 |
+--------------------------------------+----------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+

[stack@<director node> ~]$ heat resource-list overcloud -n5 |grep -vi complete
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+
| resource_name            | physical_resource_id                  | resource_type                     | resource_status | updated_time         | stack_name                                       |
+--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+
| 0                        | ab442d42-bf3c-428a-992b-ffdfd336aefd  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow    |
| 1                        | 7b30b509-f178-4111-92e0-ee56a223feb1  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow    |
| 2                        | cc63971c-2bec-4b9e-a0a7-144187af9810  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow    |
| 3                        | 7dcfb52d-ed65-4b73-b412-f210333a1474  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow    |
| 4                        | 707715f9-a5ef-408d-b744-652a70b8733d  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud-ComputeHostsDeployment-27c7sp7lw7ow    |
| ComputeHostsDeployment   | 6b2cb14b-92e1-4745-bcb9-b078a13493b2  | OS::Heat::StructuredDeployments   | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud                                        |
| ControllerHostsDeployment| 7680285e-6884-45e2-9a53-735b1a2a987b  | OS::Heat::StructuredDeployments   | CREATE_FAILED   | 2017-04-06T15:42:10Z | overcloud                                        |
| 0                        | 738fc79b-0f13-4bfa-8869-b283d9bf8ca2  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we |
| 1                        | 8884f752-c2d6-4da9-a42f-07dd4b8d52da  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we |
| 2                        | cc2147a4-8ac0-49d2-8923-cf6e8e0d605d  | OS::Heat::StructuredDeployment    | CREATE_FAILED   | 2017-04-06T15:42:11Z | overcloud-ControllerHostsDeployment-e2oo5f3bo4we |
+--------------------------+---------------------------------------+-----------------------------------+-----------------+----------------------+--------------------------------------------------+
[stack@<director node> ~]$


[stack@<director node> ~]$ heat resource-show  6b2cb14b-92e1-4745-bcb9-b078a13493b2 0
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                     |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                         |
|                        |   "deploy_stdout": null,                                                                                                                                                  |
|                        |   "deploy_stderr": null,                                                                                                                                                  |
|                        |   "deploy_status_code": null                                                                                                                                              |
|                        | }                                                                                                                                                                         |
| creation_time          | 2017-04-06T15:42:10Z                                                                                                                                                      |
| description            |                                                                                                                                                                           |
| links                  | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ComputeHostsDeployment-27c7sp7lw7ow/6b2cb14b-92e1-4745-bcb9-b078a13493b2/resources/0 (self) |
|                        | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ComputeHostsDeployment-27c7sp7lw7ow/6b2cb14b-92e1-4745-bcb9-b078a13493b2 (stack)            |
| logical_resource_id    | 0                                                                                                                                                                         |
| parent_resource        | ComputeHostsDeployment                                                                                                                                                    |
| physical_resource_id   | ab442d42-bf3c-428a-992b-ffdfd336aefd                                                                                                                                      |
| required_by            |                                                                                                                                                                           |
| resource_name          | 0                                                                                                                                                                         |
| resource_status        | CREATE_FAILED                                                                                                                                                             |
| resource_status_reason | CREATE aborted                                                                                                                                                            |
| resource_type          | OS::Heat::StructuredDeployment                                                                                                                                            |
| updated_time           | 2017-04-06T15:42:10Z                                                                                                                                                      |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@<director node> ~]$ heat resource-show  7680285e-6884-45e2-9a53-735b1a2a987b 0
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                        |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                            |
|                        |   "deploy_stdout": null,                                                                                                                                                     |
|                        |   "deploy_stderr": null,                                                                                                                                                     |
|                        |   "deploy_status_code": null                                                                                                                                                 |
|                        | }                                                                                                                                                                            |
| creation_time          | 2017-04-06T15:42:11Z                                                                                                                                                         |
| description            |                                                                                                                                                                              |
| links                  | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ControllerHostsDeployment-e2oo5f3bo4we/7680285e-6884-45e2-9a53-735b1a2a987b/resources/0 (self) |
|                        | http://172.16.20.21:8004/v1/60220f2d4a4540fd912d1fb59172e634/stacks/overcloud-ControllerHostsDeployment-e2oo5f3bo4we/7680285e-6884-45e2-9a53-735b1a2a987b (stack)            |
| logical_resource_id    | 0                                                                                                                                                                            |
| parent_resource        | ControllerHostsDeployment                                                                                                                                                    |
| physical_resource_id   | 738fc79b-0f13-4bfa-8869-b283d9bf8ca2                                                                                                                                         |
| required_by            |                                                                                                                                                                              |
| resource_name          | 0                                                                                                                                                                            |
| resource_status        | CREATE_FAILED                                                                                                                                                                |
| resource_status_reason | CREATE aborted                                                                                                                                                               |
| resource_type          | OS::Heat::StructuredDeployment                                                                                                                                               |
| updated_time           | 2017-04-06T15:42:11Z                                                                                                                                                         |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@<director node> ~]$


[stack@<director node> ~]$ sudo mysql -D heat -e "select * from stack where status!='COMPLETE'\G;"
*************************** 1. row ***************************
                   id: 5b093501-60aa-4b32-91a6-cd9003787b92
           created_at: 2016-11-30 11:27:13
           updated_at: 2017-04-18 14:05:16
           deleted_at: NULL
                 name: overcloud
      raw_template_id: 33412
        user_creds_id: 2
             username: admin
             owner_id: NULL
               action: UPDATE
               status: FAILED
        status_reason: resources.allNodesConfig.properties.enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]
              timeout: 240
               tenant: 60220f2d4a4540fd912d1fb59172e634
     disable_rollback: 1
stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb
               backup: 0
         nested_depth: 0
          convergence: 0
 prev_raw_template_id: 34007
    current_traversal: NULL
         current_deps: null
 parent_resource_name: NULL
*************************** 2. row ***************************
                   id: 6b2cb14b-92e1-4745-bcb9-b078a13493b2
           created_at: 2017-04-06 15:42:10
           updated_at: NULL
           deleted_at: NULL
                 name: overcloud-ComputeHostsDeployment-27c7sp7lw7ow
      raw_template_id: 33981
        user_creds_id: 2
             username: admin
             owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92
               action: CREATE
               status: FAILED
        status_reason: Resource CREATE failed: Operation cancelled
              timeout: 14400
               tenant: 60220f2d4a4540fd912d1fb59172e634
     disable_rollback: 1
stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb
               backup: 0
         nested_depth: 1
          convergence: 0
 prev_raw_template_id: NULL
    current_traversal: NULL
         current_deps: null
 parent_resource_name: ComputeHostsDeployment
*************************** 3. row ***************************
                   id: 7680285e-6884-45e2-9a53-735b1a2a987b
           created_at: 2017-04-06 15:42:11
           updated_at: NULL
           deleted_at: NULL
                 name: overcloud-ControllerHostsDeployment-e2oo5f3bo4we
      raw_template_id: 33985
        user_creds_id: 2
             username: admin
             owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92
               action: CREATE
               status: FAILED
        status_reason: Resource CREATE failed: Operation cancelled
              timeout: 14400
               tenant: 60220f2d4a4540fd912d1fb59172e634
     disable_rollback: 1
stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb
               backup: 0
         nested_depth: 1
          convergence: 0
 prev_raw_template_id: NULL
    current_traversal: NULL
         current_deps: null
 parent_resource_name: ControllerHostsDeployment
*************************** 4. row ***************************
                   id: 8b89f408-01dd-426c-8775-8e80a1f61fd6
           created_at: 2017-04-06 15:37:25
           updated_at: NULL
           deleted_at: NULL
                 name: overcloud*
      raw_template_id: 33414
        user_creds_id: 2
             username: admin
             owner_id: 5b093501-60aa-4b32-91a6-cd9003787b92
               action: CREATE
               status: FAILED
        status_reason:
              timeout: 240
               tenant: 60220f2d4a4540fd912d1fb59172e634
     disable_rollback: 1
stack_user_project_id: bbf5bc43542344c392e2c329da6ce2bb
               backup: 1
         nested_depth: 0
          convergence: 0
 prev_raw_template_id: NULL
    current_traversal: NULL
         current_deps: null
 parent_resource_name: NULL
[stack@<director node> ~]$


[stack@<director node> ~]$ openstack stack failures list overcloud
overcloud.ComputeHostsDeployment.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 7b30b509-f178-4111-92e0-ee56a223feb1
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment 9cf9432a-bad9-43fc-acc8-54b3ce963a59 completed
  deploy_stderr: |
None
overcloud.ComputeHostsDeployment.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: ab442d42-bf3c-428a-992b-ffdfd336aefd
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment f7f52fb8-704f-405f-b0c4-3079bb241c96 completed
  deploy_stderr: |
None
overcloud.ComputeHostsDeployment.3:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 7dcfb52d-ed65-4b73-b412-f210333a1474
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment a3d3f6b2-d472-49e2-9291-b29d2ea525bd completed
  deploy_stderr: |
None
overcloud.ComputeHostsDeployment.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cc63971c-2bec-4b9e-a0a7-144187af9810
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment ac2b6c2b-ed6d-4a81-8da8-6a1102c853ef completed
  deploy_stderr: |
None
overcloud.ComputeHostsDeployment.4:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 707715f9-a5ef-408d-b744-652a70b8733d
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment 83b08742-7054-48c3-b6ae-ddcdbc651d09 completed
  deploy_stderr: |
None
overcloud.ControllerHostsDeployment.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 8884f752-c2d6-4da9-a42f-07dd4b8d52da
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment 8c39594a-b3e3-4bcd-8d30-0398fece696b completed
  deploy_stderr: |
None
overcloud.ControllerHostsDeployment.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 738fc79b-0f13-4bfa-8869-b283d9bf8ca2
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment 98c5208a-de68-4ccc-a243-01bbf0f0f871 completed
  deploy_stderr: |
None
overcloud.ControllerHostsDeployment.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cc2147a4-8ac0-49d2-8923-cf6e8e0d605d
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
    os-apply-config deployment e828e50d-58d9-4b2f-84e5-c94c5dc03654 completed
  deploy_stderr: |
None
[stack@<director node> ~]$
Comment 15 Irina Petrova 2017-05-02 08:54:51 UTC 
Closing the BZ now. 

For resolution: check out the attached Red Hat Knowledge Base (Solution) [to be updated].
Comment 17 mathieu bultel 2017-05-15 09:41:45 UTC 
Hi,

The issue is fixed in heat with:
https://review.openstack.org/458497
and cherry picked into ocata and newton:
https://review.openstack.org/#/c/464585/
https://review.openstack.org/464584

And for OSP 10:
https://code.engineering.redhat.com/gerrit/#/c/106170/

So this fix, once release into the next 10 puddle, should prevent from this failure in the future.
Comment 18 Sofer Athlan-Guyot 2017-06-30 12:05:08 UTC 
*** Bug 1466725 has been marked as a duplicate of this bug. ***
Comment 22 Ronnie Rasouli 2017-09-03 12:41:28 UTC 
ran tempest on latest Z stream:
{0} heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_with_new_version [19.477346s] ... ok
{0} heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_with_old_version [15.206239s] ... ok
Comment 24 errata-xmlrpc 2017-09-06 17:13:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2655