Bug 1379716 - After failed openstack overcloud update, stack is stuck in "Resource update already requested "
Summary: After failed openstack overcloud update, stack is stuck in "Resource update a...
Keywords:
Status: CLOSED DUPLICATE of bug 1314080
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: James Slagle
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-27 13:43 UTC by Andreas Karis
Modified: 2019-12-16 06:54 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-27 14:31:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Andreas Karis 2016-09-27 13:43:47 UTC
Description of problem:
After failed openstack overcloud update, stack is stuck in "Resource update already requested "

Need guidance or a remote session with engineering to fix this, please.

Version-Release number of selected component (if applicable):
OSP 7.3, latest packages

How reproducible:
openstack overcloud update command failed initially, here is the customer's description, unfortunately, we do not have much more details:

+++++++++++++++++++++++++++++++

"It appears the update went well until the last node, the primary controller.  After rebooting everything many of the pcs resources are stopped."

+++++++++++++++++++++++++++++++

customer then changed the password for the overcloud because he thought they were wrong, eventually, he rolled back his password changes. but during that phase, he had the following error messages:
~~~
[stack@atl1ospcl1dir01 scripts]$ heat resource-list -n5 overcloud | grep
-iv complete

+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+

| resource_name                                 | physical_resource_id
                    | resource_type                                     |
resource_status    | updated_time         | parent_resource
              |

+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+

| ControllerPostPuppet                          |
007fb4d0-bc6a-466b-b5a0-a6e2603d86c7          |
OS::TripleO::Tasks::ControllerPostPuppet          | UPDATE_FAILED      |
2016-09-26T19:38:30Z | ControllerNodesPostDeployment                 |

| ControllerPostPuppetRestartDeployment         |
60725fb2-33fd-4ff3-943a-a2e0e5bff534          |
OS::Heat::SoftwareDeployments                     | UPDATE_FAILED      |
2016-09-26T19:39:31Z | ControllerPostPuppet                          |

| 0                                             |
09829694-1f38-442e-8807-c8601a730fb3          |
OS::Heat::SoftwareDeployment                      | UPDATE_FAILED      |
2016-09-26T19:39:32Z | ControllerPostPuppetRestartDeployment         |

| ComputeNodesPostDeployment                    |
71aa6481-7336-4cf0-ac0c-e02e3ecdcba9          |
OS::TripleO::ComputePostDeployment                | UPDATE_FAILED      |
2016-09-26T20:48:04Z |                                               |

| ComputePuppetDeployment                       |
5c9dec67-d993-4956-8b71-300ee7a9868b          |
OS::Heat::StructuredDeployments                   | UPDATE_IN_PROGRESS |
2016-09-26T20:48:11Z | ComputeNodesPostDeployment                    |

| 1                                             |
b8b55e38-336c-41b9-8d94-20505d2f1c68          |
OS::Heat::StructuredDeployment                    | UPDATE_IN_PROGRESS |
2016-09-26T20:48:12Z | ComputePuppetDeployment                       |

| ControllerNodesPostDeployment                 |
8f8e9335-40f5-497c-aaa9-c706a8ae042f          |
OS::TripleO::ControllerPostDeployment             | UPDATE_FAILED      |
2016-09-26T20:48:12Z |                                               |

| 2                                             |
bf489efb-ce93-4b93-8d6a-c2e8012ab0f9          |
OS::Heat::StructuredDeployment                    | UPDATE_IN_PROGRESS |
2016-09-26T20:48:13Z | ComputePuppetDeployment                       |

| 3                                             |
b4f0e2e6-62f1-4da2-a73e-30b97fd2400f          |
OS::Heat::StructuredDeployment                    | UPDATE_IN_PROGRESS |
2016-09-26T20:48:13Z | ComputePuppetDeployment                       |

| 0                                             |
d70e3896-328d-4362-907e-24fb954e88e1          |
OS::Heat::StructuredDeployment                    | UPDATE_IN_PROGRESS |
2016-09-26T20:48:14Z | ComputePuppetDeployment                       |

| ControllerOvercloudServicesDeployment_Step6   |
0e49662b-aa06-4243-8598-dffbacb9ad33          |
OS::Heat::StructuredDeployments                   | UPDATE_FAILED      |
2016-09-26T20:54:00Z | ControllerNodesPostDeployment                 |

| 0                                             |
94131e99-6e6d-4777-8387-980637f315bf          |
OS::Heat::StructuredDeployment                    | UPDATE_FAILED      |
2016-09-26T20:54:02Z | ControllerOvercloudServicesDeployment_Step6   |

| Compute                                       |
107cde1f-bdc0-45c6-bfb8-3f7518052777          | OS::Heat::ResourceGroup
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:14Z |
                                      |

| 2                                             |
9a3f3c2a-b6a3-4d82-b504-fa6ff5f6cab4          | OS::TripleO::Compute
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:19Z | Compute
                                      |

| Controller                                    |
8ececb8f-2391-40cb-8d05-6f6c17d74f9e          | OS::Heat::ResourceGroup
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:19Z |
                                      |

| 2                                             |
d805d967-76fb-40d3-b3a8-7ea7bee27d3c          | OS::TripleO::Controller
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:24Z |
Controller                                    |

| 3                                             |
ec7c969b-ce08-429f-9842-5feac6b06d4d          | OS::TripleO::Compute
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:28Z | Compute
                                      |

| 0                                             |
f2ed8f99-c845-488a-9236-0f22f6a8b953          | OS::TripleO::Controller
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:35Z |
Controller                                    |

| 0                                             |
911acae5-b570-4813-8846-1151ab171af2          | OS::TripleO::Compute
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:40Z | Compute
                                      |

| 1                                             |
5b04dc3d-cabb-4cf0-a716-4acd98588d74          | OS::TripleO::Controller
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:26:58Z |
Controller                                    |

| 1                                             |
bf22dd67-466b-494e-8f88-c797842561b4          | OS::TripleO::Compute
                      | UPDATE_IN_PROGRESS | 2016-09-26T21:27:01Z | Compute
                                      |

| UpdateDeployment                              |
991ffc43-d0de-40a3-82c2-03db1bc52a9c          |
OS::Heat::SoftwareDeployment                      | UPDATE_IN_PROGRESS |
2016-09-26T21:27:08Z | 2                                             |

+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+
~~~

+++++++++++++++++++++++++++++++

now, heat stack-update goes well through all of the breakpoints for the computes. on the controllers, it either goes through an endless loop (going back to the same controller breakpoint all of the time). Or, the latest result was a plain FAILED:

~~~
openstack overcloud update (...)
(...)
DEBUG: tripleo_common.stack_update overcloud status: FAILED
FAILED
update finished with status FAILED
DEBUG: openstackclient.shell clean_up UpdateOvercloud
~~~

The reason seems to be `Resource update already requested`:
~~~
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+
| resource_name                                 | physical_resource_id                          | resource_type                                     | resource_status    | updated_time         | parent_resource                               |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+
| ControllerNodesPostDeployment                 | 8f8e9335-40f5-497c-aaa9-c706a8ae042f          | OS::TripleO::ControllerPostDeployment             | UPDATE_FAILED      | 2016-09-27T13:23:36Z |                                               |
| ControllerPostPuppet                          | 007fb4d0-bc6a-466b-b5a0-a6e2603d86c7          | OS::TripleO::Tasks::ControllerPostPuppet          | UPDATE_FAILED      | 2016-09-27T13:33:23Z | ControllerNodesPostDeployment                 |
| ControllerPostPuppetRestartDeployment         | 60725fb2-33fd-4ff3-943a-a2e0e5bff534          | OS::Heat::SoftwareDeployments                     | UPDATE_FAILED      | 2016-09-27T13:34:23Z | ControllerPostPuppet                          |
| 0                                             | a504bf08-d0ca-4b12-af99-7831a6e26f91          | OS::Heat::SoftwareDeployment                      | CREATE_FAILED      | 2016-09-27T13:34:25Z | ControllerPostPuppetRestartDeployment         |
| 1                                             | 5b04dc3d-cabb-4cf0-a716-4acd98588d74          | OS::TripleO::Controller                           | UPDATE_IN_PROGRESS | 2016-09-27T13:40:18Z | Controller                                    |
| 0                                             | f2ed8f99-c845-488a-9236-0f22f6a8b953          | OS::TripleO::Controller                           | UPDATE_IN_PROGRESS | 2016-09-27T13:40:43Z | Controller                                    |
| 2                                             | d805d967-76fb-40d3-b3a8-7ea7bee27d3c          | OS::TripleO::Controller                           | UPDATE_IN_PROGRESS | 2016-09-27T13:41:00Z | Controller                                    |
| Compute                                       | 107cde1f-bdc0-45c6-bfb8-3f7518052777          | OS::Heat::ResourceGroup                           | UPDATE_FAILED      | 2016-09-27T14:03:25Z |                                               |
| Controller                                    | 8ececb8f-2391-40cb-8d05-6f6c17d74f9e          | OS::Heat::ResourceGroup                           | UPDATE_FAILED      | 2016-09-27T14:03:31Z |                                               |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+

[stack@atl1ospcl1dir01 ~]$ heat resource-show overcloud Controller
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                    |
|                        |   "attributes": null,                                                                                                                                |
|                        |   "refs": null                                                                                                                                       |
|                        | }                                                                                                                                                    |
| description            |                                                                                                                                                      |
| links                  | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud/03389902-78de-497e-98f3-1357eb77dc70/resources/Controller (self)      |
|                        | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud/03389902-78de-497e-98f3-1357eb77dc70 (stack)                          |
|                        | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud-Controller-iwoc7uscaebb/8ececb8f-2391-40cb-8d05-6f6c17d74f9e (nested) |
| logical_resource_id    | Controller                                                                                                                                           |
| physical_resource_id   | 8ececb8f-2391-40cb-8d05-6f6c17d74f9e                                                                                                                 |
| required_by            | ControllerBootstrapNodeDeployment                                                                                                                    |
|                        | ControllerBootstrapNodeConfig                                                                                                                        |
|                        | ControllerAllNodesDeployment                                                                                                                         |
|                        | CephClusterConfig                                                                                                                                    |
|                        | AllNodesExtraConfig                                                                                                                                  |
|                        | ControllerClusterDeployment                                                                                                                          |
|                        | VipDeployment                                                                                                                                        |
|                        | ControllerIpListMap                                                                                                                                  |
|                        | ControllerNodesPostDeployment                                                                                                                        |
|                        | SwiftDevicesAndProxyConfig                                                                                                                           |
|                        | ControllerAllNodesValidationDeployment                                                                                                               |
|                        | ControllerCephDeployment                                                                                                                             |
|                        | ControllerClusterConfig                                                                                                                              |
|                        | allNodesConfig                                                                                                                                       |
|                        | ControllerSwiftDeployment                                                                                                                            |
|                        | AllNodesValidationConfig                                                                                                                             |
| resource_name          | Controller                                                                                                                                           |
| resource_status        | UPDATE_FAILED                                                                                                                                        |
| resource_status_reason | resources.Controller: Exception: resources[2]: Resource update already requested                                                                     |
| resource_type          | OS::Heat::ResourceGroup                                                                                                                              |
| updated_time           | 2016-09-27T14:03:31Z                                                                                                                                 |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
~~~

Comment 3 Zane Bitter 2016-09-27 14:02:47 UTC

*** This bug has been marked as a duplicate of bug 1314080 ***

Comment 5 Andreas Karis 2016-09-27 14:31:16 UTC

*** This bug has been marked as a duplicate of bug 1314080 ***


Note You need to log in before you can comment on or make changes to this bug.