Bug 1379716
| Summary: | After failed openstack overcloud update, stack is stuck in "Resource update already requested " | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Andreas Karis <akaris> |
| Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> |
| Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 (Kilo) | CC: | mburns, rhel-osp-director-maint, zbitter |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-27 14:31:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1314080 *** *** This bug has been marked as a duplicate of bug 1314080 *** |
Description of problem: After failed openstack overcloud update, stack is stuck in "Resource update already requested " Need guidance or a remote session with engineering to fix this, please. Version-Release number of selected component (if applicable): OSP 7.3, latest packages How reproducible: openstack overcloud update command failed initially, here is the customer's description, unfortunately, we do not have much more details: +++++++++++++++++++++++++++++++ "It appears the update went well until the last node, the primary controller. After rebooting everything many of the pcs resources are stopped." +++++++++++++++++++++++++++++++ customer then changed the password for the overcloud because he thought they were wrong, eventually, he rolled back his password changes. but during that phase, he had the following error messages: ~~~ [stack@atl1ospcl1dir01 scripts]$ heat resource-list -n5 overcloud | grep -iv complete +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | parent_resource | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ | ControllerPostPuppet | 007fb4d0-bc6a-466b-b5a0-a6e2603d86c7 | OS::TripleO::Tasks::ControllerPostPuppet | UPDATE_FAILED | 2016-09-26T19:38:30Z | ControllerNodesPostDeployment | | ControllerPostPuppetRestartDeployment | 60725fb2-33fd-4ff3-943a-a2e0e5bff534 | OS::Heat::SoftwareDeployments | UPDATE_FAILED | 2016-09-26T19:39:31Z | ControllerPostPuppet | | 0 | 09829694-1f38-442e-8807-c8601a730fb3 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-09-26T19:39:32Z | ControllerPostPuppetRestartDeployment | | ComputeNodesPostDeployment | 71aa6481-7336-4cf0-ac0c-e02e3ecdcba9 | OS::TripleO::ComputePostDeployment | UPDATE_FAILED | 2016-09-26T20:48:04Z | | | ComputePuppetDeployment | 5c9dec67-d993-4956-8b71-300ee7a9868b | OS::Heat::StructuredDeployments | UPDATE_IN_PROGRESS | 2016-09-26T20:48:11Z | ComputeNodesPostDeployment | | 1 | b8b55e38-336c-41b9-8d94-20505d2f1c68 | OS::Heat::StructuredDeployment | UPDATE_IN_PROGRESS | 2016-09-26T20:48:12Z | ComputePuppetDeployment | | ControllerNodesPostDeployment | 8f8e9335-40f5-497c-aaa9-c706a8ae042f | OS::TripleO::ControllerPostDeployment | UPDATE_FAILED | 2016-09-26T20:48:12Z | | | 2 | bf489efb-ce93-4b93-8d6a-c2e8012ab0f9 | OS::Heat::StructuredDeployment | UPDATE_IN_PROGRESS | 2016-09-26T20:48:13Z | ComputePuppetDeployment | | 3 | b4f0e2e6-62f1-4da2-a73e-30b97fd2400f | OS::Heat::StructuredDeployment | UPDATE_IN_PROGRESS | 2016-09-26T20:48:13Z | ComputePuppetDeployment | | 0 | d70e3896-328d-4362-907e-24fb954e88e1 | OS::Heat::StructuredDeployment | UPDATE_IN_PROGRESS | 2016-09-26T20:48:14Z | ComputePuppetDeployment | | ControllerOvercloudServicesDeployment_Step6 | 0e49662b-aa06-4243-8598-dffbacb9ad33 | OS::Heat::StructuredDeployments | UPDATE_FAILED | 2016-09-26T20:54:00Z | ControllerNodesPostDeployment | | 0 | 94131e99-6e6d-4777-8387-980637f315bf | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-09-26T20:54:02Z | ControllerOvercloudServicesDeployment_Step6 | | Compute | 107cde1f-bdc0-45c6-bfb8-3f7518052777 | OS::Heat::ResourceGroup | UPDATE_IN_PROGRESS | 2016-09-26T21:26:14Z | | | 2 | 9a3f3c2a-b6a3-4d82-b504-fa6ff5f6cab4 | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2016-09-26T21:26:19Z | Compute | | Controller | 8ececb8f-2391-40cb-8d05-6f6c17d74f9e | OS::Heat::ResourceGroup | UPDATE_IN_PROGRESS | 2016-09-26T21:26:19Z | | | 2 | d805d967-76fb-40d3-b3a8-7ea7bee27d3c | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-26T21:26:24Z | Controller | | 3 | ec7c969b-ce08-429f-9842-5feac6b06d4d | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2016-09-26T21:26:28Z | Compute | | 0 | f2ed8f99-c845-488a-9236-0f22f6a8b953 | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-26T21:26:35Z | Controller | | 0 | 911acae5-b570-4813-8846-1151ab171af2 | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2016-09-26T21:26:40Z | Compute | | 1 | 5b04dc3d-cabb-4cf0-a716-4acd98588d74 | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-26T21:26:58Z | Controller | | 1 | bf22dd67-466b-494e-8f88-c797842561b4 | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2016-09-26T21:27:01Z | Compute | | UpdateDeployment | 991ffc43-d0de-40a3-82c2-03db1bc52a9c | OS::Heat::SoftwareDeployment | UPDATE_IN_PROGRESS | 2016-09-26T21:27:08Z | 2 | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ ~~~ +++++++++++++++++++++++++++++++ now, heat stack-update goes well through all of the breakpoints for the computes. on the controllers, it either goes through an endless loop (going back to the same controller breakpoint all of the time). Or, the latest result was a plain FAILED: ~~~ openstack overcloud update (...) (...) DEBUG: tripleo_common.stack_update overcloud status: FAILED FAILED update finished with status FAILED DEBUG: openstackclient.shell clean_up UpdateOvercloud ~~~ The reason seems to be `Resource update already requested`: ~~~ +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | parent_resource | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ | ControllerNodesPostDeployment | 8f8e9335-40f5-497c-aaa9-c706a8ae042f | OS::TripleO::ControllerPostDeployment | UPDATE_FAILED | 2016-09-27T13:23:36Z | | | ControllerPostPuppet | 007fb4d0-bc6a-466b-b5a0-a6e2603d86c7 | OS::TripleO::Tasks::ControllerPostPuppet | UPDATE_FAILED | 2016-09-27T13:33:23Z | ControllerNodesPostDeployment | | ControllerPostPuppetRestartDeployment | 60725fb2-33fd-4ff3-943a-a2e0e5bff534 | OS::Heat::SoftwareDeployments | UPDATE_FAILED | 2016-09-27T13:34:23Z | ControllerPostPuppet | | 0 | a504bf08-d0ca-4b12-af99-7831a6e26f91 | OS::Heat::SoftwareDeployment | CREATE_FAILED | 2016-09-27T13:34:25Z | ControllerPostPuppetRestartDeployment | | 1 | 5b04dc3d-cabb-4cf0-a716-4acd98588d74 | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-27T13:40:18Z | Controller | | 0 | f2ed8f99-c845-488a-9236-0f22f6a8b953 | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-27T13:40:43Z | Controller | | 2 | d805d967-76fb-40d3-b3a8-7ea7bee27d3c | OS::TripleO::Controller | UPDATE_IN_PROGRESS | 2016-09-27T13:41:00Z | Controller | | Compute | 107cde1f-bdc0-45c6-bfb8-3f7518052777 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-09-27T14:03:25Z | | | Controller | 8ececb8f-2391-40cb-8d05-6f6c17d74f9e | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-09-27T14:03:31Z | | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+-----------------------------------------------+ [stack@atl1ospcl1dir01 ~]$ heat resource-show overcloud Controller +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "attributes": null, | | | "refs": null | | | } | | description | | | links | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud/03389902-78de-497e-98f3-1357eb77dc70/resources/Controller (self) | | | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud/03389902-78de-497e-98f3-1357eb77dc70 (stack) | | | http://<heat-ip>:8004/v1/aa1488468f9740eda9dae6bdfda17070/stacks/overcloud-Controller-iwoc7uscaebb/8ececb8f-2391-40cb-8d05-6f6c17d74f9e (nested) | | logical_resource_id | Controller | | physical_resource_id | 8ececb8f-2391-40cb-8d05-6f6c17d74f9e | | required_by | ControllerBootstrapNodeDeployment | | | ControllerBootstrapNodeConfig | | | ControllerAllNodesDeployment | | | CephClusterConfig | | | AllNodesExtraConfig | | | ControllerClusterDeployment | | | VipDeployment | | | ControllerIpListMap | | | ControllerNodesPostDeployment | | | SwiftDevicesAndProxyConfig | | | ControllerAllNodesValidationDeployment | | | ControllerCephDeployment | | | ControllerClusterConfig | | | allNodesConfig | | | ControllerSwiftDeployment | | | AllNodesValidationConfig | | resource_name | Controller | | resource_status | UPDATE_FAILED | | resource_status_reason | resources.Controller: Exception: resources[2]: Resource update already requested | | resource_type | OS::Heat::ResourceGroup | | updated_time | 2016-09-27T14:03:31Z | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ ~~~