Bug 1397945

Summary: Cannot delete overcloud stack from undercloud because of 409 Conflict in overcloud swift container
Product: Red Hat OpenStack Reporter: Kevin Jones <kejones>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: aschultz, dbecker, ebarrera, kobi.ginon, mburns, morazi, rhel-osp-director-maint, therve
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-04 14:12:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
RHOSP10 templates for deployment none

Description Kevin Jones 2016-11-23 15:55:44 UTC 
Created attachment 1223328 [details]
RHOSP10 templates for deployment

Description of problem:
On a hung deployment of RHOSP10 on latest puddle, I attempted to delete the overcloud stack and cannot due to 409 conflict response from swift.

[stack@undercloud ~]$ heat stack-delete overcloud
WARNING (shell) "heat stack-delete" is deprecated, please use "openstack stack delete" instead
Are you sure you want to delete this stack(s) [y/N]? y
Request to delete stack overcloud has been accepted.
[stack@undercloud ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+----------------------+--------------+
| id                                   | stack_name | stack_status       | creation_time        | updated_time |
+--------------------------------------+------------+--------------------+----------------------+--------------+
| b762d5b6-fd0a-45de-b852-8ea885ed25ee | overcloud  | DELETE_IN_PROGRESS | 2016-11-23T22:25:35Z | None         |
+--------------------------------------+------------+--------------------+----------------------+--------------+
[stack@undercloud ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+----------------------+--------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| b762d5b6-fd0a-45de-b852-8ea885ed25ee | overcloud  | DELETE_FAILED | 2016-11-23T22:25:35Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+

2016-11-23 10:48:04.841 4041 INFO heat.engine.resource [req-f26cdd4e-35bf-459a-a9cd-a6ab70f7ded2 978dc87dca644f73a35cea8be3c964d7 5bac9e3719ab449294f55051dad935e7 - - -] DELETE: ServerUpdateAllowed "Controller" [4da114de-e14f-49a8-8bcd-f576912648c4] Stack "overcloud-Controller-g5gkj7yc222r-0-tii3a3fmtc4j" [e157347f-5efd-43f1-ae49-1b5fcda97dce]
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     yield
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1669, in delete
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     *action_args)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 336, in wrapper
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     step = next(subtask)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 800, in action_handler_task
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     handler_data = handler(*args)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1529, in handle_delete
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     return self._delete()
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1500, in _delete
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     self._delete_temp_url()
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1480, in _delete_temp_url
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     swift.delete_object(container, object_name)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1815, in delete_object
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     response_dict=response_dict)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     service_token=self.service_token, **kwargs)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1456, in delete_object
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource     raise ClientException.from_response(resp, 'Object DELETE failed', body)
2016-11-23 10:48:04.841 4041 ERROR heat.engine.resource ClientException: Object DELETE failed: http://192.168.2.10:8080/v1/AUTH_5bac9e3719ab449294f55051dad935e7/ov-5gkj7yc222r-0-tii3a3fmtc4j-Controller-2nmw7tfd4fqu/68067377-c5f9-4974-ac82-e693b879de4f 409 Conflict  [first 60 chars of response] <html><h1>Conflict</h1><p>There was a conflict when trying t


Version-Release number of selected component (if applicable):
RHOSP10 latest from rhos-release
RHEL 7.3 on undercloud

How reproducible:
Every issue of heat stack-delete overcloud

Steps to Reproduce:
1. Issue overcloud deployment
2. Hung on controller deployment step3 (with no obvious error)
3. Issue heat stack-delete overcloud or openstack stack delete overcloud

Actual results:
Overcloud stack deletion goes to delete_failed

Expected results:
Overcloud stack is fully deleted

Additional info:
Deploy command

[stack@undercloud ~]$ cat deploy.sh 
#!/usr/bin/env bash

cd ~
source ~/stackrc

openstack overcloud deploy --templates \
	-e /home/stack/templates/cloudname.yaml \
	-e /home/stack/templates/storage-environment.yaml \
	-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
	-e /home/stack/templates/network-environment.yaml \
	-e /home/stack/templates/firstboot-environment.yaml \
	--control-flavor control \
	--compute-flavor compute \
	--control-scale 1 \
	--compute-scale 4 \
	--ceph-storage-scale 0 \
	--ntp-server 192.168.0.3 \
	--neutron-tunnel-types vxlan \
	--neutron-network-type vxlan


templates attached
Comment 2 kobi ginon 2017-12-30 19:08:01 UTC 
Is there a solution for that ?
i'm suffering from similar issue
Comment 3 Thomas Hervé 2018-01-04 14:12:23 UTC 
This is an issue with NTP sync. It generally happens if the initial deployment was made and then the clock goes back. NTP needs to be setup properly on the undercloud.
Comment 4 kobi ginon 2018-01-04 14:15:21 UTC 
yes - well found it the hard way.
We made sure to force sync before and after undercloud deployment,
to fix this issue.
was 'bleeding' till found out the rrot cause

thx for the info