Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1290585

Summary:	rhel-osp-director: the stack status is UPDATE_FAILED after attempting to remove a compute from the overcloud deployment
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	rhosp-director	Assignee:	Marios Andreou <mandreou>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	jcoufal, mandreou, mburns, rhel-osp-director-maint
Target Milestone:	y2	Keywords:	Regression, TestOnly
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-21 16:54:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2015-12-10 21:14:02 UTC

rhel-osp-director:  the stack status is UPDATE_FAILED after attempting to remove a compute from the overcloud deployment.

Environment:
instack-undercloud-2.1.2-34.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-87.el7ost.noarch

Steps to reproduce:
1. Deploy HA overcloud with 2 computes using rhel-osp-direcror 7.1
2. Update the setup to 7.2
3. Try to remove one node using this procedure:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Removing_Nodes_from_the_Overcloud.html


Result:
The node is being deleted right away and the prompt is returned as if the operation was successful.
You can learn about the failed deletion only by using heat stack-list.
There are puppet errors on the controller:
Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 CRITICAL cinder [req-f1cd1fd2-7089-4463-b9f8-c94862b7b09d - - - - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")

Comment 2 Marios Andreou 2015-12-11 15:10:57 UTC

i am getting setup to dig into this sasha (7.1 env with 2 computes and updating to 7.2)... any more logs you have would be useful here please. in particular /var/log/messages from the compute that fails and the controllers please. I  don't think the errors in the description are relevant yet.

Comment 4 Alexander Chuzhoy 2015-12-16 21:53:31 UTC

Verified:

Environment:
instack-undercloud-2.1.2-36.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch



Works fine.

Deployed 7.1
Updated to 7.2
Removed one compute node according to the doc in comment #0.

heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 3c0ea183-d43a-4125-8764-b23a51784215 | overcloud  | UPDATE_COMPLETE | 2015-12-16T16:22:26Z |

Comment 5 Marios Andreou 2015-12-17 07:10:49 UTC

thanks Sasha, I guess that verifies this shares the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1290572 (same fix)

Comment 8 errata-xmlrpc 2015-12-21 16:54:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651