Bug 1290585 - rhel-osp-director: the stack status is UPDATE_FAILED after attempting to remove a compute from the overcloud deployment
rhel-osp-director: the stack status is UPDATE_FAILED after attempting to rem...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: y2
: 7.0 (Kilo)
Assigned To: Marios Andreou
Alexander Chuzhoy
: Regression, TestOnly
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-10 16:14 EST by Alexander Chuzhoy
Modified: 2015-12-21 11:54 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-21 11:54:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Chuzhoy 2015-12-10 16:14:02 EST
rhel-osp-director:  the stack status is UPDATE_FAILED after attempting to remove a compute from the overcloud deployment.

Environment:
instack-undercloud-2.1.2-34.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-87.el7ost.noarch

Steps to reproduce:
1. Deploy HA overcloud with 2 computes using rhel-osp-direcror 7.1
2. Update the setup to 7.2
3. Try to remove one node using this procedure:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Removing_Nodes_from_the_Overcloud.html


Result:
The node is being deleted right away and the prompt is returned as if the operation was successful.
You can learn about the failed deletion only by using heat stack-list.
There are puppet errors on the controller:
Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 CRITICAL cinder [req-f1cd1fd2-7089-4463-b9f8-c94862b7b09d - - - - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
Comment 2 Marios Andreou 2015-12-11 10:10:57 EST
i am getting setup to dig into this sasha (7.1 env with 2 computes and updating to 7.2)... any more logs you have would be useful here please. in particular /var/log/messages from the compute that fails and the controllers please. I  don't think the errors in the description are relevant yet.
Comment 4 Alexander Chuzhoy 2015-12-16 16:53:31 EST
Verified:

Environment:
instack-undercloud-2.1.2-36.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch



Works fine.

Deployed 7.1
Updated to 7.2
Removed one compute node according to the doc in comment #0.

heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 3c0ea183-d43a-4125-8764-b23a51784215 | overcloud  | UPDATE_COMPLETE | 2015-12-16T16:22:26Z |
Comment 5 Marios Andreou 2015-12-17 02:10:49 EST
thanks Sasha, I guess that verifies this shares the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1290572 (same fix)
Comment 8 errata-xmlrpc 2015-12-21 11:54:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651

Note You need to log in before you can comment on or make changes to this bug.