rhel-osp-director: the stack status is UPDATE_FAILED after attempting to remove a compute from the overcloud deployment. Environment: instack-undercloud-2.1.2-34.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-87.el7ost.noarch Steps to reproduce: 1. Deploy HA overcloud with 2 computes using rhel-osp-direcror 7.1 2. Update the setup to 7.2 3. Try to remove one node using this procedure: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Removing_Nodes_from_the_Overcloud.html Result: The node is being deleted right away and the prompt is returned as if the operation was successful. You can learn about the failed deletion only by using heat stack-list. There are puppet errors on the controller: Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:40 overcloud-controller-0 cinder-api: 2015-12-10 16:06:40.864 48878 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 CRITICAL cinder [req-f1cd1fd2-7089-4463-b9f8-c94862b7b09d - - - - -] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:42 overcloud-controller-0 cinder-scheduler: 2015-12-10 16:06:42.799 49003 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:43 overcloud-controller-0 heat-engine: DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 CRITICAL cinder [-] DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)") Dec 10 16:06:45 overcloud-controller-0 cinder-volume: 2015-12-10 16:06:45.327 49092 TRACE cinder DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '10.19.94.17' (111)")
i am getting setup to dig into this sasha (7.1 env with 2 computes and updating to 7.2)... any more logs you have would be useful here please. in particular /var/log/messages from the compute that fails and the controllers please. I don't think the errors in the description are relevant yet.
Verified: Environment: instack-undercloud-2.1.2-36.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch Works fine. Deployed 7.1 Updated to 7.2 Removed one compute node according to the doc in comment #0. heat stack-list +--------------------------------------+------------+-----------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+-----------------+----------------------+ | 3c0ea183-d43a-4125-8764-b23a51784215 | overcloud | UPDATE_COMPLETE | 2015-12-16T16:22:26Z |
thanks Sasha, I guess that verifies this shares the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1290572 (same fix)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2651