Bug 1280011

Summary:	Overcloud update fails due to nova-consoleauth service restart failure
Product:	Red Hat OpenStack	Reporter:	Jiri Stransky <jstransk>
Component:	rhosp-director	Assignee:	Jiri Stransky <jstransk>
Status:	CLOSED ERRATA	QA Contact:	Gabriel Szasz <gszasz>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	calfonso, jslagle, kbasil, mburns, mcornea, ohochman, rhel-osp-director-maint
Target Milestone:	y2	Keywords:	TestOnly, Triaged
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Note: the doc text is the same as for BZ 1275986, i don't think we want to release this text twice, so i'm setting requires_doc_text-. Cause: Pacemaker prevented Puppet from restarting services during overcloud update. Consequence: Puppet run failed, which caused overcloud update to fail. Fix: Pacemaker is put into maintenance mode for the duration of Puppet run during the overcloud update. Result: Puppet is allowed to restart services, and it doesn't cause the overcloud update to fail anymore.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-21 16:58:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jiri Stransky 2015-11-10 17:41:51 UTC

This was hit together with bug 1275986 and the cause and the fix could be similar or the same for these two bugs, but we're not sure of that yet, so we agreed to report this separately for now. We hit this issue on two environments.

os-collect-config journal:

lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Failed to call refresh: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.

nova-consoleauth journal:

[root@overcloud-controller-0 ~]# journalctl -u openstack-nova-consoleauth
-- Logs begin at Út 2015-11-10 07:17:50 EST, end at Út 2015-11-10 12:11:16 EST. --
lis 10 07:31:02 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 07:31:09 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:09:50 overcloud-controller-0.localdomain systemd[1]: Stopping OpenStack Nova VNC console auth Server...
lis 10 09:09:51 overcloud-controller-0.localdomain systemd[1]: Stopped OpenStack Nova VNC console auth Server.
lis 10 09:24:36 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:24:38 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:52 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:53 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopped Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:57 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:59 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.

Comment 1 James Slagle 2015-11-16 20:18:13 UTC

jistr, assigning this one to you as I think it will be fixed under the larger umbrella of fixing pacemaker/puppet service restart issues

Comment 3 Marius Cornea 2015-12-15 15:00:37 UTC

I did several updates for both 7.0 -> 7.2 and 7.1 -> 7.2 and I didn't hit the
issue reported here so I will move this to verified.

Comment 4 Gabriel Szasz 2015-12-15 15:05:30 UTC

This bug is almost same issue as bug 1275986. The use case has been already covered by Marius (see Comment 3).

Comment 6 errata-xmlrpc 2015-12-21 16:58:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651