Bug 1280011 - Overcloud update fails due to nova-consoleauth service restart failure
Summary: Overcloud update fails due to nova-consoleauth service restart failure
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: y2
: 7.0 (Kilo)
Assignee: Jiri Stransky
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-10 17:41 UTC by Jiri Stransky
Modified: 2015-12-21 16:58 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Note: the doc text is the same as for BZ 1275986, i don't think we want to release this text twice, so i'm setting requires_doc_text-. Cause: Pacemaker prevented Puppet from restarting services during overcloud update. Consequence: Puppet run failed, which caused overcloud update to fail. Fix: Pacemaker is put into maintenance mode for the duration of Puppet run during the overcloud update. Result: Puppet is allowed to restart services, and it doesn't cause the overcloud update to fail anymore.
Clone Of:
Environment:
Last Closed: 2015-12-21 16:58:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2651 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory 2015-12-21 21:50:26 UTC

Description Jiri Stransky 2015-11-10 17:41:51 UTC
This was hit together with bug 1275986 and the cause and the fix could be similar or the same for these two bugs, but we're not sure of that yet, so we agreed to report this separately for now. We hit this issue on two environments.

os-collect-config journal:

lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Failed to call refresh: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.

nova-consoleauth journal:

[root@overcloud-controller-0 ~]# journalctl -u openstack-nova-consoleauth
-- Logs begin at Út 2015-11-10 07:17:50 EST, end at Út 2015-11-10 12:11:16 EST. --
lis 10 07:31:02 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 07:31:09 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:09:50 overcloud-controller-0.localdomain systemd[1]: Stopping OpenStack Nova VNC console auth Server...
lis 10 09:09:51 overcloud-controller-0.localdomain systemd[1]: Stopped OpenStack Nova VNC console auth Server.
lis 10 09:24:36 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:24:38 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:52 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:53 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopped Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:57 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:59 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.

Comment 1 James Slagle 2015-11-16 20:18:13 UTC
jistr, assigning this one to you as I think it will be fixed under the larger umbrella of fixing pacemaker/puppet service restart issues

Comment 3 Marius Cornea 2015-12-15 15:00:37 UTC
I did several updates for both 7.0 -> 7.2 and 7.1 -> 7.2 and I didn't hit the
issue reported here so I will move this to verified.

Comment 4 Gabriel Szasz 2015-12-15 15:05:30 UTC
This bug is almost same issue as bug 1275986. The use case has been already covered by Marius (see Comment 3).

Comment 6 errata-xmlrpc 2015-12-21 16:58:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651


Note You need to log in before you can comment on or make changes to this bug.