Bug 1280011 - Overcloud update fails due to nova-consoleauth service restart failure
Overcloud update fails due to nova-consoleauth service restart failure
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: y2
: 7.0 (Kilo)
Assigned To: Jiri Stransky
Gabriel Szasz
: TestOnly, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-10 12:41 EST by Jiri Stransky
Modified: 2015-12-21 11:58 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Note: the doc text is the same as for BZ 1275986, i don't think we want to release this text twice, so i'm setting requires_doc_text-. Cause: Pacemaker prevented Puppet from restarting services during overcloud update. Consequence: Puppet run failed, which caused overcloud update to fail. Fix: Pacemaker is put into maintenance mode for the duration of Puppet run during the overcloud update. Result: Puppet is allowed to restart services, and it doesn't cause the overcloud update to fail anymore.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-21 11:58:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jiri Stransky 2015-11-10 12:41:51 EST
This was hit together with bug 1275986 and the cause and the fix could be similar or the same for these two bugs, but we're not sure of that yet, so we agreed to report this separately for now. We hit this issue on two environments.

os-collect-config journal:

lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Failed to call refresh: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Nova::Consoleauth/Nova::Generic_service[consoleauth]/Service[nova-consoleauth]: Could not restart Service[nova-consoleauth]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart openstack-nova-consoleauth' returned 1: Job for openstack-nova-consoleauth.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Error: /Stage[main]/Apache::Service/Service[httpd]: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Wrapped exception:
lis 10 09:32:32 overcloud-controller-0.localdomain os-collect-config[1259]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service canceled.

nova-consoleauth journal:

[root@overcloud-controller-0 ~]# journalctl -u openstack-nova-consoleauth
-- Logs begin at Út 2015-11-10 07:17:50 EST, end at Út 2015-11-10 12:11:16 EST. --
lis 10 07:31:02 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 07:31:09 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:09:50 overcloud-controller-0.localdomain systemd[1]: Stopping OpenStack Nova VNC console auth Server...
lis 10 09:09:51 overcloud-controller-0.localdomain systemd[1]: Stopped OpenStack Nova VNC console auth Server.
lis 10 09:24:36 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:24:38 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:52 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:53 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopping Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:55 overcloud-controller-0.localdomain systemd[1]: Stopped Cluster Controlled openstack-nova-consoleauth.
lis 10 09:31:57 overcloud-controller-0.localdomain systemd[1]: Starting Cluster Controlled openstack-nova-consoleauth...
lis 10 09:31:59 overcloud-controller-0.localdomain systemd[1]: Started Cluster Controlled openstack-nova-consoleauth.
Comment 1 James Slagle 2015-11-16 15:18:13 EST
jistr, assigning this one to you as I think it will be fixed under the larger umbrella of fixing pacemaker/puppet service restart issues
Comment 3 Marius Cornea 2015-12-15 10:00:37 EST
I did several updates for both 7.0 -> 7.2 and 7.1 -> 7.2 and I didn't hit the
issue reported here so I will move this to verified.
Comment 4 Gabriel Szasz 2015-12-15 10:05:30 EST
This bug is almost same issue as bug 1275986. The use case has been already covered by Marius (see Comment 3).
Comment 6 errata-xmlrpc 2015-12-21 11:58:24 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651

Note You need to log in before you can comment on or make changes to this bug.