Bug 1123303 - Rubygem-staypuft: HA: Systemd unit MUST be disabled! (all of them, including extra services like mariadb)
Summary: Rubygem-staypuft: HA: Systemd unit MUST be disabled! (all of them, including ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z1
: Installer
Assignee: Jason Guiditta
QA Contact: Leonid Natapov
URL:
Whiteboard:
: 1093759 (view as bug list)
Depends On: 1129397 1129399 1129404 1129407 1129408 1129409 1129410
Blocks: 1040649
TreeView+ depends on / blocked
 
Reported: 2014-07-25 09:27 UTC by Leonid Natapov
Modified: 2016-04-26 19:47 UTC (History)
10 users (show)

Fixed In Version: openstack-foreman-installer-2.0.23-1.el6ost
Doc Type: Bug Fix
Doc Text:
Puppet module configured High Availability services when 'systemd' was disabled. As a result, Pacemaker and 'systemd' fight for control of services causing intermittent issues depending on which one got the control. With this fix, 'systemd' is disabled on High Availability services allowing Pacemaker to control them. As a result, Packmaker controls the High Availability services and 'systemd' is disabled for those services.
Clone Of:
: 1129397 1129399 1129404 1129407 1129408 1129409 1129410 (view as bug list)
Environment:
Last Closed: 2014-10-01 13:25:39 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1350 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Bug Fix Advisory 2014-10-01 17:22:34 UTC

Description Leonid Natapov 2014-07-25 09:27:43 UTC
Rubygem-staypuft: HA: Systemd unit MUST be disabled! (all of them, including extra services like mariadb)

[root@maca25400868092 ~]# systemctl status openstack-cinder-api
openstack-cinder-api.service - OpenStack Cinder API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-api.service; enabled)

For every systemd pcs resource, you must make sure that they systemd unit is disabled or you will have some serious pacemaker/systemd interaction problems.

Comment 2 Jason Guiditta 2014-08-12 16:39:57 UTC
The openstack puppet modules currently have support for some services doing this, but not all.  The modules that appear to support this currently are:
* nova
* neutron
* cinder
* qpid
* rabbitmq

I am going to open BZs against opm for each of the remaining services for tracking.  Once they exist there, we can add them to quick stack.

Patch in progress for the services we have current support for is here:
https://github.com/redhat-openstack/astapor/pull/349

Comment 5 Crag Wolfe 2014-08-21 17:23:07 UTC
As a meta-pattern, it is tempting to use https://github.com/redhat-openstack/astapor/pull/357 in the top level if for each pacemaker service, i.e., if glance is already set up, don't include any of the glance manifests.  Then, we could "manually" shutdown glance (via an Exec) and disable its systemd resource (via an Exec) right before we add the glance pacemaker resource.  Then, we wouldn't care about whether we have puppet support for managing the glance service or not.  Times the dozen or so things we are managing in puppet.

Comment 6 Mike Burns 2014-08-27 22:54:27 UTC
Work continues on this BZ, but it's more involved and taking longer than planned, moving to A1.

Comment 7 Mike Burns 2014-09-08 17:43:58 UTC
*** Bug 1093759 has been marked as a duplicate of this bug. ***

Comment 8 Jason Guiditta 2014-09-08 21:04:22 UTC
This fix take a two run puppet agent approach to fully configure the nodes.  The first run sets up all service and hands them over to pacemaker, while the second disables the systems unit so it does not conflict with pacemaker.  Originally we tried to do this all in one run, but there were race conditions we could not work around in the time available, and after some discussion, it was agreed that 2 runs to achieve the fully desired state was an acceptable direction (others using puppet do this on purpose as well).  After the second run, all openstack services plus https, rabbit and galera should be disabled on boot and puppet will not try to alter their running state.

Comment 11 Leonid Natapov 2014-09-21 10:30:50 UTC
openstack-foreman-installer-2.0.24-1.el6ost.noarch

Content of multi-user.target.wants directory right after deployment.

[root@macf04da2732fb1 multi-user.target.wants]# cd /etc/systemd/system/multi-user.target.wants/
[root@macf04da2732fb1 multi-user.target.wants]# ls -all
total 8
drwxr-xr-x.  2 root root 4096 Sep 21 08:45 .
drwxr-xr-x. 10 root root 4096 Sep 21 08:16 ..
lrwxrwxrwx.  1 root root   38 Sep 21 08:14 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx.  1 root root   40 Sep 21 08:24 corosync.service -> /usr/lib/systemd/system/corosync.service
lrwxrwxrwx.  1 root root   37 Sep 21 08:03 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx.  1 root root   39 Sep 21 08:26 haproxy.service -> /usr/lib/systemd/system/haproxy.service
lrwxrwxrwx.  1 root root   42 Sep 21 08:04 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx.  1 root root   37 Sep 21 08:04 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx.  1 root root   41 Sep 21 08:18 memcached.service -> /usr/lib/systemd/system/memcached.service
lrwxrwxrwx.  1 root root   34 Sep 21 08:16 nfs.target -> /usr/lib/systemd/system/nfs.target
lrwxrwxrwx.  1 root root   43 Sep 21 08:22 openvswitch.service -> /usr/lib/systemd/system/openvswitch.service
lrwxrwxrwx.  1 root root   41 Sep 21 08:24 pacemaker.service -> /usr/lib/systemd/system/pacemaker.service
lrwxrwxrwx.  1 root root   36 Sep 21 08:24 pcsd.service -> /usr/lib/systemd/system/pcsd.service
lrwxrwxrwx.  1 root root   39 Sep 21 08:04 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx.  1 root root   38 Sep 21 08:11 puppet.service -> /usr/lib/systemd/system/puppet.service
lrwxrwxrwx.  1 root root   40 Sep 21 08:03 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx.  1 root root   41 Sep 21 08:04 rhsmcertd.service -> /usr/lib/systemd/system/rhsmcertd.service
lrwxrwxrwx.  1 root root   39 Sep 21 08:15 rpcbind.service -> /usr/lib/systemd/system/rpcbind.service
lrwxrwxrwx.  1 root root   39 Sep 21 08:04 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx.  1 root root   36 Sep 21 08:04 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx.  1 root root   37 Sep 21 08:04 tuned.service -> /usr/lib/systemd/system/tuned.service
lrwxrwxrwx.  1 root root   38 Sep 21 08:18 xinetd.service -> /usr/lib/systemd/system/xinetd.service

Comment 12 Jason Guiditta 2014-09-25 19:02:42 UTC
I am unclear why this doc text was added.  The functionality this patch added removes the need for the stated workaround in the Fixed-in version.  In fact, using the workaround will now potentially cause an incomplete deployment.

Comment 15 errata-xmlrpc 2014-10-01 13:25:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1350.html


Note You need to log in before you can comment on or make changes to this bug.