Rubygem-staypuft: HA: Systemd unit MUST be disabled! (all of them, including extra services like mariadb)
[root@maca25400868092 ~]# systemctl status openstack-cinder-api
openstack-cinder-api.service - OpenStack Cinder API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-api.service; enabled)
For every systemd pcs resource, you must make sure that they systemd unit is disabled or you will have some serious pacemaker/systemd interaction problems.
The openstack puppet modules currently have support for some services doing this, but not all. The modules that appear to support this currently are:
I am going to open BZs against opm for each of the remaining services for tracking. Once they exist there, we can add them to quick stack.
Patch in progress for the services we have current support for is here:
As a meta-pattern, it is tempting to use https://github.com/redhat-openstack/astapor/pull/357 in the top level if for each pacemaker service, i.e., if glance is already set up, don't include any of the glance manifests. Then, we could "manually" shutdown glance (via an Exec) and disable its systemd resource (via an Exec) right before we add the glance pacemaker resource. Then, we wouldn't care about whether we have puppet support for managing the glance service or not. Times the dozen or so things we are managing in puppet.
Work continues on this BZ, but it's more involved and taking longer than planned, moving to A1.
*** Bug 1093759 has been marked as a duplicate of this bug. ***
This fix take a two run puppet agent approach to fully configure the nodes. The first run sets up all service and hands them over to pacemaker, while the second disables the systems unit so it does not conflict with pacemaker. Originally we tried to do this all in one run, but there were race conditions we could not work around in the time available, and after some discussion, it was agreed that 2 runs to achieve the fully desired state was an acceptable direction (others using puppet do this on purpose as well). After the second run, all openstack services plus https, rabbit and galera should be disabled on boot and puppet will not try to alter their running state.
Content of multi-user.target.wants directory right after deployment.
[root@macf04da2732fb1 multi-user.target.wants]# cd /etc/systemd/system/multi-user.target.wants/
[root@macf04da2732fb1 multi-user.target.wants]# ls -all
drwxr-xr-x. 2 root root 4096 Sep 21 08:45 .
drwxr-xr-x. 10 root root 4096 Sep 21 08:16 ..
lrwxrwxrwx. 1 root root 38 Sep 21 08:14 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 40 Sep 21 08:24 corosync.service -> /usr/lib/systemd/system/corosync.service
lrwxrwxrwx. 1 root root 37 Sep 21 08:03 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 39 Sep 21 08:26 haproxy.service -> /usr/lib/systemd/system/haproxy.service
lrwxrwxrwx. 1 root root 42 Sep 21 08:04 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Sep 21 08:04 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root 41 Sep 21 08:18 memcached.service -> /usr/lib/systemd/system/memcached.service
lrwxrwxrwx. 1 root root 34 Sep 21 08:16 nfs.target -> /usr/lib/systemd/system/nfs.target
lrwxrwxrwx. 1 root root 43 Sep 21 08:22 openvswitch.service -> /usr/lib/systemd/system/openvswitch.service
lrwxrwxrwx. 1 root root 41 Sep 21 08:24 pacemaker.service -> /usr/lib/systemd/system/pacemaker.service
lrwxrwxrwx. 1 root root 36 Sep 21 08:24 pcsd.service -> /usr/lib/systemd/system/pcsd.service
lrwxrwxrwx. 1 root root 39 Sep 21 08:04 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root 38 Sep 21 08:11 puppet.service -> /usr/lib/systemd/system/puppet.service
lrwxrwxrwx. 1 root root 40 Sep 21 08:03 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 41 Sep 21 08:04 rhsmcertd.service -> /usr/lib/systemd/system/rhsmcertd.service
lrwxrwxrwx. 1 root root 39 Sep 21 08:15 rpcbind.service -> /usr/lib/systemd/system/rpcbind.service
lrwxrwxrwx. 1 root root 39 Sep 21 08:04 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root 36 Sep 21 08:04 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 37 Sep 21 08:04 tuned.service -> /usr/lib/systemd/system/tuned.service
lrwxrwxrwx. 1 root root 38 Sep 21 08:18 xinetd.service -> /usr/lib/systemd/system/xinetd.service
I am unclear why this doc text was added. The functionality this patch added removes the need for the stated workaround in the Fixed-in version. In fact, using the workaround will now potentially cause an incomplete deployment.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.