Bug 1446669

+++ This bug was initially created as a clone of Bug #1316130 +++

Description of problem:

during a cluster node reboot, the node get's always fenced if pacemaker ist not stopped before.


Version-Release number of selected component (if applicable):

pacemaker-1.1.13-10.el7_2.2.x86_64
resource-agents-3.9.5-54.el7_2.6.x86_64

How reproducible:

with three gfs2 volumes nearly always. 


Steps to Reproduce:
1. configure multipath
2. configure gfs2 according to https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Global_File_System_2/ch-clustsetup-GFS2.html
3. reboot a node

Actual results:

node gets fenced


Expected results:

node reboots


Additional info:

using rhel 7.2, pacemaker, dlm, clvmd, gfs2 on SAN based block devices. Block devices are multipath devices. 


cluster log:


notice: Scheduling Node nodeb for shutdown
notice: Initiating action 70: stop clvmd_stop_0 on nodeb
warning: Action 70 (clvmd_stop_0) on nodeb failed (target: 0 vs. rc: 1): Error
warning: Node nodeb will be fenced because of resource failure(s)
warning: Scheduling Node nodeb for STONITH



the journal log on nodeb:


ERROR: Volume group "vg_sys" not found Cannot process volume group vg_sys


Analyse:

The ocf::heartbeat::clvm resource agent determines the available volume groups, and then proceeds to shut stop them. It is a shell script and not an atom operation. See https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/clvm#L269

On nodeb shutdown the systemd units  blk-availability.service and/or multipathd.service will remove the multipath block devices. Without the block devices, the volume group vanishes. 

If the volume groups vanish after the resource agent determined the volumes (line 269), it can't stop them in line 273.

problem can be solved by adding a dependency in the pacemaker.service unit.

/etc/systemd/system/pacemaker.service.d/order.conf

[Unit]
After=multipathd.service
After=blk-availability.service

<snip>

--- Additional comment from Ken Gaillot on 2016-12-19 13:46:58 EST ---

(In reply to Jan Pokorný from comment #18)
> (it could also be s/\.service/\.target/ though no experience here)

Yes, a .target is essentially identical to a .service, but with only [Unit] dependency information (Before/After/Wants/Requires), no [Service] section.

I'm leaning to this solution:

* resource-agents would deploy a systemd target for agent dependencies (basically just a name, no actual dependencies listed)

* pacemaker's systemd unit file would add After= and Wants= with the new target

* If a particular resource agent has a systemd unit dependency for something that cannot be managed by pacemaker as a resource, that agent could create a drop-in adding the dependency to the target when it is started. For example, clvmd and LVM require blk-availability, but blk-availability would never be a pacemaker resource. I would avoid automating any other dependencies, because we don't know whether pacemaker will manage them -- for example, LVM might depend on iSCSI or multipathd, but we wouldn't want drop-in dependencies for them if pacemaker is managing them.

* System administrators would be required to manually create drop-ins for the new target for any local dependencies. Resource agent man pages and meta-data, and any relevant online documentation, would be updated to mention how to do this. Resource agents could mention common dependencies (such as iSCSI and multipathd for Filesystem).