Bug 1263348

Looking at the code, there are tons of conditions inside the agent that can lead to file temporary file leaks beside the normal runtime monitoring issue.

I have never used mysql is master/slave mode (where this code path is leaking). Can you please share your configuration so i can set it up easily to verify any fix?

Thanks

I'm currently unable to produce a test case. I will try to find the time in the next week, so please do not autoclose this bug.

OCF_CHECK_LEVEL=10 is required to reproduce the issue:
# pcs resource op add MDB monitor interval=15s OCF_CHECK_LEVEL=10

I tested and verified that the following patch solves the issue:
https://github.com/ClusterLabs/resource-agents/pull/722

Setup:
# pcs resource create MySQL mysql
# pcs resource master MySQL
# pcs resource op add MySQL monitor interval=15s OCF_CHECK_LEVEL=10


Run on the slave node.

Before:
# rpm -q resource-agents
resource-agents-3.9.5-54.el7_2.6.x86_64
# ls /var/run/resource-agents/check_slave.*
/var/run/resource-agents/check_slave.MySQL.uCeh9f

Wait at least 15 seconds.
# ls /var/run/resource-agents/check_slave.*
/var/run/resource-agents/check_slave.MySQL.uCeh9f
/var/run/resource-agents/check_slave.MySQL.uxRvQo


After:
# rpm -q resource-agents
resource-agents-3.9.5-61.el7.x86_64
# ls /var/run/resource-agents/check_slave.*
/var/run/resource-agents/check_slave.MySQL.uCeh9f
/var/run/resource-agents/check_slave.MySQL.uxRvQo

Wait at least 15 seconds.
# ls /var/run/resource-agents/check_slave.*
/var/run/resource-agents/check_slave.MySQL.uCeh9f
/var/run/resource-agents/check_slave.MySQL.uxRvQo

The same number of files (from before the patch still exist, but will be removed on reboot).

I have verified that there are no more leftover files on slave after the patch
in resource-agents-3.9.5-81.el7.x86_64

---

common setup
 * setup cluster (1)
 * setup mysql resource as master/slave clone and verify that mysql is 
    running (2)

before the patch (resource-agents-3.9.5-54.el7.x86_64)
======================================================
> on slave virt-022, there are a lot of leftover files

[root@virt-022 ~]# find /var/run/resource-agents
/var/run/resource-agents
/var/run/resource-agents/check_slave.mysql.FtU5zA
/var/run/resource-agents/check_slave.mysql.vIqLEy
/var/run/resource-agents/check_slave.mysql.rGDFFZ
/var/run/resource-agents/check_slave.mysql.8X9GpU
/var/run/resource-agents/check_slave.mysql.Nns5S0
/var/run/resource-agents/check_slave.mysql.jRlGo9
/var/run/resource-agents/check_slave.mysql.pHXcvw
/var/run/resource-agents/check_slave.mysql.YM8ofW
/var/run/resource-agents/check_slave.mysql.ej6his
/var/run/resource-agents/check_slave.mysql.BFzDwJ
/var/run/resource-agents/check_slave.mysql.lvI3t5
/var/run/resource-agents/check_slave.mysql.iQr2Xt
/var/run/resource-agents/check_slave.mysql.4unyu5
/var/run/resource-agents/threads.mysql.XGarNE
/var/run/resource-agents/master_status.mysql
/var/run/resource-agents/cmirrord-clvmd.pid
/var/run/resource-agents/clvmd-clvmd.pid


after the patch (resource-agents-3.9.5-81.el7.x86_64)
=====================================================

> when mysql resource is active on virt-005, virt-022 is slave:

[root@virt-022 ~]# find /var/run/resource-agents
/var/run/resource-agents
/var/run/resource-agents/cmirrord-clvmd.pid
/var/run/resource-agents/clvmd-clvmd.pid


-----

>> (1)
[root@virt-005 ~]# pcs status
Cluster name: STSRHTS2803
Stack: corosync
Current DC: virt-021 (version 1.1.15-10.el7-e174ec8) - partition with quorum
Last updated: Fri Sep  2 15:40:09 2016          Last change: Fri Sep  2 15:34:26 2016 by root via cibadmin on virt-005

4 nodes and 14 resources configured

Online: [ virt-004 virt-005 virt-021 virt-022 ]

Full list of resources:

 fence-virt-004 (stonith:fence_xvm):    Started virt-004
 fence-virt-005 (stonith:fence_xvm):    Started virt-005
 fence-virt-021 (stonith:fence_xvm):    Started virt-021
 fence-virt-022 (stonith:fence_xvm):    Started virt-022
 Clone Set: dlm-clone [dlm]
     Started: [ virt-004 virt-005 virt-021 virt-022 ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ virt-004 virt-005 virt-021 virt-022 ]
 Master/Slave Set: mysql-clone [mysql]
     Masters: [ virt-005 ]
     Slaves: [ virt-022 ]

Failed Actions:

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

>> (2)
[root@virt-005 ~]# pcs resource show --full
 Clone: dlm-clone
  Meta Attrs: interleave=true ordered=true
  Resource: dlm (class=ocf provider=pacemaker type=controld)
   Operations: start interval=0s timeout=90 (dlm-start-interval-0s)
               stop interval=0s timeout=100 (dlm-stop-interval-0s)
               monitor interval=30s on-fail=fence (dlm-monitor-interval-30s)
 Clone: clvmd-clone
  Meta Attrs: interleave=true ordered=true
  Resource: clvmd (class=ocf provider=heartbeat type=clvm)
   Attributes: with_cmirrord=1
   Operations: start interval=0s timeout=90 (clvmd-start-interval-0s)
               stop interval=0s timeout=90 (clvmd-stop-interval-0s)
               monitor interval=30s on-fail=fence (clvmd-monitor-interval-30s)
 Master: mysql-clone
  Meta Attrs: master-node-max=1 clone-max=2 notify=true master-max=1 clone-node-max=1
  Resource: mysql (class=ocf provider=heartbeat type=mysql)
   Operations: start interval=0s timeout=240 (mysql-start-interval-0s)
               stop interval=0s timeout=120 (mysql-stop-interval-0s)
               promote interval=0s timeout=120 (mysql-promote-interval-0s)
               demote interval=0s timeout=120 (mysql-demote-interval-0s)
               monitor interval=30s (mysql-monitor-interval-30s)
               monitor interval=15s OCF_CHECK_LEVEL=10 (mysql-monitor-interval-15s)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2174.html