Bug 1039734

Moving bug to pacemaker since this issue is caused by crm_resource (which pcs calls) hanging.

While I investigate, can you try running:

   crm_resource -C -r clusterfs-clone

and see if that behaves the same way?

(Never to refer to the thing inside the clone/group)

I tried with Dummy resource and this behaves the same way. 
Other thing to notice: The cleaning of the resource(s) is done but 1 is
returned presumably because of the crmd not replying.

I believe that the problem might be that crm_resource is waiting for +1 crmd
replies than it should:

# pcs status
Cluster name: STSRHTS19810
Last updated: Fri Jan 31 14:53:55 2014
Last change: Fri Jan 31 14:51:48 2014 via crmd on virt-139
Stack: corosync
Current DC: virt-139 (2) - partition with quorum
Version: 1.1.10-22.el7-368c726
3 Nodes configured
6 Resources configured


Online: [ virt-138 virt-139 virt-140 ]

Full list of resources:

 virt-fencing   (stonith:fence_xvm):    Started virt-138 
 Resource Group: ha-apache-0
     ip-0       (ocf::heartbeat:IPaddr2):       Started virt-140 
     lvm-0      (ocf::heartbeat:LVM):   Started virt-140 
     fs-0       (ocf::heartbeat:Filesystem):    Started virt-140 
     apache-0   (ocf::heartbeat:apache):        Started virt-140 
 le-dummy       (ocf::heartbeat:Dummy): Started virt-139 

PCSD Status:
  virt-138: Online
  virt-139: Online
  virt-140: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

# crm_resource -C -r le-dummy
Cleaning up le-dummy on virt-138.cluster-qe.lab.eng.brq.redhat.com
Cleaning up le-dummy on virt-139.cluster-qe.lab.eng.brq.redhat.com
Cleaning up le-dummy on virt-140.cluster-qe.lab.eng.brq.redhat.com
Waiting for 4 replies from the CRMd...No messages received in 60 seconds.. aborting

# crm_resource -C -r ha-apache-0
Cleaning up ip-0 on virt-138.cluster-qe.lab.eng.brq.redhat.com
Cleaning up ip-0 on virt-139.cluster-qe.lab.eng.brq.redhat.com
Cleaning up ip-0 on virt-140.cluster-qe.lab.eng.brq.redhat.com
Cleaning up lvm-0 on virt-138.cluster-qe.lab.eng.brq.redhat.com
Cleaning up lvm-0 on virt-139.cluster-qe.lab.eng.brq.redhat.com
Cleaning up lvm-0 on virt-140.cluster-qe.lab.eng.brq.redhat.com
Cleaning up fs-0 on virt-138.cluster-qe.lab.eng.brq.redhat.com
Cleaning up fs-0 on virt-139.cluster-qe.lab.eng.brq.redhat.com
Cleaning up fs-0 on virt-140.cluster-qe.lab.eng.brq.redhat.com
Cleaning up apache-0 on virt-138.cluster-qe.lab.eng.brq.redhat.com
Cleaning up apache-0 on virt-139.cluster-qe.lab.eng.brq.redhat.com
Cleaning up apache-0 on virt-140.cluster-qe.lab.eng.brq.redhat.com
Waiting for 13 replies from the CRMd............No messages received in 60 seconds.. aborting
12 resources 13 crmd replies expected.

I encountered this as well today -- it's a bug that looks to already be fixed upstream in tools/crm_resource.c 1.1.11-rc5 but there's a patch in the RPM making it a bit hidden.

https://github.com/ClusterLabs/pacemaker/blob/master/tools/crm_resource.c

https://github.com/ClusterLabs/pacemaker/commit/145c782e432d8108ca865f994640cf5a62406363

In the beta RPM (pacemaker-1.1.10-19.el7.src.rpm) the code sets crmd_replies_needed=0, then loops through and adds one for every node, then also in the start_mainloop() increments it one more time. The counter is decremented for each valid reply, however we're still left over with that one extra one added in start_mainloop() that wasn't decremented.

The patch in the SRPM:

bz720543-pcmk-crm_resource_wait_for_all_replies_when_cleaning_up_resources.patch

...is adding the loop increment which resulted in this bug due to the extra increment in start_mainloop():

+        if(node->details->remote_rsc == NULL) {
+            crmd_replies_needed++;
+        }

At least, that's my take on it -- I'm new to Pacemaker. :)

(In reply to tengel from comment #5)
> https://github.com/ClusterLabs/pacemaker/commit/
> 145c782e432d8108ca865f994640cf5a62406363

This is definitely the correct patch, well need to backport it

This is still not fixed in RHEL7 RC1 - I have two nodes configured and get:

[root@linclus3a heartbeat]# crm_resource -C -r db_ip
Cleaning up db_ip on linclus3a
Cleaning up db_ip on linclus3b
Waiting for 3 replies from the CRMd..No messages received in 60 seconds.. aborting

(In reply to tengel from comment #8)
> This is still not fixed in RHEL7 RC1

Right. The fix will be available post GA.

This appears to be fixed - I rebuilt the cluster with the GA and performed a full RPM update:

[root@linclus3a ~]# crm_resource -C -r db_ip
Cleaning up db_ip on linclus3a
Cleaning up db_ip on linclus3b
Waiting for 2 replies from the CRMd.. OK

[root@linclus3a ~]# rpm -q pacemaker
pacemaker-1.1.10-31.el7_0.x86_64

Looking spiffy, thanks.