Bug 772380

Summary:	Manual fencing get's failed \| can't open /tmp/fence_manual.fifo: No such file or directory
Product:	Red Hat Enterprise Linux 5	Reporter:	Anuj Singh <anuj>
Component:	cman	Assignee:	Lon Hohberger <lhh>
Status:	CLOSED NOTABUG	QA Contact:	Cluster QE <mspqa-list>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	5.7	CC:	cluster-maint, edamato, mgrac, mkelly
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-09 17:47:51 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Anuj Singh 2012-01-07 08:44:29 UTC

Description of problem:
Manual fencing is not working. RHEL5.5


Version-Release number of selected component (if applicable):
cman-2.0.115-34.el5

[root@rhel5node2 ~]# rpm -qf `which fence_ack_manual`
cman-2.0.115-34.el5


How reproducible:
Configure manual fencing for the cluster, hard reboot the other node and then try to manual fence the other node.

Jan  7 04:03:13 rhel5node2 kernel: dlm: closing connection to node 1
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] New Configuration: 
Jan  7 04:03:13 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com"
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] 	r(0) ip(172.16.217.4)  
Jan  7 04:03:13 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] Members Left: 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] 	r(0) ip(172.16.217.3)  
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] Members Joined: 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] CLM CONFIGURATION CHANGE 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] New Configuration: 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] 	r(0) ip(172.16.217.4)  
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] Members Left: 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] Members Joined: 
Jan  7 04:03:13 rhel5node2 openais[2430]: [SYNC ] This node is within the primary component and will provide service. 
Jan  7 04:03:13 rhel5node2 openais[2430]: [TOTEM] entering OPERATIONAL state. 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CLM  ] got nodejoin message 172.16.217.4 
Jan  7 04:03:13 rhel5node2 openais[2430]: [CPG  ] got joinlist message from node 2 
Jan  7 04:03:18 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com"
Jan  7 04:03:18 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed
Jan  7 04:03:25 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com"
Jan  7 04:03:25 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed
Jan  7 04:03:31 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com"
Jan  7 04:03:31 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed

[root@rhel5node2 ~]# fence_ack_manual -n rhel5node1.clust.com

Warning:  If the node "rhel5node1.clust.com" has not been manually fenced
(i.e. power cycled or disconnected from shared storage devices)
the GFS file system may become corrupted and all its data
unrecoverable!  Please verify that the node shown above has
been reset or disconnected from storage.

Are you certain you want to continue? [yN] y
can't open /tmp/fence_manual.fifo: No such file or directory
[root@rhel5node2 ~]# 


Steps to Reproduce:
1. Configure manual fencing for the cluster
2. While the nodes are up, hard reboot one node and check /var/log/messages for fencing other node message.
3.Fence the other node using
 fence_ack_manual -n <node of which fencing failed message coming in logs>

root@rhel5node2 ~]# fence_ack_manual -n rhel5node1.clust.com

Warning:  If the node "rhel5node1.clust.com" has not been manually fenced
(i.e. power cycled or disconnected from shared storage devices)
the GFS file system may become corrupted and all its data
unrecoverable!  Please verify that the node shown above has
been reset or disconnected from storage.

Are you certain you want to continue? [yN] y
can't open /tmp/fence_manual.fifo: No such file or directory

  
Actual results:
Manual fencing get's failed with an error message:
can't open /tmp/fence_manual.fifo: No such file or directory

Expected results:
Should get manual fencing done.

Additional info:
Tried on the clone of rhel as well. I updated Clustering group from RHN as well.

Thanks,
Anuj Singh

Comment 1 Marek Grac 2012-01-11 14:53:19 UTC

@Anuj:

This looks more like a problem in yours security configuration. Is it possible to create a non-block file in /tmp?

Line in the code looks really solid in this place and only other thing that can be tested if the filename of file is correct. But as it is a hard-coded, I don't expect problem there.

Comment 2 RHEL Program Management 2012-02-10 18:14:37 UTC

Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

Comment 3 Anuj Singh 2012-04-06 15:56:09 UTC

(In reply to comment #1)
> @Anuj:
> 
> This looks more like a problem in yours security configuration. Is it possible
> to create a non-block file in /tmp?
> 
> Line in the code looks really solid in this place and only other thing that can
> be tested if the filename of file is correct. But as it is a hard-coded, I
> don't expect problem there.

@Marek

I tried with disabling SELinux as well, yes i can create non-block files under /tmp (for example touch /tmp/foo). Please let me know if more information required?

Comment 4 RHEL Program Management 2012-04-06 15:59:16 UTC

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.

Comment 5 Lon Hohberger 2012-04-09 17:47:51 UTC

Do not use fence_manual.

Instead, remove it from cluster.conf.  When a node fails to fence such as in your configuration, run:

# fence_ack_manual -e -n rhel5node1.clust.com
                   ^^

Comment 7 Lon Hohberger 2012-04-09 17:56:17 UTC

More information available here:

https://access.redhat.com/knowledge/articles/36302

Also in the manual page for 'fence_ack_manual' on RHEL5.

Comment 8 Anuj Singh 2012-04-11 17:06:45 UTC

Thanks Lon/Marek for help.

(In reply to comment #1)
> @Anuj:
> 
> This looks more like a problem in yours security configuration. Is it possible
> to create a non-block file in /tmp?
> 
> Line in the code looks really solid in this place and only other thing that can
> be tested if the filename of file is correct. But as it is a hard-coded, I
> don't expect problem there.

@Marek

I tried with disabling SELinux as well, yes i can create non-block files under /tmp (for example touch /tmp/foo). Please let me know if more information required?