Hide Forgot
Description of problem: Manual fencing is not working. RHEL5.5 Version-Release number of selected component (if applicable): cman-2.0.115-34.el5 [root@rhel5node2 ~]# rpm -qf `which fence_ack_manual` cman-2.0.115-34.el5 How reproducible: Configure manual fencing for the cluster, hard reboot the other node and then try to manual fence the other node. Jan 7 04:03:13 rhel5node2 kernel: dlm: closing connection to node 1 Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] New Configuration: Jan 7 04:03:13 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com" Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] r(0) ip(172.16.217.4) Jan 7 04:03:13 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] Members Left: Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] r(0) ip(172.16.217.3) Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] Members Joined: Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] CLM CONFIGURATION CHANGE Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] New Configuration: Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] r(0) ip(172.16.217.4) Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] Members Left: Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] Members Joined: Jan 7 04:03:13 rhel5node2 openais[2430]: [SYNC ] This node is within the primary component and will provide service. Jan 7 04:03:13 rhel5node2 openais[2430]: [TOTEM] entering OPERATIONAL state. Jan 7 04:03:13 rhel5node2 openais[2430]: [CLM ] got nodejoin message 172.16.217.4 Jan 7 04:03:13 rhel5node2 openais[2430]: [CPG ] got joinlist message from node 2 Jan 7 04:03:18 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com" Jan 7 04:03:18 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed Jan 7 04:03:25 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com" Jan 7 04:03:25 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed Jan 7 04:03:31 rhel5node2 fenced[2449]: fencing node "rhel5node1.clust.com" Jan 7 04:03:31 rhel5node2 fenced[2449]: fence "rhel5node1.clust.com" failed [root@rhel5node2 ~]# fence_ack_manual -n rhel5node1.clust.com Warning: If the node "rhel5node1.clust.com" has not been manually fenced (i.e. power cycled or disconnected from shared storage devices) the GFS file system may become corrupted and all its data unrecoverable! Please verify that the node shown above has been reset or disconnected from storage. Are you certain you want to continue? [yN] y can't open /tmp/fence_manual.fifo: No such file or directory [root@rhel5node2 ~]# Steps to Reproduce: 1. Configure manual fencing for the cluster 2. While the nodes are up, hard reboot one node and check /var/log/messages for fencing other node message. 3.Fence the other node using fence_ack_manual -n <node of which fencing failed message coming in logs> root@rhel5node2 ~]# fence_ack_manual -n rhel5node1.clust.com Warning: If the node "rhel5node1.clust.com" has not been manually fenced (i.e. power cycled or disconnected from shared storage devices) the GFS file system may become corrupted and all its data unrecoverable! Please verify that the node shown above has been reset or disconnected from storage. Are you certain you want to continue? [yN] y can't open /tmp/fence_manual.fifo: No such file or directory Actual results: Manual fencing get's failed with an error message: can't open /tmp/fence_manual.fifo: No such file or directory Expected results: Should get manual fencing done. Additional info: Tried on the clone of rhel as well. I updated Clustering group from RHN as well. Thanks, Anuj Singh
@Anuj: This looks more like a problem in yours security configuration. Is it possible to create a non-block file in /tmp? Line in the code looks really solid in this place and only other thing that can be tested if the filename of file is correct. But as it is a hard-coded, I don't expect problem there.
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.
(In reply to comment #1) > @Anuj: > > This looks more like a problem in yours security configuration. Is it possible > to create a non-block file in /tmp? > > Line in the code looks really solid in this place and only other thing that can > be tested if the filename of file is correct. But as it is a hard-coded, I > don't expect problem there. @Marek I tried with disabling SELinux as well, yes i can create non-block files under /tmp (for example touch /tmp/foo). Please let me know if more information required?
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux release for currently deployed products. This request is not yet committed for inclusion in a release.
Do not use fence_manual. Instead, remove it from cluster.conf. When a node fails to fence such as in your configuration, run: # fence_ack_manual -e -n rhel5node1.clust.com ^^
More information available here: https://access.redhat.com/knowledge/articles/36302 Also in the manual page for 'fence_ack_manual' on RHEL5.
Thanks Lon/Marek for help. (In reply to comment #1) > @Anuj: > > This looks more like a problem in yours security configuration. Is it possible > to create a non-block file in /tmp? > > Line in the code looks really solid in this place and only other thing that can > be tested if the filename of file is correct. But as it is a hard-coded, I > don't expect problem there. @Marek I tried with disabling SELinux as well, yes i can create non-block files under /tmp (for example touch /tmp/foo). Please let me know if more information required?