Bug 219633 - fenced fails to execute fence secondary fence methods if its ccs connection timesout
fenced fails to execute fence secondary fence methods if its ccs connection t...
Status: CLOSED ERRATA
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: fence (Show other bugs)
4
All Linux
medium Severity medium
: ---
: ---
Assigned To: Chris Feist
Cluster QE
:
Depends On:
Blocks: 354421
  Show dependency treegraph
 
Reported: 2006-12-14 10:39 EST by Josef Bacik
Modified: 2010-10-22 03:25 EDT (History)
6 users (show)

See Also:
Fixed In Version: RHBA-2007-0138
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-05-10 17:25:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch that resolves the problem (4.19 KB, patch)
2006-12-14 13:23 EST, Josef Bacik
no flags Details | Diff
test src.rpm with the fix. (315.92 KB, application/octet-stream)
2006-12-14 13:29 EST, Josef Bacik
no flags Details

  None (edit)
Description Josef Bacik 2006-12-14 10:39:33 EST
Description of problem:
Customer has multiple fence levels setup, one being something that can be 
turned off, like ipmi or ilo, and the second being something else.  When the 
ipmi interface fails, fenced goes to do a ccs_get() on the next fence level 
and fails with these messages

Dec 11 17:52:47 node_b ccsd[9390]: process_get: Invalid connection descriptor 
received. 
Dec 11 17:52:47 node_b ccsd[9390]: Error while processing get: Invalid request 
descriptor

I have yet to be able to reproduce this myself.  It has also been reported in 
the wild

https://www.redhat.com/archives/linux-cluster/2006-December/msg00096.html

I have not been able to reproduce this.

Version-Release number of selected component (if applicable):
ccs-1.0.7-0-x86_64

How reproducible:
every time

Steps to Reproduce:
1.setup multiple fence levels
2.disable the first fence level so it will fail
3.echo c > /proc/sysrq-trigger or some other method of forcing the box to be 
fenced
  
Actual results:
The second fence level is not executed

Expected results:
The second fence level should be executed

Additional info:
I'm building a debug ccs package now to try and narrow down what is happening.  
I will have that shortly.
Comment 1 Josef Bacik 2006-12-14 10:57:24 EST
you know what, looking at this now, there is a _large_ delay between the 
fencing action and the fence action returning a failure

Dec 11 17:50:28 node_b fenced[3240]: fencing node "node_a"
Dec 11 17:52:47 node_b fenced[3240]: agent "fence_ipmilan" reports: Rebooting 
machine @ IPMI:172.18.56.17...ipmilan: Failed to connect after 30 seconds 
Failed 

I bet the connection to ccsd is timing out and thats why ccsd freaks out when 
we go to do the next ccs_get().  I will try and reproduce this on my test 
setup.
Comment 2 Marcos David 2006-12-14 11:10:57 EST
Well, it tries to fence the device twice before it fails. Between tries there is
a 30 second (or maybe a minute) delay until ipmilan exists. Maybe this is the
reason for the timeout. Should it try to fence only once?
Comment 3 Josef Bacik 2006-12-14 11:14:00 EST
yeah any delay that long is going to cause this problem.  I just reproduced on 
my test cluster.  I will try and get a solution together for you to test out 
as soon as i fix it.
Comment 4 Josef Bacik 2006-12-14 11:49:56 EST
ok the reason this is happening is because each connection has a default 
expire time of 30 seconds.  One solution would be to bump that timeout, but i 
think thats a bad idea.  I think the best solution would be to check the error 
from ccs_get() in fenced and if its -EBADR, to reconnect and then try again.  
I will write this up and test it out and see how it goes.
Comment 5 Josef Bacik 2006-12-14 13:23:20 EST
Created attachment 143668 [details]
patch that resolves the problem

Ok here is the patch that I used to solve the problem.	Bascially this moves
the ccs_connect/disconnect into dispatch_fence_agent(), so the caller doesn't
have to handle any of that.  If we get an -EBADR when doing a ccs_get() we try
to reinstantiate the connection and call the ccs_get() again.  I have tested
this on my cluster and it works perfectly.
Comment 6 Josef Bacik 2006-12-14 13:25:27 EST
fixing summary and component.
Comment 7 Josef Bacik 2006-12-14 13:29:23 EST
Created attachment 143669 [details]
test src.rpm with the fix.

please test this patch to make sure it works.  Just download this src.rpm and
run

rpmbuild --rebuild fence-1.32.25-1.bz219633.test.src.rpm

and then install the rpm that it creates and test it out to make sure it works
alright.
Comment 8 Marcos David 2006-12-14 13:38:05 EST
I'm probably wrong, but won't this block if it can't connect to ccs without a
ccs_force_connect? It will sleep forever...
In this case the if (cd <0 ) 
will never be executed because of the sleep while (cd < 0)?


+	if (force)
+		cd = ccs_force_connect(NULL, 0);
+	else {
+		while ((cd = ccs_connect()) < 0)
+			sleep(1);
+	}
+
+	if (cd < 0) {
+		syslog(LOG_ERR, "cannot connect to ccs %d\n", cd);
+		return -1;
+	}
Comment 9 Marcos David 2006-12-14 13:48:17 EST
rpmbuild --rebuild fence-1.32.25-1.bz219633.test.src.rpm
doesn't work:
error: Architecture is not included: i386
Comment 10 Josef Bacik 2006-12-14 15:13:20 EST
yeah if you cannot connect to ccs then you will sleep forever.  That is left 
over from the original ccs_connect() that was done in fence_victims(), I just 
copied it over, why fix whats not broke?  During normal operations force will 
be 0, so you will do that while loop, and if you cannot connect to ccs during 
normal operations, you have much larger problems than not being able to 
fence :).  The force option is provided simply for the fence_node facility, 
which will force a connection to ccs even if the cluster is without quorum.  
If you are on i686 you need to use

rpmbuild --rebuild fence-1.32.25-1.bz219633.test.src.rpm --target i686

in order to make it build.
Comment 11 Marcos David 2006-12-15 05:45:44 EST
Ok, I've tested this and it seems to work, it no longer has the message:
"process_get: Invalid connection descriptor received."
repeating over and over, instead it just tries the next fence level.

Thanks for the help.
Comment 12 Chris Feist 2006-12-20 13:15:46 EST
Patch looks good and I've committed it to CVS.  It will be included in the next
fence build (>=1.32.27).
Comment 19 Red Hat Bugzilla 2007-05-10 17:25:45 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0138.html
Comment 20 Issue Tracker 2007-06-15 10:05:03 EDT
Hello,

As per your previous comments, I'll go ahead and close this issue.

Best Regards,

Jon

Internal Status set to 'Resolved'
Status set to: Closed by Tech
Resolution set to: 'NotABug'

This event sent from IssueTracker by jfautley 
 issue 107933

Note You need to log in before you can comment on or make changes to this bug.