Bug 1013062

Created attachment 804098 [details]
session.log with output from Node 1: ps ax -o pid,stat,cmd,wchan

I don't think I ever asked for a ps ax -o pid,stat,cmd,wchan from node1 while things were stuck, could you collect that?

session.log attached with requested information.

I believe both nodes (1, 15) are in ping_members(), and have sent a status message to the other via dlm_rcom_status().

The tcpdump shows the status message from 15 to 1, but it shows nothing from 1 to 15.  Node 1 should also have replied to the status message it received from node 15, but we don't see any reply either.  We need to figure out where these messages are being dropped.  The dlm message debugging does not go any lower, so we need to either add that, or implicate a layer below it.

(In reply to David Teigland from comment #4)
> I believe both nodes (1, 15) are in ping_members(), and have sent a status
> message to the other via dlm_rcom_status().
> 
> The tcpdump shows the status message from 15 to 1, but it shows nothing from
> 1 to 15.  Node 1 should also have replied to the status message it received
> from node 15, but we don't see any reply either.  We need to figure out
> where these messages are being dropped.  The dlm message debugging does not
> go any lower, so we need to either add that, or implicate a layer below it.

Thanks David,
do you need us to do anything further or provide more information?

btw. looking at the network interface there are 0 dropped packets on both nodes.
Also, there are no iptables rules applied.

Created attachment 805321 [details]
systemtap

Here's a systemtap script to show any messages the dlm receives.
I'm still trying to get a system set up to test it, so I don't know if it works.

Created attachment 805393 [details]
systemtap

Fixed some %d/%u prints in case systemtap cares.

Created attachment 805450 [details]
systemtap

I debugged this one on rhel7, hopefully it works on rhel6.

Adding the systemtap steps to the steps we've run before:

Node1: dlm_tool join fooN

Node1:
tcpdump -w /tmp/node1-tcpdump -i eth0 host 10.16.100.34 and port 21064
Node15:
tcpdump -w /tmp/node15-tcpdump -i eth0 host 10.16.100.6 and port 21064

Node1:
stap dlm-recv.stp > /tmp/node1-stap
Node15:
stap dlm-recv.stp > /tmp/node15-stap

Node15: dlm_tool join fooN  (This doesn't complete.)

Stop tcpdump and stap and attached the captured data.

Unfortuantely node1 has been fenced in the meanwhile and after that the problem disappeared.

Tomas,

have customer experienced this issue again? Otherwise i´ll need to close this one. Clearly we can re-open it again if necessary.

We would need to reproduce the problem while running tcpdump and systemtap.
That data would show whether the problem was in the dlm or outside the dlm.

(In reply to Fabio Massimo Di Nitto from comment #12)
> Tomas,
> 
> have customer experienced this issue again? Otherwise i´ll need to close
> this one. Clearly we can re-open it again if necessary.

Fabio/Tomas,

Since we have increased the Token time, and set secauth to off for
the tokens. The cluster seems to be more stable. It has stayed up
since early January with no nodes being fenced off.

I think that this can be closed and if we encounter the issue
again a new ticket can be initiated.

Thanks for you assistance with this bug.
-Tony

Closing based on comment #16.

Please reopen if problem arise again.