Thank you for your bug report. This issue was evaluated for inclusion
in the current release of Red Hat Enterprise Linux. Unfortunately, we
are unable to address this request in the current release. Because we
are in the final stage of Red Hat Enterprise Linux 6 development, only
significant, release-blocking issues involving serious regressions and
data corruption can be considered.

If you believe this issue meets the release blocking criteria as
defined and communicated to you by your Red Hat Support representative,
please ask your representative to file this issue as a blocker for the
current release. Otherwise, ask that it be evaluated for inclusion in
the next minor release of Red Hat Enterprise Linux.

We need to know if this happens with:

a) firewall disabled, and
b) selinux in permissive mode.

I don't understand this part of the report:

> However, the custmoer says the fencing did work:"After NodeB has been
> successfully fenced, we try to reboot it." 

The fencing appears to be ilo and power fencing, both of which should be able to reboot the node, so if fencing was successful, why did the node not reboot automatically?

Also, with respect to the log messages:

> Sep 28 10:13:25 lilc052b gfs_controld[7791]: mount_done: lt_products not found
> Sep 28 10:13:25 lilc052b gfs_controld[7791]: do_leave: lt_products not found

The mount done message is sent to gfs_controld by mount.gfs2 after the mount syscall has returned in order to inform it of the result. gfs_controld is complaining that it has no record of that particular mount group, so chances are that something has failed after the original join of the mount group, or maybe the join failed but was not reported for some reason.

Output from gfs_control dump etc. from a stuck cluster would help us figure out what is going on here.

Can we please get answers to the questions in https://bugzilla.redhat.com/show_bug.cgi?id=639025#c5 ?

If this is still an issue, we would like to try and fix it. If no update or current information is posted, we will likely close this BZ this week.

Thanks!

the issue occured during an internal qa-test of rhel6 and rhcs including gfs2. 
I'm sorry, but I cannot provide any more information, as this project is already closed.
Nevertheless, in my opinion this issue needs further investigation. In our tests the filesystem on the surviving node was unresponive for more than 30 minutes.

Mark, I have a hunch about the causes of this which you might be able to help me answer. When you saw this happen, did the nodes in question have a large amount of RAM (say >32G) and is it likely that the nodes were caching a large number of inodes (based on the number of inodes on the fs, and whether each node is likely to have accessed a large number of them just before the issue occurred)

Assuming that both of these things are true, then we have another report which may have provided the clue required in order to fix this.

I'd still be rather wary of saying that particular problem was the one originally reported though, without a bit more information from the original reporter, but it does seem likely.

Steve,
I can confirm your guesses. The nodes have a large amount of RAM (>32G) and they where both caching a large number of inodes on the fs, as the failover test has been done during a file creation test on both nodes.
Best Regards,
Mark

Mark, if we give you a test kernel, are you in a position to run a test to see whether it resolves the issue?

Steve,
I'm sorry, but I don't have the required test setup available at the moment.

Ok, thats not a problem. We are definitely working on this and we'll be testing internally anyway to try and ensure that we resolve the issue. I believe we have a fix, we just need to verify that there are no side-effects to it.

Be advised that there is not even a remote possibility that the patch in comment 14 is acceptable.

Dave, do you have an alternative patch in mind or revisions to the patch proposed in https://bugzilla.redhat.com/show_bug.cgi?id=639025#c14?

I just found out from Steve what the specific performance issue was.  Now that I know that, I can develop a patch targetted at the specific issue.  Steve's patch fundamentally changes the entire design of the dlm -- it's like destroying your house and building a new one because a light bulb burned out.

I do not believe Steve's comments (and my replies) are related to whatever problem the customer was having.  Steve's concern is about slow lock recovery.
That issue belongs in a different bug.

The reported problems seem to have been related to gfs_controld, but without more information we cannot say what the problem was, so this will have to wait for more information.

We have had no further updates from the customer, and we've not got a reproducer. I think we have little option but to close this. If it happens again, or we have some further insight into what happened, then please reopen.