Red Hat Bugzilla – Bug 506140
GFS2: Filesystem deadlock when running SPECsfs on BIGI test bed.
Last modified: 2009-09-03 10:09:41 EDT
Description of problem: Running the SPECsfs workload from bz #504335 on a single node GFS2 filesystem on the BIGI test bed occassionally causes the machine to hang. Version-Release number of selected component (if applicable): kernel-2.6.18-152.el5 How reproducible: Occasionally on the BIGI test bed. Has not been reproduced elsewhere. Additional info: Looking at the stack traces and glock dumps, it appears that the main problem is a process that is stuck waiting for a resource group lock. However the glock is in a compatible state, so it would be granded to that process if the glock was processed. However, for some reason, nothing has run through the holder queue on the glock to promote it.
Created attachment 347978 [details] crash and glock info The important process is pid 15509. All the other stuck processes are waiting for a glock that it it holding, and it is waiting for glock (3/8034), which nobody is holding. All of the glock_workqueue processes are idle. So there is nothing to run the glock queue.
A new potential blocker that fell out of debugging Barry's original test case.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Created attachment 348028 [details] debugging patch This patch will keep a list of enqueues, dequeues and calls to glock_work_func. It stores the last 2048 actions per filesystem. Hopefully this will let us narrow down why the glock isn't getting promoted.
If this bug is a dup of the other one, can we close it as such?
I thought we could keep the original bug open for the performance issue, and track the actual hang with this one. We might want to open one more bug for the panic.
Steve found a place where this hang can happen. gfs2_shrink_glock_memory() locks the glock's GLF_LOCK bit, but doesn't always call reschedule a glock_workqueue process to perform the promotions related t the glock. If a glock_workqueue process tries to work on the glock while it is locked by gfs2_shrink_glock_memory(), it will see the that GLF_LOCK bit is locked and assume that whoever locked it is going to deal with the lock themselves. I believe that Steve is working on a patch to fix this and keep the iopen glocks off the lru list to help solve 504335.
Created attachment 349021 [details] Patch to always queue work when we lock GLF_LOCK
Posted
in kernel-2.6.18-156.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified.
Patch is in -158.el5. Adding SanityOnly.
Patch was tested thoroughly in -156 with postmark. We also tested SPECsfs and saw no deadlock condition. Barry
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1243.html