Bug 502531
Summary: | GFS2: smbd proccess hangs with flock() call. | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 5 | Reporter: | Flávio do Carmo Júnior <flaviocj> | ||||||||||
Component: | kernel | Assignee: | Abhijith Das <adas> | ||||||||||
Status: | CLOSED ERRATA | QA Contact: | Cluster QE <mspqa-list> | ||||||||||
Severity: | medium | Docs Contact: | |||||||||||
Priority: | low | ||||||||||||
Version: | 5.4 | CC: | bmarzins, cluster-maint, cward, czhang, dzickus, edamato, flaviocj, gdeschner, lwang, nstraz, obnox, rpeterso, ryan.suarez, sbose, ssorce, swhiteho, tao, teigland | ||||||||||
Target Milestone: | --- | ||||||||||||
Target Release: | --- | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Linux | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2010-03-30 07:40:05 UTC | Type: | --- | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 526775, 526947, 533192 | ||||||||||||
Attachments: |
|
Description
Flávio do Carmo Júnior
2009-05-25 21:56:17 UTC
I doubt that smbstatus triggers the hanging smbd. It rather uncovers the existing problem, because smbstatus tries to walk the locks and can't pass the lock held by the hanging smbd. That smbd seems to try to perform an operation that fails under the lock. Michael Created attachment 345533 [details]
strace, sysrq-t, gfs2_tool lockdumps
Here is the data into the package:
On one node:
# strace -tt -s 256 -v -o /tmp/strace-$(uname -n).out smbstatus
Where smbstatus is the command I've been using to cause the hang. Now while it is in the hung state, I do this from every node:
# echo 't' > /proc/sysrq-trigger
# gfs2_tool lockdump <mountpoint> > /tmp/lockdump-<mountpoint>-$(uname -n).out
# echo 't' > /proc/sysrq-trigger
So is attached:
- strace*.out
- /var/log/messages
- lockdump-geral-*.out
If there is anything more to help you, please let me know.
Created attachment 345534 [details]
output of net conf list = samba conf
Ok, I've noticed that using that config attached for samba services the GFS2 distributed lock wasn't working correctly, as you can see on this paste: [root@athos samba-3.3.4]# smbstatus | grep cap-04 3:3863 1522 DENY_WRITE 0x2019f RDWR NONE /geral/troca Backup-Recuperados/juliana/cap-04.odt Wed May 27 12:32:46 2009 0:27752 1442 DENY_WRITE 0x2019f RDWR NONE /geral/troca Backup-Recuperados/juliana/cap-04.odt Wed May 27 12:32:09 2009 I'm with 2 files opened on two different nodes and both have RDWR oplock. Samba community advise me to use, on smb.conf: fileid:algorithm = fsname vfs objects = fileid This solves THAT (oplocks, dlm) problem. I don't know if this is relative for GFS2 and if it is going to be the solution of the problem, but I tought it could help. Again, samba community guide me to disable flock() calls for smbd (smbd/open.c) and recompile it. Until this moment is everything working as expected, without zombie process. But the system is working a few hours. This seems to be a WORKAROUND for the issue. I'll wait some days to post it as workaround. I don't think I understand the issue from the above reports. Can you state clearly what the problem is? There is nothing in the strace attached to indicate any issue with flock that I can see. I didn't think that smb used flock anyway. Hi Steve, Well, the problem is that I get a zombie smbd process after a few hours using Samba+CTDB with GFS2. I can't reproduce the problem now, once I've patched smbd/open.c file to test the workaround proposed by a member of samba team, IIRC the flock() call on open.c is just for use with GPFS and will not hurt have it disabled when using another filesystem. A filtered output of "echo 't' > /proc/sysrq-trigger" while a zombie smbd process was running can be view at this paste: http://pastebin.ca/1436324 If you need more info, just let me know and I'll try to provide it. As the servers are in production use I can't play much with it but I'll try to create an equal situation to reproduce this situation. (In reply to comment #6) > I don't think I understand the issue from the above reports. Can you state > clearly what the problem is? There is nothing in the strace attached to > indicate any issue with flock that I can see. I didn't think that smb used > flock anyway. Samba normally does not use flock(), if I recall correctly this flock is a GPFS specific optimization that have do do with share modes. It's not strictly required, but it seem to show we have a problem with gfs2 if a process get stuck in D state when using it. Hi folks, as promised I'm sending that patch to comment flock() call into smbd/open.c for samba-3.3.4 version. Follow as plain text here, and I'll attach it too. diff -Nur samba-3.3.4.orig/source/smbd/open.c samba-3.3.4.noflock/source/smbd/open.c --- samba-3.3.4.orig/source/smbd/open.c 2009-05-28 14:21:55.000000000 -0300 +++ samba-3.3.4.noflock/source/smbd/open.c 2009-05-27 11:55:59.000000000 -0300 @@ -2005,7 +2005,7 @@ locking database for permission to set this deny mode. If the kernel refuses the operations then the kernel is wrong. note that GPFS supports it as well - jmcd */ - +/* if (fsp->fh->fd != -1) { ret_flock = SMB_VFS_KERNEL_FLOCK(fsp, share_access); if(ret_flock == -1 ){ @@ -2016,7 +2016,7 @@ return NT_STATUS_SHARING_VIOLATION; } } - +*/ /* * At this point onwards, we can guarentee that the share entry * is locked, whether we created the file or not, and that the Created attachment 345954 [details]
This is a workaround for the problem described here.
As mentioned before, this is not a solution for the bug, this is a Workaround to use samba-3.3.4+ctdb with GFS2 filesystem. What the patch does is just comment a flock() call for smbd when opening files.
Created attachment 346051 [details]
Patch to return error on LOCK_MAND
Browsing through the smbd code, I noticed that it uses LOCK_MAND with LOCK_READ, LOCK_WRITE or LOCK_RW instead of the LOCK_SH and LOCK_EX that gfs2 recognizes.
GFS2's locking is undefined (probably erroneous) with LOCK_MAND+friends and this could be the reason why smdb hangs.
According to Steve Whitehouse, this should've been caught by a condition check on the setgid (S_ISGID) bit on the inode in question, as a LOCK_MAND should not be issued against an inode that doesn't have this bit set.
This patch removes the check against S_ISGID bit of the inode->i_mode and directly checks for LOCK_MAND in fl->fl_type.
Could you verify that this is indeed the problem? If so, this patch should gracefully return an error code instead of crashing smbd.
Thanks!
--Abhi
Hi Abhi, OK. I'll test this changes and post results. --Flavio The patch for this is upstream. I think we should go ahead and post this for 5.5 and if we get some positive feedback about it, we can also dup for 5.4 (maybe .z depending on timing). Hi, I'm using this patch without any problem. As we've (Abhi and me) talked with 'vl' from Samba team, samba code doesn't any verification about -EOPNOTSUPP at flock() call and so this works as that samba workaround. Since 'vl' could think this is a "bug" by samba, probably we need to know how samba will treat with this on newer versions. As I said on IRC, I'd needed to remove my users from system, was really unstable. So, now I've no more than 10 users doing tests on it. Even with none of this patches/changes, 10 users doesn't seem to be a problem. I don't know how to test this setup now, I can't put all my users to work on it, maybe is possible to write some benchmark/testing tool to simulate users access. I'll be glad to use it on my testing setup. Posted patch in comment #11 to rhkernel-list for inclusion in RHEL 5.5 This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. in kernel-2.6.18-169.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified. ~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~ RHEL 5.5 Beta has been released! There should be a fix present in this release that addresses your request. Please test and report back results here, by March 3rd 2010 (2010-03-03) or sooner. Upon successful verification of this request, post your results and update the Verified field in Bugzilla with the appropriate value. If you encounter any issues while testing, please describe them and set this bug into NEED_INFO. If you encounter new defects or have additional patch(es) to request for inclusion, please clone this bug per each request and escalate through your support representative. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0178.html The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |