Bug 280311
Summary: | circular locking in NLM subsystem | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Jeff Layton <jlayton> | ||||||||
Component: | kernel | Assignee: | Jeff Layton <jlayton> | ||||||||
Status: | CLOSED UPSTREAM | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
Severity: | low | Docs Contact: | |||||||||
Priority: | medium | ||||||||||
Version: | rawhide | CC: | dzu, staubach, steved | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | All | ||||||||||
OS: | Linux | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2007-09-22 21:34:01 UTC | Type: | --- | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Attachments: |
|
Description
Jeff Layton
2007-09-06 11:15:50 UTC
I'm not exactly clear on what lockdep is complaining about, but I think it's lock ordering. It had already seen lockd take the f_mutex then the nlm_host_mutex, but apparently the codepath above is now doing this in the reverse order. Created attachment 196301 [details]
kernel log
As an additional data point, I am seeing the same problem with 2.6.22.5-49.fc6debug: ======================================================= [ INFO: possible circular locking dependency detected ] 2.6.22.5-49.fc6debug #1 ------------------------------------------------------- lockd/3100 is trying to acquire lock: (&file->f_mutex){--..}, at: [<c0646a8e>] mutex_lock+0x21/0x24 but task is already holding lock: (nlm_host_mutex){--..}, at: [<c0646a8e>] mutex_lock+0x21/0x24 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (nlm_host_mutex){--..}: [<c0446a92>] __lock_acquire+0x9ea/0xb75 [<c044700f>] lock_acquire+0x56/0x6f [<c06468f6>] __mutex_lock_slowpath+0xf7/0x26e [<c0646a8e>] mutex_lock+0x21/0x24 [<f8ee491c>] nlm_lookup_host+0x8f/0x2d9 [lockd] [<f8ee4b96>] nlmsvc_lookup_host+0x30/0x37 [lockd] [<f8ee6393>] nlmsvc_lock+0xc1/0x2f6 [lockd] [<f8ee9a25>] nlm4svc_proc_lock+0x8b/0xd3 [lockd] [<f8f4abc5>] svc_process+0x336/0x66d [sunrpc] [<f8ee5363>] lockd+0x14c/0x225 [lockd] [<c0405c8f>] kernel_thread_helper+0x7/0x10 [<ffffffff>] 0xffffffff -> #0 (&file->f_mutex){--..}: [<c0446976>] __lock_acquire+0x8ce/0xb75 [<c044700f>] lock_acquire+0x56/0x6f [<c06468f6>] __mutex_lock_slowpath+0xf7/0x26e [<c0646a8e>] mutex_lock+0x21/0x24 [<f8ee6120>] nlmsvc_traverse_blocks+0x22/0x8c [lockd] [<f8ee7418>] nlm_traverse_files+0x1b1/0x20c [lockd] [<f8ee74f5>] nlmsvc_mark_resources+0x27/0x29 [lockd] [<f8ee446c>] nlm_gc_hosts+0x49/0x19b [lockd] [<f8ee4933>] nlm_lookup_host+0xa6/0x2d9 [lockd] [<f8ee4b96>] nlmsvc_lookup_host+0x30/0x37 [lockd] [<f8ee9570>] nlm4svc_retrieve_args+0x38/0xb6 [lockd] [<f8ee99f9>] nlm4svc_proc_lock+0x5f/0xd3 [lockd] [<f8f4abc5>] svc_process+0x336/0x66d [sunrpc] [<f8ee5363>] lockd+0x14c/0x225 [lockd] [<c0405c8f>] kernel_thread_helper+0x7/0x10 [<ffffffff>] 0xffffffff other info that might help us debug this: 1 lock held by lockd/3100: #0: (nlm_host_mutex){--..}, at: [<c0646a8e>] mutex_lock+0x21/0x24 stack backtrace: [<c04061ac>] show_trace_log_lvl+0x1a/0x2f [<c0406c30>] show_trace+0x12/0x14 [<c0406c89>] dump_stack+0x16/0x18 [<c04451c3>] print_circular_bug_tail+0x5f/0x68 [<c0446976>] __lock_acquire+0x8ce/0xb75 [<c044700f>] lock_acquire+0x56/0x6f [<c06468f6>] __mutex_lock_slowpath+0xf7/0x26e [<c0646a8e>] mutex_lock+0x21/0x24 [<f8ee6120>] nlmsvc_traverse_blocks+0x22/0x8c [lockd] [<f8ee7418>] nlm_traverse_files+0x1b1/0x20c [lockd] [<f8ee74f5>] nlmsvc_mark_resources+0x27/0x29 [lockd] [<f8ee446c>] nlm_gc_hosts+0x49/0x19b [lockd] [<f8ee4933>] nlm_lookup_host+0xa6/0x2d9 [lockd] [<f8ee4b96>] nlmsvc_lookup_host+0x30/0x37 [lockd] [<f8ee9570>] nlm4svc_retrieve_args+0x38/0xb6 [lockd] [<f8ee99f9>] nlm4svc_proc_lock+0x5f/0xd3 [lockd] [<f8f4abc5>] svc_process+0x336/0x66d [sunrpc] [<f8ee5363>] lockd+0x14c/0x225 [lockd] [<c0405c8f>] kernel_thread_helper+0x7/0x10 ======================= Detlev, Did you happen to notice whether lockd was hung after seeing this warning? Unfortunately I've not been able to reproduce this since (though I haven't tried too hard)... After asking someone more knowledgeable about lockdep, I think the issue is that the kernel thinks there is a problem with lock ordering. The only takers of these locks seem to be lockd though, so ordering shouldn't really be much of an issue. From the stack traces above, the Supposedly, lockdep can get this wrong sometimes...maybe this is one of those cases? I'd be less inclined to think so if lockd was really hung after these messages popped... Jeff, lockd did not hang after the message. In fact it is still running nicely. From what I understand of lockdep, it warns us that one codepath took the 2 locks already in a different order than what is currently tried - so it _may_ happen that if both these codepaths should ever run concurrently with just the right timing, they deadlock. So even if the lockd on my machine is still running (Pentium IV, so 2 CPUs), that is not really a proof that we have a false positive - the kernel doesn't even use PREEMPT so the conditions may actually be very friendly to that problem. But CONFIG_PREEMPT and more CPUs could possibly provoke the error quickly. Pondering some more about the problem I cannot understand how this can be only harmless. Maybe it is neccessary to order the locks on levels (maybe even introduce "an upper" level) and teach lockdep about that usage (like described in Documentation/lockdep-design.txt) but unfortunately I won't be able to do that :( Thanks for the confirmation. That was my interpretation too. I think I might see what lockdep is complaining about (well, at least one possible ordering violation): First call chain (more or less the one in the stack traces above): nlm4svc_proc_lock nlm4svc_retrieve_args nlmsvc_lookup_host nlm_lookup_host (nlm_host_mutex is taken) nlm_gc_hosts nlmsvc_mark_resources nlm_traverse_files nlm_inspect_file nlmsvc_traverse_blocks (file->f_mutex is taken) ...but later... nlm4svc_proc_lock nlmsvc_lock (file->f_mutex is taken) nlmsvc_create_block nlmsvc_lookup_host nlm_lookup_host (nlm_host_mutex is taken) ...so there does seem to be some ordering violation with this. It's not clear to me that any of these functions are called outside of lockd however. A potential fix might be to rejigger nlm_gc_hosts() such that we don't need to hold the nlm_host_mutex when nlmsvc_mark_resources() is called, but I'd welcome suggestions on this :-) Created attachment 202831 [details]
NLM: Fix a circular lock dependency in lockd
The problem is that the garbage collector for the 'host' structures
nlm_gc_hosts(), holds nlm_host_mutex while calling down to
nlmsvc_mark_resources, which, eventually takes the file->f_mutex.
We cannot therefore call nlmsvc_lookup_host() from within
nlmsvc_create_block, since the caller will already hold file->f_mutex, so
the attempt to grab nlm_host_mutex may deadlock.
Fix the problem by calling nlmsvc_lookup_host() outside the file->f_mutex.
Created attachment 203001 [details]
reproducer program
I've been able to reproduce this more or less at will by running this program
against the same file on the client and server. Eventually the lockdep warnings
pop.
Going to test Trond's patch now and see if it helps...
Patch seems to correct the problem. Closing this with a resolution of UPSTREAM. |