Bug 546700
| Summary: | Deadlock in aio | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Matt Cross <matt.cross> | ||||||||||
| Component: | kernel | Assignee: | Jeff Moyer <jmoyer> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Igor Zhang <yugzhang> | ||||||||||
| Severity: | medium | Docs Contact: | |||||||||||
| Priority: | low | ||||||||||||
| Version: | 5.4 | CC: | cward, jarod, yugzhang | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | All | ||||||||||||
| OS: | Linux | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 587402 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2011-01-13 20:57:32 UTC | Type: | --- | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 587402 | ||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Matt Cross
2009-12-11 18:08:07 UTC
Forgot to mention that there is a workaround - if the application waits for all I/O to complete before calling io_destroy() then the deadlock does not occur. If you run the test app as './aio_test 1 100 /tmp/testfile' it enables the workaround and does not deadlock. Thanks for the nice write-up, Matt. I agree that the commit you referenced would indeed solve the problem.
What happens is that your program closes the file descriptor before calling io_destroy:
if (fd >= 0)
close(fd);
if (ioctx_initted)
io_destroy(ioctx);
Because of this, the last reference to the filp will be dropped in interrupt (I/O completion) context, and the final put on the filp must be scheduled for process context.
Then, any one of the other aio_test threads can call io_destroy and it will trigger the deadlock, since io_destroy calls flush_workqueue, which takes the workqueue_mutex, and that will in turn wait for the aio_fput_routine to run, which does the last put on the ioctx, which in turn calls flush_workqueue for the aio_wq.
I agree, Jeff. Implementing the workaround in my real application showed this - my application was closing the fd and calling io_destroy in two different pthread thread-specific data destructors, and it turned out that the one that did the close was running before the io_destroy. Once I modified my application to always do the close() after the io_destroy(), the hang went away. Let me know if you'd like me to test a fix. I can't actually trigger this hang. Did you test on bare metal, or just in a VM? If the former, what is the I/O performance of your device? While running your test, I'm getting between 150 and 200 MB/s. Maybe my device is too fast to trigger the race? It's also an 8 cpu system, so I tried using the same multiplier as you, 50 threads per cpu. Still, no hang. I know that I've seen the hang on bare metal with my real application, I can't recall if I reproduced it with this test code on bare metal or just on a VM. I'll try it again on bare metal with this test code and I'll let you know what I find. Created attachment 388587 [details] do not call flush_workqueue from wthin a workqueue A kernel with this patch applied can be found here: http://people.redhat.com/jmoyer/aio/rhel5/kernel-2.6.18-186.el5.jmoyer.aio.1.x86_64.rpm Since I can't reproduce this, would you mind giving it a try? Thanks! Created attachment 388642 [details]
improved testcase
I was not able to reproduce the problem on real hardware using my original test case. I tweaked the test case and am now able to reproduce the problem on real hardware.
Jeff, I am testing your kernel and I can't reproduce the problem. However, on reviewing the patch I'm not sure it's correct. Shouldn't the check in aio_fput_routine() be "if (!wq_context)"? As currently implemented it looks like it only does the flush_workqueue() when running in the context of a workqueue handler, I think the right thing is to do the opposite. I think your patch removes the race condition by never calling flush_workqueue() from the mainline code, which is bad because items could still be on the workqueue related to the io context being removed. Boy, how did I miss that? Thanks for pointing out that flaw, you are definitely right. Are you able to build kernels for testing? If so, could you invert that logic and see if the problem is still addressed? Thanks! Sure, I'll put that patch with the corrected logic against 164.6.1 (which is what I have handy) and let you know how it goes. Created attachment 388656 [details]
Updated patch that fixes the logic inversion
I kicked off a build with that patch, but it will be a while before I'm able to upload the kernel to my people page. In the mean time, I'd love to hear your results. Thanks for all of the help! I built a kernel by correcting the original patch (I just added the ! as I described). My test hung after 29 seconds... I'll take a look at it tomorrow and let you know what I find. Never mind, I managed to compile the kernel but not build the RPM, so I was running the old code without your patch. I just reran the test and it looks good. I ran this kernel: http://people.redhat.com/jmoyer/aio/rhel5/kernel-2.6.18-186.el5.jmoyer.aio.2.x86_64.rpm This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. in kernel-2.6.18-225.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0017.html |