Description of problem: Driver and other code that uses the iounmap function may deadlock. This happens because the iounmap contains a code path wherein a writelock is taken, followed by a semaphore, which can lead to a sleep-while-holding spinlock sort of deadlock. See additional info below: Version-Release number of selected component (if applicable): RHEL4.6 (and earlier) How reproducible: Moderately Steps to Reproduce: 1. load/unload a driver that uses ioremap_nocache/iounmap. 2. Put a load on the machine that causes some scheduling activity. 3. Actual results: After a while, some kernel threads will deadlock. Expected results: No deadlock Additional info: This problem is fixed in RHEL5 by dropping the writelock before calling the code that can sleep. Here is the RHEL4 iounmap function: void iounmap(void __iomem *addr) { struct vm_struct *p, **pprev; if (addr <= high_memory) return; write_lock(&vmlist_lock); for (p = vmlist, pprev = &vmlist; p != NULL; pprev = &p->next, p = *pprev) if (p->addr == (void *)(PAGE_MASK & (unsigned long)addr)) break; if (!p) { printk("__iounmap: bad address %p\n", addr); goto out_unlock; } *pprev = p->next; unmap_vm_area(p); if (p->flags >> 20) { /* p->size includes the guard page, but cpa doesn't like that */ ioremap_change_attr(p->phys_addr, (p->size - PAGE_SIZE), 0); } out_unlock: write_unlock(&vmlist_lock); kfree(p); } Note that the vmlist_lock is still locked when ioremap_change_attr is called. ioremap_change_attr calls change_page_attr_addr, which in turn, does this: down_write(&init_mm.mmap_sem); And now we may sleep while holding the vmlist_lock.
This is a very timing-and-scheduling-sensitive bug. It has been around for a long time, and we never hit it through RHEL4.4 and RHEL4.5 until last week. One of our drivers, at system startup, does an ioremap_nocache, followed by an iounmap. Meanwhile, we must have changed something in the system that changes the timing of events such that the init_mm.mmap_sem is taken by some other thread at just the wrong time. Now our system hits this bug frequently on bootup, but changing anything in the way the system starts can make this go away again. So it's hard to assess the exposure acurately.
Created attachment 245981 [details] Patch to fix this problem This patch fixes this problem:
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Simon, can you confirm that the patch below fixes the issue on 4.6? (Since this may get proposed for 4.6.z). Thanks!
Will do patching, rebuilding and will submit to Systems System Test group for verification tomorrow morning.
Good, Stratus Systems Test group have been running the patched 2.6.9-67.0.4 kernel for several hours now, and are unable to repro the problem. Previously, on one system, the problem would occur in about an hour. They will continue tests for 24 hours before giving a definiative sign-off on the patch.
Stratus Systems Test group have run tests successfully for 24 hours without reproducing problem, or encountering any new ones. Therefore, patch is GOOD.
On Friday Feb 15th Vivek G. agreed to include patch in next 4.7 build, with no objections from Larry W. This bug is cloned for inclusion in the 4.6.z stream with: https://bugzilla.redhat.com/show_bug.cgi?id=433267
Committed in 68.12. RPMS are available at http://people.redhat.com/vgoyal/rhel4/
What test procedures did Stratus Systems Test group use to test this?
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2008-0665.html