Description of problem: Using the system ibm-firefly.lab.boston.redhat.com, i386, RHEL5-Server-GOLD while running the kernel smoke tests. Test /kernel/stress/racer causes the following BUG: Version-Release number of selected component (if applicable): 2.6.21-28.el5rt How reproducible: Unknown Actual results: BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-18/0x00000000/465 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= Expected results: Ssytem should not BUG: Additional info: Note: This is _not_ the same issue as: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236375 and https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=237463
puma1.lab.boston.redhat.com, i386, RHEL5-Server-GOLD http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=169826 BUG: scheduling with irqs disabled: IRQ-14/0x00000000/422 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-14/0x00000000/422 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 =======================
ibm-defiant.rhts.boston.redhat.com, i386, RHEL5-Server-GOLD http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=169920 BUG: scheduling with irqs disabled: IRQ-16/0x00000000/507 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-16/0x00000000/507 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 =======================
dl585-01.rhts.boston.redhat.com, i386, RHEL5-Server-GOLD http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=169915 BUG: scheduling with irqs disabled: IRQ-17/0x00000000/677 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-17/0x00000000/677 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c06310db>] schedule+0x86/0xfa [<c06311b4>] wait_for_completion+0x65/0x9b [<c0423775>] set_cpus_allowed+0x6e/0x8c [<c042d3e1>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459300>] do_irqd+0x202/0x24e [<c043a00b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 =======================
Running test /kernel/stress/racer on i386 groucho.lab.boston.redhat.com RHEL5-Server-GOLD BUG: scheduling with irqs disabled: IRQ-20/0x00000000/457 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c0630c3b>] schedule+0x86/0xfa [<c0630d14>] wait_for_completion+0x65/0x9b [<c04236b5>] set_cpus_allowed+0x6e/0x8c [<c042d321>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459240>] do_irqd+0x202/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-20/0x00000000/457 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c0630c3b>] schedule+0x86/0xfa [<c0630d14>] wait_for_completion+0x65/0x9b [<c04236b5>] set_cpus_allowed+0x6e/0x8c [<c042d321>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459240>] do_irqd+0x202/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= irq 16: nobody cared (try booting with the "irqpoll" option) [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c04592e8>] __report_bad_irq+0x39/0x79 [<c04594b4>] note_interrupt+0x18c/0x1bc [<c0458aa6>] thread_simple_irq+0x5e/0x6c [<c04590f6>] do_irqd+0xb8/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= handlers: [<c058eb76>] (usb_hcd_irq+0x0/0x52) irq 16: nobody cared (try booting with the "irqpoll" option) [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c04592e8>] __report_bad_irq+0x39/0x79 [<c04594b4>] note_interrupt+0x18c/0x1bc [<c0458aa6>] thread_simple_irq+0x5e/0x6c [<c04590f6>] do_irqd+0xb8/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= handlers: [<c058eb76>] (usb_hcd_irq+0x0/0x52) BUG: scheduling with irqs disabled: IRQ-20/0x00000000/457 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c0630c3b>] schedule+0x86/0xfa [<c0630d14>] wait_for_completion+0x65/0x9b [<c04236b5>] set_cpus_allowed+0x6e/0x8c [<c042d321>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459240>] do_irqd+0x202/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= BUG: scheduling with irqs disabled: IRQ-20/0x00000000/457 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c0630c3b>] schedule+0x86/0xfa [<c0630d14>] wait_for_completion+0x65/0x9b [<c04236b5>] set_cpus_allowed+0x6e/0x8c [<c042d321>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459240>] do_irqd+0x202/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= irq 16: nobody cared (try booting with the "irqpoll" option) [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c04592e8>] __report_bad_irq+0x39/0x79 [<c04594b4>] note_interrupt+0x18c/0x1bc [<c0458aa6>] thread_simple_irq+0x5e/0x6c [<c04590f6>] do_irqd+0xb8/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 ======================= handlers: [<c058eb76>] (usb_hcd_irq+0x0/0x52) BUG: scheduling with irqs disabled: IRQ-20/0x00000000/457 caller is wait_for_completion+0x65/0x9b [<c0405fec>] dump_trace+0x5f/0x107 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f [<c04066d0>] show_trace+0x12/0x14 [<c040675a>] dump_stack+0x16/0x18 [<c0630c3b>] schedule+0x86/0xfa [<c0630d14>] wait_for_completion+0x65/0x9b [<c04236b5>] set_cpus_allowed+0x6e/0x8c [<c042d321>] do_softirq_from_hardirq+0xb7/0xd5 [<c0459240>] do_irqd+0x202/0x24e [<c0439f4b>] kthread+0xb3/0xdc [<c0405c03>] kernel_thread_helper+0x7/0x10 =======================
This is a regression from my fix to keep hardirqs pinned to local cpu. I'll work on a fix on Monday for it.
Patch sent for inclusion
Patch applied; fixed