Bug 246586 - Ooops in queue_work on boot.
Summary: Ooops in queue_work on boot.
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel-xen   
(Show other bugs)
Version: 4.5
Hardware: i686 Linux
Target Milestone: ---
: ---
Assignee: Rik van Riel
QA Contact: Martin Jenner
Depends On:
TreeView+ depends on / blocked
Reported: 2007-07-03 08:47 UTC by Ian Campbell
Modified: 2007-11-17 01:14 UTC (History)
1 user (show)

Fixed In Version: RHBA-2007-0791
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-11-15 16:29:48 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2007:0791 normal SHIPPED_LIVE Updated kernel packages available for Red Hat Enterprise Linux 4 Update 6 2007-11-14 18:25:55 UTC

Description Ian Campbell 2007-07-03 08:47:34 UTC
If a timer interrupt is received between time_init() and init_workqueues() and
HYPERVISOR_shared_info->wc_version ticks over and causes clock_was_set() to be
called from timer_interrupt() the kernel can oops because clock_was_set() will
attempt to defer work to the keventd_wq which has not yet been initialised. 

        Checking if this processor honours the WP bit even in supervisor mode... Ok.
        printing eip:
        02669000 -> *pde = 00000000:00000000
        Oops: 0000 [#1]
        Modules linked in:
        CPU:    0
        EIP:    0061:[<c012a724>]    Not tainted VLI
        EFLAGS: 00010046   (2.6.9-55.ELxenU)
        EIP is at queue_work+0x21/0x53
        eax: 00001004   ebx: 00000000   ecx: 00000000   edx: c029b600
        esi: 00000000   edi: 00000000   ebp: c0338f6c   esp: c0338f58
        ds: 007b   es: 007b   ss: 0068
        Process swapper (pid: 0, threadinfo=c0338000 task=c0297a40)
        Stack: 006fae09 00000000 c012dff0 00000000 006fae09 c0338f6c c0338f6c
               00000000 00000000 00000000 c010dbd0 00cf1f60 00000000 00000000
               c10000c0 c1000080 00000000 00000000 45cf494b 0000c74f 00000000
        Call Trace:
         [<c012dff0>] clock_was_set+0x2d/0x180
         [<c010dbd0>] timer_interrupt+0x26e/0x3f5
         [<c01094aa>] handle_IRQ_event+0x44/0x85
         [<c0109a38>] do_IRQ+0x122/0x1b5
         [<c01f7150>] evtchn_do_upcall+0x84/0xb8
         [<c0107538>] hypervisor_callback+0x2c/0x34
         [<c01020e6>] calibrate_delay+0xe3/0x1a8
         [<c02f36f3>] start_kernel+0x14f/0x1b6
        Code: 89 fa 5b 5e 5f e9 73 ea 13 00 56 31 f6 53 89 c3 b8 00 f0 ff ff 21
e0 8b 48 10 f0 0f ba 2a 00 19 c0 85 c0 75 33 8d 83 04 10 00 00 <39> 83 04 10 00
00 8d 42 04 0f 44 ce 39 42 04 74 08 0f 0b 68 00
         <0>Kernel panic - not syncing: Fatal exception in interrupt

It happens quite rarely but can be made to trigger quite reliably by adding a
very large (multiple tens of seconds) delay after or during calibrate_delay() to
increase the window of opportunity.

The fix is in:

The indirection via the workqueue is not strictly necessary in 2.6.9 since
clock_was_set() will do this itself. It is there because we wanted the fix to
apply to a wide variety of kernels.

Ian Campbell, XenSource.

Comment 1 Rik van Riel 2007-07-03 14:30:38 UTC
In RHEL4 things look like this:

void clock_was_set(void)
        struct k_itimer *timr;
        struct timespec new_wall_to;
        unsigned long seq;

        if (unlikely(in_interrupt())) {

This seems to indicate that your patch will result in the kernel doing exactly 
what it is doing today, but maybe a little more efficiently.  Your patch also 
makes a possible (though highly unlikely in RHEL4) future change to 
clock_was_set() safe.

However, I do not understand how your patch fixes the bug.  If schedule_work 
is called before keventd_wq is initialized, surely it does not matter whether 
schedule_work is called from clock_was_set or directly from timer_interrupt?

What exactly is going on here?

Comment 2 RHEL Product and Program Management 2007-07-03 14:35:39 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update

Comment 3 Ian Campbell 2007-07-03 14:45:39 UTC
The important bit of the patch is the addition of the "if (keventd_up())" which
prevents clock_was_set() (or in this case schedule_work() directly) from being
called before keventd_wq is set.

The workqueue added by the patch is indeed unecessary for the 2.6.9 kernel since
clock_was_set() does the same thing itself. It's just there so the patch is
useful on a variety of kernels (e.g. in 2.6.21 clock_was_set() cannot be called
from interrupt context and does not defer itself). If you wanted you could
simplify to:
   if (keventd_up()) clock_was_set();

Comment 4 Rik van Riel 2007-07-03 14:57:40 UTC
Good point, doh!

Thanks for this patch Ian, I'll try to get it folded into the RHEL 4.6 tree 

Comment 5 Rik van Riel 2007-07-03 15:49:36 UTC
I have posted the patch on our internal kernel mailing list.

Comment 6 Red Hat Bugzilla 2007-07-25 00:53:32 UTC
change QA contact

Comment 7 Jason Baron 2007-07-25 14:32:20 UTC
committed in stream U6 build 55.22. A test kernel with this patch is available
from http://people.redhat.com/~jbaron/rhel4/

Comment 11 errata-xmlrpc 2007-11-15 16:29:48 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.