From a -128.el5 kernel (though I think the problem started in -127.el5). The message seems to reliably pop during udev init. The kernel here has some extra patches, but they're mostly VFS related stuff -- nothing that involves Xen. ------------------------[snip]------------------------- type=1403 audit(1229615656.565:3): policy loaded auid=4294967295 ses=4294967295 input: PC Speaker as /class/input/input2 GSI 17 sharing vector 0xB1 and IRQ 17 ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 28 (level, low) -> IRQ 177 Xen version 3.1. Hypercall area is 1 pages. Grant table initialized xen_mem: Initialising balloon driver. netfront: Initialising virtual ethernet driver. Registering block device major 3 register_blkdev: cannot get major 3 for ide xen_blk: can't get major 3 with name ide vbd vbd-768: 19 xlvbd_add at /local/domain/0/backend/vbd/8/768 Registering block device major 3 register_blkdev: cannot get major 3 for ide xen_blk: can't get major 3 with name ide vbd vbd-768: 19 xlvbd_add at /local/domain/0/backend/vbd/8/768 netfront: device eth0 has copying receive path. XENBUS: Timeout connecting to device: device/vbd/768 (state 6) ======================================================= [ INFO: possible circular locking dependency detected ] 2.6.18-128.el5.jtltest.57debug #1 ------------------------------------------------------- modprobe/915 is trying to acquire lock: (xenwatch_mutex){--..}, at: [<ffffffff88170b01>] unregister_xenbus_watch+0x10c/0x121 [xen_platform_pci] but task is already holding lock: (old_style_spin_init#20){--..}, at: [<ffffffff881710ad>] xvd_dev_shutdown+0x1a/0x8a [xen_platform_pci] which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (old_style_spin_init#20){--..}: [<ffffffff800a7cf9>] __lock_acquire+0x9a9/0xadf [<ffffffff800a85ef>] lock_acquire+0x55/0x70 [<ffffffff881711ce>] xendev_rem_add+0xb1/0x10b [xen_platform_pci] [<ffffffff80068576>] _spin_lock+0x26/0x52 [<ffffffff881711ce>] xendev_rem_add+0xb1/0x10b [xen_platform_pci] [<ffffffff8819f3eb>] xlvbd_add+0x48a/0x4af [xen_vbd] [<ffffffff800a251d>] keventd_create_kthread+0x0/0xc9 [<ffffffff8819edbc>] backend_changed+0xb0/0x192 [xen_vbd] [<ffffffff88172129>] xenbus_read_driver_state+0x26/0x3b [xen_platform_pci] [<ffffffff88170e4d>] xenwatch_thread+0x0/0x140 [xen_platform_pci] [<ffffffff8817027f>] xenwatch_handle_callback+0x15/0x48 [xen_platform_pci] [<ffffffff88170f74>] xenwatch_thread+0x127/0x140 [xen_platform_pci] [<ffffffff800a2734>] autoremove_wake_function+0x0/0x2e [<ffffffff800a251d>] keventd_create_kthread+0x0/0xc9 [<ffffffff80034306>] kthread+0xfe/0x132 [<ffffffff8006808c>] trace_hardirqs_on_thunk+0x35/0x37 [<ffffffff80061079>] child_rip+0xa/0x11 [<ffffffff8006898d>] _spin_unlock_irq+0x24/0x27 [<ffffffff800606a8>] restore_args+0x0/0x30 [<ffffffff80034208>] kthread+0x0/0x132 [<ffffffff8006106f>] child_rip+0x0/0x11 [<ffffffffffffffff>] 0xffffffffffffffff -> #0 (xenwatch_mutex){--..}: [<ffffffff800a7c0f>] __lock_acquire+0x8bf/0xadf [<ffffffff88170b01>] unregister_xenbus_watch+0x10c/0x121 [xen_platform_pci] [<ffffffff800a85ef>] lock_acquire+0x55/0x70 [<ffffffff88170b01>] unregister_xenbus_watch+0x10c/0x121 [xen_platform_pci] [<ffffffff800672c6>] mutex_lock_nested+0x104/0x29c [<ffffffff88170b01>] unregister_xenbus_watch+0x10c/0x121 [xen_platform_pci] [<ffffffff88171298>] free_otherend_watch+0x14/0x27 [xen_platform_pci] [<ffffffff88171f9f>] xenbus_dev_remove+0x19/0x46 [xen_platform_pci] [<ffffffff801c5247>] __device_release_driver+0xa2/0xc6 [<ffffffff801c555b>] device_release_driver+0x3d/0x5f [<ffffffff801c4a0c>] bus_remove_device+0x9b/0xb0 [<ffffffff801c36a2>] device_del+0x13a/0x1ba [<ffffffff801c373f>] device_unregister+0x9/0x12 [<ffffffff881710e8>] xvd_dev_shutdown+0x55/0x8a [xen_platform_pci] [<ffffffff88190014>] xlblk_init+0x14/0x18 [xen_vbd] [<ffffffff800adf3f>] sys_init_module+0xb1/0x1ee [<ffffffff80060116>] system_call+0x7e/0x83 [<ffffffffffffffff>] 0xffffffffffffffff other info that might help us debug this: 1 lock held by modprobe/915: #0: (old_style_spin_init#20){--..}, at: [<ffffffff881710ad>] xvd_dev_shutdown+0x1a/0x8a [xen_platform_pci] stack backtrace: Call Trace: [<ffffffff800a65f8>] print_circular_bug_tail+0x65/0x6e [<ffffffff800a7c0f>] __lock_acquire+0x8bf/0xadf [<ffffffff88170b01>] :xen_platform_pci:unregister_xenbus_watch+0x10c/0x121 [<ffffffff800a85ef>] lock_acquire+0x55/0x70 [<ffffffff88170b01>] :xen_platform_pci:unregister_xenbus_watch+0x10c/0x121 [<ffffffff800672c6>] mutex_lock_nested+0x104/0x29c [<ffffffff88170b01>] :xen_platform_pci:unregister_xenbus_watch+0x10c/0x121 [<ffffffff88171298>] :xen_platform_pci:free_otherend_watch+0x14/0x27 [<ffffffff88171f9f>] :xen_platform_pci:xenbus_dev_remove+0x19/0x46 [<ffffffff801c5247>] __device_release_driver+0xa2/0xc6 [<ffffffff801c555b>] device_release_driver+0x3d/0x5f [<ffffffff801c4a0c>] bus_remove_device+0x9b/0xb0 [<ffffffff801c36a2>] device_del+0x13a/0x1ba [<ffffffff801c373f>] device_unregister+0x9/0x12 [<ffffffff881710e8>] :xen_platform_pci:xvd_dev_shutdown+0x55/0x8a [<ffffffff88190014>] :xen_vbd:xlblk_init+0x14/0x18 [<ffffffff800adf3f>] sys_init_module+0xb1/0x1ee [<ffffffff80060116>] system_call+0x7e/0x83
Don is working on some patches for 5.4 that should remove this code path completely. Assuming that all works out, this should go away. We'll leave it open until then, though, just to make sure. Chris Lalancette
Created attachment 332936 [details] Proposed, to-be posted patch Attached is proposed patch. brew built. based on backport of 4.8. This *removes* the hacks for anaconda installation failures when xen-platform-pci.ko was built into the (bare-metal, FV) kernel. With xen-platform-pci.ko now a loadable, and not part of the 5.3-anaconda initrd, the anaconda failure cannot occur. The removal of the hacky code gets rid of the debug-kernel warning, which pops up due to the extra code. Will report back on testing results later today.
Tested the following with the patch & all worked well: (a) x86_64 FV; block attach & detach multiple times (b) x86_64 pv kernel; block attach & detach multiple times (c) i686 FV kernel (d) x86_64 -debug kernel -- none of the debug messages listed in Description of problem / bug
Patch in c#3 posted for 5.4 inclusion.
in kernel-2.6.18-133.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1243.html