Hide Forgot
Description of problem: This is on a 4 X 4 core Opteron system with 64GB of ram (with NFS mounted home directories). The kernel version is 3.1.7-1.fc16.x86_64. After couple of days uptime & heavy load, I see the following dmesg (the systems appears to run OK though): [1389863.137022] INFO: rcu_bh_state detected stall on CPU 9 (t=0 jiffies) [1389863.137022] sending NMI to all CPUs: [1389863.137135] NMI backtrace for cpu 0 [1389863.137144] CPU 0 [1389863.137147] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137192] [1389863.137196] Pid: 6579, comm: imag_time Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137202] RIP: 0033:[<000000000047d57a>] [<000000000047d57a>] 0x47d579 [1389863.137213] RSP: 002b:00007f7f23f36bb8 EFLAGS: 00000287 [1389863.137216] RAX: 00007f80099f8090 RBX: 00007f80099f8010 RCX: 00007f7f1c008d60 [1389863.137218] RDX: 0000000000000080 RSI: 0000000000000008 RDI: 3a647adc1d780861 [1389863.137221] RBP: 0000000000000010 R08: bb28e8f0e4a82a0e R09: 0000000000001060 [1389863.137223] R10: 000000000000001a R11: 0000000000100000 R12: 00007f7f1c000a60 [1389863.137225] R13: 0000000000000010 R14: 0000000000000100 R15: 000000000047d820 [1389863.137228] FS: 00007f7f23f37700(0000) GS:ffff880417c00000(0000) knlGS:000000005574c6c0 [1389863.137231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137233] CR2: 00000000010d2000 CR3: 000000040b2b4000 CR4: 00000000000006f0 [1389863.137236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137238] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137242] Process imag_time (pid: 6579, threadinfo ffff88040c7f8000, task ffff88040bd4dcc0) [1389863.137244] [1389863.137245] Call Trace: [1389863.137260] NMI backtrace for cpu 3 [1389863.137267] CPU 3 [1389863.137269] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137315] [1389863.137320] Pid: 24480, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137327] RIP: 0033:[<0000000001114821>] [<0000000001114821>] 0x1114820 [1389863.137339] RSP: 002b:00007fffe855d8e0 EFLAGS: 00000287 [1389863.137341] RAX: 00000000000023d2 RBX: 0000000000000060 RCX: 0000000000000052 [1389863.137344] RDX: 0000000000000520 RSI: 00000000155d6740 RDI: 0000000000262e00 [1389863.137346] RBP: 0000000023ec2c70 R08: 0000000023ec2c88 R09: 0000000023ec2c40 [1389863.137349] R10: 0000000023ec2c10 R11: 0000000023ec2bf8 R12: 0000000000000cba [1389863.137351] R13: 0000000023ec2c28 R14: 0000000023ec2c58 R15: 0000000024125488 [1389863.137354] FS: 0000000012f38870(0063) GS:ffff880417d80000(0000) knlGS:000000005574c6c0 [1389863.137357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137360] CR2: 0000000001e8ff0a CR3: 00000008731f5000 CR4: 00000000000006e0 [1389863.137363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137365] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137369] Process molprop_2010_1_ (pid: 24480, threadinfo ffff880c37892000, task ffff880c68324590) [1389863.137371] [1389863.137373] Call Trace: [1389863.137390] NMI backtrace for cpu 2 [1389863.137397] CPU 2 [1389863.137399] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137448] [1389863.137453] Pid: 19427, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137460] RIP: 0033:[<0000000001843458>] [<0000000001843458>] 0x1843457 [1389863.137474] RSP: 002b:00007fff70e48040 EFLAGS: 00000283 [1389863.137476] RAX: 00000000e4777948 RBX: 00000000e4408948 RCX: 00000000000002e8 [1389863.137479] RDX: 00000000e4776948 RSI: 00000000000006d8 RDI: 00000000e4774948 [1389863.137482] RBP: 00000000e4775948 R08: 0000000000004000 R09: 0000000000000800 [1389863.137485] R10: 00000000229b6dc0 R11: 00000000229a92c0 R12: 00000000229ac980 [1389863.137487] R13: 0000000000174000 R14: 000000000111c000 R15: 00000000e48eb948 [1389863.137491] FS: 000000001488e870(0063) GS:ffff880417d00000(0000) knlGS:000000005574c6c0 [1389863.137494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137496] CR2: 0000000036e3f058 CR3: 0000000870a0f000 CR4: 00000000000006e0 [1389863.137500] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137502] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137506] Process molprop_2010_1_ (pid: 19427, threadinfo ffff88040ca0e000, task ffff88040bcdc590) [1389863.137508] [1389863.137510] Call Trace: [1389863.137529] NMI backtrace for cpu 5 [1389863.137537] CPU 5 [1389863.137539] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137590] [1389863.137595] Pid: 6581, comm: imag_time Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137602] RIP: 0033:[<000000000047d567>] [<000000000047d567>] 0x47d566 [1389863.137616] RSP: 002b:00007f7f22f34bd8 EFLAGS: 00000287 [1389863.137618] RAX: 00007f7f04027050 RBX: 00007f7f04001270 RCX: 00007f8011ac2260 [1389863.137621] RDX: 0000000000000080 RSI: 0000000000000025 RDI: baf1b7380f670811 [1389863.137624] RBP: 0000000000100000 R08: 3d603aaeee0d1b7c R09: 0000000000000010 [1389863.137626] R10: 000000000000009b R11: 0000000000000010 R12: 00007f8011ac2010 [1389863.137629] R13: 0000000000001060 R14: 0000000000000100 R15: 000000000047d8a0 [1389863.137633] FS: 00007f7f22f35700(0000) GS:ffff880817c80000(0000) knlGS:000000005574c6c0 [1389863.137636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137638] CR2: 00007f7f0c03f000 CR3: 000000040b2b4000 CR4: 00000000000006e0 [1389863.137642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137645] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137649] Process imag_time (pid: 6581, threadinfo ffff88040c844000, task ffff88040c462e60) [1389863.137651] [1389863.137653] Call Trace: [1389863.137676] NMI backtrace for cpu 4 [1389863.137687] CPU 4 [1389863.137689] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137754] [1389863.137760] Pid: 7177, comm: classical Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137768] RIP: 0033:[<0000000000419459>] [<0000000000419459>] 0x419458 [1389863.137787] RSP: 002b:00007fb00109cdd0 EFLAGS: 00000202 [1389863.137789] RAX: 0000000000000263 RBX: 0000000000000200 RCX: 00007fb4b9d7a060 [1389863.137794] RDX: 000000000000014a RSI: 000000000001c269 RDI: 000000000001c26a [1389863.137796] RBP: 0000000000000200 R08: fffffffa7b7ec000 R09: 00007fb4b9d7b1f0 [1389863.137798] R10: 00007faf355671f0 R11: 0000000000000119 R12: 00007fb4818a8010 [1389863.137801] R13: 00007faefd094010 R14: 0000000000002000 R15: 0000000000000003 [1389863.137806] FS: 00007fb00109d700(0000) GS:ffff880817c00000(0000) knlGS:000000005574c6c0 [1389863.137809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137811] CR2: 00007f10102b3760 CR3: 000000023ac52000 CR4: 00000000000006e0 [1389863.137815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137818] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137822] Process classical (pid: 7177, threadinfo ffff880c680b6000, task ffff880c683f2e60) [1389863.137824] [1389863.137826] Call Trace: [1389863.137022] NMI backtrace for cpu 9 [1389863.137022] CPU 9 [1389863.137022] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.137022] [1389863.137022] Pid: 19428, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.137022] RIP: 0010:[<ffffffff812b97f2>] [<ffffffff812b97f2>] find_next_bit+0x12/0xd0 [1389863.137022] RSP: 0000:ffff880c17c83d68 EFLAGS: 00000083 [1389863.137022] RAX: 0000000000000100 RBX: 0000000000000002 RCX: 000000000000000b [1389863.137022] RDX: 000000000000000b RSI: 0000000000000100 RDI: ffffffff81b60d60 [1389863.137022] RBP: ffff880c17c83d68 R08: ffffffff81b60d60 R09: 000000000000000b [1389863.137022] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000d402 [1389863.137022] R13: ffffffff81b60d60 R14: 0000000000000400 R15: 0000000000000096 [1389863.137022] FS: 0000000014197870(0063) GS:ffff880c17c80000(0000) knlGS:000000005574c6c0 [1389863.137022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.137022] CR2: 0000000084dabfd8 CR3: 0000000ad048b000 CR4: 00000000000006e0 [1389863.137022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.137022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.137022] Process molprop_2010_1_ (pid: 19428, threadinfo ffff8801ab4ec000, task ffff88040dcd5cc0) [1389863.137022] Stack: [1389863.137022] ffff880c17c83db8 ffffffff81031693 000000000000000e 000000000000000a [1389863.137022] ffff880c17c83dd8 0000000000002710 ffffffff81a30180 ffffffff81a30280 [1389863.137022] 0000000000000000 0004f012ba916259 ffff880c17c83dc8 ffffffff81035987 [1389863.137022] Call Trace: [1389863.137022] <IRQ> [1389863.137022] [<ffffffff81031693>] default_send_IPI_mask_sequence_phys+0x53/0xe0 [1389863.137022] [<ffffffff81035987>] physflat_send_IPI_all+0x17/0x20 [1389863.137022] [<ffffffff810318a1>] arch_trigger_all_cpu_backtrace+0x61/0xa0 [1389863.137022] [<ffffffff810e07ef>] __rcu_pending+0x1bf/0x380 [1389863.137022] [<ffffffff810e0dbb>] rcu_check_callbacks+0x10b/0x120 [1389863.137022] [<ffffffff8107cb08>] update_process_times+0x48/0x90 [1389863.137022] [<ffffffff8109f6b4>] tick_sched_timer+0x64/0xc0 [1389863.137022] [<ffffffff81092100>] __run_hrtimer+0x70/0x1e0 [1389863.137022] [<ffffffff8109f650>] ? tick_nohz_handler+0x100/0x100 [1389863.137022] [<ffffffff81019fd9>] ? read_tsc+0x9/0x20 [1389863.137022] [<ffffffff81092a7b>] hrtimer_interrupt+0xeb/0x210 [1389863.137022] [<ffffffff815df989>] smp_apic_timer_interrupt+0x69/0x99 [1389863.137022] [<ffffffff815dd85e>] apic_timer_interrupt+0x6e/0x80 [1389863.137022] <EOI> [1389863.137022] Code: ea 01 48 8b 34 d7 48 85 f6 74 ee 48 c1 e2 06 48 0f bd f6 48 8d 04 32 5d c3 55 48 39 f2 48 89 f0 48 89 e5 73 79 48 89 d1 49 89 d1 <48> c1 e9 06 49 83 e1 c0 4c 8d 04 cf 48 89 f7 4c 29 cf 83 e2 3f [1389863.137022] Call Trace: [1389863.137022] <IRQ> [<ffffffff81031693>] default_send_IPI_mask_sequence_phys+0x53/0xe0 [1389863.137022] [<ffffffff81035987>] physflat_send_IPI_all+0x17/0x20 [1389863.137022] [<ffffffff810318a1>] arch_trigger_all_cpu_backtrace+0x61/0xa0 [1389863.137022] [<ffffffff810e07ef>] __rcu_pending+0x1bf/0x380 [1389863.137022] [<ffffffff810e0dbb>] rcu_check_callbacks+0x10b/0x120 [1389863.137022] [<ffffffff8107cb08>] update_process_times+0x48/0x90 [1389863.137022] [<ffffffff8109f6b4>] tick_sched_timer+0x64/0xc0 [1389863.137022] [<ffffffff81092100>] __run_hrtimer+0x70/0x1e0 [1389863.137022] [<ffffffff8109f650>] ? tick_nohz_handler+0x100/0x100 [1389863.137022] [<ffffffff81019fd9>] ? read_tsc+0x9/0x20 [1389863.137022] [<ffffffff81092a7b>] hrtimer_interrupt+0xeb/0x210 [1389863.137022] [<ffffffff815df989>] smp_apic_timer_interrupt+0x69/0x99 [1389863.137022] [<ffffffff815dd85e>] apic_timer_interrupt+0x6e/0x80 [1389863.137022] <EOI> [1389863.138002] NMI backtrace for cpu 8 [1389863.138002] CPU 8 [1389863.138002] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.138002] [1389863.138002] Pid: 24479, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.138002] RIP: 0033:[<00000000010fbe7b>] [<00000000010fbe7b>] 0x10fbe7a [1389863.138002] RSP: 002b:00007fff95c34710 EFLAGS: 00000283 [1389863.138002] RAX: 0000000000000000 RBX: 000000000000002c RCX: 0000000000000138 [1389863.138002] RDX: 0000000000000001 RSI: 0000000000001373 RDI: 00000000143e0078 [1389863.138002] RBP: 00000000264d61a0 R08: 0000000000009c10 R09: 0000000000000080 [1389863.138002] R10: 0000000000004000 R11: 00000000264da1a0 R12: 000000000000001d [1389863.138002] R13: 00000000143cb308 R14: 000000000011add0 R15: 000000000000001d [1389863.138002] FS: 000000001339c870(0063) GS:ffff880c17c00000(0000) knlGS:000000005574c6c0 [1389863.138002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.138002] CR2: 0000000124576000 CR3: 0000000e1e7f9000 CR4: 00000000000006e0 [1389863.138002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.138002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.138002] Process molprop_2010_1_ (pid: 24479, threadinfo ffff880e1e7e6000, task ffff880c68385cc0) [1389863.138002] [1389863.138002] Call Trace: [1389863.138002] NMI backtrace for cpu 6 [1389863.138002] CPU 6 [1389863.138002] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.138002] [1389863.138002] Pid: 24481, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.138002] RIP: 0033:[<0000000001114899>] [<0000000001114899>] 0x1114898 [1389863.138002] RSP: 002b:00007fff11c377e0 EFLAGS: 00000202 [1389863.138002] RAX: 00000000000022e0 RBX: 0000000000000088 RCX: 0000000000000030 [1389863.138002] RDX: 0000000000000300 RSI: 0000000022ca2ef8 RDI: 000000000065a280 [1389863.138002] RBP: 0000000024a2cf38 R08: 0000000024a2cf70 R09: 0000000024a2cec8 [1389863.138002] R10: 0000000024a2ce58 R11: 0000000024a2ce20 R12: 00000000000017ec [1389863.138002] R13: 0000000024a2ce90 R14: 0000000024a2cf00 R15: 00000000250871f0 [1389863.138002] FS: 0000000013aa2870(0063) GS:ffff880817d00000(0000) knlGS:000000005574c6c0 [1389863.138002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.138002] CR2: 00007f10103cc1ab CR3: 0000000c3788b000 CR4: 00000000000006e0 [1389863.138002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.138002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.138002] Process molprop_2010_1_ (pid: 24481, threadinfo ffff8808e447a000, task ffff8808e47f0000) [1389863.138002] [1389863.138002] Call Trace: [1389863.138009] NMI backtrace for cpu 10 [1389863.138009] CPU 10 [1389863.138009] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.138009] [1389863.138009] Pid: 7175, comm: classical Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.138009] RIP: 0033:[<0000000000419435>] [<0000000000419435>] 0x419434 [1389863.138009] RSP: 002b:00007fffe79105c0 EFLAGS: 00000246 [1389863.138009] RAX: 0000000000000268 RBX: 0000000000000200 RCX: 00007fb4b9d7c060 [1389863.138009] RDX: 000000000000014b RSI: 000000000001c26a RDI: 000000000001c26b [1389863.138009] RBP: 0000000000000200 R08: fffffffa7b7ec000 R09: 00007fb4b9d7d230 [1389863.138009] R10: 00007faf35569230 R11: 000000000000011d R12: 00007fb4818a8010 [1389863.138009] R13: 00007faefd094010 R14: 0000000000002000 R15: 00000000006d8f20 [1389863.138009] FS: 00007fb704958780(0000) GS:ffff880c17d00000(0000) knlGS:000000005574c6c0 [1389863.138009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.138009] CR2: 000000019d806000 CR3: 000000023ac52000 CR4: 00000000000006e0 [1389863.138009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.138009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.138009] Process classical (pid: 7175, threadinfo ffff88012c15a000, task ffff880104c48000) [1389863.138009] [1389863.138009] Call Trace: [1389863.138023] NMI backtrace for cpu 1 [1389863.138023] CPU 1 [1389863.138023] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.138023] [1389863.138023] Pid: 19430, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.138023] RIP: 0033:[<0000000001843462>] [<0000000001843462>] 0x1843461 [1389863.138023] RSP: 002b:00007fff52a0f2c0 EFLAGS: 00000206 [1389863.138023] RAX: 00000000e2d191d8 RBX: 00000000e29aa1d8 RCX: 0000000000000000 [1389863.138023] RDX: 00000000e2d181d8 RSI: 00000000000006d8 RDI: 00000000e2d161d8 [1389863.138023] RBP: 00000000e2d171d8 R08: 0000000000004000 R09: 0000000000000800 [1389863.138023] R10: 0000000020d56ff0 R11: 0000000020d494f0 R12: 0000000020d50270 [1389863.138023] R13: 0000000000004000 R14: 0000000000f1ab80 R15: 00000000e2d191d8 [1389863.138023] FS: 0000000012e3b870(0063) GS:ffff880417c80000(0000) knlGS:000000005574c6c0 [1389863.138023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.138023] CR2: 00007f7f1c03f000 CR3: 00000008a803a000 CR4: 00000000000006e0 [1389863.138023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.138023] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.138023] Process molprop_2010_1_ (pid: 19430, threadinfo ffff88031879a000, task ffff88040cf0c590) [1389863.138023] [1389863.138023] Call Trace: [1389863.138026] NMI backtrace for cpu 7 [1389863.138026] CPU 7 [1389863.138026] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.138026] [1389863.138026] Pid: 19429, comm: molprop_2010_1_ Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.138026] RIP: 0033:[<0000000001843903>] [<0000000001843903>] 0x1843902 [1389863.138026] RSP: 002b:00007fff42b353c0 EFLAGS: 00000216 [1389863.138026] RAX: 00000000e29cdbe0 RBX: 00000000001fc000 RCX: 0000000000004000 [1389863.138026] RDX: 0000000000093918 RSI: 0000000000000800 RDI: 000000000020a438 [1389863.138026] RBP: 000000001faf4470 R08: 00000000e29cebe0 R09: 00000000e29cfbe0 [1389863.138026] R10: 0000000000000400 R11: 00000000000006d8 R12: 0000000000093d10 [1389863.138026] R13: 000000000020a830 R14: 00000000e29d0be0 R15: 00000000e2bcbbe0 [1389863.138026] FS: 0000000012af3870(0063) GS:ffff880817d80000(0000) knlGS:000000005574c6c0 [1389863.138026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.138026] CR2: 0000000035155078 CR3: 00000008a816e000 CR4: 00000000000006e0 [1389863.138026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.138026] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.138026] Process molprop_2010_1_ (pid: 19429, threadinfo ffff880318756000, task ffff88040cef9730) [1389863.138026] [1389863.138026] Call Trace: [1389863.139007] NMI backtrace for cpu 11 [1389863.139007] CPU 11 [1389863.139007] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.139007] [1389863.139007] Pid: 7184, comm: classical Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.139007] RIP: 0033:[<00000000004194a6>] [<00000000004194a6>] 0x4194a5 [1389863.139007] RSP: 002b:00007faffd895dd0 EFLAGS: 00000287 [1389863.139007] RAX: 0000000000000234 RBX: 0000000000000200 RCX: 00007fb4b9f2c060 [1389863.139007] RDX: 0000000000000223 RSI: 000000000001c342 RDI: 000000000001c343 [1389863.139007] RBP: 0000000000000200 R08: fffffffa7b7ec000 R09: 00007fb4b9f2c180 [1389863.139007] R10: 00007faf35718180 R11: 0000000000000012 R12: 00007fb4818a8010 [1389863.139007] R13: 00007faefd094010 R14: 0000000000002000 R15: 0000000000000003 [1389863.139007] FS: 00007faffd896700(0000) GS:ffff880c17d80000(0000) knlGS:000000005574c6c0 [1389863.139007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.139007] CR2: 000000014e670000 CR3: 000000023ac52000 CR4: 00000000000006e0 [1389863.139007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.139007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.139007] Process classical (pid: 7184, threadinfo ffff880e9f164000, task ffff88100c34c590) [1389863.139007] [1389863.139007] Call Trace: [1389863.139006] NMI backtrace for cpu 14 [1389863.139006] CPU 14 [1389863.139006] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.139006] [1389863.139006] Pid: 7183, comm: classical Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.139006] RIP: 0033:[<0000000000419417>] [<0000000000419417>] 0x419416 [1389863.139006] RSP: 002b:00007faffe096dd0 EFLAGS: 00000206 [1389863.139006] RAX: 0000000000000410 RBX: 0000000000000200 RCX: 00007fb4b9f28060 [1389863.139006] RDX: 0000000000000221 RSI: 000000000001c340 RDI: 000000000001c341 [1389863.139006] RBP: 0000000000000200 R08: fffffffa7b7ec000 R09: 00007fb4b9f29f40 [1389863.139006] R10: 00007faf35715f50 R11: 00000000000001ef R12: 00007fb4818a8010 [1389863.139006] R13: 00007faefd094010 R14: 0000000000002000 R15: 0000000000000003 [1389863.139006] FS: 00007faffe097700(0000) GS:ffff881027d00000(0000) knlGS:000000005574c6c0 [1389863.139006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.139006] CR2: 00007f7f0403f000 CR3: 000000023ac52000 CR4: 00000000000006e0 [1389863.139006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.139006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.139006] Process classical (pid: 7183, threadinfo ffff880c5ac9a000, task ffff88100c34ae60) [1389863.139006] [1389863.139006] Call Trace: [1389863.139009] NMI backtrace for cpu 13 [1389863.139009] CPU 13 [1389863.139009] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.139009] [1389863.139009] Pid: 6578, comm: imag_time Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.139009] RIP: 0033:[<000000000047d57f>] [<000000000047d57f>] 0x47d57e [1389863.139009] RSP: 002b:00007f7f24737bd8 EFLAGS: 00000202 [1389863.139009] RAX: 00007f7f300838d0 RBX: 00007f7f300008d0 RCX: 00007f800811d810 [1389863.139009] RDX: 0000000000000080 RSI: 0000000000000080 RDI: 3ad2597e8e01a000 [1389863.139009] RBP: 0000000000100000 R08: 3d3b677b81b74100 R09: 0000000000000010 [1389863.139009] R10: 0000000000000002 R11: 0000000000000010 R12: 00007f800811d010 [1389863.139009] R13: 0000000000001060 R14: 0000000000000100 R15: 000000000047d8a0 [1389863.139009] FS: 00007f7f24738700(0000) GS:ffff881027c80000(0000) knlGS:000000005574c6c0 [1389863.139009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.139009] CR2: 00007f7f2c03f000 CR3: 000000040b2b4000 CR4: 00000000000006f0 [1389863.139009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.139009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.139009] Process imag_time (pid: 6578, threadinfo ffff88040b1ee000, task ffff88007d4f4590) [1389863.139009] [1389863.139009] Call Trace: [1389863.139006] NMI backtrace for cpu 15 [1389863.139006] CPU 15 [1389863.139006] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.139006] [1389863.139006] Pid: 6582, comm: imag_time Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.139006] RIP: 0033:[<000000000047d570>] [<000000000047d570>] 0x47d56f [1389863.139006] RSP: 002b:00007f7f22733bb8 EFLAGS: 00000202 [1389863.139006] RAX: 00007f800b998b10 RBX: 00007f800b998810 RCX: 00007f7f2c030e00 [1389863.139006] RDX: 0000000000000080 RSI: 0000000000000030 RDI: bb0926d97713a8f9 [1389863.139006] RBP: 0000000000000010 R08: bb571233afc9af43 R09: 0000000000001060 [1389863.139006] R10: 000000000000003a R11: 0000000000100000 R12: 00007f7f2c000c60 [1389863.139006] R13: 0000000000000010 R14: 0000000000000100 R15: 000000000047d820 [1389863.139006] FS: 00007f7f22734700(0000) GS:ffff881027d80000(0000) knlGS:000000005574c6c0 [1389863.139006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.139006] CR2: 00007f7f2803f000 CR3: 000000040b2b4000 CR4: 00000000000006e0 [1389863.139006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.139006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.139006] Process imag_time (pid: 6582, threadinfo ffff88040c87c000, task ffff88040dc8dcc0) [1389863.139006] [1389863.139006] Call Trace: [1389863.139009] NMI backtrace for cpu 12 [1389863.139009] CPU 12 [1389863.139009] Modules linked in: virtio_net kvm_amd kvm nfs fscache auth_rpcgss nfs_acl lockd microcode nv_tco amd64_edac_mod edac_core edac_mce_amd forcedeth serio_raw k10temp i2c_nforce2 i2c_core sunrpc uinput ata_generic pata_acpi sata_nv pata_amd [last unloaded: scsi_wait_scan] [1389863.139009] [1389863.139009] Pid: 7182, comm: classical Not tainted 3.1.7-1.fc16.x86_64 #1 empty empty/S4980 [1389863.139009] RIP: 0033:[<0000000000419491>] [<0000000000419491>] 0x419490 [1389863.139009] RSP: 002b:00007faffe897dd0 EFLAGS: 00000206 [1389863.139009] RAX: 00000000000002eb RBX: 0000000000000200 RCX: 00007fb4b9f2a060 [1389863.139009] RDX: 0000000000000222 RSI: 000000000001c341 RDI: 000000000001c342 [1389863.139009] RBP: 0000000000000200 R08: fffffffa7b7ec000 R09: 00007fb4b9f2acf0 [1389863.139009] R10: 00007faf35716d00 R11: 00000000000000ca R12: 00007fb4818a8010 [1389863.139009] R13: 00007faefd094010 R14: 0000000000002000 R15: 0000000000000003 [1389863.139009] FS: 00007faffe898700(0000) GS:ffff881027c00000(0000) knlGS:000000005574c6c0 [1389863.139009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1389863.139009] CR2: 00007f7f3003f000 CR3: 000000023ac52000 CR4: 00000000000006e0 [1389863.139009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1389863.139009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1389863.139009] Process classical (pid: 7182, threadinfo ffff880c5accc000, task ffff881008fddcc0) [1389863.139009] [1389863.139009] Call Trace:
Please install kernel-debug, it should print some more useful information where the problem is.
[mass update] kernel-3.3.0-4.fc16 has been pushed to the Fedora 16 stable repository. Please retest with this update.