Bug 367141

Summary: Kernel 2.6.23.1-21 panics randomly after install
Product: [Fedora] Fedora Reporter: Gilbert Sebenste <sebenste>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED DEFERRED QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 7CC: ron
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-01-15 19:58:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
This is my /var/log/dmesg from my last boot from 2.6.23.-12.53 none

Description Gilbert Sebenste 2007-11-05 17:44:08 UTC
Description of problem: Kernel 2.6.23.1-21 panics randomly after install


Version-Release number of selected component (if applicable):
2.6.23.1-21

How reproducible: Always, but randomly, if that makes sense...


Steps to Reproduce:
1. Wait a few hours to several days...
2. get an oops.

  
Actual results: A kernel oops:

Nov  4 07:01:00 weather kernel: BUG: unable to handle kernel paging request
at virtual address 80040001
Nov  4 07:01:00 weather kernel:  printing eip:
Nov  4 07:01:00 weather kernel: 080fb294
Nov  4 07:01:00 weather kernel: *pde = 00000000
Nov  4 07:01:00 weather kernel: Oops: 0002 [#1]

No error or warning mesages precede or post-date this.

Expected results:
No oops!

Additional info: This bug has been going on since 2.6.23.1-10. It supercedes bug
#344821; see the oops message there before closing it off. 2.6.23-8 worked fine
and did not produce any oops messages.

Comment 1 Chuck Ebbert 2007-11-05 19:58:28 UTC
This does not look like the same bug. But the oops message is incomplete, so
there's no way to be sure. Can you set up netconsole to get the complete message?

Comment 2 Gilbert Sebenste 2007-11-05 20:31:04 UTC
Pardon my ignorance, but how do I do this? On a console window I had open, it 
just showed the last line when I walked into the office Sunday morning
to reboot it, IE:

kernel: Oops: 0002 [#1]

Comment 4 Gilbert Sebenste 2007-11-11 18:38:21 UTC
Hello Chuck,

I sincerely apologize for not getting back to you this week. Over the last 8 
days, I've had a nasty cold that has put me on my back, unable to do anything 
with this, or be at work, where my machines are, until this weekend.

What I can tell you is this. Early Saturday morning, I finally felt well enough 
to update the kernel to -26 on my machines, and throw all the backlogged F7 
patches on there as well. Sicne then, and so far, it hasn't oopsed...but it has 
only been a few days, and this last oops was a week apart from the last time I 
booted. And having said that...

My messages file had a few kernel errors from a piece of weather software I am 
running on it. Aha! It didn't cause an oops, but did cause this:

Nov  7 07:39:11 weather kernel: sfccalc[22046]: segfault at 00000000 eip 
0810319d esp bfa6e9a0 error 4

Nov  9 16:09:20 weather kernel: sfccalc[27890]: segfault at 00000000 eip 
0810319d esp bf81d6d0 error 4

I didn't see this until early Saturday as well, and the programmer is out until 
Monday. I don't know if this is part of or entirely the problem, but software 
still should not cause the kernel to oops. I do note that with -26, at least 
one or two things that cause an oops have been fixed. So with that, I'll keep 
you posted and post any more errors that I get, or news from the software 
programmer above.


Comment 5 Gilbert Sebenste 2007-11-19 04:23:22 UTC
Hi Chuck,

This hasn't happened so far since the hang up last week. I do not know if it is 
fixed, but it has been about 8 days since I last rebooted the machine. And it 
is running with full data at full throttle. If it is fixed by the recent -28 
kernel which I have on there now, I don't know what could have fixed it. Should 
I give it another week to be sure? Or should we wait until 2.6.23.8 for F7 is 
released, or something else? Guidance, please.

Comment 6 Gilbert Sebenste 2007-11-20 22:23:49 UTC
Aha! Look at this. From -28:

Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: Oops: 0000 [#1]
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: SMP
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: CPU:    1
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: EIP:    0060:[<c0463e9e>]    Not tainted VLI
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: EFLAGS: 00010286   (2.6.23.1-28.fc7 #1)
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: EIP is at try_to_release_page+0x20/0x42
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: eax: 40000801   ebx: ffff0718   ecx: c1627aa0   edx: 00000000
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: esi: 000000d0   edi: 00000001   ebp: c31ccf80   esp: c31ccdfc
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: Process kswapd0 (pid: 303, ti=c31cc000 task=c3230c20 
task.ti=c31cc000)
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: Stack: c1627aa0 ffff0718 c04699da 38412e7c 00000ab9 00000000 
c31ccf10 0001d240
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:        00000000 0000000c 0000000c 00000001 c1624b40 c1625ca0 
c16231a0 c163bbe0
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:        c164ad60 c164b360 c148e160 c1648d60 c16399e0 c162a560 
c164bea0 c16283c0
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: Call Trace:
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c04699da>] shrink_page_list+0x409/0x4ea
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c0468fb3>] isolate_lru_pages+0x6b/0x16b
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c0469bb3>] shrink_inactive_list+0xf8/0x2d8
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c0469e5f>] shrink_zone+0xcc/0xf1
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c046a297>] kswapd+0x289/0x40a
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c043d3c1>] autoremove_wake_function+0x0/0x35
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c046a00e>] kswapd+0x0/0x40a
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c043d2fa>] kthread+0x38/0x5e
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c043d2c2>] kthread+0x0/0x5e
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  [<c0405dbb>] kernel_thread_helper+0x7/0x10
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel:  =======================
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: Code: ff e8 6f 45 00 00 89 d8 5a 5b c3 56 89 c1 53 8b 58 10 89 
d6 8b 00 a8 01 75 04 0f 0b eb fe 8b 01 31 d2 f6 c4 10 75 21 85 db 74 14 <8b> 43 
38 8b 58 28 85
db 74 0a 89 f2 89 c8 ff d3 89 c2 eb 09 5b
Message from syslogd@ at Tue Nov 20 15:57:01 2007 ...
weather kernel: EIP: [<c0463e9e>] try_to_release_page+0x20/0x42 SS:ESP 
0068:c31ccdfc

---
Does that help you?

Comment 7 Chuck Ebbert 2007-11-20 23:13:56 UTC
Oops: 0000 [#1]
SMP
CPU:    1
EIP:    0060:[<c0463e9e>]    Not tainted VLI
EFLAGS: 00010286   (2.6.23.1-28.fc7 #1)
EIP is at try_to_release_page+0x20/0x42
eax: 40000801   ebx: ffff0718   ecx: c1627aa0   edx: 00000000
esi: 000000d0   edi: 00000001   ebp: c31ccf80   esp: c31ccdfc
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process kswapd0 (pid: 303, ti=c31cc000 task=c3230c20 task.ti=c31cc000)
Stack: c1627aa0 ffff0718 c04699da 38412e7c 00000ab9 00000000 c31ccf10 0001d240
       00000000 0000000c 0000000c 00000001 c1624b40 c1625ca0 c16231a0 c163bbe0
       c164ad60 c164b360 c148e160 c1648d60 c16399e0 c162a560 c164bea0 c16283c0
Call Trace:
 [<c04699da>] shrink_page_list+0x409/0x4ea
 [<c0468fb3>] isolate_lru_pages+0x6b/0x16b
 [<c0469bb3>] shrink_inactive_list+0xf8/0x2d8
 [<c0469e5f>] shrink_zone+0xcc/0xf1
 [<c046a297>] kswapd+0x289/0x40a
 [<c043d3c1>] autoremove_wake_function+0x0/0x35
 [<c046a00e>] kswapd+0x0/0x40a
 [<c043d2fa>] kthread+0x38/0x5e
 [<c043d2c2>] kthread+0x0/0x5e
 [<c0405dbb>] kernel_thread_helper+0x7/0x10
 =======================
Code: ff e8 6f 45 00 00 89 d8 5a 5b c3 56 89 c1 53 8b 58 10 89 
d6 8b 00 a8 01 75 04 0f 0b eb fe 8b 01 31 d2 f6 c4 10 75 21 85 db 74 14 <8b> 43 
38 8b 58 28 85 db 74 0a 89 f2 89 c8 ff d3 89 c2 eb 09 5b


Comment 8 Gilbert Sebenste 2007-11-20 23:22:21 UTC
OK...now what?


Comment 9 Gilbert Sebenste 2007-11-20 23:23:26 UTC
BTW, are you going to release .9-RC1 for F7? Do you think it fixes this?

Comment 10 Chuck Ebbert 2007-11-21 00:06:20 UTC
8b 43 38 : mov 0x38(%ebx),%eax

mm/filemap.c : 2236
        if (mapping && mapping->a_ops->releasepage)

mapping == ebx == ffff0718


Comment 11 Gilbert Sebenste 2007-11-21 00:13:02 UTC
Hi Chuck,

Sorry, I wish I could say I am a programmer, but what you wrote doesn't make 
sense to me. Are you saying you found a bug in the kernel?


Comment 12 Gilbert Sebenste 2007-11-23 02:01:07 UTC
BTW, just updated to .8 from Koji, will update to .9-RC1 when it is done
building tonight.

Comment 13 Gilbert Sebenste 2007-11-30 16:38:54 UTC
Another oops from .9-39:

Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: Oops: 0000 [#1]
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: SMP
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: CPU:    0
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: EIP:    0060:[<c0463efc>]    Not tainted VLI
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: EFLAGS: 00010286   (2.6.23.9-39.fc7 #1)
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: EIP is at try_to_release_page+0x20/0x42
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: eax: 40000809   ebx: fff89318   ecx: c1260520   edx: 00000000
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: esi: 000000d0   edi: 00000001   ebp: c318af80   esp: c318adfc
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: Process kswapd0 (pid: 312, ti=c318a000 task=c322c610
task.ti=c318a000)
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: Stack: c1260520 fff89318 c0469a3a c3014300 c3014278 00000000
c318af10 00440424
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:        00000000 00000005 00000005 00000001 c1609e60 c127b360
c116a7a0 c12dff00
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:        c1100280 c3032180 c042ab61 ce125230 00000000 c318ae74
c04253fd c3029180
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: Call Trace:
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c0469a3a>] shrink_page_list+0x409/0x4ea
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c042ab61>] load_balance_start_fair+0x18/0x21
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c04253fd>] balance_tasks+0x77/0x11c
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c0469013>] isolate_lru_pages+0x6b/0x16b
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c0469c13>] shrink_inactive_list+0xf8/0x2d8
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c061b444>] __sched_text_start+0x594/0x638
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<f883c5fc>] mb_cache_shrink_fn+0x1e/0xb7 [mbcache]
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c0469ebf>] shrink_zone+0xcc/0xf1
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c046a2f7>] kswapd+0x289/0x40a
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c043d3d1>] autoremove_wake_function+0x0/0x35
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c046a06e>] kswapd+0x0/0x40a
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c043d30a>] kthread+0x38/0x5e
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c043d2d2>] kthread+0x0/0x5e
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  [<c0405dbb>] kernel_thread_helper+0x7/0x10
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel:  =======================
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: Code: ff e8 71 45 00 00 89 d8 5a 5b c3 56 89 c1 53 8b 58 10 89
d6 8b 00 a8 01 75 04 0f 0b eb fe 8b 01 31 d2 f6 c4 10 75 21 85 db 74 14 <8b> 43
38 8b 58 28 85
db 74 0a 89 f2 89 c8 ff d3 89 c2 eb 09 5b
Message from syslogd@ at Fri Nov 30 07:57:02 2007 ...
weather kernel: EIP: [<c0463efc>] try_to_release_page+0x20/0x42 SS:ESP 0068:c318adfc

Comment 14 Gilbert Sebenste 2007-11-30 16:39:40 UTC
By the way, this last one was a "soft oops"...everything still seems to be working.

Comment 15 Gilbert Sebenste 2007-12-07 21:29:10 UTC
This is a hard oops I just got with -45:

Dec  7 12:06:56 weather kernel: BUG: unable to handle kernel paging request at
virtual address fe622f08
Dec  7 12:06:56 weather kernel:  printing eip:
Dec  7 12:06:56 weather kernel: c0466b45
Dec  7 12:06:56 weather kernel: *pde = 00000000
Dec  7 12:06:56 weather kernel: Oops: 0000 [#1]
Dec  7 12:06:56 weather kernel: SMP
Dec  7 12:06:56 weather kernel: Modules linked in: lp autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_hda_intel 
snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1
snd_seq_dummy snd_rawmidi snd_ac97_codec ac97_bus snd_seq_oss snd_seq_midi_event
snd_seq 
snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_util_mem snd_timer
snd_hwdep pcspkr firewire_ohci snd e1000 parport_pc soundcore parport emu10k1_gp
gameport 
firewire_core crc_itu_t snd_page_alloc forcedeth button i2c_nforce2 i2c_core
sr_mod cdrom sg dm_snapshot dm_zero dm_mirror dm_mod ahci pata_amd ata_generic
sata_nv libata 
sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd
Dec  7 12:06:56 weather kernel: CPU:    0

Comment 16 Gilbert Sebenste 2007-12-08 17:04:36 UTC
And again:

Dec  8 05:56:04 weather kernel: BUG: unable to handle kernel paging request at
virtual address facdd178
Dec  8 05:56:04 weather kernel:  printing eip:
Dec  8 05:56:04 weather kernel: c0466b45
Dec  8 05:56:04 weather kernel: *pde = 00000000
Dec  8 05:56:04 weather kernel: Oops: 0000 [#1]

What in the world is going on here?!? I don't understand this, nor how to fix it!

Comment 17 Gilbert Sebenste 2007-12-09 20:17:05 UTC
Dec  9 06:09:54 weather kernel: Code: 00 31 c0 c3 89 c1 0f ae f0 89 f6 8b 50 10
8b 00 66 85 c0 79 07 ba 40 1d 70 c0 eb 0f 8b 01 84 c0 78 1b f6 c2 01 75 16 85 d2
74 12 <8b> 
42 38 85 c0 74 0b 8b 50 08 85 d2 74 04 89 c8 ff d2 e8 dc a1
Dec  9 06:09:54 weather kernel: EIP: [<c04623ad>] sync_page+0x27/0x41 SS:ESP
0068:e2d29bcc
Dec  9 06:09:54 weather kernel: BUG: unable to handle kernel paging request at
virtual address 00050041
Dec  9 06:09:54 weather kernel:  printing eip:
Dec  9 06:09:54 weather kernel: 00050041
Dec  9 06:09:54 weather kernel: *pde = 439bc067
Dec  9 06:09:54 weather kernel: Oops: 0000 [#2]
Dec  9 06:09:54 weather kernel: SMP
Dec  9 06:09:54 weather kernel: Modules linked in: autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_emu10k1_synth snd_esnd_emux_synth snd_seq_virmidi snd_seq_midi_emul
snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus snd_seq_dummy snd_hda_intel
snd_seq_oss snd_seq_midi_event snd_seq 
snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device parport_pc snd_util_mem
snd_timer snd_hwdep parport emu10k1_gp e1000 i2c_nforce2 firewire_ohci button
snd i2c_core 
firewire_core forcedeth soundcore snd_page_alloc crc_itu_t gameport sr_mod cdrom
sg dm_snapshot dm_zero dm_mirror dm_mod ahci pata_amd ata_generic sata_nv libata
sd_mod 
scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd
Dec  9 06:09:54 weather kernel: CPU:    3
Dec  9 06:09:54 weather kernel: EIP:    0060:[<00050041>]    Tainted: G      D VLI
Dec  9 06:09:54 weather kernel: EFLAGS: 00210002   (2.6.23.1-8.fc7 #1)
Dec  9 06:09:54 weather kernel: EIP is at 0x50041
Dec  9 06:09:54 weather kernel: eax: e2d29bf0   ebx: e2d29bf0   ecx: 00000000  
edx: 00000003
Dec  9 06:09:54 weather kernel: esi: 02600005   edi: 00000001   ebp: eae03cdc  
esp: eae03cbc
Dec  9 06:09:54 weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Dec  9 06:09:54 weather kernel: Process mv (pid: 14629, ti=eae03000
task=f7e95840 task.ti=eae03000)
Dec  9 06:09:54 weather kernel: Stack: c04244c6 eae03d0c 00000003 c300cfac
facdfccd c300cfac eae03d0c 00000001
Dec  9 06:09:54 weather kernel:        eae03d00 c042649e 00000000 eae03d0c
00000003 00200286 c300cfac 00000000
Dec  9 06:09:54 weather kernel:        fff59fd8 f0985700 c043d45a eae03d0c
c2be0620 00000000 c2531020 c046c684
Dec  9 06:09:54 weather kernel: Call Trace:
Dec  9 06:09:54 weather kernel:  [<c04244c6>] __wake_up_common+0x32/0x55
Dec  9 06:09:54 weather kernel:  [<c042649e>] __wake_up+0x32/0x43  
Dec  9 06:09:54 weather kernel:  [<c043d45a>] __wake_up_bit+0x2e/0x33
Dec  9 06:09:54 weather kernel:  [<c046c684>] __do_fault+0x365/0x394
Dec  9 06:09:54 weather kernel:  [<c046e9fa>] handle_mm_fault+0x3a0/0x78b
Dec  9 06:09:54 weather kernel:  [<c046217e>] file_read_actor+0x0/0xe0
Dec  9 06:09:54 weather kernel:  [<c061ed44>] do_page_fault+0x26a/0x5ef
Dec  9 06:09:54 weather kernel:  [<c0471865>] mmap_region+0x31c/0x3d8
Dec  9 06:09:54 weather kernel:  [<c061eada>] do_page_fault+0x0/0x5ef
Dec  9 06:09:54 weather kernel:  [<c061d7c2>] error_code+0x72/0x78
Dec  9 06:09:54 weather kernel:  [<c04f632c>] clear_user+0x52/0x60
Dec  9 06:09:54 weather kernel:  [<c04a73dd>] padzero+0x16/0x24
Dec  9 06:09:54 weather kernel:  [<c04a8386>] load_elf_binary+0xbfd/0x151d
Dec  9 06:09:54 weather kernel:  [<c046bc98>] kmap_high+0x1a/0x178
Dec  9 06:09:54 weather kernel:  [<c0483f3e>] get_arg_page+0x42/0x8c
Dec  9 06:09:54 weather kernel:  [<c046b9fe>] page_address+0x78/0x98
Dec  9 06:09:54 weather kernel:  [<c04840f1>] copy_strings+0x169/0x173
Dec  9 06:09:54 weather kernel:  [<c04841b7>] search_binary_handler+0x95/0x1ce
Dec  9 06:09:54 weather kernel:  [<c048546e>] do_execve+0x13b/0x1af
Dec  9 06:09:54 weather kernel:  [<c04031ab>] sys_execve+0x2f/0x4f
Dec  9 06:09:54 weather kernel:  [<c040518a>] syscall_call+0x7/0xb
Dec  9 06:09:54 weather kernel:  [<c0610000>] xfrm_send_policy_notify+0x268/0x4fd
Dec  9 06:09:54 weather kernel:  =======================
Dec  9 06:09:54 weather kernel: Code:  Bad EIP value.
Dec  9 06:09:54 weather kernel: EIP: [<00050041>] 0x50041 SS:ESP 0068:eae03cbc

Comment 19 Gilbert Sebenste 2007-12-10 23:51:45 UTC
Any chance that could be incorporated into a new F7 kernel?


Comment 20 Chuck Ebbert 2007-12-11 23:07:59 UTC
Nevermind, that patch is just cosmetic.

Comment 21 Gilbert Sebenste 2007-12-14 17:37:10 UTC
Dec 14 06:31:23 weather kernel: BUG: unable to handle kernel paging request at
virtual address 0010fff0
Dec 14 06:31:23 weather kernel:  printing eip:
Dec 14 06:31:23 weather kernel: c046d237
Dec 14 06:31:23 weather kernel: *pde = 00000000
Dec 14 06:31:23 weather kernel: Oops: 0002 [#1]
Dec 14 06:31:23 weather kernel: SMP
Dec 14 06:31:23 weather kernel: Modules linked in: lp autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm 
snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1
snd_rawmidi snd_ac97_codec ac97_bus snd_hda_intel snd_seq_dummy snd_seq_oss 
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device
snd_timer snd_util_mem snd_hwdep sr_mod cdrom e1000 emu10k1_gp gameport
firewire_ohci 
i2c_nforce2 i2c_core sg pcspkr snd firewire_core crc_itu_t parport_pc forcedeth
soundcore parport button snd_page_alloc dm_snapshot dm_zero dm_mirror dm_mod
ahci pata_amd 
ata_generic sata_nv libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd
uhci_hcd
Dec 14 06:31:23 weather kernel: CPU:    3

Comment 22 Gilbert Sebenste 2007-12-14 17:38:10 UTC
This is almost happening daily now.


Comment 23 Gilbert Sebenste 2007-12-14 22:27:11 UTC
Think the new 2.6.23.10 will fix it? Even so, can you put out one for F7?

Comment 24 Gilbert Sebenste 2007-12-15 06:15:46 UTC
2.6.23.10-51 is on. We'll see what happens.

Comment 25 Gilbert Sebenste 2007-12-15 06:18:41 UTC
Gahhh! 2.6.23.11 is out, but you caught one of the two bugs and put it
into .10. Is the other bug trivial?


Comment 26 Gilbert Sebenste 2007-12-15 19:04:16 UTC
Dec 15 09:03:40 weather kernel: BUG: unable to handle kernel paging request at
virtual address fe452748
Dec 15 09:03:40 weather kernel:  printing eip:
Dec 15 09:03:40 weather kernel: c04623b1
Dec 15 09:03:40 weather kernel: *pde = 00000000
Dec 15 09:03:40 weather kernel: Oops: 0000 [#1]
Dec 15 09:03:40 weather kernel: SMP
Dec 15 09:03:40 weather kernel: Modules linked in: autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_hda_intel snd_emu10$
Dec 15 09:03:40 weather kernel: CPU:    1
Dec 15 09:03:40 weather kernel: EIP:    0060:[<c04623b1>]    Not tainted VLI
Dec 15 09:03:40 weather kernel: EFLAGS: 00210282   (2.6.23.10-51.fc7 #1)
Dec 15 09:03:40 weather kernel: EIP is at sync_page+0x27/0x41
Dec 15 09:03:40 weather kernel: eax: 8001007d   ebx: f5344dfc   ecx: c1a632a0  
edx: fe452710
Dec 15 09:03:40 weather kernel: esi: f5344dfc   edi: c3007a80   ebp: c046238a  
esp: f5344de0
Dec 15 09:03:40 weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Dec 15 09:03:40 weather kernel: Process pqsurf (pid: 3273, ti=f5344000
task=f6522610 task.ti=f5344000)
Dec 15 09:03:40 weather kernel: Stack: c061bcaa f5344dfc c1a632a0 f5344e18
000481ad c046237c 00000002 c1a632a0
Dec 15 09:03:40 weather kernel:        00000000 00000001 f6522610 c043d3fe
c3007a84 c3007a84 f6452720 f6452710
Dec 15 09:03:40 weather kernel:        c0462448 00000000 f6452668 f5fa1000
f6ae4e00 c04641eb c1758a20 c0466265
Dec 15 09:03:40 weather kernel: Call Trace:
Dec 15 09:03:40 weather kernel:  [<c061bcaa>] __wait_on_bit_lock+0x2a/0x52
Dec 15 09:03:40 weather kernel:  [<c046237c>] __lock_page+0x58/0x5e
Dec 15 09:03:40 weather kernel:  [<c043d3fe>] wake_bit_function+0x0/0x3c
Dec 15 09:03:40 weather kernel:  [<c0462448>] find_lock_page+0x5a/0x90
Dec 15 09:03:40 weather kernel:  [<c04641eb>] filemap_fault+0x9f/0x391
Dec 15 09:03:40 weather kernel:  [<c0466265>] get_page_from_freelist+0x25d/0x2db
Dec 15 09:03:40 weather kernel:  [<c046c3ac>] __do_fault+0x59/0x394
Dec 15 09:03:40 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Dec 15 09:03:40 weather kernel:  [<c046aa02>] vma_prio_tree_insert+0x17/0x2a
Dec 15 09:03:40 weather kernel:  [<c0471899>] mmap_region+0x31c/0x3d8
Dec 15 09:03:40 weather kernel:  [<c061e30c>] do_page_fault+0x26a/0x5ef
Dec 15 09:03:40 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Dec 15 09:03:40 weather kernel:  [<c061e0a2>] do_page_fault+0x0/0x5ef
Dec 15 09:03:40 weather kernel:  [<c061cd8a>] error_code+0x72/0x78
Dec 15 09:03:40 weather kernel:  [<c0610000>] xfrm_do_migrate+0x1e/0x10b
Dec 15 09:03:40 weather kernel:  =======================
Dec 15 09:03:40 weather kernel: Code: 00 31 c0 c3 89 c1 0f ae f0 89 f6 8b 50 10
8b 00 66 85 c0 79 07 ba 40 0d 70 c0 eb 0f 8b 01 84 c0 78 1b f6 c2 01 75 16 85 d2
74 12 <8b> 42 38$
Dec 15 09:03:40 weather kernel: EIP: [<c04623b1>] sync_page+0x27/0x41 SS:ESP
0068:f5344de0
Dec 15 09:03:59 weather last message repeated 9 times

---------------------------------------------------------------------
It seems like random things on my computer are causing it to crash. Pqsurf is
part of a weather data ingest and management software program collectively known
as the LDM. That isn't causing problems on my other machines. This is only
occurring on this one.

One thing I have noticed is that I have 4 GB of RAM on this machine, but only
3.6 is usable. I noticed this on another machine which has the same hardware setup:

top - 13:02:55 up 39 min,  3 users,  load average: 0.72, 0.81, 1.44
Tasks: 239 total,   1 running, 238 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6%us,  1.8%sy,  0.0%ni, 96.1%id,  0.3%wa,  0.1%hi,  0.1%si,  0.0%st
Mem:   3630632k total,  3337948k used,   292684k free,   166528k buffers
Swap:  2031608k total,        0k used,  2031608k free,  2832632k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
                                                                               
                  
 2889 root     -99   0 16696 2264 1724 S    3  0.1   1:08.75 darkice           
                                                                               
                  
 3525 ldm       20   0 23280  21m  740 S    1  0.6   0:40.79 pqact             
                                                                               
                  
 3526 ldm       20   0 13948  12m  756 S    1  0.4   0:56.70 pqact             
                                                                               
                  
 1310 root      20   0  2388 1120  800 R    0  0.0   0:00.04 top               
                                                                               
                  
 3522 ldm       20   0  2148 1020  860 S    0  0.0   0:12.93 rtstats           
                                                                               
                  
 3524 ldm       20   0  2304 1196  740 S    0  0.0   0:12.77 pqact             
                                                                               
                  
 3527 ldm       20   0  2304 1128  740 S    0  0.0   0:15.29 pqact             
                                                                               
                  
 3529 ldm       20   0  2172 1060  740 S    0  0.0   0:15.21 pqact             
                                                                               
                  
 3531 ldm       20   0  2304 1148  740 S    0  0.0   0:15.24 pqact             
                                                                               
                  
 3532 ldm       20   0  2040  856  728 S    0  0.0   0:15.18 pqact             
                                                                               
                  
 3533 ldm       20   0  1984  764  656 S    0  0.0   0:45.43 pqsurf            
                                                                               
                  
 3535 ldm       20   0  6704 4396  708 S    0  0.1   0:09.46 rpc.ldmd          
                                                                               
                  
25504 ldm       20   0  3936 1480  708 S    0  0.0   0:01.36 rpc.ldmd          
                                                                               
                  
    1 root      20   0  2136  672  584 S    0  0.0   0:01.15 init              
                                                                               
                  
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd          
                                                                               
                  
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0       
                                                                               
                  
    4 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0       
                                                                               
                  
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0        
                                                                               
                  
    6 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1       
            

ANY idea as to what could be causing this? Thanks.


Comment 27 Gilbert Sebenste 2007-12-20 06:10:23 UTC
Here's another. FYI, this is from the original F7 kernel, 2.6.21. I thought to
use that to see if it still oopses. It does.

BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
 [<c042b0cf>] local_bh_enable+0x45/0x92
 [<c06002bd>] cond_resched_softirq+0x2c/0x42
 [<c059adf3>] release_sock+0x4f/0x9d
 [<c05c670d>] tcp_sendmsg+0x90b/0x9f9
 [<c04e7350>] copy_to_user+0x3c/0x50
 [<c059f285>] memcpy_toiovec+0x27/0x4a
 [<c059ddb8>] __kfree_skb+0xb5/0x113
 [<c06016f4>] _spin_lock_bh+0x8/0x18
 [<c05dec95>] inet_sendmsg+0x3b/0x45
 [<c0598731>] sock_aio_write+0xf6/0x102
 [<c04753ea>] do_sync_readv_writev+0xc1/0xfe
 [<c0436e71>] autoremove_wake_function+0x0/0x35
 [<c04e7100>] copy_from_user+0x3a/0x66
 [<c04752a5>] rw_copy_check_uvector+0x5c/0xb0
 [<c0475b33>] do_readv_writev+0xbc/0x187
 [<c059863b>] sock_aio_write+0x0/0x102
 [<c059a7dc>] sock_common_setsockopt+0x1d/0x22
 [<c0475c3b>] vfs_writev+0x3d/0x48
 [<c04760a4>] sys_writev+0x41/0x95
 [<c0404f70>] syscall_call+0x7/0xb
 [<c0600000>] __sched_text_start+0x6e8/0x89e
 =======================

Comment 28 Gilbert Sebenste 2007-12-21 16:50:18 UTC
Also seeing this in dmesg file:

audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=1896 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded
audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=1897 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded
audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=1898 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded
audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=1899 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded
audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=1900 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded

Comment 29 Gilbert Sebenste 2007-12-21 16:52:02 UTC
And this...

EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 6553849
ext3_orphan_cleanup: deleting unreferenced inode 83395159
ext3_orphan_cleanup: deleting unreferenced inode 83887893
ext3_orphan_cleanup: deleting unreferenced inode 37257272
ext3_orphan_cleanup: deleting unreferenced inode 37257266
ext3_orphan_cleanup: deleting unreferenced inode 37257255
ext3_orphan_cleanup: deleting unreferenced inode 37257265
ext3_orphan_cleanup: deleting unreferenced inode 37257252
ext3_orphan_cleanup: deleting unreferenced inode 37257385
ext3_orphan_cleanup: deleting unreferenced inode 37257241
ext3_orphan_cleanup: deleting unreferenced inode 37257236
ext3_orphan_cleanup: deleting unreferenced inode 37257230
EXT3-fs: dm-0: 12 orphan inodes deleted
EXT3-fs: recovery complete.    
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks

P.S. I have replaced the hard drive in this machine, so I do not believe it to
be hardware failure. I got this to happen on two different drives, both brand new.

Comment 30 Gilbert Sebenste 2007-12-21 18:56:35 UTC
I am getting this error every time on bootup, from dmesg:

BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
 [<c042b0cf>] local_bh_enable+0x45/0x92
 [<c06002bd>] cond_resched_softirq+0x2c/0x42
 [<c059adf3>] release_sock+0x4f/0x9d
 [<c05c670d>] tcp_sendmsg+0x90b/0x9f9
 [<c04e7350>] copy_to_user+0x3c/0x50
 [<c059f285>] memcpy_toiovec+0x27/0x4a
 [<c059ddb8>] __kfree_skb+0xb5/0x113
 [<c06016f4>] _spin_lock_bh+0x8/0x18
 [<c05dec95>] inet_sendmsg+0x3b/0x45
 [<c0598731>] sock_aio_write+0xf6/0x102
 [<c04753ea>] do_sync_readv_writev+0xc1/0xfe
 [<c0436e71>] autoremove_wake_function+0x0/0x35
 [<c04e7100>] copy_from_user+0x3a/0x66
 [<c04752a5>] rw_copy_check_uvector+0x5c/0xb0
 [<c0475b33>] do_readv_writev+0xbc/0x187
 [<c059863b>] sock_aio_write+0x0/0x102
 [<c059a7dc>] sock_common_setsockopt+0x1d/0x22
 [<c0475c3b>] vfs_writev+0x3d/0x48
 [<c04760a4>] sys_writev+0x41/0x95
 [<c0404f70>] syscall_call+0x7/0xb
 [<c0600000>] __sched_text_start+0x6e8/0x89e
 =======================

Comment 31 Chuck Ebbert 2007-12-21 19:13:36 UTC
(In reply to comment #30)
> I am getting this error every time on bootup, from dmesg:
> 
> BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
>  [<c042b0cf>] local_bh_enable+0x45/0x92
>  [<c06002bd>] cond_resched_softirq+0x2c/0x42
>  [<c059adf3>] release_sock+0x4f/0x9d
>  [<c05c670d>] tcp_sendmsg+0x90b/0x9f9
>  [<c04e7350>] copy_to_user+0x3c/0x50
>  [<c059f285>] memcpy_toiovec+0x27/0x4a
>  [<c059ddb8>] __kfree_skb+0xb5/0x113
>  [<c06016f4>] _spin_lock_bh+0x8/0x18
>  [<c05dec95>] inet_sendmsg+0x3b/0x45
>  [<c0598731>] sock_aio_write+0xf6/0x102
>  [<c04753ea>] do_sync_readv_writev+0xc1/0xfe
>  [<c0436e71>] autoremove_wake_function+0x0/0x35
>  [<c04e7100>] copy_from_user+0x3a/0x66
>  [<c04752a5>] rw_copy_check_uvector+0x5c/0xb0
>  [<c0475b33>] do_readv_writev+0xbc/0x187
>  [<c059863b>] sock_aio_write+0x0/0x102
>  [<c059a7dc>] sock_common_setsockopt+0x1d/0x22
>  [<c0475c3b>] vfs_writev+0x3d/0x48
>  [<c04760a4>] sys_writev+0x41/0x95
>  [<c0404f70>] syscall_call+0x7/0xb
>  [<c0600000>] __sched_text_start+0x6e8/0x89e
>  =======================

That is bug 240982

Comment 32 Gilbert Sebenste 2007-12-21 19:47:45 UTC
Ah, OK. Thanks!

Comment 33 Chuck Ebbert 2007-12-21 21:45:55 UTC
(In reply to comment #28)
> Also seeing this in dmesg file:
> 
> audit: audit_backlog=321 > audit_backlog_limit=320
> audit: audit_lost=1896 audit_rate_limit=0 audit_backlog_limit=320
> audit: backlog limit exceeded
> audit: audit_backlog=321 > audit_backlog_limit=320
> audit: audit_lost=1897 audit_rate_limit=0 audit_backlog_limit=320
> audit: backlog limit exceeded
> audit: audit_backlog=321 > audit_backlog_limit=320
> audit: audit_lost=1898 audit_rate_limit=0 audit_backlog_limit=320
> audit: backlog limit exceeded
> audit: audit_backlog=321 > audit_backlog_limit=320
> audit: audit_lost=1899 audit_rate_limit=0 audit_backlog_limit=320
> audit: backlog limit exceeded
> audit: audit_backlog=321 > audit_backlog_limit=320
> audit: audit_lost=1900 audit_rate_limit=0 audit_backlog_limit=320
> audit: backlog limit exceeded


You are getting a lot of audit messages. Try using the aureport and ausearch
tools to find out what is happening.


Comment 34 Chuck Ebbert 2007-12-21 21:47:59 UTC
Some other things:

1. Attach a copy of /var/log/dmesg from the failing machine to this bugzilla.

2. Try reducing the amount of memory with option "mem=2G"


Comment 35 Gilbert Sebenste 2007-12-21 21:57:21 UTC
Created attachment 290277 [details]
This is my /var/log/dmesg from my last boot from 2.6.23.-12.53

Here it is...

Comment 36 Gilbert Sebenste 2007-12-21 21:58:24 UTC
And in re: 

> 2. Try reducing the amount of memory with option "mem=2G"

How can I automagically do this upon reboot?

Comment 37 Chuck Ebbert 2007-12-21 23:52:25 UTC
Just add 

  mem=2000M

to the right line in /etc/grub.conf



Comment 38 Gilbert Sebenste 2007-12-23 00:57:33 UTC
OK, before I do that...I found something.

Some of my weather programs and other programs that I run require libf2c and
libgcc. They weren't installed; I thought they were and I had error messages
from some programs that use them, but I didn't notice anything wrong. Such as
libg2c.so.0 missing errors. Those went away when I did a yum install *libf2c*
and yum install *libgcc*.

Could that be causing the kernel panics that I saw? I have once again put
everything running back as it was. And I am using the -53 kernel now.
And if this was the problem, why would it cause the panics? Pure speculation, I
know.


Comment 39 Gilbert Sebenste 2007-12-24 01:22:58 UTC
I believe I have just found the problem. This machine has been up for more than
2 days now, and I am intentionally trying to crash it by flooding it with data,
which would normally take it out.

A number of the programs I use require FORTAN and GCC libraries. To my surprise,
most of them didn't get installed several months ago. The net result was this:
data that was coming in for post processing were routed to programs that were to
process it to a file. The programs didn't display any errors, but when I
manually ran them, I got a bunch of whatever.libso.0 errors...sure enough, GCC,
C++ and Fortran and their libraries were largely uninstalled. I figure that they
were causing I/o errors that caused the kernel to freak out. It shouldn't have
done that, but it still did.

If this is the case, about a week from now I will close this ticket and it's
sister that I created for it. Should I close them both and then open a new one,
or is a kernel panic expected in something like this, when I/O is under a lot of
stress and processes croak?


Comment 40 Gilbert Sebenste 2007-12-27 22:48:56 UTC
Well, scratch the above comment.
Dec 27 16:02:24 weather kernel: BUG: unable to handle kernel paging request at
virtual address fe11bf08
Dec 27 16:02:24 weather kernel:  printing eip:
Dec 27 16:02:24 weather kernel: c0466b69
Dec 27 16:02:24 weather kernel: *pde = 00000000
Dec 27 16:02:24 weather kernel: Oops: 0000 [#1]
Dec 27 16:02:24 weather kernel: SMP
Dec 27 16:02:24 weather kernel: Modules linked in: autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_hda_intel snd_emu10$
Dec 27 16:02:24 weather kernel: CPU:    0
Dec 27 16:02:24 weather kernel: EIP:    0060:[<c0466b69>]    Not tainted VLI
Dec 27 16:02:24 weather kernel: EFLAGS: 00210282   (2.6.23.12-53.fc7 #1)
Dec 27 16:02:24 weather kernel: EIP is at set_page_dirty+0x22/0x52
Dec 27 16:02:24 weather kernel: eax: 8001086c   ebx: c1be19a0   ecx: c1be19a0  
edx: fe11bed0

Comment 41 Gilbert Sebenste 2007-12-27 22:51:04 UTC
I also saw this in dmesg:

PCI: Setting latency timer of device 0000:04:00.0 to 64
scsi6 : ahci
ata7: SATA max UDMA/133 cmd 0xf8844100 ctl 0x00000000 bmdma 0x00000000 irq 20
ata7: SATA link down (SStatus 0 SControl 300)
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 37257352
ext3_orphan_cleanup: deleting unreferenced inode 6553739
ext3_orphan_cleanup: deleting unreferenced inode 6553896
ext3_orphan_cleanup: deleting unreferenced inode 6553619
ext3_orphan_cleanup: deleting unreferenced inode 6553674
ext3_orphan_cleanup: deleting unreferenced inode 6553661
ext3_orphan_cleanup: deleting unreferenced inode 6553868
ext3_orphan_cleanup: deleting unreferenced inode 6553653
ext3_orphan_cleanup: deleting unreferenced inode 6553895
ext3_orphan_cleanup: deleting unreferenced inode 6553845
ext3_orphan_cleanup: deleting unreferenced inode 6553874
ext3_orphan_cleanup: deleting unreferenced inode 6553684
ext3_orphan_cleanup: deleting unreferenced inode 6553905
ext3_orphan_cleanup: deleting unreferenced inode 37257258
ext3_orphan_cleanup: deleting unreferenced inode 37257259
ext3_orphan_cleanup: deleting unreferenced inode 37257260
ext3_orphan_cleanup: deleting unreferenced inode 37257256
ext3_orphan_cleanup: deleting unreferenced inode 37257261
ext3_orphan_cleanup: deleting unreferenced inode 37257254
ext3_orphan_cleanup: deleting unreferenced inode 37257247
ext3_orphan_cleanup: deleting unreferenced inode 37257244
EXT3-fs: dm-0: 21 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
audit(1198795158.093:2): selinux=0 auid=4294967295
sd 1:0:0:0: Attached scsi generic sg0 type 0

Comment 42 Gilbert Sebenste 2007-12-27 22:57:21 UTC
Could this be it?

http://forum.soft32.com/linux/BUG-unable-handle-paging-19-git-ftopict337184.html

Comment 43 Gilbert Sebenste 2008-01-07 15:24:29 UTC
Double oops this morning:

Jan  7 06:27:53 weather kernel: BUG: unable to handle kernel paging request at
virtual address feb2e938
Jan  7 06:27:53 weather kernel:  printing eip:
Jan  7 06:27:53 weather kernel: c04623b1
Jan  7 06:27:53 weather kernel: *pde = 00000000
Jan  7 06:27:53 weather kernel: Oops: 0000 [#1]
Jan  7 06:27:53 weather kernel: SMP
Jan  7 06:27:53 weather kernel: Modules linked in: lp autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_hda_intel snd_em$
Jan  7 06:27:53 weather kernel: CPU:    3
Jan  7 06:27:53 weather kernel: EIP:    0060:[<c04623b1>]    Not tainted VLI
Jan  7 06:27:53 weather kernel: EFLAGS: 00210286   (2.6.23.12-55.fc7 #1)
Jan  7 06:27:53 weather kernel: EIP is at sync_page+0x27/0x41
Jan  7 06:27:53 weather kernel: eax: 8001086d   ebx: f614ddfc   ecx: c21a98e0  
edx: feb2e900
Jan  7 06:27:53 weather kernel: esi: f614ddfc   edi: c300d4a4   ebp: c046238a  
esp: f614dde0
Jan  7 06:27:53 weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Jan  7 06:27:53 weather kernel: Process pqact (pid: 3297, ti=f614d000
task=f48c4000 task.ti=f614d000)
Jan  7 06:27:54 weather kernel: Stack: c061bc2a f614ddfc c21a98e0 f614de18
00048068 c046237c 00000002 c21a98e0
Jan  7 06:27:54 weather kernel:        00000000 00000001 f48c4000 c043d3fe
c300d4a8 c300d4a8 f6b2e910 f6b2e900
Jan  7 06:27:54 weather kernel:        c0462448 00000000 f6b2e858 f6227b40
f6d8ea80 c04641eb c27e87a0 c0466265
Jan  7 06:27:54 weather kernel: Call Trace:
Jan  7 06:27:54 weather kernel:  [<c061bc2a>] __wait_on_bit_lock+0x2a/0x52
Jan  7 06:27:54 weather kernel:  [<c046237c>] __lock_page+0x58/0x5e
Jan  7 06:27:54 weather kernel:  [<c043d3fe>] wake_bit_function+0x0/0x3c
Jan  7 06:27:54 weather kernel:  [<c0462448>] find_lock_page+0x5a/0x90
Jan  7 06:27:54 weather kernel:  [<c04641eb>] filemap_fault+0x9f/0x391
Jan  7 06:27:54 weather kernel:  [<c0466265>] get_page_from_freelist+0x25d/0x2db
Jan  7 06:27:54 weather kernel:  [<c046c3ac>] __do_fault+0x59/0x394
Jan  7 06:27:54 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:27:54 weather kernel:  [<c046aa02>] vma_prio_tree_insert+0x17/0x2a
Jan  7 06:27:54 weather kernel:  [<c0471899>] mmap_region+0x31c/0x3d8
Jan  7 06:27:54 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:27:54 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:27:54 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:27:54 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:27:54 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:27:54 weather kernel:  =======================
Jan  7 06:27:54 weather kernel: Code: 00 31 c0 c3 89 c1 0f ae f0 89 f6 8b 50 10
8b 00 66 85 c0 79 07 ba 40 0d 70 c0 eb 0f 8b 01 84 c0 78 1b f6 c2 01 75 16 85 d2
74 12 <8b> 42 38$
Jan  7 06:27:54 weather kernel: EIP: [<c04623b1>] sync_page+0x27/0x41 SS:ESP
0068:f614dde0

And then....

Jan  7 06:30:01 weather kernel: invalid opcode: 0000 [#2]
Jan  7 06:30:01 weather kernel: SMP
Jan  7 06:30:01 weather kernel: Modules linked in: lp autofs4 hidp rfcomm l2cap
bluetooth sunrpc dm_multipath video output sbs battery ac ipv6 kvm_intel kvm
snd_hda_intel snd_em$
Jan  7 06:30:01 weather kernel: CPU:    0  
Jan  7 06:30:01 weather kernel: EIP:    0060:[<c27e87a8>]    Tainted: G      D VLI
Jan  7 06:30:01 weather kernel: EFLAGS: 00210082   (2.6.23.12-55.fc7 #1)
Jan  7 06:30:01 weather kernel: EIP is at 0xc27e87a8
Jan  7 06:30:01 weather kernel: eax: f614de94   ebx: f614de94   ecx: 00000000  
edx: 00000003
Jan  7 06:30:01 weather kernel: esi: f614df30   edi: 00000001   ebp: d67f4e38  
esp: d67f4e18
Jan  7 06:30:01 weather kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Jan  7 06:30:01 weather kernel: Process grep (pid: 22667, ti=d67f4000
task=f4610000 task.ti=d67f4000)
Jan  7 06:30:01 weather kernel: Stack: c04244ba d67f4e68 00000003 c300d4a4
00000000 c300d4a4 d67f4e68 00000001
Jan  7 06:30:01 weather kernel:        d67f4e5c c0426492 00000000 d67f4e68
00000003 00200282 c300d4a4 00000000
Jan  7 06:30:01 weather kernel:        fffaa12c f481f540 c043d3ae d67f4e68
c2bee480 00000000 c2bee480 c046c6b8
Jan  7 06:30:01 weather kernel: Call Trace:
Jan  7 06:30:01 weather kernel:  [<c04244ba>] __wake_up_common+0x32/0x55
Jan  7 06:30:01 weather kernel:  [<c0426492>] __wake_up+0x32/0x43
Jan  7 06:30:01 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:01 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:01 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:01 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:01 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:01 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:01 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:01 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:01 weather kernel:  =======================
Jan  7 06:30:01 weather kernel: Code: ee 2c c2 2c 08 00 80 02 00 00 00 ff ff ff
ff e0 67 dc e9 30 43 83 c6 00 00 00 00 d8 05 0e c2 38 13 5a c2 00 00 08 80 00 00
00 00 <ff> ff ff$
Jan  7 06:30:01 weather kernel: EIP: [<c27e87a8>] 0xc27e87a8 SS:ESP 0068:d67f4e18

Comment 44 Gilbert Sebenste 2008-01-07 15:26:42 UTC
Whoops, wait, there was more...

Jan  7 06:30:13 weather kernel: BUG: soft lockup - CPU#0 stuck for 11s!
[grep:22701] 
Jan  7 06:30:13 weather kernel:
Jan  7 06:30:13 weather kernel: Pid: 22701, comm:                 grep
Jan  7 06:30:13 weather kernel: EIP: 0060:[<c061ca41>] CPU: 0
Jan  7 06:30:13 weather kernel: EIP is at _spin_lock_irqsave+0x34/0x4e
Jan  7 06:30:13 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:13 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:13 weather kernel: ESI: c9fdee68 EDI: 00000001 EBP: c9fdee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:13 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 1bc12000 CR4:
000026d0
Jan  7 06:30:13 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:13 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:13 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:13 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:13 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:13 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:13 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:13 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:13 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef   
Jan  7 06:30:13 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:13 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:13 weather kernel:  =======================
Jan  7 06:30:13 weather kernel: BUG: soft lockup - CPU#1 stuck for 11s! [grep:22670]
Jan  7 06:30:13 weather kernel:
Jan  7 06:30:13 weather kernel: Pid: 22670, comm:                 grep
Jan  7 06:30:13 weather kernel: EIP: 0060:[<c061ca3e>] CPU: 1
Jan  7 06:30:13 weather kernel: EIP is at _spin_lock_irqsave+0x31/0x4e
Jan  7 06:30:13 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:13 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:13 weather kernel: ESI: ea98ee68 EDI: 00000001 EBP: ea98ee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:13 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 13430000 CR4:
000026d0
Jan  7 06:30:13 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:13 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:13 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:13 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:13 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394 
Jan  7 06:30:13 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:13 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef 
Jan  7 06:30:13 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:13 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:13 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:13 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:13 weather kernel:  =======================
Jan  7 06:30:18 weather rpcbind: connect from 128.163.192.19 to
getport/addr(300029): request from unauthorized host
Jan  7 06:30:18 weather rpcbind: connect from 128.163.192.19 to
getport/addr(300029): request from unauthorized host
Jan  7 06:30:24 weather kernel: BUG: soft lockup - CPU#0 stuck for 11s! [grep:22701]
Jan  7 06:30:24 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:24 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:24 weather kernel: ESI: c9fdee68 EDI: 00000001 EBP: c9fdee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:24 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 1bc12000 CR4:
000026d0
Jan  7 06:30:24 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:24 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:24 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:24 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:24 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:24 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:24 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:24 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:24 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:24 weather kernel:  [<c061cd0a>] error_code+0x72/0x78   
Jan  7 06:30:24 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:24 weather kernel:  =======================
Jan  7 06:30:25 weather kernel: BUG: soft lockup - CPU#1 stuck for 11s! [grep:22670]
Jan  7 06:30:25 weather kernel:
Jan  7 06:30:25 weather kernel: Pid: 22670, comm:                 grep  
Jan  7 06:30:25 weather kernel: EIP: 0060:[<c061ca3e>] CPU: 1
Jan  7 06:30:25 weather kernel: EIP is at _spin_lock_irqsave+0x31/0x4e  
Jan  7 06:30:25 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:25 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:25 weather kernel: ESI: ea98ee68 EDI: 00000001 EBP: ea98ee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:25 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 13430000 CR4:
000026d0
Jan  7 06:30:25 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:25 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:25 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:25 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:25 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:25 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:25 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:25 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:25 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:25 weather kernel:  [<c061cd0a>] error_code+0x72/0x78   
Jan  7 06:30:25 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:25 weather kernel:  =======================
Jan  7 06:30:36 weather kernel: BUG: soft lockup - CPU#0 stuck for 11s! [grep:22701]
Jan  7 06:30:36 weather kernel:
Jan  7 06:30:36 weather kernel: Pid: 22701, comm:                 grep
Jan  7 06:30:36 weather kernel: EIP: 0060:[<c061ca3e>] CPU: 0
Jan  7 06:30:36 weather kernel: EIP is at _spin_lock_irqsave+0x31/0x4e  
Jan  7 06:30:36 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:36 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:36 weather kernel: ESI: c9fdee68 EDI: 00000001 EBP: c9fdee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:36 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 1bc12000 CR4:
000026d0
Jan  7 06:30:36 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:36 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:36 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:36 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:36 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:36 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:36 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:36 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:36 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:36 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:36 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:36 weather kernel:  =======================
Jan  7 06:30:36 weather kernel: BUG: soft lockup - CPU#1 stuck for 11s! [grep:22670]
Jan  7 06:30:36 weather kernel:
Jan  7 06:30:36 weather kernel: Pid: 22670, comm:                 grep 
Jan  7 06:30:36 weather kernel: EIP: 0060:[<c061ca41>] CPU: 1
Jan  7 06:30:36 weather kernel: EIP is at _spin_lock_irqsave+0x34/0x4e
Jan  7 06:30:36 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:36 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:36 weather kernel: ESI: ea98ee68 EDI: 00000001 EBP: ea98ee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:36 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 13430000 CR4:
000026d0
Jan  7 06:30:36 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:36 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:36 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:36 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33   
Jan  7 06:30:36 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:36 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:36 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:36 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:36 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:36 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:36 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:36 weather kernel:  =======================
Jan  7 06:30:48 weather rpcbind: connect from 128.163.192.19 to
getport/addr(300029): request from unauthorized host
Jan  7 06:30:48 weather rpcbind: connect from 128.163.192.19 to
getport/addr(300029): request from unauthorized host
Jan  7 06:30:48 weather kernel: BUG: soft lockup - CPU#0 stuck for 11s! [grep:22701]
Jan  7 06:30:48 weather kernel:
Jan  7 06:30:48 weather kernel: Pid: 22701, comm:                 grep
Jan  7 06:30:48 weather kernel: EIP: 0060:[<c061ca3e>] CPU: 0
Jan  7 06:30:48 weather kernel: EIP is at _spin_lock_irqsave+0x31/0x4e  
Jan  7 06:30:48 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:48 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:48 weather kernel: ESI: c9fdee68 EDI: 00000001 EBP: c9fdee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:48 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 1bc12000 CR4:
000026d0
Jan  7 06:30:48 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:48 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:48 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:48 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:48 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:48 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:48 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:48 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:48 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:48 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:30:48 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:48 weather kernel:  =======================
Jan  7 06:30:48 weather kernel: BUG: soft lockup - CPU#1 stuck for 11s! [grep:22670]
Jan  7 06:30:48 weather kernel:
Jan  7 06:30:48 weather kernel: Pid: 22670, comm:                 grep
Jan  7 06:30:48 weather kernel: EIP: 0060:[<c061ca41>] CPU: 1
Jan  7 06:30:48 weather kernel: EIP is at _spin_lock_irqsave+0x34/0x4e  
Jan  7 06:30:48 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:30:48 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:30:48 weather kernel: ESI: ea98ee68 EDI: 00000001 EBP: ea98ee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:30:48 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 13430000 CR4:
000026d0
Jan  7 06:30:48 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:30:48 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:30:48 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:30:48 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33
Jan  7 06:30:48 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:30:48 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:30:48 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:30:48 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:30:48 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:30:48 weather kernel:  [<c061cd0a>] error_code+0x72/0x78      
Jan  7 06:30:48 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:30:48 weather kernel:  =======================
Jan  7 06:31:00 weather kernel: BUG: soft lockup - CPU#0 stuck for 11s! [grep:22701]
Jan  7 06:31:00 weather kernel:
Jan  7 06:31:00 weather kernel: Pid: 22701, comm:                 grep
Jan  7 06:31:00 weather kernel: EIP: 0060:[<c061ca3c>] CPU: 0
Jan  7 06:31:00 weather kernel: EIP is at _spin_lock_irqsave+0x2f/0x4e  
Jan  7 06:31:00 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:31:00 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:31:00 weather kernel: ESI: c9fdee68 EDI: 00000001 EBP: c9fdee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:31:00 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 1bc12000 CR4:
000026d0
Jan  7 06:31:00 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:31:00 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:31:00 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:31:00 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33   
Jan  7 06:31:00 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:31:00 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:31:00 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:31:00 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:31:00 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:31:00 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:31:00 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:31:00 weather kernel:  =======================
Jan  7 06:31:00 weather kernel: BUG: soft lockup - CPU#1 stuck for 11s! [grep:22670]
Jan  7 06:31:00 weather kernel:
Jan  7 06:31:00 weather kernel: Pid: 22670, comm:                 grep 
Jan  7 06:31:00 weather kernel: EIP: 0060:[<c061ca3e>] CPU: 1
Jan  7 06:31:00 weather kernel: EIP is at _spin_lock_irqsave+0x31/0x4e
Jan  7 06:31:00 weather kernel:  EFLAGS: 00200282    Tainted: G      D 
(2.6.23.12-55.fc7 #1)
Jan  7 06:31:00 weather kernel: EAX: 00200282 EBX: c300d4a4 ECX: 00200282 EDX:
c300d4a4
Jan  7 06:31:00 weather kernel: ESI: ea98ee68 EDI: 00000001 EBP: ea98ee5c DS:
007b ES: 007b FS: 00d8
Jan  7 06:31:00 weather kernel: CR0: 8005003b CR2: 0804b500 CR3: 13430000 CR4:
000026d0
Jan  7 06:31:00 weather kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3:
00000000
Jan  7 06:31:00 weather kernel: DR6: ffff0ff0 DR7: 00000400
Jan  7 06:31:00 weather kernel:  [<c0426478>] __wake_up+0x18/0x43
Jan  7 06:31:00 weather kernel:  [<c043d3ae>] __wake_up_bit+0x2e/0x33   
Jan  7 06:31:00 weather kernel:  [<c046c6b8>] __do_fault+0x365/0x394
Jan  7 06:31:00 weather kernel:  [<c046ea2e>] handle_mm_fault+0x3a0/0x78b
Jan  7 06:31:00 weather kernel:  [<c061e28c>] do_page_fault+0x26a/0x5ef
Jan  7 06:31:00 weather kernel:  [<c0458fc6>] audit_syscall_exit+0x2aa/0x2c6
Jan  7 06:31:00 weather kernel:  [<c061e022>] do_page_fault+0x0/0x5ef
Jan  7 06:31:00 weather kernel:  [<c061cd0a>] error_code+0x72/0x78
Jan  7 06:31:00 weather kernel:  [<c0610000>] xfrm_do_migrate+0x9e/0x10b
Jan  7 06:31:00 weather kernel:  =======================

Comment 45 Gilbert Sebenste 2008-01-15 19:56:02 UTC
*** Bug 344821 has been marked as a duplicate of this bug. ***

Comment 46 Gilbert Sebenste 2008-01-15 19:58:01 UTC
I have decided to close this and another bug that I believe is a duplicate. The
problem may have been found, but I want to test it before I open another bug
report. if I am correct, this one is worded wrong.