From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Description of problem: Running a simple RAID1 configuration on 2 IDE devices. When booted into 2.6.9-1.681_FC3, any heavy IO, (although writing seems to cause more problems) causes a kernel panic, then endless errors to console. Version-Release number of selected component (if applicable): 2.6.9-1.681_FC3 How reproducible: Always Steps to Reproduce: 1. mdadm --create -l 1 -n 2 /dev/md0 /dev/hdc1 /dev/hdd1 2. mkfs.ext3/mount /dev/md0 /raid 3. cd /raid; tar -xvpf bigtarfile.tar Actual Results: after a few mins, kernel dumps LOTS of data to console (too fast to read) then endlessly dumps below error to the console and keyboard lights flash. "atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly." Expected Results: complete writing to the raid device without issue. Additional info: Worth noting, this happens even in single user mode (ie, no X running). Have not tested to see if this is a problem on i386 build. The above results occur about 90% of the time. I've also encountered complete lockups, and sometimes massive amounts of memeory being consumed until the kernel starts killing off processes, and eventually freezes.
not sure if this is obvious in the ticket or not, but experencing this problem on the x86_64 platform.
so, i've tracked this problem down to the libata kernel module. I do not have any SATA devices, but just with the module loaded in the kernel, panics soon follow. panics are less in 2.6.9-1.667, but still happen. Definately a huge problem in 2.6.9-1.681_FC3.
any chance you can capture and paste one of the panic messages ?
Finally caught some kernel messages... Seem to be catching mostly all Oops messages. login: Unable to handle kernel paging request at 000000fff40561d4 RIP: <ffffffff80166f88>{free_pages_bulk+575} PML4 0 Oops: 0002 [1] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext 3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 48, comm: kswapd0 Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166f88>] <ffffffff80166f88>{free_pages_bulk+575} RSP: 0018:0000010037c85b18 EFLAGS: 00010093 RAX: ffffffff80432fb8 RBX: ffffffff80432dc0 RCX: 0000000000000000 RDX: 0000010001e10680 RSI: ffffffff80432eb0 RDI: ffffffff80432dc0 RBP: 6db6db692245b567 R08: 0000000000000000 R09: 0000010037c85b90 R10: 000001002d532088 R11: ffffffffa00624f8 R12: ffffffff80432e88 R13: 000001000160c848 R14: ffffffff80432eb0 R15: 36db6db49122dab3 FS: 0000002a95577a20(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000fff40561d4 CR3: 0000000000101000 CR4: 00000000000006e0 Process kswapd0 (pid: 48, threadinfo 0000010037c84000, task 0000010037c831d0) Stack: 0000010037c85b78 ffffffffffffffff 0000000001788138 0000000000000000 0000000900000056 0000010001038000 ffffffff80432eb0 000000003c6bd460 ffffffff80432fe0 0000000631958898 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801676ac>{__pagevec_free+38} <ffffffff8016f41a>{release_pages+642} <ffffffff80162659>{__remove_from_page_cache+25} <ffffffff8016f9f6>{__pagevec_release+23} <ffffffff8016ff8c>{invalidate_mapping_pages+163} <ffffffff801adf60>{shrink_icache_memory+571} <ffffffff8017027f>{shrink_slab+188} <ffffffff80172021>{balance_pgdat+504} <ffffffff801721d8>{kswapd+224} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80133068>{schedule_tail+11} <ffffffff8011124f>{child_rip+8} <ffffffff80155343>{kthread+0} <ffffffff801720f8>{kswapd+0} <ffffffff80111247>{child_rip+0} Code: 44 0f bb 3a 19 c0 85 c0 0f 84 a4 00 00 00 0f b6 4c 24 14 b8 RIP <ffffffff80166f88>{free_pages_bulk+575} RSP <0000010037c85b18> CR2: 000000fff40561d4 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0) already locked by mm/page_alloc.c/259 Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 c7f8067 PGD 30577067 PMD 0 Oops: 0002 [2] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext 3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 1655, comm: kjournald Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:000001003bc87908 EFLAGS: 00010006 RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 000000000000eddd R09: 000000000000eddd R10: 0000010038f43bc0 R11: ffffffff8028b44c R12: ffffffff804332c0 R13: 000001000175cfb8 R14: 000001003bed3610 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process kjournald (pid: 1655, threadinfo 000001003bc86000, task 000001003bed3610) Stack: 0000010037cf2c00 0000000000000246 0000000000000000 000002003bed3610 0000000000000046 000001003bc87960 0000000000000056 0000000000000000 0000000000000200 ffffffff804332c0 Call Trace:<ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80284d5b>{submit_bio+219} <ffffffff8019071c>{bio_alloc+28} <ffffffff8018e015>{submit_bh+144} <ffffffff8018e0fd>{ll_rw_block+93} <ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f9ee>{:jbd:kjournald+524} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f7dc>{:jbd:commit_timeout+0} <ffffffff80133068>{schedule_tail+11} <ffffffff8011124f>{child_rip+8} <ffffffffa004f7e2>{:jbd:kjournald+0} <ffffffff80111247>{child_rip+0} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003bc87908> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 c7f8067 PGD 30577067 PMD 0 Oops: 0002 [3] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext 3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:0000010037c0d7b8 EFLAGS: 00010006 RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 0000010037c0da50 R09: 00000000fffffffa R10: 00000100013a7408 R11: 0000000000000000 R12: ffffffff804332c0 R13: 000001000175cfb8 R14: 0000010037c08920 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process pdflush (pid: 46, threadinfo 0000010037c0c000, task 0000010037c08920) Stack: 0000010037c0d7e0 0000010037ce1c00 0000000000021a01 0000020000000220 0000000000000046 000001003a66cda0 0000000000000056 0000000000000000 0000000000000200 ffffffff804332c0 Call Trace:<ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8019071c>{bio_alloc+28} <ffffffff8019096e>{bio_clone+13} <ffffffffa003f1fb>{:raid1:make_request+1303} <ffffffff80284c6b>{generic_make_request+361} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80284d5b>{submit_bio+219} <ffffffff80190820>{bio_alloc+288} <ffffffff8018e080>{submit_bh+251} <ffffffff8018f520>{__block_write_full_page+432} <ffffffff8019433b>{blkdev_get_block+0} <ffffffff801ba791>{mpage_writepages+561} <ffffffff801bad45>{mpage_writepages+2021} <ffffffff80194412>{blkdev_writepage+0} <ffffffff8010ebbc>{__switch_to+254} <ffffffff803557da>{thread_return+41} <ffffffff801b89bd>{__writeback_single_inode+662} <ffffffff801b96e4>{sync_sb_inodes+470} <ffffffff801b9d6e>{writeback_inodes+605} <ffffffff80169055>{wb_kupdate+147} <ffffffff803557da>{thread_return+41} <ffffffff8016a085>{pdflush+0} <ffffffff8016a4a7>{pdflush+1058} <ffffffff80168fc2>{wb_kupdate+0} <ffffffff801553ff>{kthread+188} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff8011124f>{child_rip+8} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <0000010037c0d7b8> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 c7f8067 PGD 30577067 PMD 0 Oops: 0002 [4] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext 3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:ffffffff8049fa68 EFLAGS: 00010006 RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 0000000000000000 R09: ffffffff8049fbb8 R10: 000000002640f942 R11: 000001003c245900 R12: ffffffff804332c0 R13: 000001000175cfb8 R14: 0000010037c08920 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process pdflush (pid: 46, threadinfo 0000010037c0c000, task 0000010037c08920) Stack: 0000000000000206 ffffffff8030d164 000001003c54f708 000002200000000f 0550000000000046 ffffffff8049fbb0 0000000000000056 0000000000000000 0000000000000220 ffffffff804332c0 Call Trace:<IRQ> <ffffffff8030d164>{__ip_route_output_key+1880} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff802e953a>{sk_alloc+42} <ffffffff8032be09>{tcp_create_openreq_child+50} <ffffffff803281d2>{tcp_v4_syn_recv_sock+85} <ffffffffa0262001>{:ipv6:tcp_v6_syn_recv_sock+64} <ffffffff80327b8f>{tcp_v4_send_synack+202} <ffffffff8032c705>{tcp_check_req+526} <ffffffff803020e9>{qdisc_restart+26} <ffffffff802f12f7>{dev_queue_xmit+1223} <ffffffff802f6e0c>{neigh_resolve_output+516} <ffffffff802fb1f2>{nf_iterate+82} <ffffffff803117ec>{ip_finish_output2+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff80311b68>{ip_finish_output+473} <ffffffff803114ac>{dst_output+0} <ffffffff803114c2>{dst_output+22} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff80311f62>{ip_queue_xmit+1007} <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33} <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} <ffffffff80122312>{dma_map_sg+622} <ffffffff80132a95>{recalc_task_prio+337} <ffffffff802ac382>{dma_timer_expiry+0} <ffffffff802abfae>{ide_dma_intr+0} <ffffffffa01f4388>{:iptable_nat:do_bindings+169} <ffffffff803285f8>{tcp_v4_do_rcv+207} <ffffffff80328edd>{tcp_v4_rcv+1712} <ffffffff8030ea83>{ip_local_deliver_finish+236} <ffffffff8030e997>{ip_local_deliver_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030ecec>{ip_local_deliver+468} <ffffffff8030eecb>{ip_rcv_finish+472} <ffffffff8030ecf3>{ip_rcv_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030f2fd>{ip_rcv+1008} <ffffffff802f191f>{netif_receive_skb+572} <ffffffffa009baaf>{:e100:e100_poll+710} <ffffffff802f191f>{netif_receive_skb+572} <ffffffff802f19ce>{process_backlog+125} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff80111da2>{oops_end+159} <ffffffff80111d15>{oops_end+18} <ffffffff80124595>{do_page_fault+1074} <ffffffff80111099>{error_exit+0} <ffffffff80167885>{buffered_rmqueue+446} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8019071c>{bio_alloc+28} <ffffffff8019096e>{bio_clone+13} <ffffffffa003f1fb>{:raid1:make_request+1303} <ffffffff80284c6b>{generic_make_request+361} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80284d5b>{submit_bio+219} <ffffffff80190820>{bio_alloc+288} <ffffffff8018e080>{submit_bh+251} <ffffffff8018f520>{__block_write_full_page+432} <ffffffff8019433b>{blkdev_get_block+0} <ffffffff801ba791audit(1104659019.304:0): initialized Sometimes after a kernel paging request oops messages, also get lots of these at random blocks: __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096 __find_get_block_slow() failed. block=1715, b_blocknr=4448602 b_state=0x00000019, b_size=4096 device blocksize: 4096
got another one this morning. [snip] lockd: couldn't shutdown host module! Bad page state at prep_new_page (in process 'rsync', page 0000010001a0b4f8) flags:0x2000103c mapping:00000100230ff450 mapcount:0 count:2 Backtrace: Call Trace:<ffffffff80166d1d>{bad_page+114} <ffffffff80167a57>{buffered_rmqueue+912} <ffffffff80167b85>{__alloc_pages+200} <ffffffff8016482b>{generic_file_buffered_write+320} <ffffffff8016503f>{generic_file_aio_write_nolock+735} <ffffffff801650ef>{generic_file_aio_write+125} <ffffffffa0060388>{:ext3:ext3_file_write+22} <ffffffff8018a9d0>{do_sync_write+173} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff801c3a5c>{compat_sys_select+1428} <ffffffff8018aacb>{vfs_write+207} <ffffffff8018abb3>{sys_write+69} <ffffffff80125dc3>{cstar_do_call+27} Trying to fix it up, but a reboot is needed Unable to handle kernel paging request at 0000000000200200 RIP: <ffffffff80166e63>{free_pages_bulk+282} PML4 5a5d067 PGD 28852067 PMD 0 Oops: 0002 [1] CPU 0 Modules linked in: nfs lockd parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_c onntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 27745, comm: rsync Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e63>] <ffffffff80166e63>{free_pages_bulk+282} RSP: 0018:0000010007d336c8 EFLAGS: 00010017 RAX: 000001000149a408 RBX: ffffffff80432dc0 RCX: 00000100010b0e58 RDX: 0000000000200200 RSI: 0000010001a0b520 RDI: ffffffff80432dc0 RBP: 000000000002cea8 R08: 0000000000000000 R09: 0000000000000040 R10: 0000000000000040 R11: 0000000000040000 R12: 0000010001a0b4f8 R13: 0000010001a0b4f8 R14: ffffffff80432ec8 R15: 000000000000b3aa FS: 0000002a95577a20(0000) GS:ffffffff80511980(005b) knlGS:00000000f7fdf6c0 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 0000000000200200 CR3: 0000000000101000 CR4: 00000000000006e0 Process rsync (pid: 27745, threadinfo 0000010007d32000, task 0000010038cfb750) Stack: 0000010000000005 fffffffffffffffe 000000013fb95400 0000000000000000 000000018bab5fdf 0000010001038000 ffffffff80432eb0 0000000007d339d0 ffffffff80432fe0 0000000e3fb95400 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801676ac>{__pagevec_free+38} <ffffffff8016f4af>{release_pages+791} <ffffffff801635e4>{find_get_pages_tag+179} <ffffffff8016f9f6>{__pagevec_release+23} <ffffffff801bad2d>{mpage_writepages+1997} <ffffffffa0048a6f>{:jbd:do_get_write_access+1776} <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0} <ffffffffa0063053>{:ext3:ext3_mark_iloc_dirty+802} <ffffffff801b89bd>{__writeback_single_inode+662} <ffffffff801b96e4>{sync_sb_inodes+470} <ffffffff801b9d6e>{writeback_inodes+605} <ffffffff80168e41>{balance_dirty_pages_ratelimited+197} <ffffffff80164aff>{generic_file_buffered_write+1044} <ffffffff8016503f>{generic_file_aio_write_nolock+735} <ffffffff801650ef>{generic_file_aio_write+125} <ffffffffa0060388>{:ext3:ext3_file_write+22} <ffffffff8018a9d0>{do_sync_write+173} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff801c3a5c>{compat_sys_select+1428} <ffffffff8018aacb>{vfs_write+207} <ffffffff8018abb3>{sys_write+69} <ffffffff80125dc3>{cstar_do_call+27} Code: 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 10 00 8b 54 RIP <ffffffff80166e63>{free_pages_bulk+282} RSP <0000010007d336c8> CR2: 0000000000200200 <1>Unable to handle kernel paging request at 0000000000200200 RIP: <ffffffff80167889>{buffered_rmqueue+450} PML4 5a5d067 PGD 28852067 PMD 0 Oops: 0002 [2] CPU 0 Modules linked in: nfs lockd parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_c onntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 sata_via libata sd_mod scsi_mod Pid: 27745, comm: rsync Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167889>] <ffffffff80167889>{buffered_rmqueue+450} RSP: 0018:ffffffff8049fc88 EFLAGS: 00010006 RAX: 0000010001a0b520 RBX: ffffffff80432dc0 RCX: 0000000000200200 RDX: 000001000149a408 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 0101a8c0ed01a8c0 R09: 0000000000020000 R10: 0000000000000003 R11: 0000000000000000 R12: ffffffff804332c0 R13: 0000010001a0b4f8 R14: 0000010038cfb750 R15: 0000000000000000 FS: 0000002a95577a20(0000) GS:ffffffff80511980(005b) knlGS:00000000f7fdf6c0 CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b CR2: 0000000000200200 CR3: 0000000000101000 CR4: 00000000000006e0 Process rsync (pid: 27745, threadinfo 0000010007d32000, task 0000010038cfb750) Stack: 000001001f6d4800 000001001f6d4800 0000000000000000 0000022000001000 00000100063ebf68 ffffffffa02620df 0000000000000056 0000000000000000 0000000000000220 ffffffff804332c0 Call Trace:<IRQ> <ffffffffa02620df>{:ipv6:tcp_v6_syn_recv_sock+286} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80327c90>{tcp_v4_conn_request+170} <ffffffffa01e6ecb>{:ip_conntrack:ip_ct_refresh_acct+164} <ffffffff802f0dd9>{__skb_linearize+252} <ffffffff803020e9>{qdisc_restart+26} <ffffffff802f12f7>{dev_queue_xmit+1223} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff80311b68>{ip_finish_output+473} <ffffffffa00a93f3>{:via_velocity:velocity_intr+1738} <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33} <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33} <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} <ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121} <ffffffffa01f41cc>{:iptable_nat:ip_nat_setup_info+2282} <ffffffffa01f41df>{:iptable_nat:ip_nat_setup_info+2301} <ffffffff80113176>{handle_IRQ_event+40} <ffffffff801137fb>{do_IRQ+588} <ffffffff8031f9b9>{tcp_rcv_state_process+114} <ffffffff803287e6>{tcp_v4_do_rcv+701} <ffffffff80328edd>{tcp_v4_rcv+1712} <ffffffff8030ea83>{ip_local_deliver_finish+236} <ffffffff8030e997>{ip_local_deliver_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030ecec>{ip_local_deliver+468} <ffffffff8030eecb>{ip_rcv_finish+472} <ffffffff8030ecf3>{ip_rcv_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030f2fd>{ip_rcv+1008} <ffffffff802f191f>{netif_receive_skb+572} <ffffffff802f19ce>{process_backlog+125} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff80111da2>{oops_end+159} <ffffffff80111d15>{oops_end+18} <ffffffff80124595>{do_page_fault+1074} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff80111099>{error_exit+0} <ffffffff80166e63>{free_pages_bulk+282} <ffffffff80166fe5>{free_pages_bulk+668} <ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801676ac>{__pagevec_free+38} <ffffffff8016f4af>{release_pages+791} <ffffffff801635e4>{find_get_pages_tag+179} <ffffffff8016f9f6>{__pagevec_release+23} <ffffffff801bad2d>{mpage_writepages+1997} <ffffffffa0048a6f>{:jbd:do_get_write_access+1776} <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0} <ffffffffa0063053>{:ext3:ext3_mark_iloc_dirty+802} [/snip] then the machine auto-rebooted (configured to reboot after 30 seconds on panic) No find_get_block_slow messages this time.
all of the above oops/panic messages are in kernel 2.6.9-1.681_FC3 WITH sata_via/libata/sd_mod/scsi_mod unloaded from the kernel and see if the symptoms change.
2 more reboots from oops/panics in the last 24 hours. The "Spurious ACK on isa0060/serio0." message is back. These are with 2.6.9-1.681_FC3 and the sata_via/libata/sd_mod/scsi_mod kernel modules NOT loaded... When the modules are not loaded, it seems to change the behaviour of the oops/panics. Btw, let me know when you guys have had enough.... Otherwise i'll just try various kernel/module combinations to see if i can get them to stop.... and continue posting results. Karl Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80166e5f>{free_pages_bulk+278} PML4 1dcff067 PGD 32592067 PMD 0 Oops: 0002 [1] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278} RSP: 0018:0000010037c0d9f8 EFLAGS: 00010017 RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: 0000010001a811a8 RDX: 0000000000200200 RSI: 00000100015ba390 RDI: ffffffff80432dc0 RBP: 00000000000192e8 R08: 0000000000000000 R09: 0000000000000040 R10: 0000000000000040 R11: 0000000000040000 R12: 00000100015ba368 R13: 00000100015ba330 R14: ffffffff80432ee0 R15: 000000000000325d FS: 0000002a95578460(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process pdflush (pid: 46, threadinfo 0000010037c0c000, task 0000010037c08920) Stack: 000001003e8ac840 fffffffffffffffc 000000023e8ac840 0000000000000000 0000000100000050 0000010001038000 ffffffff80432eb0 0000000037c0dd00 ffffffff80432fe0 0000000e3fb91800 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801676ac>{__pagevec_free+38} <ffffffff8016f4af>{release_pages+791} <ffffffff801635e4>{find_get_pages_tag+179} <ffffffff8016f9f6>{__pagevec_release+23} <ffffffff801bad2d>{mpage_writepages+1997} <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0} <ffffffff8010ebbc>{__switch_to+254} <ffffffff803557da>{thread_return+41} <ffffffff803566d3>{schedule_timeout+295} <ffffffff801b89bd>{__writeback_single_inode+662} <ffffffff801b96e4>{sync_sb_inodes+470} <ffffffff801b9d6e>{writeback_inodes+605} <ffffffff8016a085>{pdflush+0} <ffffffff80168f5e>{background_writeout+108} <ffffffff8016a4a7>{pdflush+1058} <ffffffff80168ef2>{background_writeout+0} <ffffffff801553ff>{kthread+188} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff8011124f>{child_rip+8} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037c0d9f8> CR2: 0000000000100108 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0) already locked by mm/page_alloc.c/259 mm/slab.c:2204: spin_is_locked on uninitialized spinlock 0000010001a81218. kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40) already locked by kernel/printk.c/551 kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40) already locked by kernel/printk.c/551 kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40) already locked by kernel/printk.c/551 kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40) already locked by kernel/printk.c/551 kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40) already locked by kernel/printk.c/551 Unable to handle kernel paging request at 00000000ffffffff RIP: <ffffffff8021331b>{strnlen+12} PML4 1dcff067 PGD 0 Oops: 0000 [2] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 3, comm: events/0 Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff8021331b>] <ffffffff8021331b>{strnlen+12} RSP: 0018:ffffffff804a0160 EFLAGS: 00010097 RAX: 00000000ffffffff RBX: ffffffff804a0238 RCX: 000000000000000a RDX: ffffffff804a0270 RSI: fffffffffffffffe RDI: 00000000ffffffff RBP: 00000000ffffffff R08: 00000000ffffffff R09: 0000000000000003 R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff804ab67a R13: 00000000ffffffff R14: 0000000000000000 R15: ffffffff804aba5f FS: 0000002a95578460(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000000ffffffff CR3: 0000000000101000 CR4: 00000000000006e0 Process events/0 (pid: 3, threadinfo 0000010037e3e000, task 0000010002140030) Stack: ffffffff80213bbd 0000000000000400 ffffffff804ab660 ffffffff8036951f 0000000000000400 ffffffff8036950d 0000010001af9408 0000000000000000 000001003d88c012 0000000000000000 Call Trace:<IRQ> <ffffffff80213bbd>{vsnprintf+848} <ffffffff80213e3a>{vscnprintf+9} <ffffffff80138a46>{vprintk+421} <ffffffff802ac713>{__ide_dma_read+154} <ffffffff802ac80e>{__ide_dma_begin+40} <ffffffff802af525>{__ide_do_rw_disk+853} <ffffffff80138d8b>{printk+141} <ffffffff802a5076>{ide_do_request+977} <ffffffff8016cef8>{cache_flusharray+138} <ffffffff8016cbf5>{kfree+106} <ffffffff802ebbad>{kfree_skbmem+9} <ffffffffa009bf46>{:e100:e100_poll+1885} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff801670d7>{free_pages_bulk+910} <ffffffff801671cf>{__free_pages_ok+229} <ffffffff80134add>{free_task+18} <ffffffff803557da>{thread_return+41} <ffffffffa02171c3>{:sunrpc:do_cache_clean+0} <ffffffff8014f1bd>{worker_thread+219} <ffffffff801337a8>{default_wake_function+0} <ffffffff801337a8>{default_wake_function+0} <ffffffff8014f0e2>{worker_thread+0} <ffffffff801553ff>{kthread+188} <ffffffff8011124f>{child_rip+8} <ffffffff801fdc0f>{selinux_d_instantiate+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Code: 80 3f 00 74 11 48 ff ce 48 ff c0 48 83 fe ff 74 05 80 38 00 RIP <ffffffff8021331b>{strnlen+12} RSP <ffffffff804a0160> CR2: 00000000ffffffff <3>Debug: sleeping function called from invalid context at include/linux/rwsem.h:43 in_atomic():1[expected: 0], irqs_disabled():0 Call Trace:<IRQ> <ffffffff80133173>{__might_sleep+173} <ffffffff80139d4d>{profile_task_exit+33} <ffffffff8013b87a>{do_exit+34} <ffffffff80111da2>{oops_end+159} <ffffffff801245e6>{do_page_fault+1155} <ffffffffa01f41cc>{:iptable_nat:ip_nat_setup_info+2282} <ffffffff803406cb>{fib_lookup+188} <ffffffff80111099>{error_exit+0} <ffffffff8021331b>{strnlen+12} <ffffffff80213bbd>{vsnprintf+848} <ffffffff80213e3a>{vscnprintf+9} <ffffffff80138a46>{vprintk+421} <ffffffff802ac713>{__ide_dma_read+154} <ffffffff802ac80e>{__ide_dma_begin+40} <ffffffff802af525>{__ide_do_rw_disk+853} <ffffffff80138d8b>{printk+141} <ffffffff802a5076>{ide_do_request+977} <ffffffff8016cef8>{cache_flusharray+138} <ffffffff8016cbf5>{kfree+106} <ffffffff802ebbad>{kfree_skbmem+9} <ffffffffa009bf46>{:e100:e100_poll+1885} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff801670d7>{free_pages_bulk+910} <ffffffff801671cf>{__free_pages_ok+229} <ffffffff80134add>{free_task+18} <ffffffff803557da>{thread_return+41} <ffffffffa02171c3>{:sunrpc:do_cache_clean+0} <ffffffff8014f1bd>{worker_thread+219} <ffffffff801337a8>{default_wake_function+0} <ffffffff801337a8>{default_wake_function+0} <ffffffff8014f0e2>{worker_thread+0} <ffffffff801553ff>{kthread+188} <ffffffff8011124f>{child_rip+8} <ffffffff801fdc0f>{selinux_d_instantiate+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Kernel panic - not syncing: Aiee, killing interrupt handler! <0>Rebooting in 30 seconds..<4>atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. Losing some ticks... checking if CPU frequency changed. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. warning: many lost ticks. Your time source seems to be instable or some driver is hogging interupts rip serio_interrupt+0x18c/0x19a atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. [auto reboot] next oops attack: net/core/skbuff.c:1270: spin_lock(include/linux/skbuff.h:000001003b26faf8) already locked by net/core/skbuff.c/1195 net/core/skbuff.c:1270: spin_lock(include/linux/skbuff.h:000001003b26faf8) already locked by net/core/skbuff.c/1270 net/core/skbuff.c:1270: spin_lock(include/linux/skbuff.h:000001003edb8138) already locked by net/core/skbuff.c/1195 Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80166e5f>{free_pages_bulk+278} PML4 34904067 PGD 4d22067 PMD 0 Oops: 0002 [1] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278} RSP: 0018:0000010037c0d9e8 EFLAGS: 00010013 RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: 00000100016058a8 RDX: 0000000000200200 RSI: 00000100015539a0 RDI: ffffffff80432dc0 RBP: 0000000000017598 R08: 0000000000000000 R09: 0000000000000040 R10: 0000000000000040 R11: 0000000000040000 R12: 0000010001553978 R13: 0000010001553940 R14: ffffffff80432ee0 R15: 0000000000002eb3 FS: 0000002a95577a20(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process pdflush (pid: 46, threadinfo 0000010037c0c000, task 0000010037c08920) Stack: 0000010009b9b988 fffffffffffffffc 000000020000002e 0000000000000000 000000021deffb80 0000010001038000 ffffffff80432eb0 0000000037c0dcf0 ffffffff80432fe0 0000000d3fbd2c00 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801676ac>{__pagevec_free+38} <ffffffff8016f4af>{release_pages+791} <ffffffff801635e4>{find_get_pages_tag+179} <ffffffff8016f9f6>{__pagevec_release+23} <ffffffff801bad2d>{mpage_writepages+1997} <ffffffff801bad45>{mpage_writepages+2021} <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0} <ffffffff8010ebbc>{__switch_to+254} <ffffffff803557da>{thread_return+41} <ffffffff801b89bd>{__writeback_single_inode+662} <ffffffff801b96e4>{sync_sb_inodes+470} <ffffffff801b9d6e>{writeback_inodes+605} <ffffffff80169055>{wb_kupdate+147} <ffffffff803557da>{thread_return+41} <ffffffff8016a085>{pdflush+0} <ffffffff8016a4a7>{pdflush+1058} <ffffffff80168fc2>{wb_kupdate+0} <ffffffff801553ff>{kthread+188} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff8011124f>{child_rip+8} <ffffffff80155428>{keventd_create_kthread+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037c0d9e8> CR2: 0000000000100108 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0) already locked by mm/page_alloc.c/259 Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 34904067 PGD 4d22067 PMD 0 Oops: 0002 [2] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 214, comm: kjournald Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:000001003f22d9f8 EFLAGS: 00010006 RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 0000000000000008 R09: 0000000000000007 R10: ffffffff804d5d60 R11: 00000100197a9360 R12: ffffffff804332c0 R13: 0000010001553978 R14: 000001003fb541b0 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process kjournald (pid: 214, threadinfo 000001003f22c000, task 000001003fb541b0) Stack: 0000000000000000 ffffffff80162f40 0000000000000007 0000005000000000 0000010037ff4728 ffffffff8018c8ce 0000000000000256 0000000000000000 0000000000000050 ffffffff804332c0 Call Trace:<ffffffff80162f40>{find_get_page+156} <ffffffff8018c8ce>{__find_get_block_slow+52} <ffffffff80167b85>{__alloc_pages+200} <ffffffff801633ae>{find_or_create_page+68} <ffffffff801901d5>{__getblk_slow+246} <ffffffff801904d2>{__getblk+61} <ffffffffa0052c50>{:jbd:journal_get_descriptor_buffer+42} <ffffffffa004ba68>{:jbd:journal_commit_transaction+4582} <ffffffff803557da>{thread_return+41} <ffffffffa004f9ee>{:jbd:kjournald+524} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f7dc>{:jbd:commit_timeout+0} <ffffffff80133068>{schedule_tail+11} <ffffffff8011124f>{child_rip+8} <ffffffffa004f7e2>{:jbd:kjournald+0} <ffffffff80111247>{child_rip+0} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003f22d9f8> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80166e5f>{free_pages_bulk+278} PML4 34904067 PGD 4d22067 PMD 0 Oops: 0002 [3] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 3, comm: events/0 Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278} RSP: 0018:0000010037e3fcf8 EFLAGS: 00010003 RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000200200 RSI: 00000100015539a0 RDI: ffffffff80432dc0 RBP: 0000010001b9c7c0 R08: 0000000000000000 R09: 0000000000000005 R10: 0000010002141110 R11: 000000000000087f R12: 0000010001553978 R13: 0000010035148000 R14: ffffffff804d4460 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process events/0 (pid: 3, threadinfo 0000010037e3e000, task 0000010002140030) Stack: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000010001038000 ffffffff80432eb0 0000000000000001 ffffffff80432fe0 0000000f01038000 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff8016bdcf>{kmem_freepages+195} <ffffffff8016ccbb>{slab_destroy+114} <ffffffff8016cda1>{free_block+194} <ffffffff8016ce46>{drain_array_locked+99} <ffffffff8016e2b1>{cache_reap+256} <ffffffff8016e1b1>{cache_reap+0} <ffffffff8014f359>{worker_thread+631} <ffffffff801337a8>{default_wake_function+0} <ffffffff801337a8>{default_wake_function+0} <ffffffff8014f0e2>{worker_thread+0} <ffffffff801553ff>{kthread+188} <ffffffff8011124f>{child_rip+8} <ffffffff801fdc0f>{selinux_d_instantiate+0} <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037e3fcf8> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 34904067 PGD 4d22067 PMD 0 Oops: 0002 [4] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 1656, comm: kjournald Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:000001003eb6d908 EFLAGS: 00010006 RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 00000000000195f9 R09: 00000000000195f9 R10: 0000010037e34810 R11: ffffffff8028b44c R12: ffffffff804332c0 R13: 0000010001553978 R14: 000001003ccb95d0 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process kjournald (pid: 1656, threadinfo 000001003eb6c000, task 000001003ccb95d0) Stack: 0000010037cf2c00 0000000000000246 0000000000000000 000002003ccb95d0 ffffffff8013530f 000001003eb6d960 0000000000000056 0000000000000000 0000000000000200 ffffffff804332c0 Call Trace:<ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80284d5b>{submit_bio+219} <ffffffff8019071c>{bio_alloc+28} <ffffffff8018e015>{submit_bh+144} <ffffffff8018e0fd>{ll_rw_block+93} <ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f9ee>{:jbd:kjournald+524} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f7dc>{:jbd:commit_timeout+0} <ffffffff80133068>{schedule_tail+11} <ffffffff8011124f>{child_rip+8} <ffffffffa004f7e2>{:jbd:kjournald+0} <ffffffff80111247>{child_rip+0} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003eb6d908> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 34904067 PGD 4d22067 PMD 0 Oops: 0002 [5] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 1655, comm: kjournald Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:000001003db3b7c8 EFLAGS: 00010006 RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 000001003db13ac0 R09: 0000000000000000 R10: 0000010031697360 R11: 0000010031697360 R12: ffffffff804332c0 R13: 0000010001553978 R14: 000001003eb30070 R15: 0000000000000000 FS: 0000002a95574b00(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process kjournald (pid: 1655, threadinfo 000001003db3a000, task 000001003eb30070) Stack: 0000000000000000 0000000000000202 0000000000000001 000002003eb30070 ffffffff801337a8 0000000000100100 0000000000000056 0000000000000000 0000000000000200 ffffffff804332c0 Call Trace:<ffffffff801337a8>{default_wake_function+0} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff80165f3c>{mempool_alloc+133} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8019071c>{bio_alloc+28} <ffffffff8019096e>{bio_clone+13} <ffffffffa003f1fb>{:raid1:make_request+1303} <ffffffff8028b8b8>{cfq_dispatch_requests+104} <ffffffff80284c6b>{generic_make_request+361} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff80282e28>{generic_unplug_device+181} <ffffffff80284d5b>{submit_bio+219} <ffffffff80190820>{bio_alloc+288} <ffffffff8018e080>{submit_bh+251} <ffffffff8018e0fd>{ll_rw_block+93} <ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f9ee>{:jbd:kjournald+524} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffff8013530f>{autoremove_wake_function+0} <ffffffffa004f7dc>{:jbd:commit_timeout+0} <ffffffff80133068>{schedule_tail+11} <ffffffff8011124f>{child_rip+8} <ffffffffa004f7e2>{:jbd:kjournald+0} <ffffffff80111247>{child_rip+0} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003db3b7c8> CR2: 0000000000100108 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0) already locked by mm/page_alloc.c/259 Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80166e5f>{free_pages_bulk+278} PML4 14cc2067 PGD 0 Oops: 0002 [6] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 10799, comm: httpd Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278} RSP: 0018:0000010014ff1e18 EFLAGS: 00010003 RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000200200 RSI: 00000100015539a0 RDI: 0000000000000046 RBP: 00000100018be908 R08: 000000000000000d R09: 0000000000000002 R10: 0000000000000000 R11: 0000000000000000 R12: 0000010001553978 R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 FS: 0000002a97383140(0000) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process httpd (pid: 10799, threadinfo 0000010014ff0000, task 0000010014ffe370) Stack: 00000000000000d0 0000000000000246 2f2f2f2f2f2f2f2f 0000000000000000 00000000010b1c06 0000010001038000 ffffffff80432eb0 00000000010b1c06 ffffffff80432fe0 0000000f803566d3 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801a2a73>{poll_freewait+64} <ffffffff801a38da>{sys_poll+783} <ffffffff801a2a79>{__pollwait+0} <ffffffff801107a2>{system_call+126} Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010014ff1e18> CR2: 0000000000100108 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0) already locked by mm/page_alloc.c/259 Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80166e5f>{free_pages_bulk+278} PML4 3b0d4067 PGD 0 Oops: 0002 [7] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 2333, comm: named Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278} RSP: 0018:000001003b187d08 EFLAGS: 00010003 RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0 RDX: 0000000000200200 RSI: 00000100015539a0 RDI: 0000000000000046 RBP: 00000100015d5b50 R08: 000000000000000d R09: 0000000000000002 R10: 0000000000000000 R11: 0000000000000000 R12: 0000010001553978 R13: 0000000000000001 R14: 000000000000001d R15: 000001003b187f18 FS: 0000000041e02960(005b) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process named (pid: 2333, threadinfo 000001003b186000, task 000001003b180630) Stack: 00000000000000d0 ffffffff804332c0 0000000000000000 0000000000000000 00000000ffffffff 0000010001038000 ffffffff80432eb0 000000003b187f18 ffffffff80432fe0 0000000f80356611 Call Trace:<ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801a2a73>{poll_freewait+64} <ffffffff801a3110>{do_select+1501} <ffffffff801a2a79>{__pollwait+0} <ffffffff8019ada3>{pipe_readv+132} <ffffffff801a3469>{sys_select+820} <ffffffff801107a2>{system_call+126} Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <000001003b187d08> CR2: 0000000000100108 <1>Unable to handle kernel paging request at 0000000000100108 RIP: <ffffffff80167885>{buffered_rmqueue+446} PML4 3b0d4067 PGD 0 Oops: 0002 [8] CPU 0 Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1 Pid: 2333, comm: named Not tainted 2.6.9-1.681_FC3 RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446} RSP: 0018:ffffffff8049ffe8 EFLAGS: 00010006 RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0 RBP: ffffffff80432fd0 R08: 000001003cdbe838 R09: 000001003cdbe638 R10: 000001003cdbe5c0 R11: 0000000000000000 R12: ffffffff804332c0 R13: 0000010001553978 R14: 000001003b180630 R15: 0000000000000000 FS: 0000000041e02960(005b) GS:ffffffff80511980(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0 Process named (pid: 2333, threadinfo 000001003b186000, task 000001003b180630) Stack: 0000000000000206 ffffffffa01e6d28 ffffffff804a012c 00000220804a00b8 000001003d2c4240 ffffffff8033e8af 0000000000000056 0000000000000000 0000000000000220 ffffffff804332c0 Call Trace:<IRQ> <ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121} <ffffffff8033e8af>{fn_hash_lookup+157} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff802f51a7>{dst_alloc+43} <ffffffff8030c71a>{ip_route_input+2280} <ffffffff8030ed25>{ip_rcv_finish+50} <ffffffff8030ecf3>{ip_rcv_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030f2fd>{ip_rcv+1008} <ffffffff802f191f>{netif_receive_skb+572} <ffffffffa009baaf>{:e100:e100_poll+710} <ffffffff802f191f>{netif_receive_skb+572} <ffffffff802f19ce>{process_backlog+125} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff80111da2>{oops_end+159} <ffffffff80111d15>{oops_end+18} <ffffffff80124595>{do_page_fault+1074} <ffffffff80138d8b>{printk+141} <ffffffff80111099>{error_exit+0} <ffffffff80166e5f>{free_pages_bulk+278} <ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801a2a73>{poll_freewait+64} <ffffffff801a3110>{do_select+1501} <ffffffff801a2a79>{__pollwait+0} <ffffffff8019ada3>{pipe_readv+132} <ffffffff801a3469>{sys_select+820} <ffffffff801107a2>{system_call+126} Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <ffffffff8049ffe8> CR2: 0000000000100108 <3>Debug: sleeping function called from invalid context at include/linux/rwsem.h:43 in_atomic():1[expected: 0], irqs_disabled():0 Call Trace:<IRQ> <ffffffff80133173>{__might_sleep+173} <ffffffff80139d4d>{profile_task_exit+33} <ffffffff8013b87a>{do_exit+34} <ffffffff80111da2>{oops_end+159} <ffffffff801245e6>{do_page_fault+1155} <ffffffff802f12f7>{dev_queue_xmit+1223} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff80111099>{error_exit+0} <ffffffff80167885>{buffered_rmqueue+446} <ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121} <ffffffff8033e8af>{fn_hash_lookup+157} <ffffffff80167b85>{__alloc_pages+200} <ffffffff80167e1c>{__get_free_pages+28} <ffffffff8016bc2a>{kmem_getpages+31} <ffffffff8016c422>{cache_alloc_refill+1083} <ffffffff8016bf89>{kmem_cache_alloc+79} <ffffffff802f51a7>{dst_alloc+43} <ffffffff8030c71a>{ip_route_input+2280} <ffffffff8030ed25>{ip_rcv_finish+50} <ffffffff8030ecf3>{ip_rcv_finish+0} <ffffffff802fb561>{nf_hook_slow+184} <ffffffff8030f2fd>{ip_rcv+1008} <ffffffff802f191f>{netif_receive_skb+572} <ffffffffa009baaf>{:e100:e100_poll+710} <ffffffff802f191f>{netif_receive_skb+572} <ffffffff802f19ce>{process_backlog+125} <ffffffff802f1ac1>{net_rx_action+112} <ffffffff8013ed50>{__do_softirq+76} <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} <EOI> <ffffffff80111da2>{oops_end+159} <ffffffff80111d15>{oops_end+18} <ffffffff80124595>{do_page_fault+1074} <ffffffff80138d8b>{printk+141} <ffffffff80111099>{error_exit+0} <ffffffff80166e5f>{free_pages_bulk+278} <ffffffff8016764a>{free_hot_cold_page+188} <ffffffff801a2a73>{poll_freewait+64} <ffffffff801a3110>{do_select+1501} <ffffffff801a2a79>{__pollwait+0} <ffffffff8019ada3>{pipe_readv+132} <ffffffff801a3469>{sys_select+820} <ffffffff801107a2>{system_call+126} Kernel panic - not syncing: Aiee, killing interrupt handler! <0>Rebooting in 30 seconds..<4>atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86, might be trying access hardware directly. [snip] [auto reboot]
Installed 2.6.9-1.724_FC3. Still crashes. rebooted with "noapic" kernel boot param... And it's been up for a day strait so far.... few segfaults by applications tho (different problem tho?) zmc[22651]: segfault at 000000001fffd85c rip 0000000008059074 rsp 00000000ffffd810 error 6 zmc[3731]: segfault at 000000001fffd85c rip 0000000008059074 rsp 00000000ffffd810 error 6 zmc[4068]: segfault at 000000001fffd85c rip 0000000008059074 rsp 00000000ffffd810 error 6 php[5349]: segfault at fffffffff84c2040 rip fffffffff84c2040 rsp 0000007fbffff840 error 14 zmc[4190]: segfault at 000000001fffd85c rip 0000000008059074 rsp 00000000ffffd810 error 6 spamassassin[17993]: segfault at 0000003d52ca20e8 rip 0000003d52ca20e8 rsp 0000007fbfffe800 error 14 lease broken - owner pid = 2997 zmc[8425]: segfault at 000000001fffd85c rip 0000000008059074 rsp 00000000ffffd810 error 6
turned out to be CAS timing on the ram.... i believe this is a hardware problem with the Abit AV8